Selling House : 1st Purchase Deal Failed by Buy, Now 2nd Purhaser

Hi Respected Legal Experts I am a rterired old man of 70 years, looking for legal sound advise. Last Year in June 2014, i had a simple notarized agreement on stamp paper (not registered fees one) with a house buyer, he agreed to pay full amount till last date 31st/dec/2015 (in 6 months) and paid a very less token amount of Rs. 5 lacs. (paid via cheques) He made the agreement with written terms that if he dosent pay balance till 31st dec 2015 his token will be forfeited, and if we (seller) back out, we will given him double the amount of his token money. On 31st dec 2015 he came to our house and said he is unable to arrange money, meanwhile we sold most our household good, packed remaining house item, got water, gas connections cut, as we were shifting to another town with my son, and we never thought buyer will surprise us last moment. So that episode was over. We told him token money and deal is over now. And he went back. Now recently after 9 more month, from the last failed 31st dec 2015 episode (the last date of payment as per agreement), he today out of sudden called and said can i get my token money back, we told him we had so much loss, and as per agreement its null and void and no token in refundable. And dont call us in future. Meanshile after much efforts after 1st setback, we some how managed another buyer, and sold house at much lower price than the earlier one, who has paid token amount, and whose registry has to be done in next 3 months. So my question is - will the old failed buyer can possess any threat / nuisance to the new deal ? Thanks for your time and suggestion.