Advance received against sale of house is refundable or not?

I have to sell my flat and last month a party visited and liked it. They immediately transferred a sum of Rs.51,000/- to me as token money / advance. Then after they took photo copies of all relevant documents related to property for availing a housing loan. I kept on chasing him for signing an agreement to sale but he kept postponing. After a month he said that he will sign the agreement in next couple of days and along with that he will also pay another sum of Rs.3.5 Lakh to me so that I may get ongoing housing load closed and get him the original property documents. But just a day before the agreed date he called up and cancelled the deal stating that he fears of the increasing theft cases in the area where property is located. I refused to show the property to several other prospective buyers in this course of time stating that I have already agreed to sell it to someone else. And now I am back to square one and the buyer who cancelled the deal is asking me to refund the advance he paid. Am I liable to refund the full amount or forfeit it since I lost several opportunities over that month to sell my property? Please advise.