Deal has been cancel to sell property to buyer

I am a seller. I want to cancel my deal. Since the purchaser has given me token amount 51000. Then after they have given me amount 400000 by cheque as a down payment. I have not deposited this 400000 cheque amount in my bank. I have return back this 400000 cheque amount to purchaser because I don't trust purchaser that they will give me balance amount 2500000 by loan. Now I have only token amount 51000 to give purchaser. So my question is that my agent is now demanding that you have to pay RS 6000 to purchaser because they have already spent to make agreement. But I mean to say that I have not signed any agreement . I have only signed acknowledgement paper for taking token Amount RS 51000 and I given them my 4 chain of agreement Xerox copy to purchaser. I have not signed any agreement. No agreement is made between us . So now what to do please suggest me. Now purchaser agent are demanding that if you will not pay Rs 6000 then see what will I do. They are saying that we will sent you legal notice. So please solve my problem...