Token paid but owner said he gave the house to someone else

After searching so many houses, I found one house which was suitable to me. The rent of the house was 46k. The broker asked me for a token amount of 20k. I tried to negotiate but he said it is final. I also talked to the owner about it and he said this is the token if you want to take it. I was sure about the house and anyway I am going to sign the rent agreement in 2 days, I paid it anyways. Then the owner said he is not available for the next 5 days. It was okay till then. But after 5 days, there was some emergency in my home and I said that I can not shift the house right now. I extended my current house for a month and paid the current months rent. I called them everyday to return the money. I even asked him to send 50 percent back. But both the broker and owner said it is non refundable and it is agreement breach from your side so I will not send a single penny. Anyways, After 10 days, I talked to the broker, and said I will take the house, please talk to the owner. Till then I know that no one signed the agreement till then. But they denied me saying that someone else gave the token so I can not give you the house. Now I don’t have that house nor the token. Now that I discussed with the broker that owner denied me house after receiving the token, he clearly said ‘Token nahi milega aap kuch bhi kar lo’. Now is there any legal way to take the token amount back?