Owner not refunding token amount for cancelling within 24 hours

Hello, We visted a property yesterday around 3.30pm, saw the flat, liked it and told the owner we will decide and let him know in an hour, so he said we can pay small token amount because there were others interested in the flat. Around 5.30pm we paid 5k as token amount as we were genuinely interested, he later said we would have to make agreement ourself and that we would have to bear the cost ourselves. While checking with an agency the cost to create agreement was quoted as 3.8k split as 1.9k by us and owner(agreement was not created, just a quotation) to which i asked this agency if cost is split between owner and tenant to which they said yes. When i approached this point with the owner, he said to make the agreement online where it is cheaper(nobroker site had a quotation of around 2.8k) and that he already reduced the cost of agreement by 1k (by referring to do it cheaper online) and still expecting him to pay and that he will not pay anything for agreement, so all the way upto 11.30 in the night i was messaging him to help me with information like size of the flat and survey number etc as i dont know the exact info so i could not create the agreement as he read it but didnt reply with the informatiom, next day around 12pm i decided to not go ahead with flat as it seemed like a very laid back owner and future concerns so i informed him over call that i will not be going ahead with the flat due to personal reasons to which he said that he will not return the token money as per the book(idk which book he is referring) or once he gets another tenant he will think about it, it led to an heated argument because he didnt say he will give it he said he will think about it and later he said that he doesnt like the way i am talking so he has decided that there is nothing to think about and that he has decided he will not recieve it that we can do whatever we want. What should be the course of action?