Builder is Refusing to pay the Full Booing Amount

Hi, I have paid Initial booking amount to the Builder dated 14th March 2020 , Booking Amount was 2,10,000/- and we have Not Signed any Agreement on Legal stamp paper or anything , he has Just written on plain white paper with the initial Booking amount and remaining amount and GST Details, Now Suddenly due to Covid situation i am facing Financial issues already i am about to Loose my job and i am already facing salary Cuts from my Company, so i have Decided not to go Further as i am the only earning hand in my family, so we have Informed the Builder that we want to cancel the flat booking due to financial Condition. also we Informed the same to him on 20th May 2020 and now he is Refusing to pay full amount , Initially he is telling that his CA Has Informed them , not to Refund any Booking Amount to anyone, After couple of Hustles he agreed to pay some amount that is Only 50% I.e 1 Lac Only, and he is Not ready to Negotiate Further. I Dont understand we have not done any agreement nor Signed any Bond Neither he has Informed us initially regarding his Cancellation charges, nothing is mentioned in his Brochure or sales sheet. so now i am under Tremendous stress, as they are asking to take 1Lac Rs as a Cancellation charge which will put me in deeper Financial Crisis. Can someone please tell me how to tackle him or where i can Complaint about this Particular Builder regarding this Non Ethical Practices. Your Response will be Deeply appreciated.. Regards