Property advance

Hello, I had finalized on a flat and had paid an advance amount subject to legal formalities to be validated by the bank. The bank carried out the legal formalities and found that the flat constructed is not as per the BBMP approved plan. So I sought a second opinion from another bank and the outcome was similar and my home loan was rejected due to legal issues as in the building. The seller was informed on the development in writing on a timely basis. Despite seeking a home loan from two banks I decided to give it my last shot by approaching another bank but in vain as the outcome was again the same. I informed the seller accordingly and he agreed to pay back the advance amount but withheld part of the amount of Rs.50,000/- and has refused to take my calls despite me holding the original agreement of sale. I have done rigorous follow up but the seller has challenged me to proceed with legal proceedings to recover the same. Kindly advice appropriately as to what action can be taken to recover the advance amount. Thanks, Ravi