Validity of a stamp paper agreement

I purchased a flat in 2003 and has a stamp paper agreement for paying 6 lakhs. We paid around 5 Lakhs that time and had receiving for that. Rest balance amount we kept due for giving after registry as agreed with the seller. We also has a stamp paper agreement stating that registry cost will be beared 50% each (Purchaser & Seller). Meanwhile the balance amount was also taken almost all in small small amount many a times and we have no receiving for those. After shifting to the purchased flat we asked for the registry of the flat but he was absconding because he was arrested and was behind bar for almost a year. After that I chased him several times but could not get hold of him. Hence as per agreement, flat could not be registered within the stipulated time of six months mentioned. After complete occupancy of more than 18 years he has now sent a notice via lawyer asking me to receive the earnest money / advance I paid during that time or else legal action will be taken. Please note that Electricity bill and Holding tax is in my name from last 18 years. Sir what should be my further course of action. Please advice Now what can be done.