Your have two options one is a suit for specific performance whereby you go to court seeking direction from the court to act according to the agreement.
But since a lot of time has elapsed and you seem to have already demanded the money and he too has refunded a small part of it your better option will be to institute a suit for recovery where you seek your money back with interest along with damages. This should be more ideal to your need of the hour.
Please do bear in mind that suit proceedings are normally time consuming affairs in India and this might require you to chase your remedy with some patience.
There is one other mechanism used for speedy recovery of debts. But it might not apply to your situation. If the builder runs his business through a company and there is any document where he has admitted this money as money due to you within the last three years then you may try a winding up petition where if the company fails to pay the High Court will appoint the official liquidator upon the company's property who will sell it and disburse the amounts to petitioning creditor. But this form of litigation has many technicalities and though speedy it needs to satisfy certain conditions.