First cancel the agreement and stop the loan account..
Since the proeprty is mortgage to secure the loan let the creditor proceed to attach the proeprty which is itself in dispute. This would stop you from incurring further loss.
Now as the proeprty is litigated you have got every reason to seek refund of the advance paid so far to the builder while entering into the agreement with him.
If he refuses to do so which is very likely then file a case before the consumer forum wherein you would get both refund with interest and damages with compensaton.