House construction

Hi lawyers, We have started our construction in Mar 18, we were not very well versed with agreements hence he signed a simple agreement without any penalty clause. This was on trust basis which was surely our foolishness as this person was sugar coated we trusted him. As per the agreement we have paid him all the money approx 40 lac is the project cost and we have to only pay him another 1.5 lac for which he has loads of work pending still abt 4-5 lac work is pending . Whenever we go and ask him to complete the civil work he only says pay me I will complete everything and he takes money and completes lil work for 2 days and back to the same old story give me money I will complete( I’m not exaggerating I’m being very honest) . We have overpaid him .. he says u keep 1 rupee of mine pending i will keep 50 rs of ur work pending and thts how he operates.. We have paid him everything and he is now asking for more I don’t want to give has he doesn’t do anything. Pls advise what do we do.. he is not willing to talk to us also .. whenever we call he disconnectes.. he has mentally tortured all of us.. pls advise