Can a contractor without doing his work,threaten a owner.

Our contractor creating problem every now and then, 3 times he threatened us to settle, but after mutualy compromising we fixed one final target date and final 4 instalment amount to be given,the 4th instalment amount after completion.but after taking 3 instalments also he did not complete the work, but now he is asking for money , we said first complete the works and take ur money, now he is telling i need a final measurement and final amount to be settled for him. we said still pending works is more than 40% to be completed,first do it and we will settle everything without any delay.but now he is not ready to work without taking measurements, i m waiting for completion from 3 months, since i could not shift there or rent the house and im paying loan amount due to his delay we gave him final warning that we are not going to wait for him ,and we are going it by other party. now he is threatening that he will complain to commisioner and will see how will we finish the work. since we have not signed the contract of any terms,is that possible that we can complain on him. plz advice. we already have paid total amount of 58 lakhs for 3250 sqft and construction work started on 21st october 2014 still work is pending.plz advice how can we handle him.