Very slow work inspite of regular payments

Hi, There is a contract given to a local contractor ,who is a regular and a known one in our society, to relay plastering and painting in outer and inner walls of our existing house except flooring. Also construction of a new floor is to be done on the floor above existing house with a drawing room, bedroom, 1kitchen, 1lobby, 1bathroom and 1 balcony above the existing balcony. Design is same as the existing house except 1 bedroom with attached bathroom which is not being made now. 3ft wall was already there throughout and at the dining hall area 10ft wall was already there except RCC lenter which which was earlier covered by asebestos. Amount of Rs 13.5 lakhs was agreed to complete the work and Rs 50k were paid initially and We agreed to pay Rs 2lkhs every 15 days. Till now Rs 8.5 lakhs has been paid but only the structure with RCC lenters is complete. 65% of payment is done and only 30% of work is done. Work on outer plastering is going on just with a single labour and no work is going on at the inside. The concern is that the work done is not at par with the amount already paid. Before paying this installment of Rs 2lkh it was told to contractor to speedup the work and make it par with the money paid till now and next payment will be done only after the certain amount of work will be completed. To this he got furious and pressurized us that he will stop work till he receives next installment and just left the conversation mid way. Stopped picking up the phone. Picked up the other day coz he knew we will agree for payment and we did. Again he assured what all will be done in next 15 days but still progress on the work is very slow. Its giving me anxiety and stress as i have a doubt on his intentions now. I feel once the payment is done the work is not finished he will ask for more money else he will leave the work unfinished. He is only available when the payment is to be done. Once the payment is done he stops responding to calls and does not calls back. Currently payment has been done on time but the work is not getting done on time. He wants speedy payment but not willing to speed up the work. What is the ideal way of making the payment in these kinds of work and What benefit he will get by delaying the work. Also what can be done to push him to get the work done in speed and to make him understand that if the work gets delayed payment is bound to get delayed.