Unprofessional conduct by building contractor

Hello sir,

 I've made an agreement for construction at my plot. 
Agreement has been done on notarised 10 stamp paper of Rs. 10, with signatures of myself and contractor on all the pages. Due to my carelessness I failed to add any of the following points in the agreement: 1. Stage wise completion time target. e.g. Plinth level construction: 20 days, Ground floor slab: 30 days etc 2. Clear contract termination condition. (Only that if the work shall not happen as per the agreement, I am free to terminate the contract) 3. Any guidelines in case of disputes Only overall project completion time of 9 months 15 days is present. 
I have paid him 12 Lacs in advance before even start of the work. It has been 3 months since the work has started, and the project still hasn't reached the plinth level. He had been promising the plinth level in 12 days since the beginning and has failed to deliver on any of the verbal targets. 
Overall work has happened only for 30 days and rest of the days no work. Reasons include non-availability of material on site, labours refusing to work due to non-payment of their daily wages(80% due), contractor citing personal issues. 
He also deliberately failed to comply with structural drawings, to which I had to personally supervise the construction of foundation.

Upon multiple request to speed up the work and make up for the time lost over call, WhatsApp and email, he has started dominating and threatening me. 
After the work didn't started for 10 continuous days, I politely informed him that due my dissatisfaction over work and prolonged delays and I am terminating the contract. On this he threatened me that you cannot terminate the contract and you wont be able to lay even single brick on your site. He also told me since client is terminating the contract, client has to pay 25% of the agreed project cost to him. 

On top of this, labours and material supplier has started calling me over the payment of their dues. 

Luckily, only silver lining I've is the post dated cheque is received from him against the payment made as security. 

As per my observation, I am 100% certain that he is not in position to complete my project honestly and effectively. Therefore I dont want him to continue the work. I wish for only 3 things: 1. Contractor should vacate the site and pay all the remaining dues to labours, material suppliers etc 2. I should be able to take over the site and start the construction myself 3. After deducting his incurred cost on the project and some of his profit margin (Overall cost on project so far is near 5 Lacs) return me the rest of the money in given time (say 2 months) Considering all the above points can you please help me regarding how can I proceed with the project. 
 Also, since the original signed agreement is with me only the contractor only has the photos of the agreement. Is it possible for me to deny the existence of any signed contract. I would be very obliged for your help.