My contractor not finishing the Job(construction) even though I paid him. pleas help regarding same.

Hello sir,

 I've did not make agreement for construction of my 1st floor due to contractor came by one of my trustworthy family person. Due to my carelessness I failed to take care of the following points in the : 1. Stage wise completion time target. e.g. brick construction: 20 days, floor slab: 30 days etc 2. I have paid him 6.5 Lacs in advance before even start of the work. It has been 9 months since the work has started, and the project still hasn't completed half (as Internal plaster is in undone condition) remaining work is not completed by him. He just gave me verbal promises. Overall work has happened only for 3 months and rest of the days no work. Reasons include non-availability of material on site, labours refusing to work due to non-payment of their daily wages(80% due), contractor citing personal issues. 
He also deliberately failed to comply with structural drawings, to which I had to personally supervise the construction of foundation.

Upon multiple request to speed up the work and make up for the time lost over call, he has started dominating and afterwords rejecting me/my calls. 
After the work didn't started continuous for many days, I politely informed him that due my dissatisfaction over work and prolonged delays return my remaining money and take your part from it whatever you have constructed till date. On this he threatened me that you wont be able to lay even single brick on your site. He also told me since client is terminating the contract, client has to pay 25% of the agreed project cost to him.

As per my observation, I am 100% certain that he is not in position to complete my project honestly and effectively. Therefore I also dont want him to continue the work. I wish for only 3 things: 1. Contractor should vacate the site and pay all the remaining dues to labours, material suppliers etc 2. I should be able to take over the site and start the construction myself 3. After deducting his incurred cost on the project and some of his profit margin (Overall cost on project so far is near 5 Lacs) return me the rest of the money in given time (say 10 days) Considering all the above points can you please help me regarding how can I proceed with the project. 
 Also, since I have his photos/recording when I gave him money in his hand. Requesting you to please guide me on this issue, how can I legally proceed with the same so I can get rest of the money after deducting his incurred cost on project.