Delay in possesion of flat

Hello sir I have booked a flat in Bhubaneshwar in 2011 which was scheduled to be delivered In 6 month. I have paid 80% of total amount of the flat by 2011. As there are continuous delay and no clear answer from builder I have not given him 15% amount which was scheduled. Rest 5% at the time of possession. Till date the building is not completed, very poor quality of work.And now builder demanding double the amount which was due to him for possession. He is asking this money because of our delay in payment of rest 80% and as per agreement , he is not responsible for any delay but if we didn't pay in time them we have to pay 24% interest on due amount. Now in between this in last year Aug2015 he made the registration of flat in our name hoping to get the rest of the money, but after seeing the work we have blocked the money, so now we have paid him only 80% of the amount with registration of the flat done. Now he is forcing for double amount, and to take the possession of the flat, which is not fully complete and he is also not able to provide the various clearance letter from the authority. now I have few questions 1. as now due to huge delay can I refuse to take possession and ask for refund with interest. 2. Can I go for legal claim for delay and paid the rest after deducting the claim amount. 3. As now the builder is now even speaking in health manner and always shouting on phone for money, which is the best and faster way in which I can claim my justices. As I am staying abroad and my parents are suffering with all these tension. thank you sir jyoti ranjan