Late Payment on Flat

We booked a flat in Bangalore in Sep 2 018 for 63Lakhs where we decided that 32lakhs we will give from our savings and 30 lakhs we will take a home loan. In Oct we paid 2 Lakhs and booked the flat , Took the property documents for legal validations . Since My husband travels for 6 months and told the sales manager that we will do agreement in January after his return, the sales manager said he will check with management and let us know later he agreed, January we did a agreement and paid 30 Lakhs with agreement,Now we asked the sales manager shall we do Loan from our own and there the SM said no he will take care of Loan part as his reps are working on same fields. He sent Rep for home loan next week and the loan process starts Feb 2nd week onward. They tried PNB and came back with some reason that bank had closed application. then they started in LIC and april we got sanction letter , however when we asked disbursement SM said next week and on .. in May they said LIC agent got into some scandel and LIC had canceled all application given by him, Here again we asked we can do but he said no need. will do from HDFC, again they took 1 month and finally said we can do on our own. We took the Loan process mid july and got sanctioned sep 2019,and oct disbursement done. 2 days before disbursement the builder had sent a 2.5lakhs penalty stating we delayed paymnet . That Sales Manager had quit company and we told all this odreal to CRM they denaied listing and said we dont know that SM now , he is no more part of our company . after 2 months of chase we reached thier office to meet thier MD and he treated us as badly and said he will not reduce a single rupee , and when we asked the delay happened due to his SM , he said i dont know , as per agreement builder is not responsible for Loan , if SM has said he will do then show me Proof , and if you have proof then go do a police complaint on SM , Builder has nothing to do with SM what he did or doing and he walked away,That SM is not picking our calls to answer . Builder is threatening us saying if we take some step , he can be bad for Us and he can do anything which he will not say , and also he will cancel our flat even we had started paying EMI and payed a total of 60 laks. we dont know to whom we should approach , possession date is sep 2020 and they have just got (November 19) the OC from BBMP. if it was our mistake in delaying the loan , i would have happly paid that penality but here they were making us run behind and never gave answer for loan status .Till august they never sent any email for panelty or terminaton but now when we showed sanction letter where sanctioned amout is higher, they are asking us to pay panelty. Someone please adivce