Need advice how to deal with the Builder

Hello. My father and Uncle were the owner of our ancestral house in Kolkata. My uncle passed away in 2021 oct. our House is not partitioned and my father and uncle had equal share in the property. Therefore, after my uncle died intestate my aunt and her son inherited my uncle's undivided 50% share. On 26th September 2022 my father along with my aunt and his son entered into an development agreement with a local builder and executed a GPA with him immediately to build an apartment in place of our ancestral house. As per the agreement and the GPA the builder has been given authority to do everything required to get the building plan sanctioned. As per the agreement the builder had to obtain the sanctioned building plan within 90days from the date of execution of the development agreement. furthermore, as per the clause in the agreement only after he gets the building plan sanctioned he can ask us to vacate the house. Moreover, the same clause also clearly states that the builder can only enjoy the peaceful possession of the property only if he performs his duty as per the agreement. However, the 90 days has long been passed, we tried contacting him by visiting his office or calling him since he is a local person but of no avail. in the meantime from some of our common sources we got to know few bad details regarding the developer. Then we waited till 26th september 2023. Since, neither he got the plan sanctioned nor communicated anything to us my we issued a legal notice to him and waited for 60 days for any response. Since, there was no response by the end of Nov 2023 we published in our local newspaper that are cancelling the agreement, revoking the GPA and there lies no further obligation from our side and sent a notice to him in this regard. In February 2024, after being in hibernation for so many days he through his lawyer send a notice to us hurling false allegations that we had demanded money(no evidence annexed, therefore we are trying to frustrate the agreement), used foul language about my lawyer. in the next section he made further baseless allegations that we had prepared the building plan and submitted at his cost (it was clearly mentioned in agreement he was the one who has been given the authority to get everything done). Furthermore, he threatened to file criminal cases based on facts which are completely false( like he says my father has given false facts regarding mutation of his name, the same agreement records that he has done mutation and obtained holding number in 1978) unless we pay him 2 crore rupees for hurting his reputation. post that my lawyer had sent a strong response to him citing his nefarious attitude and saying that this has caused immense mental harassment to my 73 years old father he came to our house to say sorry and with new proposal to commence the work. But we don't want to continue with him at any cost. My questions: a) can he still enforce the agreement on us? b) what shall i do at this point