Made late payment charges but builder said charges waived off in document.

We booked a flat @Chennai in 2019. At one point of time bank(SBI) refused to release installment amount as they've already disbursed 80% of loan amount but still builder didn't register the property and also didn't send any proof regarding status of construction and bank said atleast MOD agreement is needed to release further installments. It was purely a delay in builder's side. But they charged us "Late payment charges" of rupees ₹84,680. They said this is as per RERA rules so they can't waive off this charge. So we paid this charges together with last installment. And during handover they asked us to sign "DEED OF DECLARATION and INDEMNITY" agreement. In that agreement, PARTY OF FIRST PART is me the Customer. PARTY OF SECOND PART is The Builder. In that agreement, I noticed this below two points in the paragraph: (start of quote) "2. Apart from the total consideration of the Apartment mentioned in schedule C Property the party of the second part has calculated the delayed payment charges aggregating to Rs.84680/- and provided the final checklist for handing over the Apartment to the party of the first part. 3. The party of the first part has/have agreed the delayed payment made by him/her/them and requested the party of the second part for a lumpsum discount of Rs.84680/- from and out of the delayed payment charges of Rs.84680/- and the party of the second part has considered and agreed for the same and hence the party of the first part has paid the balance of delayed payment charges of Rs.0/- to the party of the second part, with his/her/their full consent." (end of quote) Doesn't this mean they have waived off my charges? I have paid my late payment charge but they haven't refunded it. But agreement says I didn't pay any late payment charges. How do I confront the builder? Can I file a case in RERA or file a case or should fight with builder for a refund? Our apartment has 1000+ homes and some people have paid even 3 lakhs to 5 lakhs as late payment charges. Kindly share your opinion so that we residents can take it together.