The builder is adopting tactics to avoid payment of the penal amount due to delay in completion of construction.
He may quote any such reason but since he has taken penal charges for the defaulted payments, he is legall bound to pay the penalty for delay in completion of construction.
If the builder is not obliging, you may first issue a legal notice demanding the said penal amount.
If he is not complying with the demand made nor has give any reply you may drag him to consumer forum seeking to pay the penal amount as well as compensating the costs for mental agony.
The builder will come running o you with a compromise package then you can dictate terms.