The builder is trying to defend himself by giving some reason to avoid being prosecuted.
You have to prove before court that the statement made by the builder is irrelevant especially at this belated stage.
If at all he had any such claim nothing prevented him demanding the same all these years of delay and remembering about this subject at this stage especially when you approached court, is nothing but an after thought to evade punitive action by the court in this regard.
You should analyse the situation and prepare your arguments accordingly.
You can also express before the court that you are always ready and willing to pay all the relevant and authorised payments as is required by law in this regard but the builder should not take shelter under this stating that the non-payment of the same is the reason for delay.