NRI is not allowing to make TDS payment

I planned to buy a property from NRI. This NRI's PoA is the point of contact here. We did an agreement to sale as per which paid 25% amount as down payment to NRI's NRO account and PoA had assured that he will take care of all tax implications. The rest 75% is from bank loan and it shall be credited directly to Buyer's account. But after breaching 45 days; NRI's PoA says that they are not interested in paying the TDS at all. I orally requested for a refund and cancellation of agreement. But seller wants to apply for Nil/Low TDS certificate now after 45 days. But I believe that Nil/Low TDS certificate tax rate will be valid only on the transactions performed during the period from Issue date to End of financial year. Any transactions performed before or after the validity of the certificate are still taxed at normal applicable rates. So, I demanded to return the 23% TDS amount from the initial down-payment so that I can pay the TDS amount at least for that transaction. And then he can apply for certificate; and once the certificate comes we can make all future transactions as per the certificate tax rate. However the seller is reluctant to refund me even the 23% of that down-payment and thereby is not allowing me to make TDS payment. Can I cancel the agreement to sale now? and ask for full refund?