What do if broker asking brokerage for purchasing the property which was occupied as rental

I had taken property on rent which was occupied by me for last 18 months. Said property were taken through broker and broker charged brokerage twice in original deal and renewal also (Total Rs.24000). Once broker approached me to purchase the same rented property but I clearly denied because I had a doubt on credibility of broker as he told that "I should not speak with owner and every communication including payment of token should be done through me not directly to owner". Because of that condition I clearly denied for deal. After 1 week owner approached me directly and enquire about my interest for purchase of property, I told him entire story. Than owner told that broker never informed about your interest before and he didn't givn any rights to broker for taking money on behalf of owner. And than owner told since broker is not in city from last 2 month, we execute deal directly without broker. Then we executed the transaction accordingly. 3 days after execution broker comes to know about deal than he started calling me for brokerage and told me that since I shown you that property for rent, you have to pay me the 1% brokerage on purchase. I told him that owner approached me please speak with owner. Than he contact and owner clearly denied for any payment telling that we were not hired you and you were not available in city on that time, so don't call me again. Now broker continously calling me and telling I don't know about I want my brokerage, since you contacted owner through me I want my brokerage. And also threating various way. I need a suggestion whether I need to pay the brokerage for purchasing this type of rented property. What is the way to deal with broker.