Real estate agent is harassing me

Hello, whoever reads this, I hope you are doing well. I had recently seen like 4 flats through a broker and liked 1, he showed me that flat like 2 times and he even gave me the key of that flat for few hours as I wanted my family to see that flat as well. Later on, I gave him a price which I could afford and he said that owner is not agreeing on this price, so I told him that no worries I’ll look for something else. Then later on I had seen an advertisement posted by an owner on 99acres.com but it had no pictures of the flat. As I have been staying in this society for last 5 years I know it pretty well so I contacted that owner and he turned out to be the owner of the same flat I had seen before. So I negotiated a deal with him which I could afford and he agreed. Then after a month now the broker got to know about this and he’s started threatening me and demanding brokerage because he had shown me the flat before. But I never went ahead with him, he did not give me owners number, he did not negotiate with the owner of flat. He’s put a complaint against me but the police told us that we have to resolve it in civil court as they can’t interfere between money matters. I need to know that am I liable and obligated to pay the broker his commission? I’d really appreciate if someone could help me out on this