Rent Amount taken - and Security Deposit not returned

In March the lockdown was imposed and i had to leave my PG in a hurry, thus I had to leave some of my baggage in the PG in Gurugram. I called up my PG owner and informed him that i will not be able to come back soon, so he can adjust the security amount given for the month of April and thus our financial terms are over as we had no written agreement. He agreed and we also felt great as he was just like a friend and had shown no intention of fraud. Once the lockdown was lifted and people were allowed to move out. I told my brother who lives in Noida to take away my belongings (A Suitcase and some books) from the PG. He went there and started to take away my belonging but the PG owner called up some persons and told my brother that he will not let these belongings to move out till I pay him the rent for (April May June), plus one month advance with him only Since these guys were threatening my brother I had to immediately transfer the whole amount in his account. I paid and then they let him take away my belongings. Now am thinking is there any legal action which we can take?