Garden on private terrace

We live in a complex where there is a private terrace accessible to the first floor flat mates, and also a common terrace on the top floor. This first floor terrace is accessible through back door to us. Nobody except the first floor residents use this space and all of us have gardens. There is a common door for the cleaning ladies to enter which is seldom used by people incase they drop their clothes/other things. The builder didn’t legally sell us the terrace but did inflate the price of the flat in terms of better view and more privacy since this is a private terrace. Some of the first floor occupants legal documents mention that this space is to be shared by the 6-7 houses on the first floor and we do that with pleasure. The association has now asked us to remove our plants saying we are encroaching on common area. We have our own balconies which we use to dry clothes and store stuff. We are being asked to move our pots into our balconies because our balconies are bigger than other houses. Is having plant pots on private terrace area wrong? And even after all the first floor flat mates have raised their voices the Association is not willing to conduct a meeting to hear our pleas. While common corridors are being used to store cycles and shoe shelves how can private terrace on the first floor be out of bounds for first floor houses?