My tree sheds leaves on neighbours property

Sir a jackfruit tree is in my compound ever since we got this house few years back. It sheds leaves as we all know always. When monsoon season starts my neighbours abuse us verbally to cut the branch or tree itself telling our tree causes dirt here n there. Otherwise they ask for jackfruit during the fruit season. But other side also has numerous trees*( from other neighbours) and they never complain to them. Their leaves also contribute to "the dirt". Now this neighbour, without following rules or asking our permission has built outdoor steps to second floor very very close to our boundary wall and this tree now is complaining again that our tree is nuisance to their rent people living above. During contruction they have destroyed our roof and also never replaced nor given us the money. Every year this repeats. Even after trimmimg of branches, they call us names n abuses verbally. My parents r both elderly citizens. They r home alone as me n my brother goto office every 6 days/week. I want to end this abuse. As they belong to different caste n community, every1 support them and treat us as villains.