1. Legally speaking, you cannot stop her from filing more cases including for violation of protection order against you. What you can do is contest her cases fittingly. Mere filing of case by her will not prompt the court to rule in her favour. She will have to substantiate her allegations by showing the requisite evidence to the court.
2. The court has allowed her to stay in her matrimonial house as she could show it to be a property owned by you. If her stay in your house poses danger to the physical and/or mental well being of your mother then you may apply to court to recall the order giving her the right to stay in your house. If you have any proof of her subjecting your mother and other family members to verbal abuse then produce it in the court. Alternatively, you may challenge in a higher court the order of the court awarding her residence.
3. Your mother may also file a case for domestic violence against your wife to escape her victimization at latter's hands. That apart, your mother may file a criminal complaint for intimidation against your wife.
4. Neither your kids nor your wife have any share in your self acquired property as long as you are alive.
5. As regards the ancestral property, your children have an equal share therein, to cull out which their mother may move to court on their behalf, whereas your wife has no share therein.
6. If you wish to transfer your property to your children then you may make a gift deed in their favour. Get a flawless deed drafted by a lawyer so that your wife cannot do any misdemeanor.
7. For prosecuting your brother-in-law for tax evasion send a complaint addressed to Commissioner of Income Tax by annexing all the required proof.