1) I want to ask can they take back that shop acquired by me trough registered deed?\
Answer: Unfortunately they can take back their property. This is possible under the section 23 of Maintenance of Senior Citizen Act, provided your mother is above 60. So your brother cannot take back his share only your mother;
2) Can they file any suit against me to cancel Registered Transfer Deed?
Answer: Yes your mother can file a suit in the relevant court to revoke the transfer deed;
3) Is any suit maintainable in the court of law?
Answer: Yes it will be maintainable in the court.
This is my response to you:
1. Engage services of a lawyer;
2. Let him/her verify your documents;
3. If there is some loophole to bind them by their actions then use that;
4. If you provide proof that you are maintaining your mother then the suit she files will be null and void.