See firstly you cannot relinquish share you will receive after mother's demise, you can only relinquish the share presently in your name. 
Release deed cannot be conditional. See only recourse is take an agreement from brother giving life rights in property to mother.

1. i am not sue why there is any compulsion at all upon you to relinquish your share which you have rightfully inherited from your father.

2. Since you are worried about your mother then even if you decide to relinquish you share then do so in favour of your mother and not in the name of your brother.

3. Another option is to make a deed of settlement wherein you can put such conditions as you find conducive to ensure the safety and security of your mother.

After the death of your father, you are co-ower  of the property with your brother and mother i.e. 1/3 rd share each. In case you relinquish your 1/3 rd share in favor of your brother, he would become owner of 2/3rd share.  Thereafter, he will get 1/3 rd share of his mother transferred in his favor by another Relinquishment Deed.  In case you transfer your 1/3 rd share in favor of your mom, she would become owner of 2/3 rd share of the property and your brother would get her 2/3 rd share transferred in his favor. This is apparent that after getting the property transferred in his favor, your mom would be on the road.  You only can safe your mom by not transferring your share in favor of your mom and brother during the life time of your mom. After the death of your mom you would become the owner of half share and at that time you may relinquish your share in favor of your brother.  Mind it there is no transfer of property with revocation clause.  Once property is transferred, it is transferred for good.  Be aware and safe your mom by not executing any document during the lifetime of your mom.   Don't come in crocodile tears of your brother.  If you are thinking to take any monetary help from your brother, forget it and take help from other sources. Good luck.

Yes you can execute a revocable relinquishment deed in this circumstances. Or you can execute a registered will in favour of your mother

You can execute conditional gift deed that in the event brother fails to take care of your mother financially during her lifetime you would  be at liberty to revoke the gift deed 

In this case you can make WILL instead of relinquish deed or conditional GIFT deed.

You can make WILL of your inherited share as you have concern about your mother if you relinquish your share to your brother.

Assuming this is your father's self owned property. And not ancestral property.

You can put any condition the commission or ommissom of which is not barred by law. You can revoke such conditional deed on non-fulfillment of such conditions by the beneficiaries.

Release deed is irrevocable. Better execute Conditional Gift deed that gift shall be effective after mother`s death.

Your mother herself can find remedy to the maltreatment, if any, after she relinquishes her rights in the property by the beneficiary through senior citizens welfare tribunal.

 However if you apprehend that she may be deprived of her privileges or she may be subjected to tortures or troubles at a later stage, let she not relinquish her rights in the share of the property that is due to her.

You cannot ensure this until she herself is taking a decision on this especially when you are living separately away from her.