Since you both are divorced legally, neither he nor you have any rights over each other.
The assault he had done on you attracts criminal offence.
You can give a complaint to the local police and can very well leave his home for your parents' house.
Even though you both reconciled your differences and are living together, since you both are not remarried, you both cannot be considered as legally wedded couple till remarriage.
He cannot stop you from leaving the house neither he can snatch the child which is less than 5 years old from you for any reason while leaving the house.