The house belongs to you and hence he has no rights over it. You can send a legal notice to vacate the house or face criminal proceedings. Rcr notice has been sent by him therefore you should appear in court and explain why you are not living with him. If the court finds it reasonable it can dismiss the application. This can also be a ground for divorce.
Regards