Husband's divorced wife's children

Hi Sir, I need a clarity. I have married my husband who has divorced from his first wife. He has 1 son and daughter from his first wife. As per his court order of first divorce, it is clearly written that all financial settlement has been done and the children are under the custody of their mother. He has no responsibility/commitments towards his first wife and children also. Based on that our wedding happened. Slowly he has brought his daughter and son to my house. His son stayed for a year or so as he came to work in a company in Chennai. His daughter as well. When I asked him why should they stay over here which is not agreeable to me. Am their father they will be here with me as his ex-wife was living out of Chennai. After that I have strictly told him that they have to live in their own place with their mother. Also since his daughter went to study MBBS in overseas his ex-wife made my husband to borrow and spend money for her studies. Myself and my brother also gave money to the tune of Rs. 3 L each for multiple reasons to my husband including her studies as my husband committed that he will give back. Till date he has not returned to us both. He says he will say he has not borrowed from us. From then he stopped my parents and my people coming home and complained about me to all my people. Now he threatens that he will bring his married son and daughter and all of them and make them to stay in our house. Is it compulsory to allow them into my house?