Can I get sued for rejecting a guy?

I have a profile on one of the matrimony sites. This guy liked my profile and sent me a request for further contact which I accepted. We exchanged numbers in order to get to know each other better since he lived in a different city. We spoke on the phone and messaged each other from time to time for around 2 months, during that time I have not once told him that I love him or that I want to marry him or anything of that sort. In fact, I insisted that we and our families should meet before we can proceed any further. A month ago he came down with his parents to meet me; I first met him and his parent at a restaurant, during that time I realized he looked very different from his photo on the matrimony sites but I let it go. A few days after that his parents invited my parents over to their house, my parents didn’t like him especially since his father kept talking about how much more money they have than us. The day after the family meeting his father called my father to finalize things but since my parents were not interested in taking this any further he politely refused. A few days after that his father calls up my father again and threatens to sue me/us for this, he says he has all the proof (the messages, call logs) and he is going to take it to the police. I want to know if his family can actually file a complaint about this if yes, how will the police proceed in this case. Also, if they do end up suing what step can I take to deal with it legally?