Defamation of my family

My elder brother has taken money from different students and not returning them back. Those students are angry and they are coming home for their money and we had no idea that when he took money and what he did neither he is willing to tell us . Those students are coming in group and threatening me and my mother specially that they will shoot him and us. They are defaming us in every possible way they are talking crap about us to our neighbours . We don't want to live here he has left no scope for us to survive here.. My father and younger brother lives in gujarat , my father has his job there and brother studies there . My elder brother lives alone or sometimes with us.. We gave a notice in police station and local newspaper few years back about having no connection with him and that we shall not be liable for his crimes . But he came and lived with us after that notice . What should we do now? My family is going into depression we are very tensed and we are not in condition to return them money because its a huge amount of around 5-6 lakhs.. It can be more because we are unknown till the person comes home and tells that he took money