Hello,
1) If you have used the credit card you are liable to pay back. It appears that since you have not paid back the bank has given it to a recovery agency to recover the money from you.
2) However, no one is authorized to threaten you or harass you in various ways as stated by you. The bank must adopt legal ways to deal with the unpaid debt.
3) To stop the harassment to your wife, you and other members of the family, you must lodge a police company against the recovery agency. You will need to gather the mobile number, name and address of the persons calling you and harassing.
4) You also need to take steps to to repay the card provider. As you are short of money now, you can negotiate with the bank to either pay installments or settle for an amount agreeable. Write a letter to the bank/ card provider in this regard and keep a record of the same.