1) You can make complaint against him for torturing, threatening and harassing to you on phone.

2) Plus your husband can make complaint against him under IPC 497.

Section 497 in The Indian Penal Code

497. Adultery.—Whoever has sexual intercourse with a person who is and whom he knows or has reason to believe to be the wife of another man, without the consent or connivance of that man, such sexual intercourse not amounting to the offence of rape, is guilty of the offence of adultery, and shall be punished with imprisonment of either description for a term which may extend to five years, or with fine, or with both. In such case the wife shall not be punishable as an abettor. 

1) block his number 

2) if he threatens  you or calls you repeatedly file police complaint against him for stalking under section 354 D of IPC 

3) 

Any man who-

1. follows a woman and contacts, or attempts to contact such woman to foster personal interaction repeatedly despite a clear indication of disinterest by such woman; or

2. monitors the use by a woman of the internet, email or any other form of electronic communication,commits the offence of stalking;

4) 

 Any man who-

commits the offence of stalking shall be punished on first conviction with imprisonment of either description for a term which may extend to three years, and shall also be liable to fine; and be punished on a second or subsequent conviction, with imprisonment of either description for a term which may extend to five years, and shall also be liable to fine.

Just go straight to the Police and file a complaint against him.
He is taking advantage of your fear of getting defamed, but he will keep blackmailing you even if his demands are fulfilled.
So, don't let him blackmail you and your family, and fearlessly report to the Police

1. You made the mistake first by indulging in casual sex and then you repeated the offence of taking this adultery further in your marital life.

2 . Anyway , there is nothing to get worried about as he is blackmailing you for which if you lodge a complaint of extortion the police on that basis can register the FIR and then arrest the culprit as well. 

3. DO not make the mistake of getting bogged down by his empty threat. 

You committed no offence and he cannot make out a case against him legally.

Rather, you can be a real trouble for him, in case you open up and report the whole incident to the Police. It appears that he has exploited just because you kept mum all this while and kept tolerating this fellow. No need to bear him any further. Speak out about the whole episode to your husband and if need arises to Police as well. No need to succumb to his nonsense any further.  

Dear Client,

Do not accept of any such incident ever happened, and he cant do anything, on the contrary you are subjected to black mail to satisfy his desires. He can`t do anything and you can file criminal complain agaisnt him of blackmailing and demanding favors. 

You keep your husband in confidence and lodge a criminal complaint against him with the local police about his threats and indulgence of illegal activities against you.

If the police is not cooperative or not effective you may approach the top police officer of the city with a similar complaint seeking intervention and direction to the local police for proper action against the culprit.

If that step also prove futile then you may approach judicial magistrate couirt with a complaint  under section 156(3) cr,p.c. seeking direction to the concerned police  to initiate proper legal action on your complaint pending before them.

Please lodge a police complaint for stalking, extortion with the jurisdictional police station where you are residing. Note that he cannot take any action against you on the other hand you can also complain to state commission for woman if the police fail to take action against him.

you dont have to worry. he cant black mail you let him do whatever he wants to.

1. You should not have fallen prey to his misadventure in the first place.

2. Now go to the nearest police station to file a complaint against him under Section 506 IPC for criminal intimidation and under Section 323 IPC for causing hurt.

3. Preserve the phone conversation.

4. Your husband cannot prove him to be a criminal in the given facts.

messages received by you , call recordings would help yo make out a case 

Your own witness and the messages are enough to convict him for mental harassment.

Mere complain is enough and call history as an evidence of continuous stalking. Punishment may extend to three years

You have to record all calls, text and WhatsApp messages can be used in court as evidence. 

The messages sent by him to you or call records will be sufficient to prove stalking. First lodge a police complaint if police refuse to register FIR approach jurisdictional magistrate court filing private complaint under Sec. 200 Cr.P.C seeking courts direction to investigate the matter.

You lodge a criminal complaint against him in the local police station.

You can write all the incidents in your complaint.