Your husband does not care for you and is searching for excuses in order to declare and prove that you are not fit to be this wife. His family also wants the same.that is why he is behaving like this.
If you don't want to give him divorce, he will have to fight it in the family court. He will have to prove that you have been cruel incompetent unfaithful etc....these are some of the grounds under which a person gets divorce in case he proves it.
Your husband will have to prove what he says in court.
The court cannot and will not grant divorce solely upon his testimony. He has to back what he says with data and facts which he does not have.
Therefore if you haven't done anything and you do want that this marriage should last then try mediation although you have tried it once.
Also before he files a case tell him straight away that his attitude towards you has been very irresponsible callous and therefore if he filed a divorce case against you...you will immediately file a complaint against him alleging cruelty domestic violence torture...and other things which are indeed true. He wants to destroy this relationship.
If you refuse to give divorce he will go ahead and seek divorce by fighting it in court which is not easy.
You don't need to leave your job.your lawyer will take care of everything.
He will have to pay for the proceedings and for your maintenance in case you don't earn that much.