1. It is illegal and punishable under the provisions of law relating to extortion of exorbitant interest act, hence any legal action taken by you may land you in trouble if you proceed in this angle.
2. There is no reason to demand money from him in the manner you have narrated your story, hence on the basis of the evidence available with you for giving money to him you may send a legal demand notice to him demanding the money given to him after which you may file a money recovery suit.
3. He may give a complaint against you and in such a situation you may face problem from police, hence do not commit anything in writing to avoid getting trapped.