There are many aspects to be considered by court.
Firstly since you have returned half of the amount, the court will conclude it as a hand loan borrowed by you and not an amount invested by him for purchasing the property.
Secondly there's no agreement between you and him accepting him as joint owner just because he had lent the amount.
Thirdly this is a joint property purchased in the joint name of your wife and you with her contribution through her own sources of income.
Next, he never raised a claim for a share in the property when he accepted the partial amount of loan you returned earlier.
Next, your willingness expressed before court to repay the balance loan amount with interest to him is a major advantage on your side which court will consider as a valid issue to decide the case against him besides many more aspects that the court take them for deciding the dispute and to arrive at a conclusion that it was a loan amount and an investment neither it was a joint family property.