Advance money on a failed property transaction which does not have agreement

I had finalized a deal for my house in May 2017 with a relative. He had made a payment of 1L as token amount and another 6L after few days as advance for the house. The deal was to be completed in 3 months time frame. However, he did not make any payment after that citing one reason or another and neither did he say no to the house deal directly to me or asked for his money back in return. Last communication I had with him in early 2019 he still told me that he is trying to arrange the funds. Finally fed up I sold my house to another party for much lower amount in Dec 2019. Now the person is asking me for his money back and threatening me if I don't return money, he will go to court and say that he had loaned me the money and I'm not repaying him back. Since deal was within the family, there is no contract of any sort or any signed papers for all of this except bank statement for the payment he made through cheque Does he have a case here? The loss he caused me is much greater than 7L I owe him, since house remained unsold for long time and finally I have to settle for much lesser amount due to real estate going negative. But I don't want a legal battle also, as I stay in different state and the house was located in my home town (UP)