1. It appears you do not have written agreement with him nor any verbal agreement on refund of security deposit.
2. In absence of such clear understanding you are bound to return his money.
3. Within a couple of months you would find amny tenants and to avoid unnecessary dispute forget this episode as bad dream.