Divorce and not giving visitation rights to father

I have a son who is 4 yr old. Child father left us and went when my son was 11 month old. Since then we are separated. He had extra marital affairs, he took money from me to set up his business never returned, he took money from many of his friends never returned. Highly irresponsible. Initially I allowed to visit my son for 3-4 times. Later I stopped. Now he don’t come to visit us. I take/took care of all the financial expenses of my child since he is born. There is 0 contribution from his father since day 1. Now my question : I asked him for mutual consent divorce without visitation right for him and he denied it. Now I have to file the divorce, I’m worried that if he gets visitation it would be lot to toxicity from in the family. I don’t want my child to know what kind of father he had(as he don’t know who is father is now). What will happen if I don’t take divorce itself? Our marriage was not registered, do we still need to take divorce? Based on what parameters can I get divorce without giving visitation for his father?