Can a bachelor bring his lady friend to a rented flat?

I have a 2-bedroom ownership flat within a standalone building. This is not a cooperative property. I am one of the owners. Currently, 3 bachelor tenants are staying in that flat and using the flat as shared property. Now, one of the tenants is bringing his girlfriend at night and sometimes the girl is staying at night too. Personally, I don't have any issues with this as sometimes my tenant informs me that he is bringing his girlfriend or a group of friends who are girls. Now, a few other flat owners are raising issues with this. and telling me that it's an illegal activity that bachelors are staying with girls overnight. They are also trying to create chaos in flat owners association meetings and threatening me to lodge a complaint to the Police station. All of my current tenants come from good backgrounds and they are working in Software Industry (IBM). Please let me know if my other flat association members have any right to legal complaint against me or my tenants. How do I overcome this harassment?