Issue with the broker

I rented up two rooms on second floor along with two of my friends in Chandigarh through a broker. Initially the broker told me that the set is fully independent and he will give us the access to the keys of the gate. We were not told about the timing issues that the landlord has. (The landlord sleeps by 11 and locks the gate) The broker also didn’t mention to us about the dumping yard near the place, which causes extreme discomfort when the garbage is being processed. Had he told us about these issues clearly at the start, we would have never taken the place. He also agreed on the water charges being included in the rent but the agreement says otherwise. We have started living in the property but we have not signed the agreement yet. We also have the voice recordings of the broker, wherein he promises us to get the keys soon. Now the landlord is straightaway refusing to give the keys to us. We want to shift from here. The broker is not ready to face us or meet us and we have also handed over the security, rent for one month in advance and the brokerage. Why options do we have here to get out of this scene?