Now the girl's family is intimidating us by saying that they would take us to court. Is there any ground that they can file a case against us.
There is no hard and fast rule that the parties to the proposed marriage have to compulsorily marry each other even if there were some problems that cannot be solved.
There is no illegality or a crime done by calling off stopping the proposed marriage .
They have no reason to go to court, they may lodge a criminal complaint on false allegations like they could not concede to your huge dowry demand hence you have cancelled the proposal.
This can be challenged in the court during trial, however, you may obtain AB in case they are registering a FIR on this.
You can challenge their false allegations during trial proceedings on merits and documentary evidences in your side should they file a complaint.