Building plan was to sanctioned before construction. Well, no problem, if all houses are separate and not in building, all can apply separately for regularization of construction in written. If president dose not respond, keep it pending. NO effect on owners.

And for separate houses, no approval required.

you should have made application for conversion of agricultural land into NA then carried on construction 

2) un authorised construction can be demolished 

3) apply to collector for conversion of land into NA 

If panchayat is not responding you can move to high court under writ petition for speedy remedy by directions under the same. 

Dear Sir,

It is Called encroachment. You may file a suit for Possession and seeking mandatory injunction for demolition of illegal constructions.

You have to file suit in district court for injunction and possession 

At civil court having proper jurisdiction for your agricultural land

What were you doing till 20 houses were constructed on your land?

Did you not object to this or whether you had not taken legal steps through court of law?

It is not that you have not taken any action on this or have not consulted any advocate on this, hence if you are giving the details of the action by you then it will be easy to suggest you further steps on the same line or alternative lines.

However even now you can file a suit for mandatory injunction to remove the illegal structures constructed on your land. and for possession of vacant land

You can file it in the jurisdictional civil court within your geographical limits.