The property , if it was transferred to the panchayat by a registered gift deed by your grandfather then it becomes government property, which cannot be demolished by you.
You will booked for a crimninal offence for damaging government property.
How are you accessing the road so far after this property was transferred to the government by your grandfather, i.e., for the past 50 years?
This road access problem is not a new one, if at all it is a problem.
You cannot create a problem or raise a civil suit seeking easement rights for road access all of a sudden that too by allowing it to remain for 50 years.
You may discuss with a local advocate who may suggest any other plan to tackle this situation based on the prevailing situation.
Alternately you can claim re-partition of the entire property with easement rights to road access from your property after proper partition among the shareholders.