Panchayat sanctioned notice to stop construction due to setback rules

Hello, I have my ancestral house in rural area of West Bengal that is under the jurisdiction of a gram panchayat. A boundary wall is already there from long time as well as the neighbour's house is also present on that side of the house. A demarcated milestone is put in between the midway between my boundary wall and their house, which is almost at a distance of 1.5 feet from both sides. Now I'm building a structure inside(parallelly inside) the boundary wall. The neighbour have reported this to the gram panchayat and they have orally ordered to stop the construction on the basis of the 3 feet setback rule. A similar dispute also occurred earlier in this same construction cycle in which panchayat came on a middle ground conclusion, to which we agreed and continued construction. We didn't applied separately for this side's construction permission. What should I do now? As the panchayat have only provided oral instructions and no letter. The neighbour is also harrassing my construction workers due to my absence.