Am I legally bound to leave a setback/margin area of 4 feet of my 20 x 50 feet land area in village

Respected Sir/Madam, I have a small 20x50 feet area of residential land in village Arwal district of Bihar .I already left area of 3 feet in east side ,2 feet in south side, 3 feet in west side as me and my partner is agreed as same. Now as North side my neighbor is not left a single inch of margin in my side even though he made house in my land excess of 4 feet without having any land in my side since they had made their house around 15 years ago but that house is not concrete only mud and soil type with rock.Since that time my family are not knowing rule and we are single family member so we not objected that time. Now i am making a house in that land and I do not want to leave margin in north/their side then they started quarreling with us and they remove my rock and cement .They are telling leave 4 feet margin otherwise we will not allow to make you house and they are not listing us . Since they also not left any margin at my side then why i should leave . Also they having well access of street in east side as i have.I tried to manage through talking but they are not listing , then i told ok i will left 1 feet then again they are telling i take 5 feet of margin . Since they already encroached 4 feet of land in my side ,as my land is small and already i left 3 side margin with mutual agreement and north side they not left any margin then why should i leave their side since my land is small . If i will measure through land map then my land is going in their house about 4 feet .Their are no provision of street in north side as i and they already have street and all access in east side. Now thing is that i am a alone person and leaving 2500 km far from my village and working as a scientist in govt of India. My old father is leaving at my village .They having 8 son so they are threatening and harassing my family.I also do not want any quarrel with them but they are objecting to make house and threatening to working labour also, so nobody is coming to work..So i want to take legal step and want to solve situation peacefully.Please suggest me a way and clear confusions given below Now I want to ask you ,is it is true i am legally bound to leave margin in north side also even though they not left margin and also they had taken excess land of 4 feet in my side as per land map and made raw house 10 year ago ? If i will go against them in court then is it true court will ask me to leave margin of 4 feet in my side even though opposite party had not left margin and they encroached my 4 feet land then also. Is setback/margin rule is applicable in village also even though no body is following then court will ask to follow me in village ? Is Bihar building and bye law 2014 is also applicable for village also.If it is applicable then how i am legally bound to leave margin/setback for neighbor if they not left ? Please suggest me solution what should i do ? Thank you very much in advance.