Ancestor Property dispute

Hello Sir / Madam, I’m SK Patil, staying in Satara and have 2 lands in village near by Karad. Both these lands (each 850 sq. ft) is ancestor’s land and we 4 brothers have equal share in it. Around 15 years back 2 of my brothers wanted to build a house on one land, so we granted permission with the mutual understanding that the other land will be used by rest 2 brothers to build the house. As agreed, the 2 brothers built the house 15 years back and are paying corporation tax. Recently my elder brother and I decided to built the house on the land which was in our share, but now the other 2 are asking their share in it. As they already have the land and built the house, they are still asking for the share in our land which was assigned to us on mutual understanding. There is no paper agreement for this and both lands (7/12 utara) is on all 4 brother name. Now we have decided to give them their share in our land, but planning to take our share from the land which was assigned to them. Is it possible that I have a chance to get back the share from the land on which they have already built the house. The Land belongs to all 4, but they have built the house. Please advise. Thanks, SK