1. As per the information mentioned in the present query, makes it clear that you have been troubled by all without any reason.
2. It totally depends upon the area in your possession, meaning here by is that the area which is now been called as extended by you, is exactly the part of the total area of your or it has extended outside of the end of the area of yours.
3. If answer is, if that is part of yours then no body can put worstion in it.
4. In fact, then you can go before the civil court of law under a suit for injunction from taking any action against your property.

You can seek interim injunction immediately  

did you apply for and obtain sanction of building plans from panchayat?

2) did you obtain completion certificate or OC from panchayat 

3) if you have carried on construction as per sanctioned plans you have nothing to fear 

4) panchayat would not take any action against you for part extension of first floor till compound wall 

5) if no permission was obtained apply for regularisation before panchayat. Pay regularisation charges 

you have not carried on any construction outside your compound wall 

2) as mentioned herein above if you have carried on construction as per sanctioned plans you have nothing to fear 

3) as far as BBMP rules are concerned local architect can guide you 

Dear Sir,

It is common sense that when you have not extended any portion of your first floor beyond your area then how can that builder threaten you. You just approach the Civil Court and get a stay on the threatening of the builder. Please the law is as follows.

BANGALORE MAHANAGARA PALIKE BUILDING BYE-LAWS 2003 (Approved by the Government in their Order No.UDD/223/MNU/2001, Dated [deleted] – Published in pursuance of Section 428 Of the Karnataka Municipal Corporation Act 1976)

http://bbmp.gov.in/documents/10180/504904/Bangalore-Building-Byelaws-+2003.pdf/95195b55-ef62-4b68-bb9e-dc794344c18a

Sir see it is mandatory to maintain gap even in the first floor also even it is in your land limits. So it better to settle with the land owner so that there is no adverse order. Legally the gap needed to be there.

See if the plans are approved you can take stay from the civil court there no other way out .

You don't have to worry it's not.you can approach high  court for he same

Please note that if you have constructed your house as per the map duly passed by the authority then nothing to worry if not then try to get it approved by the authority because everything will be done by the Development Authority. 

First of all confirm the local rules in this regard.

If you are being threatened by anyone on this, you may first issue a legal notice to them instructing them to refrain from indulging in such illegal activities and to follow due process of law for any such thing.

Besides you can file an injunction suit against them restraining them from interfering in your peaceful possession and enjoyment of your property other than by due process of law.

once you knew that you had to leave setback, what is your doubt about the same with the construction of the first floor.

Thogh you never bothered about the rules earlier at least now you confirm without taking decisions on your presumption.