Such exchange is not legal, This could have done by executing 2 separate gift deed/sale deed. Let him approach court, he has no legal title on that land.

deed of exchange should be executed duly stamped and registered

2) if other party is objection to carrying on construction on land ask him to return your land and take his land in return 

3) since you do not want to go to court amicable settlement is best option 

Dear Sir,

You have perfected adverse title to the true  owner as such you may file suit for declaration seeking injunction. If the other party insisting for Out of court settlement , he may take some Undertakings and admissions from you and you will be thrown out of the possession since you are  not having any exchange deed . Under These circumstances it may not be possible to go for out of court settlement 

See you both can make a compromise deed you both can make a registered exchanged deed you can pay stamp duty in same and can register the properties on your names. 

you cannot carry on construction on land that you are not the owner 

2) for sanction of building plans municipal corporation or panchayat would insist on signature of owner of land 

See if the other person is disputing the title and since you are in possession of the said property from last 50 years over same then you can claim adverse possession over the property in case he files a suit. 

Who is in possession of that house, if lyign vacate without electricity and water connection can take possession and demolish it.

This is very tricky situation. You can't do anything if he goes to court. In that case you need to prove your case with witness only. 

1.  FIRSTLY House construction is illegal, IF constructed on Agricultural Land, that too on other persons land without any available registered legal documents.

2.  SECONDLY you need to regularize the exchange of land between the two owners, by executing a proper stamp duty paid registered agreement.  This is necessary for future court disputes.

3. THIRDLY, agricultural land has to be converted into non-agricultural land, to build a legal house for all futuristic purposes.

4.  IF there was no registered agreement about the exchange of lands, THEN it becomes a criminally prosecutable  matter.

CONSIDER all above and try amicable settlement, before proceeding further, for demolition or house construction.

without legal documents you cannot construct on said land. better settle dispute amicably. 

You can file case for ownership of that land by adverse possession from last 50 years.