Firslty, Sir, every state has their own law with regard to the construction of residential purpose in a portion of the agricultural land.
Secondly, in fact it was allowed long time back when it was needed to have someone watching on the crops so should have a small house on it.
Thirdly, and no permission is required only an affidavit need to be given to the local land owing agency in your area for constructing the same disclosing the reason for the same.