Show Cause Notice for possible impounding of Passport

I got my Passport reissued in Sept 2012 and after 1.5 month (Nov 2012) joined a company abroad. After 15 days of leaving India police came for verification and submitted adverse report stating I am personally not present for verification as I am abroad. In 2014, I got an objection letter and I was asked to furnish explanation and PP Form for re-verification. I have submitted my response and also clearly given proof that I am staying in abroad and my status is NRI. Also requested if anything is required from my side please do inform. I got no response to my submission in 2014 and after 2 years on 8th Aug 2016 I got show cause notice asking for explanation why the passport be not impounded. I was asked to be personally present at passport office. I flew back immediately and I have been asked to submit an affadivite explaining the reason and fill PP Form for verification again. I have informed that I am staying abroad since Nov 2012 still I was told to do the police verification. So I waited and done the verification and came back as my daughter was not keeping well. Passport officer's attitude was very rude and not even telling me the process and very threatening. My question is that in Police report they will mention that I was staying of the particular address in 2012 and they have verified me but again the report will be adverse as in 2016 I am not staying on my permanent address in India. I am worried what and how this matter is going to close.