1 yes you can refuse to go to police station since the 4 years has lapsed there is no proof of transaction he cannot recover the amount further no.criminal case is made out. Police in case of complaint gives you written notice.
2. There is no ground as such with him to file a case against you.
3. Since there is no agreement , and he has backed out of deal you can forefiet the amount.
The constable is just trying to pressurise you.