Removing soil from my property which is the boundry of my neighbor at 10 ft higher

After purchase of a plot of 10 cents in a panchayat area, I tried to erect a compound wall at 4 sides. Unfortunately, at the backend boundry, a tiny piece of land of mine, with a dim. 42 x 5 ft is lying hardly 10 ft above my plot level. This piece was kept as it is, without removing the soil, as per the warning of the neighbor at that boundry, for the reason he said was, that his boundry is at 10 feet higher, so the soil will collapse, and therefore he will go to the court and a stop memo will be issued against my house building. Actually the type of soil is hard so no chance for slipping the soil, still I pleaded him several times so politely to allow me to remove the soil and complete erecting of my compound wall, and in any condition, I will make a concrete compound wall in any strength he desires, without any loss of time. Also I showed him the model works at the neighborhood area that I am going to do after removal of my soil, but he is not agreeable to it, appears to be so adamant, and started threatening me. Sir, if I remove the soil so as to make it to the level of my plot with a sincere intention to erect it with concrete for protecting his boundry from slipping of soil, what will be the actual consequences that I likely will have to face, for the time being and also in future ? Is there any loophole to fully utilize my land lying at the extreme back end adjacent to his boundry ? Kindly advise me in this regard.