First things first, a family settlement can also be oral. Even when it is a mere memorandum memorialising the arrangement that has already been effected purusant to an oral settlement, it need not be registered.
But where a settlement is reduced to writing and ends up creating and extinguishing any rights in immovable property, it definitely needs to be registered and stamped. And unless so registered and stamped, it is inadmissible in a court (except for collateral purposes).
Please see Kale and others vs. Deputy Director of Consolidation and others, (1976) 3 SCC 119.
In case the document is registered but not stamped or stamped inadequately, it can still be presented in court subject to the first proviso of Section 35 of the Indian Stamp Act. Pu simply, the document is impounded in such cases and you are required to pay the requisite stamp duty along with any penalty. On such payment, it becomes admissible.
I would advise you to have an expert (either here or offline) have a look at the document so as to better advise you on what needs to be done in your case.
I hope that answers your question. Follow-up questions welcome.