Under the Right to Information Act, 2005, the procedure is mostly document-based, not oral. A hearing is not mandatory for the applicant. So you are not legally required to attend unless you yourself want to clarify something.
There is no provision allowing the department to force you to attend a hearing before giving the information.
You can send a reply in writing insisting that the decision on the first appeal must be given in writing, the information must be supplied as per your RTI request and a hearing cannot replace a written RTI reply.
You can send a letter in the following manner
Subject: First Appeal under RTI Act – Reg.
I refer to your notice calling me for a hearing regarding my First Appeal under the RTI Act.
Due to personal reasons I am unable to attend the hearing. I request that the appeal may kindly be decided on the basis of the records available and that the information sought in my RTI application be provided in writing as required under the RTI Act, 2005.
Kindly communicate the order of the First Appellate Authority in writing.