There are certain instances few and far between where the court imposed fine but it all depends upon the facts and circumstances of the case.

Regards 

In a one of its kind order, the Delhi High Court has imposed a cost of Rs 50,000 on a woman for filing a false complaint of sexual harassment against her senior official.

Judgement - imposition of costs is a necessary instrument to prevent the filing of frivolous cases.

You don't need a judgement for that it's a courts discretion if it primafacie feels the same to be frivolous then they will impose cost

accused may file a petition for quashing of the FIR filed by the police in the pending case. thereafter if the case is in favour he may also initiate a suit for defamation and claim damages/compensation from the complainant.. 

14. Hon'ble Supreme Court in the matter of Vineet Kumar and others Vs State of Uttar Pradesh and another reported in (2017) 13 SCC, 369 while considering the issue of scope of section 482Cr.P.C regarding quashing of criminal proceedings held in paras 22,23,24,25,26,27,28, and 29 as under:

"22. Before we enter into the facts of the present case it is necessary to consider the ambit and scope of jurisdiction under Section 482 Cr.P.C. vested in the High Court. Section 482Cr.P.C. saves the inherent power of the High Court to make such orders as may be necessary to give effect to any order under this Code, or to prevent abuse of the process of any Court or otherwise to secure the ends of justice.

"23. This Court time and again has examined scope of jurisdiction of High Court under Section 482 Cr.P.C. and laid down several principles which govern the exercise of jurisdiction of High Court under Section 482 Cr.P.C. A three-Judge Bench of this Court in State of Karnataka vs. L. Muniswamy and others, 1977 (2) SCC 699,held that the High Court is entitled to quash a proceeding if it comes to the conclusion that allowing the proceeding to continue would be an abuse of the process of the Court or that the ends of justice require that the proceeding ought to be quashed. In paragraph 7 of the judgment following has been stated:

"7....In the exercise of this wholesome power, the High Court is entitled to quash a proceeding if it comes to the conclusion that allowing the proceeding to continue would be an abuse of the process of the Court or that the ends of justice require that the proceeding ought to be quashed. The saving of the High Court's inherent powers, both in civil and criminal matters, is designed to achieve a salutary public purpose which is that a court proceeding ought not to be permitted to degenerate into a weapon of harassment or persecution. In a criminal case, the veiled object behind a lame prosecution, the very nature of the material on which the structure of the prosecution rests and the like would justify the High Court in quashing the proceeding in the interest of justice. The ends of justice are higher than the ends of mere law though justice has got to be administered according to laws made by the legislature. The compelling necessity for making these observations is that without a proper realisation of the object and purpose of the provision which seeks to save the inherent powers of the High Court to do justice, between the State and its subjects, it would be impossible to appreciate the width and contours of that salient jurisdiction."

High court can quash FIR if false case is filed 

 unable to find judgments wherein penalty imposed  

1. Yes in some cases where criminal complaint or FIR is lodged just to put pressure in civil cases court have put cost on complainant.

2. Sorry I didn't find any precedents you can search for judgements on Indiankanoon com 

Harsh Vardhan Arora v. Smt Kavita Arora, 2002 MLR 528= 2002(2) RCR (Cr) 499 (Pb and Har.)

Mukesh Rani V. State of Haryana, 2002 MLR 175=2002 (2)Cr.CC123= 2002 (1) RCR (Cr) 163= 2002 (1) CC Cases (HC) 48 (Pb. & Har.)

Anu Gill V. State and Anr., 2001 MLR 467 = 2001 (59) DRJ 417 (Delhi)

Raj Pal Singh & Ors. V. State of Haryana, 2000 MLR 594= 2000(3) Rec. C.R. 135 (Pb. & Har.)

A FIR is bible for getting evidence and eventually to secure conviction of an accused based on the evidence, therefore a FIR must contain all the material facts related to an offence. It also must contain all the specific ingredient needed for satisfying an offence and material through which investigation may proceed. if a FIR misses material facts then it is a vague FIR which can be quashed by invoking the jurisdiction of 482 CrPC. Image result for 498a quash Some believe that if a FIR satisfies the ingredients of an offence it is not the case for being fit for quashing but this preposition is untrue under the light of the judgement In R.P. Kapur v. State of Punjab (AIR 1960 SC 866) the apex Court summarized some categories of cases where inherent power can, and should be exercised to quash the proceedings. (i) where it manifestly appears that there is a legal bar against the institution or continuance e.g. want of sanction; ii) where the allegations in the first information report or complaint taken at its face value and accepted in their entirety do not constitute the offence alleged; (iii) where the allegations constitute an offence, but there is no legal evidence adduced or the evidence adduced clearly or manifestly fails to prove the charge. so lack of evidence is another ground for quashing proceedings. A FIR containing quite vague, general and sweeping, specifying no instances of criminal conduct can be quashed even if the FIR constitutes and satisfy the ingredients of an offence. It is held in Vishalbhai Niranjanbhai Adatiya … vs State Of Gujarat & on 9 December 2015 It is a matter of common experience that most of these complaints under section 498A IPC are filed in the heat of the moment over trivial issues without proper deliberations. We come across a large number of such complaints which are not even bona fide and are filed with oblique motive. At the same time, rapid increase in the number of genuine cases of dowry harassment are also a matter of serious concern.

1. In 482 Cr.P.C the courts do not levy cost on the complainant while quashing the FIR.

2. Any relief which is not sought in the petition cannot be granted in the first place.

There are certain judgements of high courts wherein penalties were imposed in false matrimonial cases. You have to be specific. What is the offence.