LLM Judges Need Strict Rules for Reliable Reasoning Pipelines

Large Language Model (LLM) judges are increasingly used within AI reasoning pipelines to evaluate and select answers. This paper argues that simply deploying an accurate LLM judge is insufficient; the decision-making framework it operates within is crucial. The cost of a wrong decision by an LLM judge can be high, and its accuracy alone doesn't guarantee better outcomes. The researchers introduce Evidence-Locked Derive-Gate-Repair (EL-DGR), a novel, task-adaptive rule that constrains the LLM judge's power. Under EL-DGR, a judge's preference can only override evidence-supported consensus if it provides an extractive evidence certificate. Repairs are only allowed if neither alternative is certified and the repair itself is. This approach significantly improves performance on tasks like GSM8K and HotpotQA without changing the judge model, candidates, or budget. The key takeaway for practitioners is to focus on bounding the judge's "blast radius" through robust admissibility rules rather than solely striving for higher judge accuracy. The study found that EL-DGR rarely overturns correct consensus and never converts a correct answer into an incorrect one, demonstrating the value of structured decision rules over unconstrained LLM judgment.