SARA Mitigates Rubric Interference in LLM Evaluation.

Large Language Models (LLMs) are increasingly used as judges to evaluate responses against detailed rubric checklists. A common practice is to assess each rubric in a separate inference call. However, evaluating multiple rubrics in a single pass, while more efficient, can lead to "rubric interference," where the judgment on one criterion is influenced by the presence of other criteria. A preliminary study revealed that only a third of samples received consistent verdicts when rubric sets varied. To address this, the researchers developed a measurement framework to systematically probe interference through operations like rubric set expansion, subsetting, reordering, and noise injection. They then propose Self-Anchored Rubric Alignment (SARA), an unsupervised method that mitigates this interference. SARA leverages the model's own single-rubric judgments as stable reference points and uses on-policy self-distillation to align the model's multi-rubric reasoning with these anchors. Validated across multiple datasets (HealthBench, FLASK, ResearchQA) and model families (Qwen3, Llama-3.1), SARA consistently improved evaluation consistency. It also maintained strong agreement with both base models and GPT-4.1, which served as a reference judge. The learned consistency was shown to transfer across different datasets, indicating that SARA teaches a general capability rather than merely fitting specific patterns.