RoPoLL Improves LLM Evaluation by Robustly Aggregating Judge Scores.

This research introduces RoPoLL, a "Robust Panel of LLM-as-Judge" system designed to significantly improve the reliability of evaluating Large Language Models (LLMs). While using a panel of LLM evaluators (LLM Jury) has become a popular alternative to single-judge evaluations, the paper highlights a critical flaw: traditional consensus methods can suffer from unbounded bias if even one judge exhibits typical LLM failures such as mode collapse, sycophancy, or safety refusals. RoPoLL addresses this by reframing jury consensus as a robust mean estimation problem. Instead of simple averaging, it employs the geometric median (GM) as its aggregation function. The geometric median is tuning-free and boasts an optimal finite-sample breakdown point of 1/2, meaning it can tolerate up to half of the input data being arbitrarily corrupted without its estimate becoming arbitrarily bad. Extensive experiments across 13 open-weight LLM judges, three reward-model benchmarks, and four corruption regimes (up to 50% contamination) demonstrate RoPoLL's superior performance. It consistently dominates traditional panel methods on all types of biased corruption, showing improvements of approximately 19% on cross-dimensional attacks and orders of magnitude on heavy-tailed Byzantine adversaries. Notably, a small 3-judge RoPoLL committee using 38B parameters even outperformed a much larger 675B parameter model (Mistral-Large-3) by 1.31x on a specific benchmark under bimodal-random corruption, showcasing its efficiency and robustness. This work provides a statistically sound and practical solution for more reliable LLM evaluation.