AI Judge Panels Need Targeted Verification for Accuracy

Large Language Model (LLM) judge panels are commonly used for evaluation, but previous studies have shown that their errors are highly correlated, meaning many judges provide little more effective information than a few. A natural solution, like adding a signal from an independent source (e.g., a test suite), was previously thought to have little impact on the panel's overall effective vote count. This new research challenges that view by demonstrating that aggregate metrics miss where verification truly helps. The study found that accuracy gains from an independent signal are entirely concentrated on decisions with a one-vote margin, which are the only ones that can be changed by a single ballot substitution. For these pivotal queries, accuracy improved significantly (10.4 to 23.3 percentage points). This targeted approach, applying a majority-side replacement rule, raised overall accuracy while only invoking the signal on a small percentage of queries. The findings suggest that focusing verification efforts on close decisions is far more effective than broad application.