New Method Accurately Attributes LLM Evaluation Drift to System or Judge

Continuous evaluation of Large Language Model (LLM) products often relies on another LLM acting as a "judge" to score performance. However, if this judge model itself changes—due to updates or prompt modifications—it creates ambiguity: a detected performance drop could be either a genuine product degradation or merely a shift in the judge's scoring criteria. Researchers have developed a new framework to resolve this ambiguity, providing "anytime-valid attribution." This method involves a fixed set of human-labeled "anchor" examples that the LLM judge re-scores periodically. By comparing the judge's scores on these anchors against human labels, the system can determine if the judge's behavior has changed. The framework employs a statistical process to attribute drift to either the LLM product ("system") or the evaluation model ("judge"). Experiments demonstrated its high accuracy in identifying judge drift from silent version updates and prompt changes, with significantly fewer false alarms compared to conventional rolling z-tests. This robust attribution mechanism was validated across different domains without re-tuning.