New Protocol Standardizes LLM Evaluator Bias Measurement

When Large Language Model (LLM) agents adapt their behavior based on evaluator feedback in closed loops, biases inherent in these evaluators can propagate and influence the agent's strategy distribution. This phenomenon, known as evaluator preference coupling, has been observed across various evaluator families and model versions. However, the field has lacked a standardized method for researchers to consistently measure, compare, and reproduce these coupling effects, especially as proprietary evaluators undergo silent updates. This paper introduces EPC (Evaluator Preference Coupling), a detailed, RFC-style protocol specification designed to address this gap. EPC outlines a four-phase isolation paradigm, covering executor and evaluator configuration, strategy and task design, the TTRL update rule, and specific metric computations (gamma, JSD, ECE, Brier). It also defines a clear output schema, ensuring consistency across studies. Accompanying the protocol is a versioned Reference Snapshot v1.0, which includes coupling measurements for eight evaluator conditions derived from five independent studies, featuring models like GPT-4o, Qwen, and DeepSeek. This snapshot is time-bound, acknowledging that proprietary evaluator updates will cause these values to decay. The authors also provide a versioning convention and a usage guide to facilitate adoption and interpretation, making EPC a crucial piece of open infrastructure for LLM agent research.