Calibrating LLM Evaluators Mitigates Preference Coupling in Feedback Loops

This research investigates a critical issue in LLM agent development: "evaluator preference coupling," where systematic biases in an LLM evaluator's feedback propagate into the agent's learned behavior. Previous work identified this problem and provided a diagnostic framework, but this study is the first to explore mitigation strategies through evaluator calibration.The authors propose applying probability calibration to the evaluator's pairwise judgments to reduce the propagation of spurious preferences. In a controlled experiment, they compared standard binary TTRL (win/loss) with a confidence-calibrated TTRL (using probability-weighted updates). DeepSeek-V4-Pro served as the executor agent, and GLM5.2 as the evaluator.The results demonstrated a significant reduction in evaluator preference coupling. Calibration decreased the coupling coefficient gamma by 20-49% and the Jensen-Shannon divergence by 45-67%. A symmetric-LR control confirmed that these effects were genuinely due to calibration and not merely reduced update asymmetry. The study releases the calibrated TTRL protocol, recommending it as a lightweight mitigation for LLM-as-judge deployment pipelines.