Rater State Bias Identified in RLHF Preference Data

Reinforcement Learning from Human Feedback (RLHF) relies heavily on pairwise preference labels to train AI models. However, new research highlights a significant, yet overlooked, source of bias: the "rater's state" during annotation. This refers to how an annotator's preferences can subtly shift over time, particularly under sustained stressful or distressing conditions, leading to preference data that reflects their emotional state alongside the actual quality of the AI response. Unlike random noise or ordinary disagreement, rater state shifts are state-dependent, can be shared among annotators working in similar conditions, and can propagate through the entire RLHF pipeline, from reward modeling to policy optimization. The paper formalizes this concept, defining rater state shift, rater state confound, and correlated rater state bias. It also introduces "survival level emotional authenticity" as a measurable response pattern. To address this, the researchers propose a comprehensive audit framework. This framework includes five falsifiable predictions and effect size thresholds for initial audits, along with a detailed audit protocol and pilot study plan applicable to publicly available instruction-tuned models. The goal is to isolate and understand this plausible source of structured bias, emphasizing that it's a testable hypothesis for improving the robustness and fairness of RLHF systems.