New Method Calibrates RLHF Reward Models with Per-Rater Shrinkage

Reinforcement Learning from Human Feedback (RLHF) relies on reward models trained from human preferences. A common challenge is that these models often use a single, global calibration across all annotators, which fails to account for systematic differences in how individual raters use rating scales. This can lead to inaccuracies because no single average fit truly represents any individual annotator's behavior. A new method, PEBS (Per-rater Empirical-Bayes Shrinkage), addresses this by introducing a post-hoc estimator. It fits unique affine calibrators for each annotator using a small, held-out portion of their ratings. These individual calibrators are then refined using Morris-James-Stein empirical-Bayes shrinkage, pulling them towards the population mean without retraining the core reward model. Evaluations on datasets like PRISM and PluriHarms showed that PEBS significantly improves accuracy, reducing within-user held-out RMSE by over 8.5% compared to the standard population-slope baseline. This approach offers a closed-form solution for annotator-specific calibration, enhancing the reliability of RLHF systems by better accounting for human variability.