New Method Improves RLHF Stability with Uncertainty-Aware Reward Models

Reinforcement Learning from Human Feedback (RLHF) is a critical technique for aligning large language models (LLMs) with human preferences. This process typically involves training reward models on human preference data and then optimizing LLM policies to maximize these predicted rewards. However, this pipeline faces two significant challenges that can lead to instability. First, standard reward models often act as deterministic point estimators, unable to signal when their predictions are unreliable. Second, modern group-based policy optimization methods, such as GRPO, treat all reward signals uniformly during advantage computation, potentially amplifying unreliable estimates. As LLM policies explore a wider range of responses, these limitations create a vulnerability where inaccurate reward estimates can exert undue influence, leading to a phenomenon known as reward hacking. This occurs when the model learns to exploit flaws in the reward function rather than genuinely aligning with human intent. To address these issues, researchers propose Uncertainty-Aware Reward Modeling (UARM). UARM enhances reward models by providing them with calibrated uncertainty estimates, achieved through quantile-based conformal prediction. Furthermore, it reweights GRPO advantages using heteroscedastic variance decomposition. Experiments conducted on datasets like HelpSteer, UltraFeedback, and PKU-SafeRLHF demonstrate that UARM substantially improves reward model calibration, effectively reduces reward hacking, and ultimately enhances the quality of downstream alignment compared to standard GRPO and other uncertainty-agnostic baselines.