RLHF Preference Averaging Causes Procedural Fairness Failures

Reinforcement Learning from Human Feedback (RLHF) is a common technique for aligning AI models with human values. However, this paper highlights a critical flaw: when human preferences are diverse, the standard method of averaging these preferences into a single reward model leads to procedural unfairness. This means that the preferences of majority groups tend to dominate, while those of minority groups are systematically overlooked. The researchers define procedural fairness in AI alignment as the preservation of distinct preference signals during the reward modeling process. They demonstrate that the conventional RLHF approach, through its preference averaging, inherently violates this principle. To counter this, they propose Preference-Aware RLHF (PA-RLHF), an alternative method that optimizes across different preference modes separately during the reward learning phase. In controlled experiments, PA-RLHF significantly improved overall alignment accuracy from 46.9% to 67.9%. Crucially, it also reduced the fairness gap between the best and worst-aligned groups from 15.9 to 9.6 percentage points. These findings underscore that fairness issues in AI alignment can stem from fundamental design choices in reward learning, even in ideal, noise-free environments, with significant implications for large language models and autonomous agents where biased reward models can exacerbate inequalities over time.