RGPO Aligns LLMs with Inconsistent Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is a cornerstone for aligning Large Language Models (LLMs) with human preferences. However, the effectiveness of RLHF is frequently undermined by the inherent inconsistencies and subjective nature of human annotations. Current preference optimization methods, such as Direct Preference Optimization (DPO), often treat all feedback equally, regardless of annotator disagreement, leading models to overfit to noisy or inconsistent supervision signals and resulting in suboptimal alignment. To address this critical limitation, researchers have proposed Reliability-Guided Preference Optimization (RGPO), a robust framework designed to reduce the impact of inconsistent human feedback. RGPO operates by first estimating the reliability of individual annotators and then inferring latent ground truth labels from the noisy human feedback. This process helps to identify and prioritize more robust preferences. Furthermore, RGPO introduces a reliability-aware consistency optimization component. This mechanism dynamically adjusts the training objective based on the consensus level of the annotations, ensuring that the model places greater emphasis on high-consensus supervision signals. Extensive experiments on LLM alignment benchmarks demonstrate that RGPO effectively reduces inconsistency and noise in training data, achieving superior performance compared to widely adopted RLHF baselines.