New DPO Method Improves LLM Alignment with Noisy Preferences

Direct Preference Optimization (DPO) has become a popular method for aligning large language models (LLMs) with human preferences, largely because it bypasses the need for explicit reward modeling and complex reinforcement learning. However, the effectiveness of DPO is highly dependent on the quality of the preference data it receives. In real-world scenarios, noisy preference labels are common and can significantly degrade the alignment performance of LLMs. To tackle this challenge, a novel bilevel optimization framework is introduced. Under specific assumptions, this framework is proven to recover the DPO optimum even when the data contains noise. The researchers also derived a prior form for a learnable weighting function, specifically designed to handle asymmetric label-flipping noise. A key innovation is a task-agnostic meta-knowledge-driven method, which enables meta-learning even when high-quality metadata, often difficult to obtain, is completely unavailable. To mitigate the high computational cost associated with higher-order gradients in LLM meta-learning, the method combines central-difference approximation with LoRA fine-tuning, resulting in a scalable training scheme. Extensive experiments on TL;DR summarization and Anthropic HH single-turn dialogue datasets demonstrate that this proposed method significantly improves training performance compared to multiple DPO baselines, even under various noise rates.