New Method Improves LLM Alignment with Robust Listwise Preference Optimization.

Existing robust preference optimization techniques for aligning language models primarily focus on pairwise supervision and apply robustness at the dataset or prompt level. This research shifts focus to listwise preference optimization, specifically addressing the uncertainty that arises in ranking labels. Such uncertainty can stem from annotator inconsistencies, near-ties in preferences, or noise within the reward model. The proposed method introduces a pointwise total-variation robust Plackett-Luce objective, which directly robustifies the ranking label conditioned on a candidate list. This robust loss can be exactly decomposed into the nominal Plackett-Luce loss plus a worst-case correction, where the worst-case ranking is efficiently found by sorting current implicit scores. This tractable structure provides strong optimization guarantees for both offline and online settings. Experiments in offline LLM alignment demonstrate that this robust correction maintains performance with clean labels and significantly enhances robustness when labels are noisy. In online alignment, it makes reward-model-ranked candidate expansion more reliable, leading to improvements in both reward-model and external GPT-4 judge metrics.