Normalized Rewards Improve LLM Preference Optimization

Direct Alignment Algorithms (DAAs), such as DPO, are widely used to align large language models (LLMs) with human preferences. However, these methods can sometimes over-optimize their implicit reward models, leading to a decrease in the overall likelihood of preferred responses and potentially undesirable model behavior. This research investigates the impact of incorporating a regularization term that maintains the total length-normalized probabilities of both chosen and rejected responses. By analyzing how response likelihood changes are distributed, the study found that a small set of outlier tokens often accounts for a significant portion of these changes, explaining how DAAs improve quality despite reducing overall likelihoods. Applying this proposed regularization to both reference-based (DPO) and reference-free (SimPO) methods yielded notable improvements. For instance, on Llama-3.1-8B-Instruct, it resulted in over 20% relative increase in AlpacaEval2 scores and more than 9% relative gains on general benchmarks. The regularization effectively mitigated likelihood displacement, particularly for outlier tokens, demonstrating a better balance between generation quality and general model capabilities.