DPO Beta Parameter Entangles Optimization and Preference Scales

Direct Preference Optimization (DPO) is a widely adopted method for aligning large language models using human preference data. A key parameter in DPO, beta (β), is commonly understood to regulate the KL divergence constraint relative to a reference policy. However, this research uncovers a more complex role for beta. The study demonstrates that beta actually serves two distinct functions: it governs the effective inverse preference-noise scale, influencing how strongly preferences are weighted, and simultaneously rescales the optimization process, linking this scale directly to the effective step size. This entanglement has significant implications, causing the achieved policy deviation to behave non-monotonically with beta, vanishing at small values, peaking at intermediate ones, and decreasing again at larger values. Furthermore, standard DPO loss values become incomparable across different beta settings, making it difficult to assess model performance consistently. To address these issues, the researchers propose a centered-softplus reformulation. This alternative objective is mathematically equivalent to DPO for positive beta but explicitly separates the inverse preference-noise scale from the learning rate effects, simplifying hyperparameter tuning and enabling more stable learning-rate schedules. It also provides a continuous endpoint for beta approaching zero, reducing to a linear preference-margin objective.