Diverse User Preferences Improve AI Alignment Through Curriculum Learning

Aligning AI systems with individual user preferences, often achieved by training a reward model from human feedback and optimizing a policy against it, faces a significant challenge: not all user reward models are equally easy to optimize. Some users' preferences might be straightforward to satisfy from an initial policy, while others are much harder, potentially leading to an "underserved group" even with accurate per-user reward models. This paper posits that a sufficiently diverse user population naturally creates a curriculum, ranging from easy-to-optimize to hard-to-optimize reward models. Building on this insight, the researchers propose CurriPO (Curriculum Policy Optimization), a method that constructs a tree-structured curriculum to manage diverse user-specific objectives. CurriPO allows the learning process to branch from existing curriculum paths and reuse previously incorporated reward models, covering the entire user population efficiently. Experiments in personalized continuous control simulations demonstrated that CurriPO achieved 1.2 to 2.1 times higher population satisfaction compared to strong baselines, while also substantially reducing training time. This improvement is largely attributed to better serving users who would typically be underserved by conventional optimization methods.