SMOPD Boosts Multi-Reward Reinforcement Learning Performance

In multi-reward reinforcement learning (RL), a common challenge is balancing different reward signals, especially when they vary in granularity (e.g., dense vs. sparse rewards). Existing methods like Group reward-Decoupled Normalization Policy Optimization (GDPO) attempt to mitigate this by normalizing rewards separately, but they can still struggle when sparse rewards provide insufficient optimization signals, preventing the corresponding capabilities from being effectively learned. To overcome this limitation, researchers propose Specialize-and-Merge Online Policy Distillation (SMOPD), a novel two-stage training methodology. The first stage, "Specialize," involves training multiple "teacher" policies, each configured with reward priorities to specialize in optimizing a single reward dimension. This ensures that each reward, regardless of its granularity, receives sufficient signal to drive effective learning for its specific capability. The second stage, "Merge," then uses online policy distillation to combine the specialized capabilities learned by these individual teachers into a single "student" policy. This merging process is designed to maintain balanced task-level optimization across all rewards. Experiments conducted on multi-reward settings, including complementary and conflicting rewards, demonstrate that SMOPD consistently outperforms GDPO across various backbone sizes (1.5B, 3B, and 7B), indicating its effectiveness in handling diverse reward structures.