SAF Improves Reinforcement Learning with On-Policy Distillation

This paper introduces SAF (Stable Advantage Fusion), a novel framework designed to effectively combine Reinforcement Learning with Verifiable Rewards (RLVR) and On-Policy Distillation (OPD). While both techniques are complementary—RLVR provides response-level rewards and OPD offers dense token-level advantages from a stronger teacher—their direct fusion often leads to "entropy collapse," where the model's exploration capacity is severely limited. The core problem lies in two mismatches: a magnitude mismatch, where OPD advantages can overwhelm RLVR signals, and a temporal mismatch, where continuous strong OPD guidance hinders exploration. SAF addresses these issues through a lightweight, four-stage pipeline applied specifically to the OPD advantage. This pipeline includes a sparsify-then-compress mechanism for magnitude control and a warm-up-then-anneal mechanism for temporal control. Each stage is independently switchable and adds minimal overhead. Evaluated with GRPO (a type of RLVR) across seven mathematical reasoning and code generation benchmarks using Qwen3 models, SAF consistently avoids entropy collapse. It outperforms fixed-coefficient GRPO+OPD fusion, improving aggregate scores by 0.51-2.70% across various model-domain settings and achieving more stable training. This demonstrates SAF's effectiveness in leveraging the strengths of both RLVR and OPD for enhanced reinforcement learning.