PowerOPD Stabilizes On-Policy Distillation for Large Language Models

On-policy distillation (OPD) is a technique used to train smaller "student" large language models (LLMs) from larger "teacher" models, typically by estimating a reverse-KL objective using student-sampled tokens. While this approach avoids computationally expensive vocabulary-wide calculations, it often suffers from severe training instabilities, including sample inefficiency and unstable generation dynamics. Researchers have identified the root cause of these pathologies: the unbounded nature of the log-ratio reward used in vanilla OPD, which generates extremely high-variance gradients concentrated early in the training process. Standard post-hoc scaling methods are ineffective because they operate after this distortion has already occurred. To mitigate this, the paper proposes PowerOPD, a novel family of natively bounded, sign-consistent rewards derived from the Box-Cox power transformation. This transformation, parameterized by alpha, effectively bounds the reward, with the log-ratio being its degenerate limit as alpha approaches zero. PowerOPD significantly improves performance on mathematical reasoning benchmarks, achieving substantial gains over vanilla and even full-vocabulary OPD, while also reducing training time and GPU memory usage.