REOPD Improves On-Policy Distillation with Adaptive Reward Extrapolation.

On-policy distillation (OPD) is a technique where a student model learns from its own generated trajectories, guided by dense, token-level supervision from a more capable teacher model. While methods like ExOPD amplify the teacher's log-likelihood ratio to push the student beyond mere imitation, they often apply a uniform amplification coefficient. This global approach can lead to "reward hacking," where the student overfits extreme reward peaks, causing instability and requiring extensive hyperparameter tuning. This research proposes REOPD (Reliability-Adaptive Reward Extrapolation for On-Policy Distillation) to address these issues. REOPD introduces a dynamic, token-wise coefficient that adapts based on the reliability of the teacher's guidance. It combines a token-level compatibility weight with a batch-level adaptive budget, allowing for selective extrapolation along directions where the teacher's reference is most reliable. Crucially, REOPD achieves this without needing additional components like verifiers, reward models, value models, or extra rollouts beyond standard OPD. Experiments demonstrated that REOPD consistently outperformed G-OPD in single-teacher mathematics and multi-teacher settings, while matching G-OPD's performance in single-teacher code generation, showcasing its effective fine-grained reliability adaptation across diverse domains.