Blockwise Gating Improves On-Policy Distillation Robustness

On-policy distillation (OPD) is a technique where a student policy learns from a teacher's signals generated on trajectories sampled by the student itself. However, for complex, long-horizon reasoning tasks, sampled-token OPD can be fragile, leading to inconsistent performance. This paper proposes "blockwise policy-drift gating" to enhance the robustness of OPD, particularly when reusing rollouts. The method involves calculating log-probability shifts between the behavior student policy (which sampled the data) and the current student policy. These shifts are then aggregated over fixed blocks of tokens and used as detached, mean-normalized gates to reweight the OPD position losses. Crucially, this technique does not alter the teacher's targets or the rollout policy. In evaluations on a Qwen3 math reasoning benchmark, fixed 64-token block gating improved the mean pass@8 score for sampled-token OPD from 0.4978 to 0.5160 across several challenging math datasets. The results highlight that controlling local old-current policy drift is a practical signal for improving solve-rate robustness in OPD.