Filtering On-Policy Distillation Improves LLM Reasoning.

On-policy distillation (OPD) is a common technique for post-training language models, where a student model learns from a teacher's token-level supervision on generated trajectories. However, a key limitation is that teacher-derived rewards don't always accurately reflect genuine reasoning progress; a reasoning step might be correct but receive a lower reward simply for deviating from the teacher's exact output. To overcome this, the researchers propose R2-OPD (Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation). This method constructs two internal rankings of reasoning spans within a trajectory: one based on teacher rewards and another based on an independently estimated "progress reward." When these two rankings disagree, the distillation rewards from the teacher are selectively suppressed. This ensures that the student model receives more accurate guidance, leading to consistent improvements in reasoning performance compared to standard OPD.