Counterfactual Recoverability Improves On-Policy LLM Distillation.

On-policy distillation (OPD) involves a student model learning from a teacher model by observing the teacher's actions on paths the student generates. A key challenge is deciding when a student's deviation from the teacher's path is a critical error that needs correction, versus a minor divergence that the student can still recover from. Current divergence-based rules often fail to make this distinction. This paper proposes the concept of "counterfactual recoverability" to address this. It involves replaying each error state through two hypothetical branches: one where the teacher continues the trajectory, and another where the student "rolls back" and tries again. By comparing the success of these branches, states are categorized as recoverable, irreversible-but-avoidable, or ambiguous. These labels then inform whether the training process should retain, roll back, or conventionally supervise the corresponding trajectory. Experiments demonstrate that this recoverability-aware control significantly outperforms existing baselines, achieving stronger performance on challenging reasoning benchmarks like AIME and GPQA-Diamond. The findings highlight that retaining teacher-correctable prefixes provides the largest individual contribution, establishing recoverability as a crucial, outcome-grounded decision variable for more effective selective supervision in OPD.