Self-Distillation Gains Questioned in LLM Training

On-policy self-distillation (OPSD) is a technique where a student model is trained on its own responses, guided by token-level supervision from the same model conditioned on "privileged reference information." The common belief is that performance improvements from OPSD stem from the student learning this additional, privileged information. This paper critically re-examines this assumption. The researchers conducted OPSD experiments using Qwen3 models (1.7B to 8B parameters) on science and mathematics datasets. Their analysis framework aimed to differentiate between supervision from the reference and supervision from the teacher model without the reference. Surprisingly, the correct reference information did not consistently provide a performance benefit across different teacher generation modes, model sizes, or datasets. Students were observed to improve even without the correct reference, and in some cases, a solution from an unrelated problem could even outperform the correct solution on mathematical reasoning benchmarks. The study found that the student's predictions aligned more strongly with the base model's inherent reasoning behavior than with the supervision induced by the reference. Furthermore, stronger alignment attributed to the correct reference did not reliably correlate with greater performance benefits. These findings suggest that OPSD gains might often be due to the recovery of existing reasoning capabilities rather than the acquisition of new knowledge from privileged information.