Self-Distillation Fails in Complex Tasks Due to Teacher Bias.

Self-distillation (SD) is a technique where a "teacher" model, privy to privileged information like a reference solution, provides dense supervision to a "student" model. While initially showing promise in simple scenarios as a compute-efficient alternative to reinforcement learning, this study reveals significant limitations when applied to more complex tasks. The research demonstrates that in challenging domains such as question answering, mathematics, coding, and multi-turn agentic tool use, self-distillation consistently fails to improve validation accuracy, often leading to degradation, even as the per-token loss decreases. This failure is attributed to a "privileged information bias." The teacher, having seen a specific correct trajectory, guides the student towards replicating that particular path rather than understanding general correctness. Consequently, the student model's objective becomes decoupled from actual task success. It learns to optimize for low-information tokens (like stopwords or punctuation) and even penalizes exploratory tokens necessary for reasoning, resulting in a less decisive model that does not genuinely improve its reasoning capabilities.