Tail-Aware Distillation Improves LLM Knowledge Transfer.

On-policy distillation (OPD) is a recognized method for transferring knowledge between language models, where a smaller student model learns to mimic the next-token distribution of a larger teacher model along its own generated trajectories. To make this process computationally feasible and provide dense supervision, many current OPD approaches minimize the reverse Kullback-Leibler (KL) divergence over only the teacher's top-$k$ most probable tokens. However, this common practice inadvertently discards crucial information about the "tail probability"—the cumulative probability of all tokens outside the top-$k$ set. The researchers found that ignoring this tail probability can lead to a steady increase in the student model's tail probability and overall entropy, ultimately degrading its accuracy on downstream tasks. To counteract this, they propose Tail-Aware Top-$k$ On-Policy Distillation (TA-OPD). This novel distillation method restores the missing tail probability signal by minimizing the reverse KL divergence over the top-$k$ tokens *plus* an additional "tail token" that encapsulates the total tail probability. By explicitly accounting for the tail, TA-OPD enables the student model to better align its next-token distribution with the teacher's, effectively preventing the undesirable increase in tail probability and entropy. Extensive experiments demonstrate the superiority of TA-OPD, showing significant improvements in average performance across common benchmarks.