Adaptive Entropy Distillation Improves LLM Knowledge Transfer.

Knowledge distillation (KD) is a widely used technique for transferring the capabilities of large language models (LLMs) to smaller, more efficient student models. However, existing KD objectives often struggle to strike the right balance between faithfully imitating the teacher model and enabling robust generation in the student. Many methods combine Forward KL (FKL) and Reverse KL (RKL) divergences, but this research argues that RKL itself contains an inherent mechanism for adjusting the student's imitation strength, which has been overlooked. The study re-evaluates on-policy Reverse Kullback-Leibler (RKL) distillation, breaking down its objective into two key components: a term that fits the teacher's distribution and a term related to the student's entropy. This decomposition reveals that the optimal token-level student distribution is a tempered version of the teacher's, where an adaptive weight controls the trade-off between mode-seeking (imitating the teacher's most likely tokens) and uncertainty preservation (maintaining a degree of randomness or diversity). Guided by this insight, the researchers propose Adaptive Entropy Distillation (AED). AED dynamically adjusts the token-level imitation strength based on the teacher model's entropy. This approach allows the student model to better align with the teacher's distribution and entropy, leading to improved overall performance. Experiments on instruction-following and mathematical reasoning benchmarks demonstrate that AED achieves superior results, suggesting a more principled way to conduct knowledge distillation.