New Method Unifies Knowledge Distillation Interpretation for LLMs.

Researchers have introduced a novel framework to better understand how knowledge distillation (KD) works in Large Language Models. By breaking down LLM outputs into a sum of interactions, which represent nonlinear relationships between input variables, they've identified a core mechanism. The key finding is that KD's success lies in "sparsification of interactions." This means student models, through distillation, learn to focus on a smaller, more essential set of interactions for inference, effectively suppressing less important ones. Furthermore, the study found that the performance differences among various KD methods are linked to their ability to manage complex interactions. Methods that achieve higher sparsity of complex interactions in student models tend to perform better. Based on these insights, a new plug-and-play loss function, Complex Interaction Penalty (CIP), is proposed to explicitly encourage this sparsity during training, showing consistent performance improvements.