New Approach Explains Knowledge Distillation in LLMs via Interactions

This paper introduces a novel framework for understanding the mechanisms behind knowledge distillation (KD) in Large Language Models (LLMs). Despite KD's widespread success in creating smaller, more efficient student models from larger teacher models, the precise reasons for its efficacy have been unclear. The proposed approach interprets KD by breaking down an LLM's output score into a sum of numerous "interactions," each representing a nonlinear relationship between input variables like words. Through this interaction-based decomposition, the study reveals a common underlying principle across various KD methods: interaction sparsification. This means student models, during distillation, learn to retain only a crucial subset of interactions for inference while effectively suppressing others to zero. Furthermore, the research indicates that the performance differences among KD methods are linked to their ability to manage complex interactions, with superior methods achieving greater sparsity in these complex relationships. Motivated by these insights, the authors developed a new plug-and-play loss function called Complex Interaction Penalty (CIP). CIP explicitly encourages the sparsity of complex interactions during the distillation process. Extensive experiments confirm that integrating CIP consistently enhances the performance of diverse KD methods across both in-domain and out-of-distribution benchmarks.