Teacher Misspecification Impacts Knowledge Distillation Task Performance.

Knowledge distillation is a common technique where a smaller "student" model learns to replicate the outputs of a larger "teacher" model. Typically, the success of this process is measured by how closely the student mimics the teacher's outputs. However, this research highlights a critical issue: a potential disconnect between the student's ability to mimic the teacher and its actual performance on the underlying task. The study introduces a minimal three-party model involving a true generative teacher, a potentially flawed teacher, and a student. It specifically investigates scenarios where the "teacher" model is "misspecified," meaning it cannot represent a crucial shared latent factor present in the true generative process. The degree of this misspecification is controlled by a scalar parameter. Through an order-parameter analysis, the researchers demonstrate that while the student's mimicry of the misspecified teacher remains invariant to the degree of misspecification, the student's true task error and the gap between mimicry and true performance strictly increase with misspecification. This gap is further amplified by the complexity of the true teacher. The findings serve as a quantitative warning against evaluating knowledge distillation solely based on how well the student replicates the teacher's outputs, emphasizing the need for diagnostics that can distinguish between teacher misspecification and capacity limitations.