New Method Adapts Temperature for Improved Knowledge Distillation

Knowledge distillation, a technique for transferring knowledge from a large "teacher" model to a smaller "student" model, often relies on temperature scaling. However, the precise role and optimal application of this temperature remain areas of active research. Transformed Teacher Matching (TTM) has previously clarified temperature's role by applying it only to the teacher distribution, interpreting the objective as standard distillation with an implicit R\'enyi entropy regularization on the student. This paper advances TTM by proposing a novel method for sample-wise inverse-temperature adaptation. Instead of a fixed temperature, the new approach dynamically adjusts the temperature for individual samples by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. This allows for more nuanced and effective knowledge transfer. The method provides closed-form derivatives for the inverse temperature, enabling an efficient, curvature-aware update that requires minimal computational overhead. Experiments on standard image classification benchmarks demonstrate that this temperature adaptation consistently improves TTM and its weighted variant (WTTM), often outperforming existing temperature-adaptive distillation baselines.