Study Compares Feature-Based Knowledge Distillation Across Student Models

This research conducts a controlled experiment to understand the effectiveness of feature-based knowledge distillation techniques when applied to different student neural network architectures. Knowledge distillation typically involves training a smaller "student" model to mimic the outputs of a larger "teacher" model. Feature-based methods add an extra constraint by aligning intermediate representations between the teacher and student. The study used a ResNet-50 teacher and evaluated various student models, including a custom ResNet family with varying widths and MobileNetV2. Each feature-based method was compared against a logit-based distillation baseline under identical training conditions. Results indicate that logit distillation consistently improved all tested student models. However, feature-based methods yielded mixed results: Attention Transfer showed no clear pattern within the custom ResNet family and was generally negative for it, but positive for MobileNetV2. FitNets consistently performed worse than logit distillation across all paired runs, with its performance gap widening for wider students within the constant-depth family. The study also highlighted that a fixed auxiliary coefficient for feature methods can lead to different gradient scales across students, suggesting that a uniform training condition is not always achieved.