Knowledge Distillation Boosts Time Series Classification Model Efficiency

Deep learning models have achieved impressive results across various domains, including time series analysis. However, their high computational and memory demands often hinder deployment in environments with limited resources. Knowledge Distillation (KD) offers a solution by transferring learned knowledge from a large, complex "teacher" model to a smaller, more efficient "student" model, aiming to preserve performance while reducing resource consumption. This study specifically explores the application of KD to Time Series Classification (TSC) tasks. Researchers evaluated its effectiveness across three distinct deep learning architectures: the Fully Convolutional Network (FCN), the Inception model, and the transformer-based ConvTran model. They systematically modified architectural components, such as convolutional filters, Inception modules, and attention heads, to create student models of varying complexities. The results, benchmarked on the extensive UCR Archive dataset, consistently showed that KD is most beneficial for student models of intermediate complexity across all three architectures. For instance, a distilled FCN student achieved a 38-fold reduction in parameters, while a distilled Inception student nearly matched teacher performance with 42% fewer parameters. The ConvTran student with two attention heads showed the most significant performance improvement through distillation. These findings highlight KD as a powerful technique for optimizing deep learning models for TSC, making them more suitable for resource-constrained deployments.