New Strategy Boosts Efficient Transfer Learning for Deep Models

Deep learning models, while powerful for image classification, often present significant computational and energy challenges, particularly during deployment and fine-tuning. A novel, lightweight training approach has been introduced to mitigate these issues, focusing on enhancing the efficiency of transfer learning. This strategy moves beyond traditional end-to-end backpropagation by decoupling the process of feature extraction from the optimization of the classifier. The core of this method involves adapting only the normalization layers of a pre-trained model to a new domain, while precomputing features just once. This significantly reduces the computational overhead. Additionally, a redesigned classifier head, incorporating a margin-based weighted loss, helps minimize ambiguity without requiring full backpropagation through the model's backbone. Extensive evaluations across various CNN and Transformer architectures, and multiple medical datasets, demonstrate that this approach drastically cuts training time and CO2 emissions, often matching or exceeding baseline accuracy.