MIITA Enables Continual Learning for Small Language Models

A new research paper introduces MIITA, a novel framework designed to enable continual learning for Small Language Models (SLMs) in resource-constrained environments. Continual learning is crucial for SLMs to adapt to new information without forgetting previously learned knowledge, a common problem known as catastrophic forgetting. Existing memory-based methods, often designed for larger models, are unsuitable for SLMs due to their limited storage and reasoning capabilities. MIITA addresses these challenges by storing supervised experiences not as raw data, but as compact "correction-direction prototypes" linked with semantic anchors. At inference time, these prototypes are retrieved using semantic and uncertainty-based cues. The retrieved directions are then applied through a gated temporary hidden-state adaptation mechanism. This approach allows SLMs to non-destructively reuse past supervision without needing to update their core parameters, extend prompts, or perform test-time backpropagation. Theoretical analysis supports its design principles, linking it to loss reduction and effective knowledge retention. Extensive experiments across various supervised continual learning scenarios demonstrate that MIITA consistently improves performance and significantly mitigates catastrophic forgetting within fixed memory budgets.