Feature-Space Regularization Boosts LLM Continual Learning

Traditional methods for preventing catastrophic forgetting in continual learning, such as Elastic Weight Consolidation (EWC), typically operate by regularizing in the model's weight space. However, these methods often fall short when applied to large language models (LLMs), a phenomenon the authors attribute to the "polysemantic" nature of LLM weights, where individual weights contribute to multiple concepts, making precise knowledge protection difficult. This research introduces a new approach that shifts regularization from the weight space to the activation space. It leverages pre-trained Sparse Autoencoders (SAEs) as a "monosemantic feature dictionary," meaning each feature represents a distinct concept. By deriving a new loss function based on constrained optimization, the method explicitly balances the need for stability (retaining old knowledge) and plasticity (learning new knowledge). A significant advantage of this feature-space regularization is its memory efficiency; unlike replay-based methods, it doesn't require storing or revisiting previous task data. Instead, it computes a compact SAE feature mask from current-task data, which is then retained. The method demonstrated superior performance on continual learning benchmarks like TRACE and MedCL, surpassing traditional weight-space regularization and other non-architectural approaches, while also providing empirical evidence for the polysemanticity thesis.