Gate-Zero Growth Enables Function-Preserving Continual Learning

This research presents "gate-zero growth," a new technique designed to facilitate continual learning in neural networks without significantly forgetting previously learned functions. The method involves adding new residual blocks to a model, initialized with a "zero gate." This gate ensures that at the point of growth, the new blocks initially have no impact on the existing function, preserving the model's performance on old tasks. The core idea is that this zero-initialization induces "rank separation" in the functional Jacobian, meaning old knowledge remains undisturbed while new parameters are introduced in a controlled manner. As the gates gradually open during subsequent training, function drift is minimized, allowing for efficient integration of new information. Experiments on Transformer models, specifically adapting LLaMA2 from WikiText-103 to BookCorpus, showed that gate-zero growth achieved remarkably low forgetting rates on old domains, significantly outperforming non-function-preserving control methods. This geometric analysis also extends to other common architectural elements like LoRA and ReZero, positioning gate-zero growth as a foundational concept for safe capacity activation in continual learning.