ReCoLoRA Improves Continual LLM Fine-Tuning Efficiency

A novel framework called ReCoLoRA (Recursive Consolidation of Low-Rank Adapters) has been introduced to enhance continual fine-tuning of large language models (LLMs). Traditional parameter-efficient fine-tuning methods, like LoRA, often suffer from catastrophic forgetting when adapting to a sequence of tasks, as new tasks tend to overwrite previously learned information. ReCoLoRA aims to mitigate this by adopting a spectrum-aware approach. The method initializes adapters from a randomized Singular Value Decomposition (SVD) of the pretrained weight, selecting per-layer effective ranks using an elbow criterion. Crucially, before each new task, ReCoLoRA re-decomposes the current effective weight (which already incorporates previous tasks) into a frozen residual, a slowly updated principal component, and a fresh adapter. This "recursive consolidation" ensures that each new task builds upon the model's accumulated knowledge. Experiments on a six-task continual GLUE sequence across four 7-8B LLM backbones demonstrated ReCoLoRA's effectiveness. It achieved the best final average score on three of the four backbones compared to several baseline methods, all while training fewer parameters. This indicates a significant step forward in making continual learning for LLMs more efficient and robust.