ReCoLoRA Improves Continual Fine-Tuning for Large Language Models.

A novel framework called ReCoLoRA (Recursive Consolidation of Low-Rank Adapters) has been introduced to enhance the continual fine-tuning of large language models (LLMs). Traditional parameter-efficient fine-tuning methods, like LoRA, often struggle with catastrophic forgetting when adapting an LLM to a sequence of tasks, as new tasks tend to overwrite knowledge from previous ones. ReCoLoRA tackles this by recursively consolidating adapters. It initializes adapters using a randomized Singular Value Decomposition (SVD) of the pretrained weights, selects effective ranks per layer, and adapts the principal subspace before opening residual capacity. Crucially, for each new task, ReCoLoRA re-decomposes the current effective weight (which already incorporates previous learning) rather than the original, allowing the model to absorb predecessors' knowledge. Evaluations on a six-task GLUE sequence across four 7-8B LLM backbones show ReCoLoRA achieving superior final average scores compared to several baselines, including rank-swept LoRA, PiSSA, AdaLoRA, and DoRA, while also training fewer parameters. This indicates a significant step forward in making LLMs more adaptable to evolving task requirements without losing prior learning.