CARE-LoRA Boosts Memory Efficiency for Large Model Fine-Tuning

Fine-tuning large language models using methods like Low-Rank Adaptation (LoRA) often faces memory constraints, primarily due to the large activations that need to be stored for backpropagation. A new approach, CARE-LoRA, addresses this by intelligently compressing these activations. The framework leverages LoRA's inherent structure to replace full input activations with their low-rank compressed counterparts. During the forward pass, it computes a small reconstruction matrix, which is then used in the backward pass to accurately reconstruct gradient signals. This allows LoRA matrices to remain fully trainable while drastically cutting down memory usage. Experiments across various models and tasks show that CARE-LoRA not only substantially reduces memory footprint but also maintains or even improves performance compared to standard LoRA and other variants. The code is publicly available.