New Sparsity-Induced Methods Enhance Parameter-Efficient Fine-Tuning Beyond LoRA

This paper delves into the realm of parameter-efficient fine-tuning (PEFT) for pre-trained models, specifically examining the effectiveness of Low-rank adaptation (LoRA) and its variants. The authors introduce novel sparsity-induced extensions to existing LoRA methods, termed Cheap LoRA (cLA) and its chained circulant variant (c^3LA). These new techniques aim to simplify and reduce the computational cost of adaptation. cLA is framed as a structured instance of asymmetric LoRA, which restricts fine-tuning to a sparse, structured column subspace. The research provides theoretical backing with information-theoretic generalization error bounds for these variants, a pioneering effort in this specific area. Empirical evaluations across 11 fine-tuning methods, 10 pre-trained models, and 14 datasets demonstrate that these sparsity-induced approaches remain competitive with parameter-matched baselines. Crucially, they offer tangible benefits, reducing training time by up to 10% and peak GPU memory by up to 15%, even with non-optimized sparse implementations. The study suggests that structured sparsity can be a viable and efficient path for model adaptation.