LAARA: Adaptive Rank Allocation for Efficient Fine-Tuning

Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of large language models. However, existing methods typically assign a uniform adapter rank across all transformer layers, despite the fact that different layers often have varying adaptation requirements. This paper theoretically and empirically demonstrates that this uniform rank allocation is fundamentally suboptimal. To address this, the researchers propose LAARA (Layer-Aware Adaptive Rank Allocation framework), a novel, search-free method that dynamically adjusts adapter ranks for each transformer layer. LAARA achieves this by utilizing lightweight diagonal Fisher estimates computed during the training process. The framework incorporates several mechanisms, including projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism, to ensure stable and efficient rank adaptation. Experimental results on GLUE and MathInstruct benchmarks show that LAARA consistently performs as well as or better than leading methods like LoRA, AdaLoRA, DyLoRA, and Bitfit, all while requiring substantially fewer trainable parameters. This highlights Fisher-guided rank allocation as a principled and effective approach for adaptive parameter-efficient fine-tuning.