New Analysis Improves LoRA Convergence for Stochastic Optimization

Low-Rank Adaptation (LoRA) is a popular technique for efficiently fine-tuning large pre-trained models by introducing a low-rank update to the base weight matrix, rather than updating all parameters. Previous theoretical analysis of LoRA's convergence in a deterministic setting suggested a high computational cost to reach a stationary point. This new research refines that understanding, demonstrating that the required number of full-gradient evaluations is significantly lower than previously thought. The paper extends its analysis to the more practical stochastic setting, where gradient estimates are noisy. It proposes a new algorithm called LoRA-NSGDM, which achieves a substantial improvement in stochastic oracle complexity for finding an approximate stationary point. This means less computation is needed when dealing with noisy gradients, common in large-scale machine learning. Furthermore, by incorporating a variance reduction strategy under specific conditions, the researchers introduce LoRA-STORM. This algorithm further enhances the stochastic oracle complexity, making LoRA fine-tuning even more efficient and robust in environments with inherent noise. These advancements provide a deeper theoretical understanding and practical improvements for optimizing LoRA-based models.