LoRA Fine-tuning: New Bounds Clarify Optimal Rank Selection.

Low-Rank Adaptation (LoRA) has become a standard and highly effective method for fine-tuning large pre-trained models, offering significant efficiency gains. However, a complete statistical understanding of its properties, particularly regarding optimal rank selection and generalization, has been lacking. Existing research provided upper bounds but a matching lower bound and formal guidance for choosing the LoRA rank `r` were missing. This research closes these gaps by establishing tight sample complexity bounds. Through a local Rademacher argument, an upper bound of O~(rd/n) is derived for the excess risk of the empirical risk minimizer using rank-`r` LoRA, assuming the target adaptation has a rank at most `r`. Crucially, a matching minimax lower bound of Omega(rd/n) is also proven, applying to any estimator within the rank-`r` LoRA class. Combining these bounds reveals a critical rank-selection dichotomy. For unregularized empirical risk minimization, the optimal rank is precisely the intrinsic rank `r*`, and choosing a rank higher than `r*` (over-ranking) actively degrades performance. In contrast, for adaptive estimators (like nuclear-norm-then-truncate), over-ranking is shown to be harmless, with the performance rate stabilizing at Theta~(r*d/n) regardless of the chosen `r`. These findings clarify that the empirically observed penalty for over-parameterization is a characteristic of unregularized empirical risk minimization, not an inherent flaw of the LoRA class itself. The theoretical predictions were validated on synthetic benchmarks and real LoRA fine-tuning tasks using DistilBERT and RoBERTa.