Unified LoRA Initialization Improves Fine-Tuning Performance.

Researchers have developed a novel approach to initializing Low-rank adaptation (LoRA) adapters, a technique used to fine-tune large pretrained models efficiently. This new method, termed Unified LoRA (ULoRA), establishes a continuous spectrum between existing initialization schemes that either project raw gradients or whiten them with loss curvature estimates. ULoRA is defined by a two-parameter family of preconditioned gradient initializations, incorporating a spectral whitening exponent and an Adam-like diagonal exponent. The study reveals that there is no single optimal preconditioning strength; instead, the best operating point for these exponents is highly task-dependent and often lies within the ULoRA family, rather than at its previously published endpoints. When tuned, ULoRA configurations were shown to match or even surpass the performance of full fine-tuning on all five GLUE tasks using RoBERTa-base. It also proved competitive with the strongest baselines on the GSM8K benchmark with LLaMA-2-7B. Furthermore, the paper introduces ULoRA-Auto, a deployable, search-free variant that automatically selects per-layer exponents based on measured spectral statistics. This automated version closely approaches the performance of the tuned ULoRA, offering a practical solution without additional search costs. The findings emphasize that curvature preconditioning in LoRA initialization should be treated as a tunable dimension rather than a fixed design choice, opening new avenues for optimizing efficient model fine-tuning.