BaRA Improves LoRA Fine-Tuning with Adaptive Rank Allocation

Low-rank adaptation (LoRA) is a popular method for efficiently fine-tuning large language models, but its fixed-rank design can limit flexibility and lead to overconfident predictions, especially with limited data. While Bayesian LoRA variants have improved uncertainty estimation, they still often rely on static or heuristically chosen ranks, failing to adapt to the specific needs of different contexts. A new framework called BaRA, or Bayesian Adaptive Rank Allocation, addresses these limitations by dynamically adjusting the adaptation capacity. Inspired by probabilistic topic models, BaRA activates a sparse, context-dependent subset of latent factors, allowing the effective rank to vary for each instance. This Bayesian approach provides a data-driven way to control model capacity, preventing over-parameterization while maintaining expressive power. The research also includes a theoretical analysis demonstrating that BaRA's generalization ability depends on the learned effective rank rather than a fixed maximum rank. This explains how sparse, adaptive rank allocation can reduce model complexity while preserving input-dependent expressiveness. Experiments on various natural language benchmarks show BaRA consistently outperforms standard LoRA and existing Bayesian LoRA methods in terms of predictive accuracy, robustness, and uncertainty calibration.