SOS-LoRA Improves LLM Fine-Tuning with Orthogonal Subspaces

Low-Rank Adaptation (LoRA) is a popular technique for efficiently fine-tuning large language models (LLMs) by adding a small number of trainable parameters. However, its single low-dimensional pathway can lead to interference and coupled behaviors during optimization. This new research proposes SOS-LoRA (Static Orthogonal Subspace LoRA) as an improved drop-in extension. SOS-LoRA re-architects the low-rank update by breaking it down into a sum of multiple "static" (always-on) low-rank experts. It distributes the total rank budget across these experts, applies a fixed multi-scale scaling to promote distinct optimization dynamics, and encourages diverse input directions through orthogonal initialization and a lightweight regularizer. Crucially, SOS-LoRA remains fully mergeable, meaning it adds no extra parameters or latency during inference after the fine-tuning process. Experiments across various LLM benchmarks, including reasoning, knowledge-intensive tasks, and math problems, consistently show that SOS-LoRA outperforms standard LoRA and its recent variants.