CT-Merging Improves LoRA Adapter Combination for Multi-task Models.

LoRA (Low-Rank Adaptation) adapters offer an efficient way to specialize large pre-trained models for numerous downstream tasks, but managing and deploying many individual adapters can be cumbersome. Model merging techniques aim to combine these independently trained adapters into a single, multi-task adapter, simplifying deployment and inference. Existing SVD-based LoRA merging methods often focus on constructing shared directions but struggle with assigning appropriate coefficients, leading to a mismatch between original task SVD magnitudes and the merged update.To address this, researchers propose CT-Merging, a new LoRA-aware merging algorithm. CT-Merging focuses on two key improvements: first, it estimates "consensus directions" by averaging task subspace projectors, effectively identifying common patterns across tasks. Second, it assigns task-level Root Mean Square (RMS) coefficient scales to the final update, rather than directly inheriting coefficients from original task SVDs. This approach reduces reliance on rank-wise SVD magnitudes after the common basis is constructed.Evaluations on the DC-Merge CLIP adapter benchmark demonstrate that CT-Merging achieves superior average normalized accuracy compared to state-of-the-art merging methods. It specifically improved over DC-Merge by 2.56 points on ViT-B/32 and 1.51 points on ViT-L/14 KnoTS-trained checkpoints, indicating a significant advancement in efficiently combining LoRA adapters for multi-task learning.