New Method Predicts LoRA Adapter Mergeability Early in Training

Training numerous domain- and task-specific language model adapters using Low-Rank Adaptation (LoRA) is cost-effective. However, determining if these adapters can be merged without negatively impacting performance typically occurs late in the development cycle, after full training and evaluation. This late feedback can be expensive, as individually strong adapters might interfere destructively when combined. A new approach, MergeProbe, aims to address this by predicting adapter mergeability much earlier. It formalizes mergeability as the degree to which an adapter retains its utility after combination and leverages signals from the first few percent of training. These signals include the alignment of low-rank updates and their gradients across tasks, and how much they disrupt shared representations. MergeProbe packages these signals into a lightweight predictor that estimates pairwise and set-level retention. It then provides a concrete decision: merge directly, reweight, prune, or route. Tested on the MERGE-PEFT benchmark, MergeProbe achieved superior average and worst-case retention compared to existing interference-aware merge baselines, while significantly reducing deployment overhead compared to full task routing. This shifts LoRA merging from a reactive engineering step to a proactive measurement problem.