Model Merging Enhanced by Probabilistic Inference in Parameter Space

Model merging is a technique designed to combine multiple single-task AI models into a unified multi-task solution without requiring additional data-driven fine-tuning. Existing methods primarily rely on the geometric properties of the local solution spaces, which often fall short in statistically evaluating the utility of each task-specific update direction during the merging process. This new research proposes a novel perspective, framing model merging as a problem of probabilistic inference. The core of this new framework is a product-of-experts (PoE) scenario, where each single-task solution is treated as an energy-based expert model (EBM) over the merged parameters. The authors demonstrate that several current model merging techniques can be seen as specific instances of their framework, implicitly assuming Gaussian distributions for the directional residuals between merged and task-specific models. However, empirical observations reveal that these residuals are frequently heavy-tailed, indicating a mismatch with the Gaussian assumption. To address this discrepancy, the researchers introduce a heavy-tailed PoE design utilizing Cauchy experts. This approach more accurately captures the observed residual behavior and offers a provably convergent inference procedure. Extensive experiments across diverse tasks and model architectures show that this Cauchy-expert-based method yields significant improvements over existing state-of-the-art baselines, enhancing the effectiveness of model merging.