Bayesian Optimization Enhances Multi-Objective Model Merging Efficiency

Combining multiple pre-trained AI models directly by manipulating their weight spaces, known as model merging, offers a computationally efficient alternative to extensive fine-tuning. However, optimizing the parameters for these merge operations is challenging due to the high cost of downstream evaluations, the absence of gradients, and potential conflicts between the capabilities of the source models. Researchers have framed this parameter selection problem as a black-box multi-objective optimization task. They developed MOBO-Merge, a framework that employs multi-objective Bayesian optimization to approximate the Pareto front of optimal merge parameters within a limited evaluation budget. This framework is designed to be agnostic to the specific merge operator used. Evaluations were conducted on models like Qwen3-4B and Llama-3.1-8B, exploring various merge operators (Linear, SLERP, TIES, block-wise) in scenarios involving two or three models with instruction, math, and code capabilities. MOBO-Merge consistently outperformed random search, achieving higher mean hypervolume in most comparisons, with particularly substantial gains for more complex merge operators and multi-objective searches. The study also highlighted that no single merge operator is universally superior, emphasizing the value of an efficient search layer like MOBO-Merge for expressive parameterizations.