CoMerge Optimizes Multi-Task LLM Merging with Conflict-Driven Preference

Model merging offers an efficient way to create multi-task large language models without extensive retraining, but it often struggles with parameter interference. Existing methods try to preserve individual model capabilities and reduce interference, but they typically don't learn directly from the degraded performance that can result from simple merging. This new research introduces CoMerge, a framework that redefines model merging as a preference optimization challenge. It employs a self-supervised, conflict-driven approach, using the shortcomings of basic merging techniques as "hard negative samples" to build preference pairs without needing external labels. CoMerge applies preference optimization to fine-tune lightweight, tensor-wise merging coefficients. This allows the model to resolve parameter-space conflicts while maintaining task-specific abilities. Extensive testing demonstrates that CoMerge achieves superior performance on MergeBench, surpassing all other evaluated data-free and data-driven model-merging baselines. Furthermore, when applied to Llama-3.1-8B-Instruct, CoMerge shows notable improvements on tasks sensitive to conflicts, such as instruction following and safety, while remaining highly competitive with full-parameter fine-tuning, despite optimizing only a small number of scalar coefficients.