CABS+ Improves Model Merging Efficiency and Performance for Multi-Task AI

A novel model merging technique, CABS+, has been introduced to address the challenges of constructing unified multi-task AI models without extensive retraining. Model merging is a promising approach, but it often suffers from performance degradation due to parameter conflicts and knowledge interference between different tasks. Previous methods like CABS attempted to mitigate this through structured pruning, but faced issues with high computational complexity and suboptimal performance for less dominant tasks. CABS+ improves upon its predecessor by incorporating Adaptive Weight Allocation (AWA), which employs a gradient-free search to optimize merging coefficients, thereby drastically reducing time complexity. Additionally, an asymmetric fitness function is utilized to ensure more balanced performance gains across all tasks, preventing high-performance tasks from dominating the optimization objective. The research also introduces a Relative Synergy Score (RSS) to quantify model mergeability and aid in model selection. Extensive empirical studies comparing CABS+ against state-of-the-art methods like AdaMerging and WUDIMerging across 27 datasets and 5 model architectures (including large language, small language, and vision models) demonstrate its effectiveness. CABS+ achieved significant performance improvements (16.97% over AdaMerging, 12.93% over WUDIMerging), exhibited superior stability, used less than 25% of AdaMerging's GPU memory, and was nearly 4x faster than WUDIMerging, making it a highly efficient and scalable solution for multi-task model deployment.