New Merging Method Improves Multi-Task AI Model Performance

Building generalist AI models from multiple specialized models often leads to performance degradation due to "task interference," where different task features become entangled within the model's parameters. This research proposes a novel merging framework designed to overcome this issue. It utilizes Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional, sparse feature space, allowing for better disentanglement of task-specific components before they are combined. To enhance efficiency, the framework also incorporates a lightweight optimizer, Group-Ranked Zeroth-Order Optimizer (GR-ZOO), which identifies and targets only the most critical layers for merging. Extensive experiments on Qwen2.5-1.5B and Qwen2.5-7B models demonstrated that this approach consistently surpassed leading baseline methods across diverse tasks like mathematical reasoning, code generation, and instruction following, particularly excelling in scenarios with high task conflict.