PACT Improves Model Merging by Preserving Core Task Knowledge

A new technique, PACT (Preserving Anchored Cores in Task-vectors), has been developed to improve the effectiveness of model merging, a training-free alternative to multi-task learning. Current model merging methods, often based on Task Arithmetic, assume that all task-specific knowledge resides solely within "task vectors" derived from fine-tuned models. However, this research argues that this assumption is flawed, as some critical task knowledge, termed "Load-Bearing Wall (LBW) dimensions," remains embedded within the pre-trained model's weights. Ignoring these LBW dimensions can lead to task conflicts and degradation during the merging process. PACT addresses this by identifying and preserving these anchored task-specific cores within task vectors. It aligns their orthogonal complements with the subspace of pre-trained weights, then removes these aligned components from the task vectors before applying standard merging algorithms. This approach consistently enhances mainstream model merging methods and achieves new state-of-the-art performance across various benchmarks. An efficient variant using randomized SVD is also proposed for scalability.