HyperFix Improves Task Vector Merging with Nonlinear Corrections

This paper presents HyperFix, a novel method designed to enhance the merging of task vectors in machine learning models without requiring extensive retraining. Current task vector merging techniques often necessitate scalar tuning for each specific subset of tasks, which becomes inefficient and restrictive, especially when dealing with varying combinations of tasks. HyperFix tackles this by formulating the merging problem as a combinatorial correction challenge. It employs a lightweight hypernetwork that learns to predict nonlinear corrections directly in the model's weight space, conditioned on the specific subset of tasks being merged. This hypernetwork is trained once on smaller subsets (e.g., singletons, pairs, triples) and then effectively generalizes to larger, unseen task subsets without needing further per-subset optimization. A local perturbation analysis supports the learning of these residual corrections beyond simple linear merging. Experiments across diverse benchmarks demonstrate that HyperFix not only surpasses existing task vector merging methods in performance but also significantly reduces the associated tuning costs, making it a more practical and scalable solution.