ReTeX Recovers Task Experts from Merged Multi-Task Models

A novel framework called Recover Task eXpert (ReTeX) has been introduced to address the challenge of parameter interference in multi-task model merging. While merging aims to consolidate multiple task-specific models into one, static merging often leads to performance degradation due to conflicting parameters. Dynamic merging attempts to mitigate this but typically requires storing and loading redundant expert components during inference, which is resource-intensive. ReTeX conceptualizes parameter interference as an affine transformation or additive offsets applied to each expert's parameters during the merging process. The framework is designed to predict these offsets, effectively "undoing" the interference and restoring the performance of individual task experts from a single, unified merged checkpoint. A key innovation is a router-free task identifier that uses SVD subspace signatures computed offline. This allows ReTeX to select the appropriate expert even when the task identity is unknown during inference, by finding the task whose subspace yields the smallest projection residual for a given input. Experiments across vision and NLP tasks show ReTeX recovers over 95% of individual expert performance and significantly improves generalization to unseen tasks by adaptively interpolating expert knowledge.