Task-Vector Interference in Merged LLMs Driven by Orientation, Not Magnitude.

Model merging techniques, like task arithmetic, sometimes fail, and the reasons are often diagnosed by looking at the magnitude of layerwise representation bias or parameter overlap. This paper challenges that view, finding that magnitude is an insufficient and inconsistent diagnostic across different model families. Through a detailed decomposition of layerwise flux and direct intervention, the researchers discovered that interference in merged LLMs is predominantly caused by the *orientation* of task-vectors, not their magnitude. The existing cross-term is amplified through the network, and simply erasing its norm is ineffective unless applied very late in the model. Crucially, erasing interference along its causal direction dose-dependently removes its effects, whereas norm-matched interventions in wrong directions fail or backfire. The study also found that instruction wrappers can gate this effect, sometimes amplifying the cross-term internally. These findings, all preregistered, highlight that the causal structure of interference is directional, not merely a matter of magnitude.