CORAM Improves Model Merging with Coherent Orthogonal Rotation

Merging finetuned models is a critical technique for combining specialized capabilities without the need for joint training or access to original data. However, most existing methods rely on linear arithmetic in Euclidean weight space, which often fails to capture the underlying geometric properties of model updates. Orthogonal Model Merging (OrthoMerge) improved upon this by using orthogonal transforms, but it couldn't alter singular values, limiting its effectiveness. This research proposes CORAM (Coherent Orthogonal Rotation for Model Merging) to overcome these limitations. CORAM partitions each target weight matrix into row slices and represents each expert slice using its singular value decomposition within the base-model SVD frame. It then merges the task-specific factors on their corresponding manifolds, preserving more of the update's geometry. To counteract the contraction that can occur during manifold averaging, CORAM introduces an amplification coefficient, lambda, estimated from expert and merged update norms. It also includes spread slicing to distribute highly updated rows and a residual pathway for non-target layers. Across diverse benchmarks involving three model families, scales from 3B to 9B, and both language and vision-language experts, CORAM consistently outperforms OrthoMerge and matches or exceeds other strong weight-space baselines.