RLVR Models Face "Sparsity Curse" in Parameter Merging

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful technique for post-training language models, often outperforming Supervised Fine-Tuning (SFT) in reasoning and preventing catastrophic forgetting. Recent observations indicate that RLVR induces highly sparse and distinct parameter updates compared to SFT. This research investigates whether this sparsity makes RLVR models more amenable to model merging, a technique for combining capabilities from independently trained models. Counterintuitively, the study reveals a "sparsity curse": the sparse RLVR updates are widely dispersed in the parameter space, forming near-orthogonal "shortcuts" that render standard merging methods ineffective and fragile. This issue is attributed to the stochastic nature of RL optimization and the diversity of emergent reasoning patterns. To overcome this, the authors propose Sensitivity-aware Resolving Merging (SAR-Merging). This new merging recipe is specifically designed for RLVR parameter spaces, resolving conflicts in overlapping update regions using Fisher Information-based sensitivity arbitration. It then applies magnitude-aware sparsification and rescaling to preserve delicate reasoning pathways. Experiments on mathematical and coding benchmarks demonstrate SAR-Merging's superior performance in enhancing single-task capabilities and fusing multiple capabilities in RLVR models.