Model Merging Rivals Joint Training in Multi-Task Reinforcement Learning.

Model merging is often proposed as a simpler alternative to training a single model for multiple tasks, especially in reinforcement learning (RL) where jointly trained multi-task models are frequently unavailable. This study directly compares model merging with a jointly trained baseline in an RL setting. Researchers trained specialist Qwen3-8B agents on the AppWorld benchmark for different difficulty levels, then merged them using methods like TIES and RAM+. The merged models achieved task-goal completion performance statistically indistinguishable from a model trained jointly on the same data. The key insight lies in the "task-vector geometry" of the specialists. The study found that the task vectors were near-orthogonal despite significant task overlap, meaning the different merging methods effectively collapsed to near-uniform averaging. This suggests that when tasks are sufficiently distinct in their learned representations, simple merging can be highly effective.