W2SPO Boosts LLM Reasoning with Weak-to-Strong RL.

Reinforcement learning with verifiable rewards (RLVR) is a common method for improving LLM reasoning, but it often faces a "support limited bottleneck." On challenging tasks, the target model's generated samples can become semantically redundant, converging into similar erroneous reasoning paths that offer little contrast for policy updates. To overcome this, the W2SPO (Weak-to-Strong Off-Policy RL via Auxiliary Branches) method proposes a weak-to-strong learning paradigm. It uses a weaker, computationally efficient auxiliary model to inform the target policy's exploration. Specifically, W2SPO injects short auxiliary segments, often as brief as 8 tokens, into the target model's intermediate trajectories. The target model then completes the reasoning path from these diverted states. Policy updates are restricted to these short inserted segments based on the final verifiable rewards. Empirically, W2SPO achieves superior performance on mathematical reasoning benchmarks for 4B-scale models, outperforming post-trained baselines. It improves Pass@1 from 62.3% to 64.2% compared to vanilla GRPO with the same sampling budget, while also achieving a 3.55 times training speedup.