CBPO Boosts RL for Language Models with Tool Integration.

Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm for training language models to interact with external tools over multiple turns. However, the sparse nature of outcome rewards in RLVR makes it challenging to pinpoint which intermediate decisions contributed to success or failure. Existing branch sampling methods, while useful, often conflate the problems of allocating a fixed rollout budget and assigning token-level credit. A new approach, Contrastive Branch Policy Optimization (CBPO), addresses these limitations by separating these two problems. It uses generation entropy to identify candidate branch positions across the entire response and employs path-level and node-level decay to distribute a fixed budget, preventing exploration from concentrating on a few paths or adjacent tokens. The core innovation is the Contrastive Branch Value (CBV), derived from reward variation within exact-prefix groups, which provides an outcome-based estimate of local decision sensitivity. CBPO also ensures that gradients are not duplicated on shared tokens by partitioning trajectories into non-overlapping credit segments. This method requires only outcome rewards, making it practical for fine-grained credit assignment in tool-integrated agent training. Extensive experiments across ten benchmarks, including mathematical reasoning and knowledge-intensive search, show CBPO consistently outperforms current policy-optimization and branch-based methods, achieving higher accuracy in both domains and across different model scales.