New RL Algorithm Boosts Language Agent Performance in Sandboxes

Traditional reinforcement learning algorithms for training large language model agents in sandboxes often treat each trajectory independently, similar to how human feedback is used. However, this approach overlooks a key characteristic of agent sandboxes: their deterministic, snapshottable, and resumable nature. This paper proposes a new algorithm, Branching Policy Optimization (BPO), which leverages this property. BPO constructs a single tree of rollouts where siblings share common prefixes, allowing for shared variance and more efficient learning. It adaptively snapshots the sandbox at high-entropy decision points, forks alternative actions, and computes advantages from sibling returns. This method is proven to be unbiased with strictly lower variance than trajectory-level baselines. Evaluations on benchmarks like WebShop, ALFWorld, and SWE-bench Verified, using Qwen2.5-7B and Llama-3.1-8B, show BPO significantly improves success rates by 3.6-6.1 absolute points over GRPO and RLOO at equivalent compute. It also halves gradient-norm variance and achieves top baseline performance with 38% fewer policy updates, demonstrating a more efficient and effective approach to training LLM agents.