New RL Method Boosts Multi-Turn Agent Performance on Complex Tasks

A new reinforcement learning (RL) framework, Process-Scorer Guided Adaptive Tree Rollout (PATR), has been developed to enhance the training of large language model (LLM) agents, particularly for long-horizon, multi-turn tasks. Traditional RL methods often waste computational budget by uniformly sampling complete trajectories, even those leading to dead ends. PATR addresses this by reframing exploration as a branching problem within a tree structure, where each turn represents a decision point. PATR leverages task-appropriate process feedback to score partial trajectories, allowing it to selectively branch from promising intermediate states. This adaptive approach reuses shared prefixes of trajectories and conservatively stops degenerate paths, thereby reducing wasted sampling and improving exploration efficiency under the same training budget. The resulting rollout groups remain compatible with standard policy optimization techniques. Evaluations on challenging benchmarks like FrozenLake and SWE-Bench, which is largely unexplored by prior tree-rollout methods, demonstrate PATR's effectiveness. It achieves performance improvements of up to 5.0 points on SWE-Bench and 9.3 points on FrozenLake, highlighting the significant benefits of process-guided tree rollouts for scalable multi-turn RL.