Adaptive Rollout Policy Optimizes LLM Agent Search Tasks.

Current reinforcement learning methods for large language model agents often waste computational resources by uniformly exploring intermediate states, even when some branches offer little value. A new framework, IGRPO, addresses this by treating intermediate-state informativeness as the core principle for budget allocation during rollouts. IGRPO employs a tree-structured rollout approach, dynamically expanding more promising branches while suppressing less informative ones. This method also defines a clear policy optimization target by inducing an explicit teacher distribution over trajectories. Experiments across several search-augmented QA benchmarks demonstrate that IGRPO consistently outperforms existing baselines under the same computational constraints, validating its effectiveness in guiding policy optimization for complex, long-horizon search agents.