GAGPO Improves Credit Assignment in Multi-Turn LLM Agent Reinforcement Learning.

Reinforcement learning (RL) is a powerful technique for post-training large language model agents, but a significant hurdle remains in "credit assignment" within multi-turn environments. Agents often receive rewards only at the end of an episode, making it difficult to pinpoint which specific intermediate actions contributed to success or failure. This challenge typically requires costly auxiliary value models to propagate delayed outcomes back to individual decision steps. This research introduces GAGPO (Generalized Advantage Grouped Policy Optimization), a novel critic-free RL method that provides precise, step-aligned temporal credit assignment. GAGPO tackles the problem by creating a non-parametric grouped value proxy from sampled rollouts. This proxy is then used to compute TD/GAE-style temporal advantages, effectively propagating outcome supervision backward through time without needing a separate critic network. By combining this with group-wise advantage normalization and an action-level importance ratio, GAGPO extracts stable and localized optimization signals directly from multi-turn trajectories. Experiments on benchmarks like ALFWorld and WebShop show that GAGPO outperforms existing strong RL baselines, demonstrating faster early-stage learning, improved interaction efficiency, and smoother optimization dynamics, offering a simpler yet effective framework for multi-turn agentic reinforcement learning.