PGPO Improves Credit Assignment for Multi-Turn AI Agents

Group-based reinforcement learning (RL) has proven effective for post-training large language models (LLMs), particularly for agentic tasks. However, in complex multi-turn scenarios where rewards are sparse and only given at the end, it struggles to assign precise credit to intermediate actions. This often means that effective actions within a failed sequence receive the same poor feedback as genuinely erroneous ones.Previous attempts to provide step-level advantages still rely on the final outcome of individual trajectories, limiting their ability to differentiate actions within unsuccessful attempts. To overcome this, researchers propose Potential-Guided Policy Optimization (PGPO). This method introduces the concept of empirical state potentials, which are estimated from return statistics within groups of rollouts.PGPO then calculates action advantages based on the potential differences between adjacent states. This innovative approach allows for credit to be propagated across different trajectories, providing much finer-grained feedback at the step level, particularly valuable for understanding and improving actions within failed sequences. Experiments on ALFWorld and WebShop benchmarks demonstrate that PGPO significantly outperforms other group-based RL methods, offering more informative credit signals without substantial training overhead.