PlanPO Optimizes Multi-Turn LLM Agents for Efficient Planning

Training agentic large language models (LLMs) for multi-turn interactive tasks often uses group-relative policy optimization. However, a common flaw in existing methods is their inability to differentiate between successful trajectories that vary significantly in efficiency. For example, a circuitous path to success might receive the same reward as a highly efficient one, leading to "advantage collapse" and hindering performance. To overcome this, researchers propose PlanPO, or Group Planning-aware Policy Optimization. This simple yet effective reinforcement learning approach is designed to teach LLMs generalizable planning skills beyond just task-specific high-quality behaviors. PlanPO introduces "coarse-to-fine advantage signals" that capture the relative differences in both trajectory length and turn-level response length within successful task completions. By integrating these nuanced advantage signals into the group-relative optimization structure, PlanPO enables agents to actively learn deliberate behaviors for interaction planning and text generation from high-quality rollouts. Experimental results show PlanPO improved performance over GRPO by an average of 27.2% across challenging multi-turn benchmarks like ALFWorld, WebShop, and SciWorld, outperforming other powerful baselines with minimal additional training cost.