New RTPO Method Stabilizes Multi-Turn Agentic RL Training

Training multi-turn agentic workflows using reinforcement learning (RL) is crucial for enabling large language models (LLMs) to perform complex reasoning, utilize external tools, and conduct iterative searches. However, this training often suffers from significant instability, leading to performance degradation as the number of turns increases. Researchers have identified three primary sources of this instability: a mismatch between rollout and training contexts, weak credit assignment for turn-level actions due to sparse terminal rewards, and asynchronous policy drift when optimizing trajectories of varying lengths. The paper posits that these issues stem from a common structural problem in flattened trajectory optimization. To resolve this, a unified reverse-turn formulation is proposed, leading to Reverse-Turn Policy Optimization (RTPO). RTPO structures multi-turn rollouts as sparse reverse trees and updates policies in reverse temporal order. This approach ensures that each decision is aligned with its subsequent continuation, enabling causally consistent turn-level credit assignment and better control over asynchronous drift. Theoretical analysis confirms that RTPO eliminates context mismatch and asynchronous drift under its turn-level formulation, reduces credit bias, and converges to recursive optimality. Experimental results on multi-turn agentic RL benchmarks show RTPO outperforming existing trajectory- and turn-level baselines by substantial margins, highlighting its potential to make training more stable for tool-using AI agents.