SAPO Boosts Agentic RL Performance, Reduces Memory Overhead.

Agentic reinforcement learning (RL) is becoming crucial for post-training large language models (LLMs). While existing critic-free, group-relative methods have shown promise by estimating policy advantages from multiple rollouts, they often suffer from limitations such as a lack of explicit value generalization, ineffective temporal credit assignment, potential advantage collapse in long-horizon tasks, and a costly trade-off between sampling budget and policy performance. To address these issues, researchers propose Single-rollout Autoregressive Policy Optimization (SAPO). This novel framework is designed to be low-memory and compute-efficient, leveraging a single autoregressive backbone that is shared between the policy and value functions. SAPO optimizes Proximal Policy Optimization (PPO) objectives and auxiliary on-policy SARSA objectives independently, while still sharing parameters, allowing for distinct policy and value predictions at different causal boundaries. Furthermore, SAPO introduces a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization to robustly estimate the contribution of each turn. Experimental results on ALFWorld and WebShop benchmarks, using Qwen2.5-1.5B/7B models, demonstrate that SAPO achieves stable training and significantly outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively. It also eliminates the memory cost of a separate critic model and reduces per-iteration runtime by 33.2% compared to PPO.