Learnijoy
ResearchAI ResearchAI Engineering & DevTools

New Policy Optimization Boosts LLM Agents on Long Tasks

Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He, Yu Li, Mengyi Liu, Pengwei Chen, Jun Xu, Lei Feng· July 28, 2026 View original

Summary

This paper introduces ProGPO, a new policy optimization method for training large language model agents on complex, long-horizon tasks. It addresses the issue of sparse rewards and all-failed rollout groups by using first-visit observation coverage to guide learning when outcome rewards are zero.

Large Language Model (LLM) agents frequently encounter difficulties with complex, multi-step tasks, often due to sparse reward signals and a phenomenon known as a "credit trap." Traditional group-based policy optimization can exacerbate this by over-sampling low-effect actions, leading to numerous failed attempts that provide no useful learning direction, while crucial state-changing actions remain under-sampled. To overcome this challenge, researchers propose Progress-conditioned Group Policy Optimization (ProGPO). This method activates specifically when all samples within a training group yield zero outcome reward. In such scenarios, ProGPO assigns higher relative advantages to trajectories that explore and visit more new states, recognizing that exploration and reaching new observations are prerequisites for eventual task success. Experimental evaluations on demanding agentic benchmarks, including ALFWorld and WebShop, using models like Qwen2.5-1.5/7B-Instruct, demonstrate that ProGPO consistently outperforms existing group-based baselines. The improvements are particularly significant on the most difficult tasks, indicating a more robust approach to training agents in environments characterized by delayed or infrequent rewards.

Why it matters

This research offers a significant advancement for developing more capable and robust AI agents, particularly for complex, multi-step tasks where traditional reward signals are sparse. Professionals building or deploying LLM agents can achieve better performance and faster learning in challenging environments.

How to implement this in your domain

  1. 1Evaluate current LLM agent performance on long-horizon tasks, identifying areas where sparse rewards hinder learning.
  2. 2Investigate integrating ProGPO's principles into existing reinforcement learning frameworks for agent training.
  3. 3Experiment with using observation coverage metrics as an auxiliary reward signal when primary task rewards are absent.
  4. 4Adapt agent training pipelines to dynamically adjust reward signals based on rollout group outcomes, prioritizing exploration in failure scenarios.
  5. 5Benchmark ProGPO-enhanced agents against current baselines on internal complex tasks to quantify performance improvements.

Who benefits

AI DevelopmentRoboticsGamingLogistics

Key takeaways

  • Sparse rewards and "credit traps" are major hurdles for LLM agents on long-horizon tasks.
  • ProGPO uses observation coverage to guide learning when outcome rewards are zero, breaking the credit trap.
  • This method significantly improves agent performance on complex benchmarks like ALFWorld and WebShop.
  • It provides a more effective way to train agents in environments with delayed or infrequent success signals.

Original post by Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He, Yu Li, Mengyi Liu, Pengwei Chen, Jun Xu, Lei Feng

"arXiv:2607.22724v1 Announce Type: new Abstract: Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group. However, on difficult long-horizon tasks, this comp…"

View on X

Originally posted by Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He, Yu Li, Mengyi Liu, Pengwei Chen, Jun Xu, Lei Feng on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Research

AI ResearchAI Engineering & DevTools

StageGuard Improves Sleep Staging by Enforcing Physiological Constraints

StageGuard is a new framework that enhances automated sleep staging by integrating physiology-informed priors, ensuring that deep learning models produce hypnograms that adhere to known biological rules. It significantly reduces physiologically implausible transitions and fragmentation while maintaining or improving accuracy.

Juntang Wang, Yihan Wang, Hao Wu, Jiayu Gao, Shixin Xu, Dongmian ZouJul 28, 2026
AI ResearchAI Engineering & DevToolsAI News & Tools

AI Model Improves Trustworthy Flood Prediction with Explainability

Researchers developed Context-Aware Concept Distillation (CACD), a framework that distills opaque Deep Learning models into interpretable, hydrology-aware surrogates for flood prediction. This method provides verifiable causal narratives required by disaster response authorities, achieving high fidelity and outperforming black-box baselines globally.

Eli Levinkopf, Efrat Morin, Claudia V. GoldmanJul 28, 2026
AI ResearchAI Engineering & DevTools

Diffusion Models' Generative Quality Gets Comprehensive Theoretical Analysis

This research provides a unified theoretical framework for understanding the generalization and convergence of score-based diffusion models. It decomposes the total generative error into four interpretable components, quantifying how training data, discretization, and optimization affect sample fidelity.

Jinshu Huang, Yiming Jiang, Chunlin WuJul 28, 2026