MileGPO Improves Long-Horizon LLM Agent Policy Optimization

This research introduces MileGPO, a novel framework designed to tackle the challenging problem of credit assignment in long-horizon agentic reinforcement learning for LLM agents. Traditional methods often struggle when supervision is only available from final rewards, overlooking crucial intermediate steps. MileGPO aims to derive process-level credit from grouped on-policy rollouts through three key innovations. First, "Milestone Discovery" identifies potential milestones in successful trajectories and recurring traps in failed ones. Second, "Reliability-Calibrated Shaping (RCS)" assigns weights to these candidates based on outcome-based confidence, reinforcing reliable milestones and traps while down-weighting uncertain ones. Third, "Progress-Contrastive Calibration (PCC)" further verifies if a candidate signifies local progress and if its associated transition outperforms alternatives from the same state. MileGPO operates without requiring auxiliary models or additional environment interactions. Experiments on ALFWorld and WebShop demonstrate state-of-the-art performance, including a small in-distribution to out-of-distribution gap on ALFWorld. Ablation studies and credit diagnostics confirm the complementary roles of reliability weighting, local progress, and same-state branch evidence in resolving ambiguous intermediate credit.