AHEAD Improves LLM Agent RL with Adaptive Hindsight Distillation

Traditional reinforcement learning for multi-turn LLM agents often applies uniform rewards, making it hard to pinpoint specific decisions leading to success or failure. While self-distillation offers finer-grained supervision, existing methods typically use the same privileged information for all steps, overlooking that routine actions need less guidance than critical errors. The AHEAD framework addresses this by introducing a step-aware approach. It provides the teacher model with dense environment feedback for all steps and, crucially, adds LLM-generated corrective hints specifically for error steps. This targeted guidance helps overcome the limitations of environment feedback alone in critical situations. AHEAD integrates minimally with standard GRPO algorithms and demonstrates substantial improvements in task success rates on benchmarks like ALFWorld and WebShop, achieving higher success with fewer training steps and tighter interaction budgets compared to outcome-only RL and previous self-distillation techniques.