PivoARL Improves LLM Agent Learning Efficiency

Large language model (LLM) agents demonstrate strong decision-making abilities in complex, long-horizon interactive tasks, yet they often struggle to effectively learn from their failures. Current methods, such as full retries, are computationally expensive, while simple experience retrieval can dilute critical learning signals. This research proposes PivoARL, a novel self-feedback retry framework designed to enhance experience exploitation in LLM agents. PivoARL's core innovation lies in its ability to identify the "pivotal erroneous turn" within a failed trajectory through structured reflection. Instead of restarting from scratch, the agent performs a local retry only from this pivotal state, thereby efficiently reusing the correct prefix of the previous attempt and significantly reducing redundant interactions. From an information-gain perspective, PivoARL concentrates useful experience signals near the error boundary, effectively mitigating the signal dilution seen in state-agnostic experience utilization. This insight led to the design of a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, further optimizing reflection quality through implicit reflection returns. Systematic evaluations across four agent tasks and seven search-based QA benchmarks showed PivoARL achieving substantial improvements, including an average gain of 11.5% on Pass@2/3 over MetaRL and over 45% improvement on Minesweeper compared to GiGPO, while reducing interaction turns by about 42% on average.