ESOpt Fine-Tunes LLM Agents for Long-Horizon Tasks with Less GPU

Fine-tuning large language models (LLMs) for complex, long-horizon agentic tasks presents significant challenges, particularly with traditional reinforcement learning (RL) methods. RL's reliance on heavyweight backpropagation and its difficulty with credit assignment over extended trajectories make it impractical for larger LLMs and longer task sequences. This research highlights these limitations and proposes an alternative approach. The paper advocates for evolution strategies (ES) as a more suitable method for fine-tuning long-horizon LLM agents. ES offers three key benefits: it enables full-parameter optimization with only inference-level GPU memory, making it feasible for very large LLMs; its black-box feedback interface provides flexibility for integration with prompt-space evolution; and it scales better with long horizons by performing trajectory-level parameter attribution without needing to decompose rewards. Based on these insights, the authors introduce Agentic ESOpt, a framework designed for flexible parameter-context co-evolution. It iteratively samples parameter perturbations, evaluates agent performance, and applies reward-weighted updates. An additional cosine decay schedule for perturbation scale improves exploration. Experiments on WebArena-Lite demonstrate that Agentic ESOpt significantly improves performance, even enabling online prompt-parameter co-evolution for automatic heuristic design.