SEAGym Environment Evaluates Self-Evolving LLM Agents' Harness Updates

Self-evolving large language model (LLM) agents primarily improve by modifying their "agent harness," which encompasses the structured execution layer around the base model, including prompts, memory, tools, and interaction loops. Existing evaluation methods often oversimplify this process, reducing it to isolated task scores or a single performance curve, which can obscure the true nature of improvements. It becomes difficult to discern if an update genuinely enhances reusability, overfits recent tasks, increases operational costs, or negatively impacts older behaviors. To address these limitations, SEAGym has been developed as a dedicated evaluation environment for self-evolving LLM agents. SEAGym provides a comprehensive framework for measuring agent harness updates across various stages: training, validation, testing, and replay, alongside cost records. It transforms Harbor-compatible benchmarks into dynamic sources for self-evolution tasks, offering distinct batches for training, frozen validation for updates, held-out views for both in-distribution (ID) and out-of-distribution (OOD) transfer, and diagnostic replay capabilities. By instantiating SEAGym on benchmarks like Terminal-Bench 2.0 and HLE, researchers compared different evolution protocols. The results demonstrate that these diverse evaluation views offer complementary signals about the evolution process. For instance, frequent updates might not always translate to improved held-out performance, useful intermediate snapshots could degrade later, and factors like source diversity and the underlying model backend can significantly influence the reliability of the agent harness.