ArbiGraph Benchmarks AI Agents' Long-Term Context Management.

Evaluating the ability of tool-assisted language agents to manage context over extended, multi-step reasoning workflows is a significant challenge. Existing benchmarks often fall short in assessing how agents retain, update, compose, and discard task-relevant information across complex dependencies. To address this, ARBIGRAPH, a novel benchmark generator, has been introduced. ARBIGRAPH represents each task as a natural-language problem paired with an executable Python solver, and it composes these tasks through typed intermediate states, such as scalar and list values. This design allows for the creation of controllable task graphs where parameters like length, dependency structure, number of distractors, and value types can be varied, all while maintaining exact automatic verification of results. The benchmark includes task categories like math problems, GSM-style word problems, and Python-tracing. Initial evaluations using a Qwen3.5-27B tool-assisted agent across four topologies revealed a substantial performance drop. While the agent showed high accuracy on isolated tasks, its performance degraded by up to 33.3% on more complex, dependent tasks, particularly on branching chains of math problems. This demonstrates that ARBIGRAPH effectively exposes critical failures in long-term context management that are not apparent from single-task evaluations, providing a valuable tool for advancing research in this area.