New Benchmark Challenges AI Agents with Long, Complex Tasks.

Researchers have introduced Long-Horizon-Terminal-Bench, a novel benchmark designed to rigorously test the capabilities of AI agents on tasks that are significantly more complex and time-consuming than those in existing benchmarks. Comprising 46 long-horizon tasks across diverse categories like software engineering, scientific computing, and interactive games, this benchmark moves beyond simple, short-duration problems. A key innovation of Long-Horizon-Terminal-Bench is its detailed grading system. Unlike previous benchmarks that only assess final outcomes, this new system decomposes tasks into fine-grained subtasks, enabling dense intermediate rewards and partial credit. This allows for a more nuanced evaluation, capturing an agent's progress and partial solutions in open-ended workflows, rather than just a binary pass/fail. Initial evaluations of 15 frontier models revealed that even the strongest agents struggled, achieving only a 15.2% pass rate at a 0.95 partial-reward threshold. Agents consumed millions of tokens and hours of execution time per task, highlighting the significant headroom for improvement in long-horizon planning, context management, and iterative debugging for current AI agents.