New Benchmark Method for Long-Horizon AI Task Failures

New research introduces a framework for more accurately evaluating AI agent performance on complex, long-horizon tasks. The paper highlights that simply observing increased failure rates in longer tasks doesn't explain the root cause, as errors can compound or later stages can become inherently harder due to earlier actions or accumulated context. To address this, the authors propose the "horizon residual" metric. This involves comparing an agent's actual success on a full long-horizon task against a predicted success baseline derived from its performance on matched short, individual stages. The difference, or residual, helps to isolate failures specifically attributable to the long-horizon nature of the task, such as "trajectory-induced degradation" or "context rot," rather than just ordinary errors. This method requires consistent agent configurations and predefined stage, checkpoint, and information parameters for valid comparison.