Beyond Accuracy: Evaluating AI Agents Post-Benchmark Saturation

Traditional AI benchmark evaluation often leads to the retirement of benchmarks once models achieve near-perfect accuracy, assuming they are no longer useful. This research challenges that paradigm, suggesting that focusing solely on accuracy overlooks several other critical aspects of AI agent performance. The authors propose evaluating agents across six additional dimensions: construct validity, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and the uplift from human-agent collaboration. The study uses CORE-Bench Hard, a benchmark for computational reproducibility of scientific code, as a practical example. Even after accuracy saturation, measuring agents along these broader dimensions provided significant insights. For instance, it revealed threats to construct validity, led to an improved benchmark (CORE-Bench v1.1), and demonstrated a statistically significant speedup in human-agent collaboration on real-world tasks. This approach advocates for a more rigorous and holistic evaluation framework for AI systems.