New Framework Predicts LLM Agent Performance, Reducing Evaluation Costs

Evaluating large language model (LLM) agents on comprehensive benchmarks like SWE-Bench and GAIA is notoriously expensive and time-consuming, often costing thousands of dollars and taking days for a single assessment. In contrast, traditional LLM benchmarks that test individual capabilities are much faster and cheaper. A new framework called PACE (Proxy for Agentic Capability Evaluation) aims to bridge this gap by predicting agentic performance using a compact, carefully chosen subset of atomic evaluation instances. PACE constructs proxy benchmarks by identifying instances from existing non-agentic evaluations that most reliably correlate with performance on target agentic benchmarks. The framework employs a regression model to map scores from this compact subset to the full agentic benchmark score. PACE-Bench, a concrete proxy benchmark developed using this method, demonstrates high accuracy in predicting agentic scores across various models and benchmarks, achieving a mean absolute error under 4% and reducing evaluation costs to less than 1% of full agentic evaluation. This enables faster, cheaper iteration in LLM agent development.