Benchmarks Underestimate LLM Capabilities by 82%, New Frontier Reveals

Current benchmarks for large language models (LLMs) typically assess a single model's accuracy from a single run, which systematically understates their actual capabilities, especially when dealing with diverse data. This underestimation arises because different models excel at different tasks, and sampling multiple generations from a model can yield better results through selective retention. To address this, researchers propose the "Capability Frontier." The Capability Frontier is a Pareto frontier that illustrates the best possible performance at various cost levels by optimally selecting across multiple models and generations, effectively using an oracle. This method corrects for both the underestimation from single-model evaluations and the potential overestimation from simply taking maxima over noisy samples. The study analyzed 21 LLMs across 16 benchmarks covering tasks like coding, reasoning, and factuality. The findings are significant: correcting for single-model evaluation alone reduced the error rate by 54%. When also accounting for single runs, the improvement soared to 82%. This means state-of-the-art accuracy could be achieved with an 85% cost reduction. Probabilistic simulations further showed that higher query topic entropy leads to a greater performance gap between oracle routing and the best single model. These results suggest that the collective capabilities of LLMs are substantially underestimated, with major implications for how LLMs are evaluated and deployed in real-world, heterogeneous environments.