Inference Compute Significantly Impacts Frontier LLM Evaluation Scores

The evaluation of advanced large language models (LLMs) is increasingly moving towards more complex tasks that require extensive tool use and iterative problem-solving. This shift means that an LLM's observed performance is highly dependent on the computational resources allocated during the inference phase. Many current evaluation methods, however, still report performance based on a single, often restrictive, compute budget, which can lead to an inaccurate representation of a model's true capabilities. To investigate this, researchers tested up to 12 frontier LLMs across seven demanding benchmarks in areas like software engineering, mathematics, medicine, and cybersecurity. They employed a controlled experimental setup that incorporated three inference-scaling techniques: increasing token budgets, compacting context, and allowing multiple submission attempts, sometimes with minimal correctness feedback. The findings highlight three key points: Firstly, providing larger token budgets significantly boosts performance across various domains. Secondly, evaluations with fixed, limited budgets increasingly fail to capture the full potential of newer, more advanced models, which achieve much higher performance with greater compute. Thirdly, the effectiveness of different inference-scaling methods varies by benchmark, though repeated submissions generally improve performance. The study concludes that benchmark scores are heavily influenced by the evaluation protocol, advocating for reporting capabilities as a function of inference-time compute and explicitly detailing protocol choices, especially in critical applications.