LLM Benchmark Gains Don't Always Reflect True Capability Expansion

The common assumption that higher LLM benchmark scores directly indicate greater model capability is challenged by new research. This study distinguishes between "realization," where a model produces an answer already within its grasp, and "reachability," where it finds an answer that was previously inaccessible. The findings suggest that many benchmark gains are due to better realization rather than a true expansion of the model's underlying knowledge or problem-solving capacity.The researchers conducted question-level audits, revealing that inference-time layer routing, even random routes, can expand reachability, but these gains are often lost without access to the correct answer during the process. Furthermore, they found that training procedures, such as DAPO, can increase deployed performance while the actual reachable ceiling of the model either remains stagnant or decreases. This highlights a critical disconnect between observed performance metrics and the fundamental capabilities of LLMs, advocating for more comprehensive evaluation methods that report both realized performance and reachability.