LLM Reasoning: When Does Extra Thinking Pay Off?

A new study investigates the economic viability of enabling extended reasoning capabilities in large language models, moving beyond mere accuracy benchmarks. Researchers introduced the "Token Economy Score" (TES), a metric that quantifies the accuracy gain from reasoning relative to the increased token cost. This allows for a marginal benchmarking approach, comparing reasoning models against non-reasoning baselines. The empirical analysis, spanning 151 model evaluations across various benchmarks, revealed that task structure is a stronger predictor of reasoning efficiency than nominal difficulty. Sequential inference-chain tasks, such as complex math or code generation, showed high TES, while knowledge-recall tasks exhibited low TES. The study also identified systematic diminishing returns with increased reasoning effort, sometimes even leading to reduced accuracy. Furthermore, deployment context significantly impacts economic viability, with on-premises deployment potentially making otherwise costly reasoning workloads more feasible due to changes in inference spend.