Oracle LLM Routing Overestimates Real-World Gains

The concept of "oracle routing" is often used to measure the maximum potential performance gain achievable by selecting the best language model (LLM) for each query from a pool. However, this paper identifies two critical flaws in this diagnostic: it often tests against a "best fixed model" selected on the same examples, which invalidates paired inference, and a full-information oracle has access to outcomes that no deployable router could observe in practice. The researchers meticulously separate three distinct estimands: the theoretical "outcome-oracle opportunity," the Bayes-optimal gain from a declared pre-answer signal, and the actual held-out gain of a learned router. They then develop and prove selection-valid confidence intervals that remain robust even when choosing the best fixed model or the best router from a family. This framework includes a signal-information sandwich and a greedy guarantee for building compact LLM pools. Applying these new diagnostics to eight LLM checkpoints across four benchmarks revealed a significant "population oracle gap" of 9.7 to 30.7 points on every task. However, the strongest deployable prompt router could only recover a small fraction, specifically 7.5% to 14.4%, of this theoretical opportunity. The analysis conclusively shows that while strong routers can outperform a single best fixed model, the realizable share of the oracle opportunity is small and certifiable, indicating that most of the potential gain remains untapped by current deployable solutions.