Optimizing LLM Allocation Under Uncertain Performance Data

Companies often face the complex task of assigning Large Language Models (LLMs) to various recurring workloads while adhering to a fixed budget. A major hurdle is the lack of a complete and accurate "quality table" that details how well each model performs on every specific task. Existing evaluation methods often fall short, either by not comparing models on identical work or by using proxy scores that don't reflect true business outcomes. The study highlights that even with more evaluation, the inherent uncertainty in how scores are produced means the quality table may remain undetermined. However, the optimal deployment decision itself might still be clear. The researchers propose a two-solve certificate approach: solve the allocation problem once with estimated performance and again with a least-favorable scenario. Agreement between these two solutions certifies the assignment; disagreement pinpoints areas where more evidence is crucial. To address this, they introduce CASE (causal active sequential experimentation), a method that strategically targets evaluation efforts to those uncertain model-workload pairs. Experiments on production logs reveal that measurement failure (inaccurate scoring) is a greater source of loss than suboptimal assignment based on existing estimates. The findings suggest that better information about model quality can yield more significant savings than simply optimizing assignments with uncertain data.