Pandora's Router Optimizes AI Model Allocation with Costly Estimation

Modern AI systems often consist of multiple specialized models, each with varying capabilities and costs. To optimize performance and efficiency, queries need to be routed to the most suitable model. This routing process typically requires estimating each model's expected return, but this estimation itself can be costly. Cheap estimators (e.g., embedding-based predictors) are fast but less accurate, while precise estimators (e.g., fine-tuned models with retrieval access) are expensive. This research frames this challenge as an instance of Pandora's Box, a classical problem involving optimal search with costly inspection. Under a Gaussian signal model, the resulting policies, termed Pandora's Router, provide closed-form expressions for the value of information. These expressions determine whether the cost of refining a value estimate for a specific model and input is justified. Experiments across diverse domains—including multi-LLM benchmarks, retrieval-augmented specialists, and LLMs with variable inference-time reasoning—demonstrate that Pandora's Router achieves the same routing quality as exhaustive estimation while significantly reducing the frequency of querying expensive estimators. The concept is also extended to a decentralized setting, Pandora's Bidder, where specialists independently decide on self-assessment before accepting a query, showing improved allocative efficiency when competing estimates are accurate.