New Bandit Algorithms Optimize LLM Routing with Surrogate Rewards.

Optimizing the routing of requests to various Large Language Models (LLMs) is a critical challenge, especially when considering trade-offs between accuracy and computational cost. Traditional contextual bandit approaches typically assume conditional independence between "arms" (e.g., different LLMs or configurations) and rely solely on direct bandit feedback. However, in real-world LLM routing, arms can be correlated, and auxiliary information, such as surrogate reward signals from a cheaper, less accurate model, might be available but potentially noisy or misspecified. This research addresses these complexities by proposing new contextual bandit algorithms designed for scenarios with correlated arms and access to surrogate rewards. Two main designs are introduced: a coupled reward-mixing approach and a decoupled prediction-mixing approach. The coupled method combines true and surrogate rewards to accelerate learning when surrogates are reliable. The decoupled prediction-mixing approach, on the other hand, maintains separate estimators for bandit feedback and surrogate rewards, adaptively combining their predictions. This decoupling provides robustness against misspecified surrogates, ensuring that in the worst case, regret guarantees are comparable to reward-only bandit methods. Crucially, it achieves improved regret when surrogate predictions are sufficiently informative. Theoretical analyses support these claims, and experiments on LLM routing benchmarks demonstrate enhanced sample efficiency and consistently better accuracy-cost trade-offs compared to standard contextual bandit baselines and static routing methods.