New Contextual Bandit Improves LLM Routing with Surrogate Rewards.

Researchers have developed new algorithms for contextual bandit problems, specifically tailored for applications like Large Language Model (LLM) routing. Unlike traditional contextual bandits that assume conditional independence between "arms" (e.g., different LLMs or routing paths) and rely solely on direct feedback, this work addresses scenarios with correlated arms and access to auxiliary "surrogate" reward signals from machine learning models. The proposed methods leverage these surrogate rewards through two designs: a coupled reward-mixing approach that combines true and surrogate rewards for faster learning when surrogates are reliable, and a decoupled prediction-mixing approach. The decoupled method maintains separate estimators for bandit feedback and surrogate predictions, adaptively combining them to ensure robustness even if surrogate signals are noisy or inaccurate. Theoretical analysis provides regret guarantees, showing improved performance when surrogates are informative, while still matching reward-only bandit methods in worst-case scenarios. Empirical evaluations on LLM routing benchmarks demonstrate enhanced sample efficiency and better accuracy-cost trade-offs compared to existing contextual bandit and static routing methods.