Online Algorithm Optimizes LLM Selection Under Dynamic Constraints.

Deploying Large Language Models (LLMs) in edge-cloud inference systems presents significant challenges due to the diverse performance profiles of different models, unpredictable user demand, and the stochastic nature of their operation. Effectively selecting the right LLM for each incoming task is crucial for maintaining service quality and optimizing resource use, especially when faced with strict monetary budgets and latency guarantees. Researchers have formulated this complex problem as a constrained stochastic bandit learning task. The goal is for a system to sequentially choose models while adhering to both hard resource limits (packing-type constraints) and soft service-level agreements (covering-type constraints), all without prior knowledge of underlying reward, cost, or latency distributions. The system must also adapt to fluctuating task demands. A novel online learning algorithm has been developed to tackle this. It employs confidence-bound estimates and predictions of future demand to intelligently balance maximizing overall reward with ensuring long-term satisfaction of all constraints. Theoretical guarantees confirm its effectiveness, showing sublinear regret and constraint violations compared to an ideal offline benchmark. Synthetic workload experiments further validate its robustness in dynamic, resource-constrained environments.