Online Linear Programming Optimizes LLM Serving Routing

Current methods for routing requests to large language model (LLM) serving infrastructure often rely on heuristics, lacking explicit ties to service-level objectives (SLOs) and offering limited control over performance trade-offs. This paper proposes a novel multi-objective optimization framework that redefines routing as an online linear programming problem, assigning interpretable decision rewards. The core of the solution is an efficient bid-price control policy, which admits requests only when their SLO-weighted benefit surpasses their shadow prices. To meet the demanding millisecond-level decision requirements, the authors developed a warm-started, projected first-order update mechanism that dynamically tracks dual shadow prices with predictable runtime. Integrated into the Vidur simulator, this new router demonstrated substantial improvements across various SLO regimes, including end-to-end latency, time-to-first-token, throughput, and tail performance, outperforming standard heuristic baselines. The findings highlight the superior performance of a science-based, optimized approach compared to traditional heuristic-driven methods in LLM serving.