SLARouter Optimizes LLM Routing with Cost and SLA Guarantees.

The inference costs associated with large language model (LLM) applications are escalating rapidly due to increasing demand and infrastructure expenses. Simultaneously, users expect high-quality responses, often formalized through Service Level Agreements (SLAs) in commercial contexts. This creates a fundamental conflict between managing operational costs and ensuring user satisfaction. While existing cost-aware LLM request routing methods show promise, they typically rely on complete feedback, offline training, extensive per-workload tuning, and often lack explicit SLA guarantees or real-time adaptability. To address these limitations, researchers have developed SLARouter, an innovative online routing algorithm. SLARouter is designed to learn a cost-optimal routing policy using only the sparse, one-sided user feedback commonly available in production systems. A key feature of SLARouter is its theoretical guarantees for both achieving cost optimality and strictly adhering to SLA compliance. Extensive experiments across various LLM benchmarks demonstrate that SLARouter effectively meets SLA constraints without requiring specific tuning for each benchmark, leading to significant operational cost reductions of up to 2.2 times compared to current baseline methods.