New LLM Router Optimizes Latency, Accuracy, and Cost Jointly.

A new research paper presents an advanced query router for large language models (LLMs) that goes beyond traditional optimization for just accuracy and cost. This router specifically incorporates latency into its decision-making process, a critical factor often overlooked by existing load-balancing policies. The challenge in integrating latency is its dynamic nature, depending on prompt length, current model workload, and serving framework policies. To address this, the researchers designed a lightweight latency estimator that simulates autoregressive token batch processing, providing accurate time-to-first-token (TTFT) estimates. By integrating this estimator, the latency-aware router can jointly optimize all three factors: latency, accuracy, and cost. Experimental results show that this joint optimization can lead to up to a 40% improvement in accuracy-cost utility while maintaining the same latency levels as standard load-balancing methods, significantly enhancing inference efficiency for dynamic LLM workloads.