Tail-Aware Scheduling Optimizes LLM Inference Latency Without Prediction.

Serving Large Language Models (LLMs) presents significant challenges due to the extreme variability in output length, which complicates size-based scheduling. Current LLM schedulers often approximate Shortest Job First (SJF) or Shortest Remaining Processing Time (SRPT) by predicting decode lengths or ranks, primarily reporting mean-centric metrics like Time-To-First-Token (TTFT) and Time-Between-Tokens (TBT). However, these prediction-driven policies can be unstable under shifts in data distribution, bursty arrival patterns, and GPU memory pressure. Crucially, they offer limited control over tail latency (P90-P99), which is a dominant factor in user experience, even with perfect knowledge of decode lengths. To address these limitations, researchers propose a novel distribution-aware, prediction-free scheduling framework. This system replaces explicit length prediction with a soft priority boosting mechanism, which is driven by lightweight statistical signals. The design also co-optimizes scheduling with cache-aware preemption, effectively accounting for the memory-coupled decode dynamics across various workload mixes. Evaluations conducted on both production and open-source traces demonstrate the framework's effectiveness. The new method reduces P99 Time-To-Last-Token (TTLT) by up to 35-50% relative to SRPT, even when SRPT has perfect length knowledge. Furthermore, it reduces TTFT by 34-47% across diverse workloads, including those heavy in reasoning and chat. These results highlight a robust alternative for optimizing tail latency in online LLM serving environments.