New WAIT Algorithm Extension Optimizes LLM Inference for Bursty Workloads

Large Language Models (LLMs) are widely used, but their inference performance is often evaluated under the assumption of constant, Poisson request arrival rates. This assumption frequently fails to reflect real-world traffic, which is inherently bursty and dynamic. Existing scheduling algorithms, while improving throughput and latency, are not optimized for these fluctuating workloads. This research introduces a lightweight extension to the state-of-the-art WAIT algorithm, designed to adapt to time-varying arrival rates without needing prior knowledge of traffic patterns. The proposed algorithm continuously estimates request intensity online by observing interarrival times. Through simulation-based evaluations using Markov Modulated Poisson Process (MMPP)-based synthetic workloads with diverse request types, the modified WAIT algorithm demonstrated superior performance. It achieved higher throughput compared to Sarathi-Serve, ORCA, and vLLM in evaluated low arrival-rate shift scenarios, all while maintaining comparable latency. This indicates a significant step towards more robust and efficient LLM inference systems in real-world, dynamic environments.