Director System Optimizes Distributed MoE Serving with Proactive Expert Placement

This paper introduces Director, a novel distributed serving system designed to significantly accelerate Mixture-of-Experts (MoE) models. MoE models rely on expert parallelism for efficient serving, but their performance is highly dependent on how experts are placed across GPUs, which impacts communication and computation latencies. Existing optimization methods for expert placement often fall short when faced with diverse and rapidly changing request patterns, as they primarily rely on historical data. Director tackles this by adopting an online, proactive approach to expert placement. It addresses several key challenges: the inherent uncertainty of incoming requests' expert activation patterns, the cost associated with migrating experts between GPUs, and the computational complexity of optimizing placement in real-time. The system employs either a lightweight cascaded predictor or a low-bit quantized replica to anticipate expert activation patterns for incoming requests. A crucial component is Director's online migration module, which executes expert changes with near-zero downtime by scheduling migrations during compute-bound phases, thus bounding disruption. At its core, a relaxation-based expert placement optimizer operates under capacity constraints, running in polynomial time and achieving a near-optimal approximation ratio. Extensive experiments with popular MoE models like Mistral, DeepSeek, and Qwen demonstrate that Director reduces end-to-end latency by 11% to 55% compared to existing serving solutions.