Director System Accelerates Distributed MoE Model Serving.

Researchers have introduced Director, a novel distributed serving system specifically designed to accelerate Mixture-of-Experts (MoE) models. MoE models rely on expert parallelism, where efficiency is heavily influenced by how experts are placed across GPUs, impacting communication and computation latencies. Existing solutions for expert placement often fall short when faced with diverse and rapidly changing request patterns, as they primarily rely on historical data. Director overcomes this by employing an online, proactive approach that predicts incoming requests' expert activation patterns using a lightweight cascaded predictor or a low-bit quantized replica. The system features an online migration module that enacts expert re-placements with near-zero downtime by scheduling migrations during compute-bound phases. At its core, a relaxation-based expert placement optimizer runs in polynomial time, achieving a strong approximation ratio. Extensive experiments show Director reduces end-to-end latency by 11-55% for popular MoE models like Mistral, DeepSeek, and Qwen.