EasyBalance Boosts MoE Inference by Cross-Layer Load Balancing

Mixture-of-Experts (MoE) models, while powerful, face significant challenges in distributed inference due to load imbalance. When routing distributions are skewed, some devices hosting lightly loaded experts end up waiting for heavily loaded ones, leading to inefficient GPU utilization. Existing solutions often involve complex expert replication or migration, which adds overhead and limits flexibility. A new approach, EasyBalance, tackles this by introducing a cross-layer load balancing strategy. Its core insight is that experts from different layers can be considered naturally redundant, and their workloads can be jointly executed to mitigate individual imbalances. EasyBalance intelligently schedules a subset of cross-layer workloads at each MoE step, deferring others for future balancing opportunities. This method requires no changes to the expert-device mapping, making it instantly adaptable and virtually overhead-free. Experiments show it consistently accelerates distributed MoE inference, often reducing GPU idling by over 40%.