FreeBalance Optimizes MoE Load Balancing with Workload Prediction

Load imbalance is a significant bottleneck in the efficiency of expert parallelism within distributed inference of Mixture-of-Experts (MoE) models. When routing distributions are skewed, the most heavily loaded expert rank can stall global execution, directly increasing inference latency. While offline expert placement can address persistent imbalances, real-world multi-task serving workloads exhibit dynamic routing patterns that vary by layer and batch, making online load balancing essential. Existing online balancing methods typically collect routing statistics *after* each MoE router, meaning expert weight loading or migration can only begin once routing decisions are available. This places migration overhead directly onto the critical inference path. This research observes that online balancing could largely overlap with preceding computations, such as attention, if routing distributions could be accurately predicted in advance. Introducing FreeBalance, a lossless online load-balancing framework that achieves this by predicting residual workloads. FreeBalance utilizes cross-layer similarities in hidden representations within the residual network to build a lightweight workload predictor. This enables proactive planning for expert migration *before* routing decisions are finalized, creating substantial overlap between weight transfer and computation-heavy pre-routing stages. A cost model further constrains the number of swaps to ensure synchronization overhead is fully hidden within the available window. Experiments show FreeBalance reduces the max-to-mean rank load ratio by 32.8% and end-to-end prefill latency by 13.1%, effectively hiding the balancing overhead for an average of 5.1 experts per layer.