New Routing Method Controls Expert Load in Frozen MoE Models

Mixture-of-Experts (MoE) models use a router to determine which experts process each token, defining a "routing law" for every token. This research investigates how to control the collective routing decisions of multiple tokens while ensuring each individual token's routing law remains unchanged. The proposed method, Hierarchical Copula-Gumbel-Top-K (CGA), offers a dual approach to managing expert traffic. Within a group of related tokens, CGA can positively correlate expert choices, increasing the coherence of expert sets. Conversely, across distinct groups, it can introduce negative dependence to balance expert loads. The key innovation is that these adjustments to coherence and load dispersion are achieved without altering the base MoE model or the individual token routing probabilities. The system uses a small, trainable controller that operates on frozen features, meaning the main MoE network is only evaluated in the forward pass, and gradients are confined to this controller. Initial small-scale tests validate the mechanism and training approach, though further work is needed to establish task-level performance gains.