Sticky Routing Boosts MoE Memory Efficiency in LLMs.

Researchers have introduced StickyMoE, a novel training approach for Mixture-of-Experts (MoE) models aimed at improving memory efficiency during inference. MoE models typically activate only a sparse subset of experts per token, but frequent expert switching between consecutive tokens leads to constant, costly weight swapping between slow storage and fast memory, especially on edge devices. StickyMoE addresses this by incorporating a differentiable routing consistency loss during the model's pretraining phase. This loss function penalizes sudden changes in expert assignments between adjacent tokens, encouraging the router to maintain the same expert for semantically coherent spans of text. The method requires no architectural changes and adds only one hyperparameter. Unlike post-hoc fine-tuning, StickyMoE allows expert representations and routing decisions to co-adapt from the start of training. Experiments show it reduces expert switch rates by up to 60% with less than 4% perplexity degradation, demonstrating a superior trade-off between quality and locality.