Sticky Routing Improves MoE Memory Efficiency During Training

Mixture-of-Experts (MoE) models are designed to activate only a sparse subset of experts for each input token, which can lead to computational efficiency. However, a significant challenge arises in inference, particularly on edge devices: consecutive tokens often activate different experts, necessitating constant and costly weight swapping between slow storage and fast memory. Existing solutions typically involve system-level caching or post-hoc fine-tuning, which don't address the root cause during the initial pretraining phase. This paper proposes StickyMoE, a novel approach that integrates a differentiable routing consistency loss directly into the MoE training process. This loss function penalizes sudden changes in expert assignments between adjacent tokens, encouraging the router to maintain the same expert selection for semantically coherent sequences. StickyMoE requires no architectural modifications and introduces only a single hyperparameter. Crucially, unlike post-hoc methods, it allows expert representations and routing decisions to co-adapt from the very beginning of training. Experiments on smaller-scale MoE language models demonstrate that StickyMoE can reduce the expert switch rate by as much as 60%, with a minimal perplexity degradation of less than 4%. This performance improvement in routing temporal locality is achieved most efficiently during the training phase, outperforming post-hoc fine-tuning methods in terms of quality-locality trade-offs.