MESH Optimizes Memory for Mixture-of-Experts Training.

Training Mixture-of-Experts (MoE) models, which are increasingly popular for scaling large language models, often faces significant memory constraints due to the optimizer state required for their numerous parameters. While memory-efficient optimizers like Sinkhorn gradient descent exist for dense Transformer matrices, directly applying them to MoE models has proven unreliable, leading to substantial performance degradation. This paper investigates the reasons behind this failure, specifically in the context of routed MoE expert matrices. It identifies that the conditional and temporally varying nature of expert gradients is poorly handled by stateless Sinkhorn normalization. To address this, the researchers propose MESH (Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training), which introduces a hidden-momentum Sinkhorn update specifically for MoE experts. MESH restores a crucial temporal first-moment signal throughout the gradient-buffer lifecycle without needing to store the full expert first moment as optimizer state. An optional block-preconditioned variant further refines the memory-quality trade-off. Experiments demonstrate that MESH reduces optimizer-state memory by over 60% and peak PyTorch CUDA allocation by about 12.6% compared to AdamW, with only a modest gap in evaluation loss. The core finding is that temporal smoothing before matrix normalization is key for MoE experts.