FlexMoE Enables Flexible Pruning for MoE Language Models

FlexMoE is a novel framework designed to optimize the deployment of Mixture-of-Experts (MoE) language models, which are increasingly prevalent in large-scale AI. While MoE models achieve impressive capabilities through sparsely activated experts, the challenge remains in efficiently storing and serving the entire model, especially given diverse deployment budget constraints across different devices and workloads. FlexMoE addresses this by enabling "one-for-all" nested intra-expert pruning. The method works by ranking expert FFN channels by importance and then allowing each expert to learn discrete actions to prune its channels. A single training run, under gradually increasing cost pressure, produces a series of nested subnetworks, each optimized for a different budget. A key aspect is a single recovery fine-tuning step at a mid-pruning budget (e.g., 40%), which then transfers performance recovery to other unseen budgets. This approach significantly surpasses existing MoE compression baselines, retaining nearly 99.8% of base performance on models like Qwen2-57B-A14B even with 50% expert parameter pruning, and offers real memory and throughput benefits, including support for real-time online budget switching.