New Pruning Method Boosts Sparse MoE LLM Performance.

Sparse Mixture-of-Experts (MoE) language models contain considerable redundancy among their routed experts, yet pruning them effectively without specific downstream calibration data remains a challenge. Existing pruning methods often rely on a single aggregated importance score, which can inadvertently bias the retained experts towards patterns dominant in the calibration data. This research proposes Generic TB-Coverage, a new approach to address this. Instead of collapsing expert utility into a single score, Generic TB-Coverage profiles each expert's utility separately across multiple generic text corpora, such as WikiText2 and C4. It then enforces a fixed-budget coverage rule, ensuring that high-utility experts from each corpus are preserved before constructing the final pruning mask. This method avoids bias by considering a broader range of expert contributions. Evaluations on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base models, with pruning budgets of 25%, 50%, and 75% retention, showed that Generic TB-Coverage improved average accuracy on six common zero-shot benchmarks. It also reduced perplexity degradation on the calibration corpora, with the most significant gains observed under aggressive pruning. This indicates that preserving cross-corpus expert coverage is an effective prior for MoE pruning using generic data.