New Pruning Method Compresses MoE Models While Retaining Accuracy

Mixture-of-Experts (MoE) models are powerful but come with substantial deployment costs due to their large memory footprint and high inference overhead. Existing compression techniques often prune entire experts or rank them coarsely, which can be inefficient because significant redundancy often exists even within important experts, concentrated in specific channels. This paper proposes a novel structural pruning framework specifically designed for MoE models. It moves beyond expert-level decisions to fine-grained channel-level pruning. The core innovation is to frame the prune-ratio allocation as a channel-score coverage maximization problem, which is then solved efficiently using an attribution-based approximation. This allows for precise identification and removal of redundant channels. Experiments on DeepSeek and Qwen MoE models demonstrate the effectiveness of this approach. When combined with 4-bit quantization, the method maintains model accuracy even with 50% or 25% structural pruning. For instance, on Qwen3-30B-A3B, it achieved a 5.27x reduction in memory footprint and consistently outperformed state-of-the-art baselines across various benchmarks.