New Pruning Method Compresses MoE Models by Targeting Channel Redundancy

Mixture-of-Experts (MoE) models are known for their computational efficiency at scale, but their substantial memory footprint and inference overhead pose deployment challenges. Existing compression techniques typically operate at a coarse expert level, either removing whole experts or ranking them by overall importance scores. However, this approach often overlooks fine-grained redundancies within individual experts, leading to suboptimal compression. This research addresses this limitation by observing that information within MoE experts is often concentrated in a small subset of channels, leaving considerable redundancy even in experts deemed important. The authors propose a novel structural pruning framework specifically designed for MoE models. Their method redefines the prune-ratio allocation as a channel-score coverage maximization problem, which is efficiently solved using an attribution-based approximation. Experiments on DeepSeek and Qwen MoE models demonstrate that this approach maintains model accuracy even with 50% or 25% structured pruning, especially when combined with 4-bit quantization. For instance, on Qwen3-30B-A3B, the method achieved a 5.27x reduction in memory footprint and consistently outperformed state-of-the-art baselines across various benchmarks.