MoE LLM Layers Show Depth-Dependent Sensitivity for Compression

Mixture-of-Experts (MoE) architectures are crucial for scaling large language models (LLMs) efficiently, but understanding the importance of individual MoE layers for compression has been challenging. This research presents a systematic layer-wise sensitivity analysis of the Qwen3.6-35B-A3B model, using a technique called magnitude-based expert masking. The study evaluated the model on a cross-lingual code translation benchmark.The core finding is that the sensitivity of MoE layers is strongly dependent on their depth within the model. Early layers (0-9) and middle layers (10-29) are found to be very fragile, meaning that masking experts in these layers significantly degrades performance. In contrast, late layers (30-39), and particularly the very-late layers (35-39), can tolerate substantial masking of low-magnitude experts without significant performance loss.This depth-aware approach to expert masking offers a more effective strategy for model compression than uniform masking. For instance, a policy focusing on masking experts in very-late layers achieved a strong quality-to-masked-expert tradeoff, retaining high performance while significantly reducing the number of active experts. These insights provide a practical foundation for optimizing MoE models through physical weight surgery, activation-based expert scoring, and targeted training recovery.