PARSER Compresses MoE LLMs by Prioritizing Output Accuracy

Mixture-of-Experts (MoE) architectures are crucial for scaling large language models (LLMs) efficiently, but they still demand substantial GPU memory, necessitating compression techniques. Residual sparsification is a common method that decomposes expert projection matrices into a shared base and per-expert residuals, then compresses these residuals. However, existing methods compress each residual independently, aiming to minimize individual matrix errors. This independent compression often leads to accuracy degradation because small errors in individual matrices can propagate and amplify across an expert's coupled computations, resulting in larger errors in the final output. To counter this, PARSER proposes shifting the compression objective to directly preserve the expert's output error. It introduces "output importance" to quantify each component's contribution to the overall expert output error, guiding a more effective sparsification. Experiments show PARSER significantly narrows the accuracy gap to uncompressed models while achieving the same memory reduction.