MODE Quantization Boosts MoE Multimodal LLM Efficiency

Mixture-of-Experts Multimodal Large Language Models (MoE-MLLMs) offer impressive performance but come with substantial GPU memory requirements, making efficient compression techniques essential. While expert-level mixed-precision quantization has proven effective for MoE-LLMs, it struggles with MoE-MLLMs due to specific biases in how expert importance is estimated. Two key biases were identified: first, the numerical dominance of vision tokens in cross-modal interactions skews expert selection frequency, overshadowing experts critical for text processing. Second, within the vision modality, a large proportion of redundant visual tokens further distorts frequency statistics, obscuring experts vital for informative visual content. To overcome these issues, a new framework called MODE (Modality-Decomposed Expert-Level Mixed-Precision Quantization) has been developed. MODE addresses these biases by explicitly decomposing expert selection frequency based on modality and by filtering out redundant vision tokens to obtain a more accurate visual frequency. These refined signals, along with quantization sensitivity per modality, are integrated into an Integer Linear Programming formulation to assign optimal bit-widths to each expert within a given memory budget. Experiments show that MODE effectively limits performance loss to under 2.9% at W3A16, with even greater gains at more aggressive 2-bit settings, making MoE-MLLMs significantly more memory-efficient.