Pruning MoE Models: Balancing Utility and Reliability in Biomedicine.

Mixture-of-Experts (MoE) models offer efficiency gains through selective expert activation but demand significant memory. Structured expert pruning is a technique to reduce deployment costs, especially in resource-constrained environments. However, previous research primarily focused on pruning's impact on model utility, largely overlooking its effect on factual reliability, a critical concern in high-stakes domains like biomedicine. This paper examines how domain-specific expert pruning influences both the utility and factual reliability of MoE models. Researchers evaluated four MoE models, six pruning methods, and various pruning ratios across generation and classification tasks in both in-domain (biomedical) and cross-domain settings. The findings indicate that moderate pruning can maintain in-domain utility without an immediate drop in reliability, though the risk of hallucinations increases with extreme pruning. Crucially, both utility and reliability degrade rapidly when models are applied to general domains. This suggests that safe compression strategies for MoE models are highly dependent on the specific task and domain, emphasizing the need for reliability assessment alongside utility evaluation for high-stakes deployments.