New Method Boosts MoE Fine-tuning Efficiency with Adaptive Pruning

Mixture-of-Experts (MoE) models are known for their scalability but can be expensive to fine-tune due to redundant experts and inefficient parameter allocation. Traditional parameter-efficient fine-tuning (PEFT) methods like LoRA often overlook the dynamic routing within MoE architectures, leading to suboptimal resource utilization. This new research proposes EPnG, an adaptive framework designed to address these limitations. EPnG intelligently reallocates LoRA capacity by evaluating expert importance based on router gate probabilities. It prunes under-utilized experts and expands high-importance experts through rank growth, all while adhering to a fixed parameter budget. Experiments on OLMoE and Qwen1.5-MoE models demonstrated that EPnG consistently surpassed LoRA in performance under the same budget, achieving results comparable to full fine-tuning while updating only 0.55%-0.72% of total parameters, representing a 140x-180x reduction. This indicates that aligning PEFT strategies with MoE routing dynamics can lead to more effective and scalable fine-tuning.