AcceptMoE Boosts MoE Speculative Decoding Throughput and Efficiency

Speculative decoding is a technique used to accelerate large language model inference by verifying a tree of draft tokens in a single forward pass. For Mixture-of-Experts (MoE) models, this process can be inefficient because it might activate a large union of experts, even if only a few tokens are ultimately accepted. This leads to high computational costs, especially when experts are offloaded, increasing data transfer traffic. This paper introduces AcceptMoE, a novel expert selection mechanism designed to address these inefficiencies. AcceptMoE dynamically determines which experts are eligible for a given verification block. It combines the target model's router scores with offline-estimated commitment probabilities, eliminating the need for a predefined expert budget. When experts are offloaded, AcceptMoE conditions eligibility on cache residency rather than prefetching based on predicted routes. Across 12 model-task pairs, AcceptMoE achieved 1.29 to 2.06 times the throughput of baseline speculative decoding, while reducing host-to-device traffic by 73.6% to 77.1%, with only a minor 0.27 percentage point reduction in accuracy.