SpecPrefetch Boosts MoE Model Inference on Edge Devices

A novel framework called SpecPrefetch has been introduced to address the deployment challenges of sparse Mixture-of-Experts (MoE) models, particularly on devices with constrained accelerator memory. MoE models, while powerful, often struggle with the overhead of offloading inactive experts to host memory, which creates a routing-dependent transfer bottleneck during inference. SpecPrefetch tackles this by using a lightweight adapter to predict next-layer expert candidates for asynchronous transfer, separate from the native router that determines the final executed experts. This separation allows for prefetching without altering the pretrained routing semantics, meaning prediction errors only affect transfer efficiency, not model outputs. Combined with a window-aware scheduler that prioritizes feasible transfers, SpecPrefetch significantly reduces exposed expert-loading latency. Empirical tests on models like Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny show superior expert recall with fewer trainable parameters. On a Snapdragon 8 Elite device, it improved decoding throughput by up to 20% over existing offloading runtimes, demonstrating practical benefits for edge deployment.