ExFold Accelerates MoE LLM Inference with Unified Expert Folding

Mixture-of-Experts (MoE) models offer high capacity while maintaining efficient per-token computation, but their inference serving faces challenges across two distinct phases: prefill, which is compute-bound, and decode, which is memory-bound. Existing acceleration techniques often optimize only one aspect or implicitly approximate the impact of excluded experts. A new framework called ExFold addresses this by introducing a unified, training-free approach to accelerate both MoE prefill and decode. It frames both phases as an output-approximation problem, selectively executing a constrained set of experts while projecting the contributions of excluded experts onto the retained ones using calibrated scalar projectors. ExFold leverages the observation that many expert outputs are directionally aligned, differing mainly in magnitude. By calibrating a pairwise scalar-projector matrix on unlabeled data, it can fold excluded expert contributions during inference. Implemented as a plug-and-play vLLM plugin with a custom CUDA kernel, ExFold demonstrates up to 1.41x speedup for Time To First Token (TTFT) and 2.45x for Tokens Per Output Token (TPOT), while maintaining approximately 99% of the original model quality.