SAEM Boosts MoE Inference for Chain-of-Thought Reasoning.

Chain-of-Thought (CoT) prompting enhances large language model (LLM) reasoning by breaking down complex problems into sequential steps. However, this sequential nature often leads to increased decoding latency and higher memory consumption. Mixture-of-Experts (MoE) models, while scaling capacity through sparse expert activation, typically require full expert weights that can exceed GPU memory, necessitating costly GPU-CPU data transfers. Existing inference runtimes often treat all tokens uniformly, overlooking a crucial characteristic of CoT traces: consecutive reasoning stages exhibit coherent and predictable patterns in expert activation. To address these inefficiencies, researchers propose SAEM (Stage-Aware Expert Management), an MoE inference runtime designed to detect reasoning stage boundaries. SAEM exploits the stage-level activation coherence to intelligently guide expert placement and optimize memory usage. It integrates stage-aware caching, expert-aligned token repacking, and in-situ CPU execution to minimize data transfer and kernel fragmentation. Evaluated on mathematical and scientific reasoning workloads, SAEM demonstrated substantial performance gains. It achieved an average 1.33x throughput improvement over leading state-of-the-art caching and offloading baselines under constrained GPU memory. This improvement rose to 1.54x when calibration data closely matched the workload, proving the effectiveness of its locality-driven, stage-aware approach for MoE inference in CoT reasoning.