MawForge Enables Local MoE Inference on Memory-Constrained Devices.

This paper introduces MawForge, a novel system designed to facilitate local inference of Sparse Mixture-of-Experts (MoE) language models on devices with constrained unified memory, such as consumer-grade machines. MoE models, while efficient in terms of active computation per token, still demand significant memory for their full parameter count, key-value caches, and runtime buffers. MawForge addresses this challenge by adopting a strategy where the complete model resides on disk, while frequently used tensors remain in memory. Crucially, it materializes only the necessary expert tensors into a bounded execution cache as they are routed and needed. This on-demand materialization allows for practical MoE serving without requiring the entire model to fit into fast memory simultaneously. The core finding is that MawForge effectively serves as a mechanism for bounded execution and measurement of local MoE inference. However, its performance is highly sensitive to the interplay between expert reuse, the size of the resident memory footprint, the key-value cache, quantization techniques, and the locality of expert routing, rather than simply maximizing cache utilization.