MoE Router Training Fails to Improve Cacheability for Edge LLMs.

This paper explores the challenge of deploying large Mixture-of-Experts (MoE) language models on edge devices, where memory bandwidth often becomes a critical bottleneck. The study specifically focuses on whether training MoE routers to prioritize expert locality can improve cache hit rates and mitigate the "memory-bandwidth wall." Using a 235B-parameter Qwen3 MoE model, measurements confirmed that decode speed is severely limited by the need to stream experts from slower storage tiers like SSDs. The researchers developed `llama-moe-trace` to analyze expert routing patterns, finding some expert reuse and traffic concentration. They then conducted pre-registered experiments, training 137M MoE models with auxiliary locality and domain router losses, aiming to reduce cache misses while maintaining perplexity. Although the mechanism successfully reduced misses by up to 60%, every configuration failed to meet the pre-registered perplexity gate (<=1% degradation), indicating a tight coupling between miss reduction and model quality. The study concludes that, for multi-domain 137M models, the "tax" on quality for cacheability is significant, a finding that persisted with larger models. However, training-free cache-aware rerouting combined with trained locality showed promising results, achieving ~80% miss reduction with acceptable perplexity trade-offs.