MoE Expert Caching Evaluation Can Be Misleading

Mixture-of-Experts (MoE) models often exceed accelerator memory, necessitating offloading expert weights to host memory. Managing this expert cache effectively is crucial for reducing traffic, but evaluating cache policies accurately proves difficult. This study identifies several factors that can lead to misleading results in trace-driven evaluations. The researchers used a simulator to analyze three MoE models, uncovering three key areas that distort evaluation outcomes. Firstly, inconsistent replay semantics in fused-event traffic contracts can inflate the perceived performance of recency-based caching policies. Secondly, workload contamination, where probe sets generate identical prefixes, can significantly skew measurements of cache friendliness. Lastly, operating regimes show that normalized miss fractions are not transferable across models, and even temporal reordering of event streams can alter the gap to optimal performance. Even after correcting for these issues, a substantial gap to the offline optimum persists, largely attributed to the challenge of predicting future expert usage. While a causal next-use predictor can help, its effectiveness in closing this gap is limited compared to the theoretical optimal. The study concludes that the potential gains from lightweight causal mechanisms are often overstated by current evaluation methods.