Elbow-Based Routing Dynamically Optimizes MoE Inference.

Mixture-of-Experts (MoE) models are designed to scale AI models while keeping inference-time computation low by activating only a subset of specialized "experts" for each input token. However, conventional MoE routing typically uses a fixed "top-k" selection, meaning the model always uses the same number of experts, regardless of whether fewer might suffice for a given token. This can lead to unnecessary computation. Researchers have developed "elbow-based routing," a novel, training-free modification that operates purely at inference time. This method dynamically adjusts the number of experts activated for each token. It works by analyzing the sorted router probability distribution and identifying an "elbow point" – an inflection point that effectively separates high-probability, highly relevant experts from lower-probability ones. The study found that most router distributions exhibit clear elbow points, making this strategy widely applicable. Both theoretical analysis and empirical experiments confirm that elbow-based routing effectively preserves expert load balance, which is crucial for efficient MoE operation. When applied to a state-of-the-art MoE model, this technique achieved an average latency reduction of 5.3% across six benchmarks, all while maintaining the model's accuracy.