SpecDrop Improves Modular AI Specialization with Parameter-Free Routing

Mixture-of-Experts (MoE) networks aim for specialized components, typically relying on learned routers and gates. However, these learned routing mechanisms can sometimes underperform simpler "No-Routing" baselines when parameter budgets are matched. This research investigates whether the bottleneck lies in the routing algorithm itself or in the alignment between the training signal's granularity and the target categories. The paper introduces SpecDrop, a novel parameter-free routing scheme. In SpecDrop, each of K branches receives a primary weight for its assigned category and a small leakage weight for others, all merged via a fixed denominator without any learned routing parameters or auxiliary losses. This method requires the category label during inference. On vision tasks like CIFAR-100 and ImageNet-1K, where each image has a single superclass label, SpecDrop significantly outperforms parameter-matched baselines that do not use the label. While direct output masking of a dense model with labels still yields higher accuracy, SpecDrop's key contribution is its ability to internalize this label-driven modular structure during training, leading to high branch-category alignment. However, for tasks with fuzzy partitions or multiple categories per unit, SpecDrop's benefits diminish, suggesting that granularity alignment is crucial for routing to be effective.