Automated Pipeline Explores Heterogeneous Mixture-of-Experts Architectures

Designing optimal Mixture-of-Experts (MoE) architectures, especially heterogeneous ones, is a complex and time-consuming task typically done manually. This research introduces an automated, large-scale search pipeline specifically for 4-Expert MoE (MoE4) architectures. The pipeline systematically combines base neural network families from the LEMUR database into MoE4 ensembles, each managed by a convolutional gating network with advanced training techniques. Over a 28-day campaign, the pipeline generated and evaluated over a thousand candidate models. A significant finding was the discovery of a coverage bias in the explored search space: due to an alphabetical enumeration method, the entire search was anchored to a single family, AirNet, exploring only a small fraction of the theoretical combinations. This bias was precisely characterized, and a fix involving stratified random sampling was proposed. Within the AirNet-anchored scope, the campaign identified ShuffleNet and MobileNetV3 as consistently producing the highest-accuracy ensembles, reaching mean accuracies up to 0.632. Conversely, FractalNet and MNASNet were flagged as low-yield families, suggesting their exclusion from future searches. The pipeline, analysis tools, and the corrected generator are now open-source, contributing to more efficient and unbiased MoE architecture exploration.