MOSAIC Framework Optimizes MoE Models for Cluster Efficiency

In the development of large-scale language models, particularly sparse Mixture-of-Experts (MoE) architectures, decisions about algorithms, architecture, and underlying systems are often made in isolation. This disconnected approach typically involves first optimizing for loss under compute constraints, followed by a separate stage to optimize implementation for hardware efficiency. This research argues for a unified approach, introducing MOSAIC (Model Architecture and Systems Co-design as an Optimization Problem). MOSAIC integrates a predictive scaling law with a calibrated performance model that estimates key system metrics like Model FLOPs Utilization (MFU), communication costs, and memory footprint, alongside identifying the best parallel layout. The framework is instantiated for sparse MoE language models, where factors like expert count and routing sparsity significantly influence both model loss and system efficiency. Through extensive scaling law sweeps on MoE models, the study reveals a critical insight: an efficiency-agnostic compute budget often suggests that sparser models are always better, pushing the optimum to the boundary of data support. However, when systems constraints of a cluster are factored in via MOSAIC, an optimal sparsity factor emerges. This demonstrates that what is "compute-optimal" in isolation is not necessarily "cluster-optimal" in a real-world deployment, advocating for a paradigm shift towards integrated architecture and systems co-design for frontier language model training.