Efficient Hyperparameter Transfer for Large MoE Models

Researchers have developed a novel method to optimize hyperparameters, specifically learning rates, for extremely large Mixture-of-Experts (MoE) models. Traditional methods of sweeping hyperparameters become computationally unfeasible at the scale of trillions of tokens and massive model sizes. The proposed framework addresses this by first adapting Maximal Update Parameterization (µP) for MoE architectures, demonstrating that optimal learning rates remain consistent across models scaled by width. The second step extends this transferability to the token dimension by establishing a predictive scaling law. By training smaller proxy models on limited data, the framework can accurately extrapolate the ideal learning rate for models trained on up to 10 trillion tokens. This approach significantly reduces the computational resources required for hyperparameter tuning, making the training of large-scale MoEs more practical and efficient.