New Optimization Policy Accelerates Large Language Model Training.

Training large language models efficiently is a significant challenge, often limited by the optimization process. While some methods explore selective updates within layers, a general and theoretically sound approach for fine-grained control has been missing. This new research introduces MGUP, a novel mechanism designed to fill this gap. MGUP works by augmenting existing momentum-based optimizers, such as AdamW, Lion, and Muon. In each training iteration, it applies larger step-sizes to a specific, fixed percentage of parameters, while still applying smaller, non-zero step-sizes to the remaining parameters. This selective update strategy is nearly plug-and-play, allowing for easy integration into current training pipelines. The researchers provide theoretical convergence guarantees for MGUP-AdamW under standard assumptions. Extensive experiments across diverse tasks, including pretraining and fine-tuning of LLMs, demonstrate that MGUP-enhanced optimizers consistently achieve superior or more stable performance compared to their original counterparts. This offers a principled and versatile strategy for accelerating and stabilizing the training of large-scale models.