New Optimizer Accelerates LLM Pretraining with Curvature-Conditioned Momentum

Pretraining large language models (LLMs) consumes a substantial portion of computational resources, yet it faces significant hurdles due to noisy gradients and highly ill-conditioned loss landscapes. While advanced adaptive optimizers such as AdamW and Muon have achieved considerable success, their gradient normalization techniques offer limited mitigation against the challenges posed by curvature. Consequently, progress along "flat directions"—eigen-directions associated with small eigenvalues—which are crucial for final loss reduction, remains relatively slow. To tackle this, researchers introduce a novel curvature-conditioned multiscale momentum method, augmented with sphere constraints, designed to provide steady acceleration during LLM pretraining. This multiscale momentum is specifically applied to flat directions, combining a slow-decay component for effective noise reduction with a fast-decay component for rapid adaptation to curvature. This synergistic approach leverages the complementary strengths of both components. A critical aspect of this method is the use of sphere constraints, which prevent parameter inflation and an overly rapid effective learning rate decay that could otherwise arise from a straightforward combination of momentum components. Extensive experiments across various LLM architectures (dense, MoE) and sizes (0.12B to 2.3B parameters) demonstrate that this proposed method significantly accelerates Muon. Theoretical analysis further validates the acceleration effect and provides insights into the design principles behind this flat-direction multiscale momentum.