Muon Optimizer Scales Efficiently for Large Diffusion Transformers.

The Muon optimizer has shown promise in improving the training of large models by balancing updates across singular directions, leading to better generative quality. However, its original implementation faced significant computational and communication overheads, particularly when scaled to very large Diffusion Transformers (DiTs) ranging from 1.3B to 15B parameters. This overhead stemmed from its 5-step Newton-Schulz iteration and full-momentum materialization at every optimization step. To address these challenges, a new method called Periodic Row-wise Muon has been developed. This approach performs a full spectral update less frequently, specifically once every K steps, and uses a more efficient, low-cost row-wise constrained update in between. The design also includes a co-designed distributed implementation that works directly with sharded momentum and accelerates spectral refreshes through optimized communication. Experimental results demonstrate that Periodic Row-wise Muon preserves the original Muon's superior generative quality, achieving 12.9-19.1% better quality than AdamW. Crucially, it reduces optimizer time by 46.9-54.3%, end-to-end step time by 15.7-24.3%, and logical communication volume by 66.7%, leading to 33.7-64.8% less active training time to reach optimal generative quality.