Dion3 Optimizer Accelerates Orthogonal Updates for AI Models

The Muon optimizer, known for its orthogonal updates, often incurs substantial overhead due to its cubic-time Newton-Schulz orthogonalization step. This computational burden is further compounded by communication overhead when model weights are sharded across multiple devices, limiting Muon's practical benefits. Dion3 is introduced as a comprehensive revision designed to tackle this overhead at every level of the software stack. It features a new Gram Newton-Schulz algorithm that reduces FLOP costs, specialized CuteDSL kernels that exploit symmetry for acceleration, and a megabatching strategy to minimize communication overhead. A crucial innovation in Dion3 is a simplified update rule that orthogonalizes only a fraction of the momentum matrix's rows at each step, further cutting costs. This approach not only improves upon previous "compressed" versions like Dion in both speed and performance but also achieves up to a 6x reduction in optimizer step time while maintaining or even improving the loss achieved by the original Muon. Dion3 is available as a drop-in replacement via the dion package.