New Muon-Style Optimizers Improve Deep Learning Training Stability

This paper delves into the geometric properties of Muon-style optimizers, which are crucial for training large deep learning models. It highlights that beyond the standard momentum rules, the way parameters are represented before orthogonalization significantly impacts the optimizer's behavior. The authors propose Muse, a new family of optimizers that maintains consistent momentum and backend operations across different parameter representations, including native, nearest-square, skinny, and vector forms. The study reveals that each representation creates a unique "steepest-descent" geometry, affecting how singular channels are supported, scaling, and convergence bounds in stochastic nonconvex settings. Experiments with large language models like LLaMA2-130M and LLaMA2-600M demonstrate that carefully chosen non-native representations can achieve performance comparable to or better than native representations. Conversely, reducing the shorter matrix dimension can weaken scaling and singular channel support, making the optimizer behave more like simpler normalized momentum methods.