Muon Optimizer Performance Reassessed on Matrix Factorization Tasks

The Muon optimizer has recently gained attention for its strong performance in large-scale deep learning, particularly in training large language models, where it reportedly surpasses Adam and AdamW. Its success has spurred theoretical work interpreting it as steepest descent under the spectral norm. However, it has been unclear whether Muon's benefits stem from its core update rule or are artifacts of the specific scale, architecture, and data found in modern deep networks. This research aims to isolate the optimizer's performance by applying Muon to a simpler, well-understood problem: low-rank matrix factorization. This controlled environment allows for a direct comparison against carefully tuned adaptive baselines, such as AdamW, without the confounding factors of complex deep learning setups. The findings indicate that Muon does not consistently outperform AdamW in this matrix factorization context. Furthermore, several advantages previously attributed to Muon are shown to be highly sensitive to the chosen hyperparameters. This study provides a more nuanced understanding of when spectrum-aware orthogonalization, a key feature of Muon, genuinely offers advantages, advocating for evaluating optimizers on controlled problems in addition to large-scale benchmarks.