Orthogonalization Drives Muon Optimizer's Grokking Speed

This new study investigates the Muon optimizer's ability to achieve "grokking" – a phenomenon where a model generalizes to unseen data long after overfitting to training data – faster than traditional optimizers like AdamW. Previous work attributed this speedup to a combination of spectral-norm constraints and orthogonalized momentum. This research isolates the specific mechanism responsible for the accelerated grokking. Through extensive ablation studies and multi-seed, multi-learning-rate sweeps, the findings clearly indicate that the speedup primarily stems from the orthogonalization component, implemented via the Newton-Schulz iteration. Optimizers that only apply orthogonalization match the performance of the full Muon optimizer, whereas those relying solely on spectral-norm constraints show no significant improvement over AdamW and are less reliable. Mechanistic analysis further reveals that orthogonalizing optimizers achieve generalization at significantly lower spectral norms. The study also explores the impact of the number of Newton-Schulz iterations, finding that fewer iterations can accelerate reaching the grokking threshold but may lead to a more fragile solution, especially at higher learning rates. The canonical five iterations offer a more robust and learning-rate-stable choice. The researchers also suggest that spectral scaling can be omitted without performance cost. The paper emphasizes the importance of stability-aware metrics for evaluating optimizer speed claims and provides code for reproducibility.