Aurora Optimizer Improves Wide MLP Layer Training Efficiency

Training neural networks, especially those with wide Multi-Layer Perceptron (MLP) layers, can be challenging due to issues with optimizer updates. Specifically, existing methods like Muon can lead to arbitrarily non-uniform row norms in matrix parameter updates, causing some neurons to receive persistently small updates and become ineffective. While row normalization can mitigate this, current techniques often distort the desired update geometry. Researchers have developed Aurora, a novel spectral optimizer designed to enforce row-uniformity in matrix parameter updates without compromising the polar factor geometry of the momentum matrix, which is crucial for effective learning. Aurora demonstrates superior performance over Muon in pre-training experiments. When combined with other state-of-the-art methods, it achieves top results among spectral optimizers on the modded-nanoGPT speedrun. Its empirical gains scale with the MLP expansion factor, suggesting it could enable more effective training of very wide MLP layers.