New Preconditioner Improves Deep Network Training Stability and Performance

Deep learning models often exhibit symmetries in their parameters, meaning certain changes don't affect the model's output. Standard optimizers like Adam can inadvertently drift along these "dead directions," hindering efficient training and leading to issues like overfitting. This new research proposes Dead-Direction Conditioners (DDC) to address this by ensuring the optimization process respects these inherent symmetries. DDC works by conditioning the optimizer's state in a way that keeps the training trajectory aligned with the true optimization space, preventing unnecessary deviations. This approach has been shown to significantly improve training stability and model performance. For instance, DDC-enhanced Adam (DDCAdam) resisted overfitting in language models much better than AdamW, and a DDC-enhanced Muon optimizer achieved superior results in vision transformers. By integrating gauge symmetry directly into the optimizer, DDC helps models find sharper minima, leading to better generalization and more interpretable training dynamics. This advancement offers a more robust and efficient way to train complex deep neural networks.