Zeta Optimizer Improves Neural Network Training with Dual Whitening

Training large neural networks often relies on matrix-aware optimizers that leverage the structural properties of weight parameters. However, a significant challenge for these methods, such as Muon, is their sensitivity to the conditioning of input matrices, particularly the severe scale heterogeneity observed in raw momentum matrices. A new optimizer named Zeta has been developed to tackle this problem. Zeta employs a dual whitening pipeline, first applying coordinate whitening to correct intra-matrix scale imbalances, and then spectral whitening. This specific ordering is crucial because coordinate whitening establishes the necessary statistical isotropy for spectral whitening to operate effectively and reliably. The developers prove that this dual approach significantly reduces orthogonalization error and improves the condition number of the input matrices. Empirical evaluations demonstrate that Zeta matches or surpasses strong baselines in language modeling, mixture-of-experts architectures, and vision tasks, resulting in faster convergence and improved generalization.