Schatten-p Norms: Optimal Use in Deep Learning Depends on Regime

This paper investigates the optimal use of Schatten-p norms in deep learning optimization, addressing previous conflicting observations about their effectiveness. The findings indicate that the choice of norm is highly dependent on the specific operational regime. While optimizers based on the Schatten-infinity norm, such as Muon, have shown strong empirical results, the study reveals that smaller Schatten-p geometries can be more advantageous. This is particularly true in low-dimensional regimes, a category that includes scenarios related to Chinchilla scaling. The conclusion stems from a new analysis demonstrating noise-robust acceleration within the SODA framework for p-values greater than 2. This analysis also explains why Muon-like methods do not require warm-up periods, naturally favor larger batch sizes, and provides a batch size scaling rule applicable to any p-value.