Muon^p Optimizer Enhances Finetuning with Fractional Spectral Powers

The Muon optimizer, widely used in machine learning, operates by replacing a gradient with its polar factor, effectively flattening the singular spectrum. While this approach is beneficial, it discards singular-value information that could be crucial for effective model adaptation. This research introduces Muon^p, a novel optimizer that extends the Muon concept by employing fractional spectral-power updates, specifically $US^pV^\top$ for rational $p \in (0,1)$. This method allows for interpolation between the full flattening of Muon and the traditional gradient descent, offering a more nuanced control over the singular spectrum. To make Muon^p practical, the researchers proved that fractional spectral powers cannot be computed using fixed univariate polynomial iterations. Instead, they derived low-degree odd bivariate recurrences that approximate $US^pV^\top$ using only matrix multiplications. This preserves Muon's original computational structure and complexity, making the new optimizer efficient to implement. The theoretical analysis shows that Muon^p maximizes the linear improvement in loss under the Schatten $q$-norm, where $q = 1 + \frac{1}{p}$. Empirical evaluations demonstrate that Muon^p is particularly effective for finetuning large-scale models, specifically those with billions of parameters. It consistently improves validation perplexity and enhances performance on various downstream tasks. The study also provides insights into scenarios where Muon^p might be less suitable, offering a spectral geometry perspective. These findings highlight the importance of selectively preserving singular spectrum information and introduce a principled method to achieve significant gains in model optimization.