Joint Affine Spectral Shaping Improves BERT-mini Training Accuracy

Traditional matrix spectral optimizers, like Muon, typically focus on reshaping the spectra of weight updates while leaving bias updates to separate optimizers. This paper investigates whether this separation is optimal. The authors propose a new method called Joint Regularized Inverse (JRI), which formulates each affine layer as a combined momentum matrix encompassing both weights and biases. A capped regularized-inverse spectral map is then applied to this complete matrix, generating updates for both weights and biases simultaneously. Through rigorous ablation studies on a BERT-mini model trained from scratch on IMDb, JRI consistently outperformed weight-only inverse shaping. It raised validation-loss-selected test accuracy from 84.903% to 85.738% and lowered test loss from 0.3479 to 0.3291. The improvements were observed across all five seeds in the initial study and confirmed in an independent 22-seed replication. JRI achieves this by preserving the eligible weight-update norm while significantly reducing the bias-update norm and altering the relationship between weight-induced boundary motion and explicit bias.