New Normalization Methods Improve Deep Neural Network Stability

Residual connections are a cornerstone for stable training of deep neural networks. Geodesic Normalization (GeoNorm) reinterprets these connections within a Riemannian manifold, ensuring that each layer's output is orthogonal to the current hidden state and applying updates via the Riemannian exponential map. This approach confines the residual stream to a hypersphere, maintaining a constant L2-norm for every hidden state. The research reveals that on a hypersphere, the entire family of retraction maps, of which the exponential map is just one, collapses to a single scalar design choice. The key differentiator among retractions is how the magnitude of an update is converted into a rotation angle within the plane defined by the hidden state and the update. This perspective unifies Euclidean residual connections and GeoNorm into a single framework. By instantiating this framework with the metric projection and Cayley retractions, the authors introduce Proj-SpheretNorm and Cay-SpheretNorm. These methods are exactly norm-preserving, require only algebraic operations, and are part of a one-parameter family called p-SpheretNorm, where the rotation angle saturates. Experiments on nanoGPT show that these new methods consistently outperform existing lightweight deep connection schemes, with optimal performance achieved at finite values of 'p', suggesting that the exponential map (used in GeoNorm) is not necessarily the best retraction for spherical residual streams.