Neural Network Symmetries Explain Near-Zero Hessian Eigenvalues

The Hessian matrix of a neural network's training loss provides crucial insights into the geometry of the loss landscape, yet the origin of its numerous vanishingly small eigenvalues has remained largely unexplained. This research proposes that these near-zero eigenvalues arise from the weakly lifted pseudo-Goldstone modes associated with continuous symmetries inherent in the network's parametrization. The paper first illustrates this concept in deep linear networks, where these symmetries are exact, leading to flat directions and precisely zero modes. The eigenvectors for these modes are explicitly constructed. When a ReLU nonlinearity is introduced, it acts as a perturbation that weakly and explicitly breaks these symmetries. By resolving the spectrum at the eigenvector level, the study finds that directions of high curvature are orthogonal to the symmetry subspace, while the bulk of the Hessian's spectrum, corresponding to the near-zero eigenvalues, lies almost entirely within this subspace. This mechanism is demonstrated in a two-layer ReLU student-teacher model and a network trained on CIFAR-10, with a convolutional example showing its applicability beyond fully connected layers.