New Method Finds Flat Minima for Better Neural Network Generalization

The "flatness hypothesis" in neural networks suggests that models converging to flatter regions of the loss landscape generalize better. While various algorithms aim to reduce the eigenvalues of the loss Hessian (a measure of flatness), the analytical direction towards these flat minima has remained elusive. Recent work provided a differentiable upper bound (Wolkowicz-Styan, WS) for the maximum eigenvalue of the cross-entropy loss Hessian in three-layer networks, but its gradient was not derived. This new research analytically derives this crucial gradient, enabling the proposal of Hessian Spectral Range (HSR) Regularization. This method updates network parameters along the steepest descent direction of the WS bound, effectively narrowing the Hessian eigenvalue spectrum. Experiments confirm that HSR Regularization helps models avoid sharp minima and saddle points, promoting convergence to flatter regions and thus improving generalization, albeit currently limited to specific network architectures and loss functions.