Data Influences Neural Network Hessian Spectrum and Sharpness

The Hessian matrix is a critical tool for understanding the loss landscape, optimization dynamics, and generalization properties of deep learning models. Previous studies often relied on empirical observations or simplified theoretical assumptions. This new work provides a more rigorous theoretical framework. Researchers have derived the eigenvalues of the Hessian for linear networks, accounting for arbitrary widths, depths, and diverse datasets. A key finding for classification tasks using Mean Squared Error (MSE) loss is that the sharpness of the solution is directly linked to the largest class imbalance within the dataset. Empirical validations support these theoretical predictions, even when relaxing some of the initial impractical assumptions and incorporating nonlinearities. The robustness of these findings suggests that the conclusions can be extended to more complex and practical learning scenarios, offering deeper insights into model behavior.