Mini-batch Noise Reduces Model Sharpness in SGD

During the training of neural networks using Stochastic Gradient Descent (SGD), gradients often align strongly with the dominant subspace, which is defined by the top eigenvectors of the loss function's Hessian. While it might seem that loss reduction primarily occurs within this space, previous work has indicated that updates within this dominant subspace do not significantly contribute to reducing the loss. This paper offers an alternative perspective, arguing that the dominant subspace is crucial for understanding the sharpness dynamics of mini-batch SGD. The researchers demonstrate how the averaged gradient, influenced by fluctuations in these dominant directions, generates a sharpness correction term. They further derive a specific sharpness correction term that is induced by the mini-batch noise within these dominant directions. Experimental results support their theory, showing that by adding this derived correction term to Gradient Descent (GD), the evolution of sharpness in GD closely mimics that observed in SGD. This work provides a deeper theoretical understanding of why mini-batch noise, a characteristic of SGD, contributes to the flatter minima (lower sharpness) often associated with better generalization.