New Theory Explains Optimizer Behavior at Deep Learning's Edge of Stability

Deep learning training often encounters a phenomenon known as the "edge of stability," where the loss function's Hessian eigenvalues remain stable despite classical theories predicting instability. Previous models linked this to the maximum Hessian eigenvalue and learning rate, but this paper observes that many first-order optimizers, like gradient descent, significantly exceed these predicted stability bounds. The deviation is systematic and depends heavily on the specific optimizer used, a factor not captured by existing formulations. To address this, the researchers developed a new stability threshold derived from the directional Hessian and the gradient-alignment score, focusing on the optimizer's actual update rather than just the maximum curvature. This new formulation provides more consistent predictions and eliminates optimizer-dependent offsets. It also offers novel diagnostic tools that highlight the unique role optimizers play in managing the trade-off between temporal and spatial aspects of first-order optimization.