SAM's Flatness Bias Quantified for Better AI Generalization

Sharpness-Aware Minimization (SAM) is a technique known to improve model generalization by seeking parameters where the loss is robust to local adversarial perturbations. However, the precise quantitative mechanism behind its implicit bias towards flatter minima has been unclear, particularly regarding the perturbation radius (ρ) as an isolated tuning parameter. A new linear stability analysis of mini-batch SAM near an interpolating minimum provides crucial insights. Under specific assumptions, it proves that linearly stable minima satisfy a bound related to the largest Hessian eigenvalue, batch size, learning rate, and perturbation radius. This quantitatively demonstrates SAM's flatness bias: smaller batch sizes, larger learning rates, or larger radii lead to flatter minima. The analysis also highlights a trade-off: ρ must be large enough for flatness but small enough for stable training. Experimental validation on CIFAR-100 confirms these predictions, showing increased ρ consistently correlates with smaller largest Hessian eigenvalues. The study also introduces Taylor-Locality Controlled SAM (TLC-SAM), which dynamically adjusts ρ, further reducing the top Hessian eigenvalue.