GEAR-SAM Improves AI Model Generalization and Robustness

Sharpness-Aware Minimization (SAM) is a technique used to improve the generalization ability of neural networks by seeking parameter configurations that minimize loss not just at a single point, but across a local neighborhood. The standard SAM approach distributes its perturbation budget across different parts of the network based on the immediate gradients from a small batch of data, which can be inconsistent and may not accurately reflect how sensitive each part of the network is over time. This new research introduces GEAR-SAM (Gradient-Energy Adaptive Radius SAM), an improved method that addresses this limitation. GEAR-SAM maintains an exponential moving average of the squared gradients for each block of parameters within the network. This moving average acts as a more stable and curvature-related signal of a block's sensitivity throughout the training process. By using this "gradient energy" signal, GEAR-SAM intelligently reallocates the fixed SAM perturbation budget. This dynamic allocation ensures that more perturbation is applied to the network blocks that are consistently more sensitive, leading to better optimization. The method is computationally efficient, requiring only scalar state beyond SAM and avoiding complex Hessian calculations. Experiments across various tasks like image classification and transfer learning demonstrate that GEAR-SAM significantly enhances generalization and robustness across different model architectures.