New Optimizer Improves Deep Learning Generalization and Robustness.

A major challenge in deep learning is generalization, where models trained with traditional optimizers often converge to "sharp" minima that perform poorly on unseen data. Sharpness-Aware Minimization (SAM) aims to find "flat" minima, which are associated with better generalization. This research proposes an enhancement called Extragradient-Inspired Sharpness-Aware Minimization (EISAM). EISAM improves upon SAM by incorporating a two-step extragradient update process. This involves a prediction step to analyze the loss landscape's geometry, followed by a perturbation step that refines the updates using a base optimizer. This approach not only achieves superior generalization performance but also significantly reduces the optimizer's sensitivity to the perturbation radius, making it more robust and easier to tune across different settings. Extensive experiments on standard datasets show EISAM consistently surpasses SGD, Adam, and SAM in both test accuracy and training efficiency. Theoretical analysis supports these empirical findings, demonstrating that EISAM guides parameters towards flatter minima, thereby tightening generalization bounds. The paper also provides practical hyperparameter tuning guidance, positioning EISAM as a robust and scalable optimization solution for deep learning.