Study Reveals Complex Factors Behind Adam-SGD Performance Differences.

Previous research has identified several isolated factors contributing to the performance disparity between Adam and Stochastic Gradient Descent (SGD) optimizers, including data characteristics, architectural design, and optimization properties. This new work revisits these hypotheses through a comprehensive empirical study. The study spans diverse tasks like vision, language, genomics, and graph processing, utilizing both modern and classical architectures with carefully designed training setups. The results demonstrate that the Adam-SGD gap is not attributable to a single factor. For instance, Adam's advantage can vary significantly based on vocabulary distribution, reverse in softmax-attention models, or increase with soft architectural changes like replacing ReLU with GeLU. These observations point to non-trivial interactions between data and architecture as the root cause of the performance gap. A consistent pattern observed is a "crossover batch size," where the relative advantage shifts between SGD and Adam as batch size changes. A theoretical model developed by the researchers captures this batch-size-dependent crossover, offering a unified perspective and practical insights for practitioners.