Vanilla SGD with Momentum Handles Heavy-Tailed Noise

Researchers have investigated the performance of standard Stochastic Gradient Descent (SGD) when combined with momentum, specifically in environments characterized by heavy-tailed noise. Unlike many existing approaches that rely on gradient clipping or normalization techniques to manage such noise, this study focuses on the fundamental behavior of vanilla SGD. The analysis provides a comprehensive convergence framework for strongly convex, convex, and nonconvex objectives. The findings indicate that while vanilla SGD with momentum does converge under these challenging conditions, its convergence rates are not as optimal as those achieved by methods that incorporate gradient control mechanisms. This highlights an inherent limitation of unadorned SGD when dealing with significant noise, suggesting that specialized techniques remain crucial for achieving peak performance in such scenarios. The theoretical conclusions are supported by practical experiments using synthetic functions.