SGD Scaling Limits at Flat Minima: Beyond Gaussian Behavior.

For Stochastic Gradient Descent (SGD) with a constant stepsize, the long-term behavior of the algorithm is described by an invariant law centered at a minimizer. In the context of strongly convex objectives, this invariant law typically exhibits a familiar $\sqrt{\alpha}$ scaling and converges to a Gaussian limit as the stepsize $\alpha$ approaches zero. However, this research demonstrates that this behavior undergoes a fundamental shift when dealing with convex objectives that possess "flat minima" and (sub)quadratic tails. Specifically, for such objectives, the study proves the existence, uniqueness, and geometric convergence to an augmented invariant law. This convergence is measured in a Wasserstein distance induced by an $\alpha$-dependent metric. When the minimizer has a local flatness exponent $m \ge 2$, the contraction bound changes to $1-c\alpha^{m-1}$. Analyzing the small-stepsize scaling limit, the invariant law is shown to concentrate on the scale $\alpha^{1/m}$, and the rescaled iterates weakly converge to the stationary distribution of a stochastic differential equation. This recovers the Gaussian limit for the quadratic case ($m=2$) but yields generally non-Gaussian stationary limits for the flat case ($m>2$), providing crucial insights into SGD's behavior in more complex optimization landscapes.