Learning Rate Cooldown Efficacy Depends on Noise and Optimizer Normalization

The effectiveness of a learning-rate cooldown phase, a common practice in large model pretraining, varies significantly across different settings. This study provides a theoretical explanation for this variability, pinpointing two critical factors: the inherent structure of the gradient noise and whether the optimizer normalizes its updates. For stochastic gradient descent (SGD) with multiplicative noise on a strongly convex objective, the learning rate effectively "anneals itself" because the step size shrinks proportionally to the gradient. In such cases, a cooldown phase offers no additional benefit. However, for sign-based or normalized optimizers, which maintain a unit-scale step, a noise floor of order η² is established, preventing convergence to the minimizer unless the learning rate is explicitly driven to zero. Any additive noise further reinforces this floor for all methods. The research provides an exact solution for the signSGD stationary law on a quadratic objective, deriving the floor constant. It also extends these findings to normalized SGD in higher dimensions and demonstrates robustness to momentum and heavy-tailed noise. Simulations confirm these predictions, and a diagnostic tool based on directly measured gradient noise is presented, offering a clear mechanism to determine when a cooldown phase will be beneficial.