Trust-Region Framework Unifies Adaptive Moment Estimation

Adaptive moment estimation algorithms, such as Adam, are fundamental to stochastic gradient optimization. This paper introduces a novel trust-region framework that provides a unified understanding of these mechanisms. Within this framework, the magnitude of each weight's update step is constrained by a moment constraint, ranging from second-order to fourth-order. This derivation leads to a new family of learning-rate mechanisms called Gmake, which leverages both second-moment and normalized p-th moment estimation. When p equals 4, this involves a kurtosis-like estimation. Gmake offers a comprehensive interpretation of various optimization techniques, including learning-rate scheduling, spectral lowpass filtering as momentum, and operator-level spectral normalization. Experiments conducted on GPT2-124M, trained on FineWeb-Edu and TinyStories, indicate that the fourth-moment realization of Gmake provides the most significant advantages when the trust-region constraints are less restrictive. As these constraints become stronger, the second-moment realization becomes increasingly competitive, sometimes achieving slightly lower validation loss.