New Trust-Region Framework Unifies Adaptive Moment Estimation

Adaptive moment estimation mechanisms, such as Adam, are fundamental to stochastic gradient optimization, but their underlying behavior can be complex. A new trust-region framework has been developed to provide a unified understanding of these mechanisms. This framework operates by constraining the magnitude of each individual weight's update step within a trust-region, which is governed by a moment constraint of order p, ranging from 2 to 4. This derivation leads to a new family of learning-rate mechanisms. These mechanisms are based on both second-moment estimation and a normalized p-th moment estimation. When p is set to 4, the framework involves a kurtosis-like estimation. The general mechanism, termed Gmake, offers a comprehensive interpretation of several optimization concepts: normalization by moment estimation, learning-rate scheduling, spectral lowpass filtering as momentum, and operator-level spectral normalization, all within a common trust-region context. Experiments conducted on GPT2-124M, trained on datasets like FineWeb-Edu and TinyStories, revealed interesting performance characteristics. The fourth-moment realization of Gmake provided its greatest advantages when the trust-region constraints were weak. However, as stronger trust-region controls were progressively introduced, the second-moment realization became increasingly competitive, often achieving slightly lower validation loss than its fourth-moment counterpart. This suggests a trade-off and context-dependency in the optimal choice of moment order.