Adam Algorithm Analysis for Nonstationary Stochastic Systems.

The Adam optimization algorithm is a cornerstone of modern machine learning due to its efficiency and strong empirical performance. However, its theoretical underpinnings have largely been confined to idealized scenarios, assuming time-invariant model parameters and independent and identically distributed (i.i.d.) data. These assumptions often do not hold in real-world applications involving time-varying and nonstationary systems. This research addresses this gap by establishing a comprehensive theory for Adam in the context of time-varying and nonstationary stochastic systems. The authors introduce novel techniques to analyze the products of nonstationary and dependent random matrices, which arise from Adam's coupled first- and second-moment recursions. They also construct a new stochastic Lyapunov function that integrates these two moment dynamics. Under a stochastic excitation condition that accommodates nonstationary and dependent data, the paper derives explicit error bounds for both parameter tracking and output prediction. These bounds quantify the impact of hyperparameters like step size, momentum parameters, gradient noise, and parameter drift, providing crucial guidance for hyperparameter tuning. Experimental results on both synthetic and real-world data validate the theoretical findings and design guidelines.