AdamW Minibatch Perturbations Show Delayed, Structured Loss Effects

This research delves into the intricate dynamics of the AdamW optimizer, specifically examining how a single minibatch's influence extends beyond its immediate gradient update. The study highlights that AdamW's internal optimizer states, which store past gradient information, cause a delayed effect on subsequent training. The authors model AdamW as a finite-horizon input-state-output (ISO) system. This system's state encompasses the model parameters and the first- and second-moment estimates. By linearizing these joint dynamics, they derive a signed response operator. This operator maps a localized gradient perturbation to its future effects on the loss function, elucidating how the optimizer's memory dictates the magnitude, timing, and direction of these influences. Further contributions include an exact multistep error decomposition and a demonstration of first-order finite-horizon accuracy under specific conditions. Experimental validation confirms the response mechanism and the impact of optimizer states, with repeated-future analyses revealing significant prospective structure in these delayed influences, which can be partially captured by ISO approximations.