Analyzing Transformer Weight Distribution Dynamics Under AdamW Training

This study delves into the dynamics of transformer weight distributions during AdamW training, specifically focusing on the behavior of the Weibull weight-scale parameter, denoted as $\lambda$. The parameter is observed to grow, overshoot, and then relax over the course of training. To understand this phenomenon, the researchers derive a leading-order three-force decomposition of the squared weight norm from the AdamW update rule. These three forces are: an alignment force, which measures the correlation between weights and the adaptive update direction; an injection force, derived from the adaptive step magnitude; and a decay force, resulting from decoupled weight decay. Experiments on self-trained Pythia-70M models with ground-truth optimizer moments reveal that the alignment force is the primary driver during the initial rise phase of $\lambda$, contributing 88-94% of the absolute force budget across different random seeds. As training approaches saturation, the alignment and decay forces balance, explaining the transition from weight-scale growth to relaxation. To extend this analysis to real-world models where optimizer moments are often unavailable, the paper introduces a spline displacement method. This method can recover the alignment force from sparsely sampled checkpoints with approximately 92-94% accuracy, significantly outperforming a naive two-point baseline. The study also notes that the peak value of $\lambda(t)$ appears to correlate with the coherence of the training data, suggesting a data-dependent component to weight-scale growth, which is slated for further investigation.