Quadratic Model Proves Predictive for LLM Optimization Dynamics

Understanding the complex loss landscapes of neural networks is critical for effective optimization, yet theoretical models often struggle to balance tractability with accuracy. This paper re-examines the simplest optimization model, the quadratic model, and finds it to be remarkably predictive even in the context of large language models (LLMs) with 150 million parameters and billions of training tokens. The study shows that Taylor expanding the model and loss function at various checkpoints during training can accurately forecast optimization dynamics for up to 10% of the total training duration. This surprising agreement allows for a deeper analysis of the local quadratic optimization problems. Through extensive probing of the Hessian spectrum using Lanczos quadrature, the researchers uncovered significant structure in both eigenvalues and eigenvectors, which varies depending on factors like batch size, preconditioner, and training time. Empirical tests of local linear stability further indicate that LLM optimization frequently occurs at a "stochastic edge of stability," a characteristic also determined by batch size. These findings suggest that the quadratic model, despite its simplicity, can serve as a valuable and theoretically tractable proxy for understanding pretraining optimization dynamics in LLMs.