LLM Learning Rate Scaling is Nonlinear, Effective Rate Key

A new study challenges the common assumption that optimal learning rates for training large language models (LLMs) scale log-linearly with model size and data volume. Empirical analysis of GPT-2 style models, ranging from 22 million to 707 million parameters trained on up to 100 billion tokens, demonstrated that the optimal learning rate exhibits an upward curvature at larger scales, leading to inaccuracies when extrapolating from smaller training runs. The researchers found that this nonlinearity largely disappears when the focus shifts from the nominal learning rate to the "effective learning rate," which represents the step size in normalized weight space. Furthermore, extrapolating based on the data scale (D) rather than model size (N) proved to be more accurate. The paper explains this nonlinearity by observing that weight-norm convergence to equilibrium is slower with smaller optimal learning rates, necessitating larger step sizes to shorten the transient training phase. Experiments with AdamH, an optimizer that directly manages the effective learning rate, further supported these findings, providing a clearer path for more efficient and predictable LLM training.