Data Predictability Predicts Transformer Weight Growth

The weight magnitudes in a trained transformer model can be characterized by a two-parameter Weibull distribution, where the shape parameter remains stable, and the scale parameter (λ) captures most of the training-induced changes. This study investigates which corpus property dictates the growth of this scale parameter. Researchers discovered a learning-rate-conditioned law that connects the bigram conditional entropy (D) of a corpus—a training-free statistic—to the growth of λ. This law, expressed as λ² - λ₀² = C₀(η) + C₁(η)(Hᵣ - D)⁰.⁵⁹, where Hᵣ is a shuffle baseline, accurately predicts weight growth. The convex exponent is derived from an independent data-side saturation relation. After normalizing for learning rate coefficients, 23 experimental runs across varying learning rates collapsed onto a single curve with high R² value, demonstrating the law's robustness. This allows for forward prediction of held-out weight growth with only 5.7% relative error. The readout holds across model and per-layer resolutions and two architectures, suggesting that data predictability is a fundamental driver of transformer weight evolution.