New Scaling Law Optimizes LLM Token Allocation

Optimizing the training of large language models (LLMs) involves carefully balancing model size, the amount of training data, and computational resources. This research introduces a novel "three-term" scaling law that explicitly incorporates training steps and batch size alongside model size and total data. This new law provides a more granular understanding of how these factors interact to influence model performance. By fitting this proposed law to a large dataset of training runs, the researchers found that it accurately predicts the scaling behavior of the optimal batch size. A key advantage of this three-term law is its robustness: it can be reliably fitted with significantly fewer training runs, even those with suboptimal batch sizes, making it more efficient for research and development. Furthermore, the law can be used to derive specific scaling relationships for scenarios where batch sizes are not optimal. These derived relationships align well with previously observed empirical findings regarding the "critical batch size," which marks a transition point in training dynamics. This work offers a more principled and efficient way to allocate computational tokens during LLM training.