Small Initialization Improves LLM Pretraining and Reasoning

While advancements in large language models (LLMs) are often attributed to increased scale, data volume, and architectural innovations, new research highlights the critical role of parameter initialization. This study reveals that the initial scale of parameters acts as a fundamental determinant of both the training process and the ultimate capacity of an LLM. Specifically, reducing the initialization scale consistently leads to improvements during pretraining, with the most substantial benefits observed in tasks that demand complex reasoning. The researchers identified two common empirical settings that inadvertently limit the advantages of small initialization and demonstrated that adjusting these settings restores its positive scaling effects. They also pinpointed an optimal initialization point that balances reasoning capabilities with overall training efficiency. Mechanistically, a smaller initialization scale appears to guide a unique developmental path for the model. Parameters initially condense into simpler structures before evolving into more complex and rich representations. This process supports the concept that compression is intrinsically linked to intelligence. Token-level analysis further indicates that the performance gains are concentrated on challenging, context-dependent predictions rather than being uniformly distributed across all tokens. These findings advocate for a straightforward "gamma-initialization" rule: make initialization range an explicit tunable parameter and default to smaller initialization, offering a nearly cost-free method to enhance pretraining and strengthen reasoning across various model sizes.