LLM Pretraining: Optimal Data Repetition for High-Quality Domain Data

As large language models continue to grow, their training budgets for tokens must also expand to maintain an effective tokens-per-parameter ratio. However, acquiring sufficient high-quality domain-specific data is challenging compared to general web data, often leading to its proportion decreasing in the overall training mixture. Repeating this valuable domain data can mitigate dilution, but excessive repetition risks overfitting. This study explores the balance of data repetition under practical LLM scaling scenarios where training token budgets increase with model size. Researchers discovered that for a given domain, the ideal repetition count slightly rises with model size, assuming a fixed tokens-per-parameter ratio. Furthermore, domains exhibiting lower validation loss generally benefit from more repetitions, indicating a strong negative correlation between optimal repetition and final validation loss. The amount of unique domain data, surprisingly, showed only a weak link to the optimal repetition count. These findings suggest that repetition counts determined using smaller proxy models, provided they maintain the same tokens-per-parameter ratio, can offer a practical estimation for training much larger models. This approach could help optimize resource allocation and improve the performance of specialized LLMs.