Data Repetition Significantly Harms Language Model Performance

As language models grow, the availability of high-quality training data is becoming a bottleneck, and even carefully deduplicated corpora still contain some level of repetition. This research revisits the impact of data repetition on language model performance, using a modern scaling law paradigm to quantify the "Compute-Equivalent Loss" incurred. This metric allows for a direct comparison of performance degradation against the computational resources that would be wasted. The study identifies three systematic ways repetition damages models. Firstly, there's an optimal "repeat count" where evaluation loss peaks, meaning repeating a moderately sized subset a moderate number of times causes more damage than extreme repetition scenarios. Secondly, this peak's location scales with model size, indicating that the most damaging repetition grows faster than compute. Finally, the research demonstrates significant compute-equivalent loss: if repeated documents consume just 10% of the FLOPs budget, the performance loss can be equivalent to running a no-repetition model with only 67% of the FLOPs. These findings are not exclusive to language models and can be explained by a statistical tradeoff between memorization and generalization, highlighting how repetition can lead to a misspecified statistical model.