Spaced Repetition Improves Continual LLM Pre-Training

Continual pre-training of large language models (LLMs) faces a fundamental challenge: acquiring new information without forgetting previously learned knowledge. Existing replay methods often use uniform sampling, which overlooks the varying rates at which different examples are forgotten. This paper introduces Spaced Repetition Training (SRT), a novel continual learning framework that draws inspiration from cognitive science principles. SRT re-frames continual pre-training as an adaptive review scheduling problem, deciding not only how much history to replay but also which specific examples to revisit at each step. By maintaining per-example review states and mapping perplexity to a recall-quality signal, SRT schedules historical examples for retention and new ones for consolidation, all while keeping the core model, objective, and optimizer unchanged. Experiments on Wikipedia and code corpora show SRT recovering substantial amounts of old-knowledge accuracy lost by naive methods, while preserving or enhancing new-knowledge acquisition, even at larger model scales.