Holistic Data Scheduler Boosts LLM Pre-training Efficiency and Capability.

Researchers have introduced the Holistic Data Scheduler (HDS), a novel framework designed to optimize data composition during Large Language Model (LLM) pre-training. Existing online data mixing methods often rely on a single optimization perspective, which fails to account for the complex, multi-dimensional requirements of LLM training. HDS addresses this by formulating data scheduling as a continuous control reinforcement learning problem. The framework leverages the Soft Actor-Critic (SAC) algorithm for its stability and sample efficiency in exploring high-dimensional policy spaces. A key innovation of HDS is its multi-objective, holistic reward function. This function integrates three critical perspectives: a data-driven reward for quality, a loss-driven reward capturing inter-domain influence, and a model-driven reward based on weight norms. Systematic experiments on LLMs of various sizes, using The Pile benchmark, demonstrated HDS's effectiveness. It achieved the same final validation perplexity as the next best method with 44% fewer training iterations. Furthermore, HDS delivered a 7.2% improvement on the MMLU 0-shot task and consistent gains across other benchmarks, showcasing its ability to enhance both training efficiency and the final capabilities of LLMs.