BLADE Boosts LLM Training Efficiency with Adaptive Data Selection

A new framework called BLADE (Bi-Level Adaptive Data sElection) has been developed to enhance the efficiency of Large Language Model (LLM) training, particularly as datasets grow to trillions of tokens. Data selection is critical for filtering noise and creating adaptive learning paths, but existing methods have limitations. Influence-based methods are principled but computationally intensive, requiring intractable inverse-Hessian calculations. Excess-loss methods are efficient but rely on a static reference model that can become misaligned during training. BLADE addresses these issues by reformulating the bi-level optimization problem of influence-based methods into a penalized single-level objective using Lagrange multipliers. This reformulation avoids inverse-Hessian computations and creates a dynamic reference model that stays synchronized with the evolving proxy model during training. The framework guarantees first-order convergence and is instantiated as a memoryless randomized block-coordinate Frank-Wolfe algorithm for efficient online batch selection. Extensive experiments show BLADE consistently outperforms state-of-the-art data selection baselines, offering a practical solution for LLM training.