GLOBE Improves Coreset Selection for On-Device Deep Learning Training

On-device training of deep neural networks faces significant challenges due to the high computational and memory demands of large datasets. Coreset selection offers a solution by identifying and retaining only a small, yet representative, subset of the original training samples. Current gradient-based coreset methods often rely on single model snapshots and simple selection procedures, which can limit their effectiveness in dynamic optimization environments or with highly correlated data. The new framework, GLOBE (Gradient Local-Balanced Extraction), addresses these limitations by formulating sample selection as a globally optimized sparse weighting problem. It constructs a gradient trajectory for each sample, capturing its influence throughout various training stages. GLOBE also introduces a multi-order matching objective to align both first-order mean and projected uncentered second-order moments of these gradient trajectories, ensuring the selected coreset preserves the full dataset's training behavior. Furthermore, GLOBE incorporates structured sparsity techniques like Group LASSO and Elastic Net regularization, along with non-negative budget constraints, to stabilize weights for correlated trajectories and induce sparsity at both group and sample levels. A class-balanced Top-K selection mechanism ensures adequate category coverage even with limited sampling budgets. Experimental results across multiple benchmarks and architectures show GLOBE consistently surpasses existing coreset selection methods, particularly when very small subsets of data are retained.