DeltaMomentum Optimizer Boosts AI Training Speed and Efficiency.

This research introduces DeltaMomentum, a novel optimization technique designed to enhance the efficiency of deep learning model training. Unlike traditional optimizers that use a fixed forgetting rate for past gradients, DeltaMomentum dynamically adjusts this rate based on the frequency with which specific input directions are encountered. By exploiting the key-value structure of linear layer gradients, it applies a canonical delta rule, allowing less frequently seen directions to be forgotten faster. The paper demonstrates that DeltaMomentum acts as a valid momentum, corrects input-side curvature without complex matrix inversions, and clears stale directions more effectively than exponential moving average (EMA) methods. It is a drop-in replacement for existing momentum buffers and shows significant speed improvements. Empirical results from pretraining on FineWeb-Edu show that AdamW with DeltaMomentum (DeltaAdamW) reaches target validation loss in significantly fewer steps (up to 46.39% fewer at 67M parameters and 22.12% fewer at 370M) compared to standard AdamW. These gains persist at 1B parameters and also apply to SGD, ResNet-18, and ViT-Tiny on CIFAR-10, indicating broad applicability and improved gradient tracking.