New Optimizer RADAR Improves Deep Learning with Adaptive Momentum

Momentum-based optimizers are fundamental to deep learning, yet their underlying mechanisms, particularly the interplay between momentum, update geometry, and acceleration, are not fully understood. This paper presents the ADMM-Inspired Momentum (AIM) framework, which reinterprets momentum as a multiplier-like correction driven by the residual of a variable splitting penalty. AIM effectively separates the residual penalty, which dictates the update geometry, from the objective approximation, which governs acceleration. Building on the AIM framework, the researchers propose RADAR (Relativistic Adaptive gradient Descent with Accelerated Residual). RADAR integrates relativistic adaptive geometry, a decoupled residual correction mechanism, and second-order momentum filtering. This combination is designed to refine the update direction and enhance momentum estimation during training. Through rigorous stochastic convergence analysis and extensive experiments across supervised vision, language modeling, and reinforcement learning, RADAR consistently demonstrates superior performance compared to leading adaptive optimizer baselines.