New Adaptive Gradient Descent Improves Optimization with One-Sided Regularity

Researchers have developed a novel adaptive gradient descent algorithm designed to enhance optimization performance, particularly for continuously differentiable, potentially nonconvex objectives. Unlike traditional methods that consider the entire gradient variation, this new approach leverages "one-sided H"older regularity." This condition specifically bounds the directional component of the gradient that influences the descent, allowing the algorithm to take more aggressive, yet stable, step sizes when large gradient changes are orthogonal to or align favorably with the update direction. The proposed method incorporates an adaptive scalar step size derived from an estimate of positive one-sided H"older curvature, coupled with a safeguard to ensure sufficient decrease. Empirical evaluations on full-batch benchmarks, specifically chosen to differentiate directional curvature from overall gradient variation, show promising results. The method achieved superior final cross-entropy, objective values, and gradient norms in a binary classification task, and the lowest objective gap and gradient norm in a nonconvex H"older regression problem, suggesting its effectiveness in scenarios where full-gradient variation might otherwise inflate step-size conservatism.