Gradient Smoothing Improves Deep Neural Network Optimization

A new optimization technique called Gradient Smoothing has been proposed to improve the training and generalization performance of deep neural networks. This method falls under a broader paradigm called Depth-wise Gradient Augmentation, which transforms collections of block-wise optimizer updates along the depth dimension of a network. Gradient Smoothing, specifically using a local Window Smoothing operator, couples updates across layers. The key advantage of Gradient Smoothing is its broad applicability and minimal overhead. It works as a drop-in enhancement for existing optimizers like SGD, Adam, or Muon, and does not require changes to model architectures or training objectives. This makes it highly compatible with current machine learning pipelines. Extensive evaluations across various models, including large language models (LLMs) for pretraining and reasoning, diffusion models, and Vision Transformers for image classification, consistently show improvements in both optimization efficiency and generalization capabilities. The method also promotes more structured representation evolution across network depth, supporting its interpretation as a structured depth-wise preconditioning technique.