Orthogonal Gradients Impact Noisy-Label Memorization in Neural Networks

Neural networks are known to memorize corrupted training labels, a phenomenon that offers insights into overfitting. This research explores a novel approach called OrthoGrad, which geometrically intervenes in the optimizer's update process. Instead of altering the objective function or architecture, OrthoGrad removes the component of each weight gradient that runs parallel to the current weight vector. Evaluated in noisy-label image classification, OrthoGrad demonstrated improved test accuracy for Convolutional Neural Networks (CNNs) in small-data environments, specifically by reducing the fitting of corrupted labels. Diagnostic analyses suggest its effectiveness is highest when the raw gradient has a significant radial component. While it can alter memorization trajectories, the method is not a universal regularizer and doesn't entirely prevent noisy-label memorization, especially in larger datasets where gradients are already nearly orthogonal to weights.