New Method Reduces "Harm" to Individual Samples in AI Training

Modern stochastic optimizers, such as SGD and AdamW, combine gradients from the current mini-batch with historical information like momentum. While highly effective for overall model training, this aggregation process can inadvertently lead to parameter updates that increase the loss for specific individual samples within the dataset, an effect termed "harm." This research formalizes this problem, framing the parameter update as an optimization challenge that explicitly aims to minimize the negative impact of both batch averaging and past optimization states on current data points. Since an exact solution to this problem is computationally intractable, the researchers introduce a highly efficient proxy. This proxy significantly reduces dimensionality by focusing the optimization on the batch size and, crucially, restricts the optimization to only the last linear layer of the neural network. This simplification is based on the unexpected finding that the last linear layer reliably captures the second-order statistics of per-sample gradients. The resulting surrogate problem can be readily integrated into standard optimizers and solved with a few GPU-friendly iterations. The method scales favorably, with computational overhead decreasing as model size or input grows. Experiments on image classification benchmarks confirm that this approach effectively reduces per-sample interference and leads to improved generalization performance.