Interleaved Noise Injection Boosts Model Robustness and Performance.

Noise injection is a known technique in stochastic optimization, but this research reveals surprising benefits from an *interleaved* (on-off-on-off) schedule rather than the typical monotonic decay. The theoretical analysis suggests that impulse noise acts as Jacobian regularization, while Gaussian noise applies a curvature penalty, both contributing to increased model robustness. The interleaved nature of the schedule is key, as it allows the optimizer to explore the loss landscape more effectively, escaping local minima without catastrophically forgetting features learned from clean data. To ensure stability during these transitions, a gradient-norm stabilization technique is introduced, scaling noisy updates based on clean gradient magnitudes. When compared against other common augmentation methods, this approach demonstrated substantial improvements in corruption tolerance and robustness to real-world distribution shifts across various benchmarks and architectures. The best results were achieved when stacking this method with other augmentations. Saliency and attention maps further indicate that interleaved noise injection works by counteracting the inherent inductive biases of models.