Stochastic Weight Averaging Boosts Data Augmentation Performance

Data augmentation is a widely used and effective technique for incorporating symmetries of a learning task into generic neural networks, thereby improving their generalization capabilities. While infinitely large deep ensembles are known to exhibit perfect symmetry when trained with augmented data, the computational cost of training multiple ensembles is prohibitive. This paper investigates Stochastic Weight Averaging (SWA) as a more efficient alternative to traditional ensembling. SWA involves averaging the model weights over the latter part of the training trajectory, effectively creating an ensemble without multiple training runs. The researchers analyze SWA by approximating the stochastic training trajectory with an Ornstein-Uhlenbeck process in the infinite-width limit. Their findings demonstrate that SWA, when applied with augmented data, provides an equivariance boost that surpasses the performance increase attributable to SWA alone. This suggests a synergistic effect between SWA and data augmentation. Extensive numerical experiments across various models in computer vision and graph classification, involving both discrete and continuous symmetries, verify these theoretical results.