Symmetrization Techniques Boost Equivariance in Bayesian Neural Networks

Symmetries are crucial in many deep learning applications, from scientific computing to medical imaging. A key debate exists on whether to enforce these symmetries through neural network architecture (equivariant networks) or to learn them via data augmentation. This study delves into data augmentation for Bayesian Neural Networks (BNNs) trained using variational inference, drawing inspiration from findings that augmented infinite deep ensembles achieve exact equivariance. The researchers derived conditions under which exact equivariance is attained for BNNs with exponential family variational distributions. Furthermore, they introduced three novel symmetrization techniques designed to amplify the effect of data augmentation in this context. Extensive experiments demonstrated that one of these methods, "orbit expansion," not only surpassed baseline performance but also significantly improved equivariance, offering a powerful approach to integrating symmetries into BNNs.