Neural Networks Exhibit Symmetries for Compression and Lifelong Learning

Artificial neural networks are often perceived as "black boxes" due to their complex internal workings. This study uncovers a fundamental property: deep neural networks spontaneously generate local symmetries, specifically fibrations and coverings, during the learning process. These symmetries, rooted in graph theory, are proven to be stable attractors for stochastic gradient descent. The researchers observed the emergence of these covering symmetries across a wide range of major network architectures, including multilayer perceptrons, convolutional networks, recurrent networks, and transformers. This widespread presence suggests a universal principle in how neural networks learn and organize information. Crucially, exploiting these newly identified symmetries offers significant practical benefits. The study demonstrates that models can be drastically compressed, reducing their size to as little as 17% of the original without compromising performance. Furthermore, by strategically breaking these covering symmetries, the networks can overcome the issue of "catastrophic forgetting" or loss of plasticity, leading to state-of-the-art performance in continual learning scenarios. This theoretical breakthrough provides a new foundation for designing more interpretable, efficient, and adaptable AI systems.