Shock-wave Theory Linked to Neural Network Training Dynamics

This research develops a rigorous mathematical connection between the principles of shock-wave theory and the learning dynamics of stochastic gradient descent (SGD) when applied to artificial neural networks. The authors leverage advanced concepts from differential geometry, Lie group theory, and fluid mechanics to establish this link. Specifically, by accounting for parameter symmetries and applying local-entropy coarse-graining, the effective dynamics of the learning process are shown to conform to a viscous Hamilton-Jacobi equation on a quotient manifold. Furthermore, if the raw parameter dynamics can be represented by a gradient field in this quotiented space, the gradient of the coarse-grained loss function follows a Burgers-type equation, which can lead to the rigorous formation of "shocks." The theory is applied to various neural network architectures, including multilayer perceptrons, convolutional neural networks, and Transformers, demonstrating their adherence to these Hamilton-Jacobi or Burgers-type equations. The authors conjecture that this framework could provide practical diagnostic tools for deep learning, suggesting that symmetry-corrected observables might offer a more principled way to monitor and control training phase transitions compared to raw parameter norms, which can be distorted by redundancy.