Shock-wave Theory Explains Neural Network Training Dynamics

Understanding the complex dynamics of deep learning optimization, particularly Stochastic Gradient Descent (SGD), is crucial for improving training efficiency and stability. This paper introduces a novel mathematical framework that connects these dynamics to concepts from differential geometry, Lie group theory, and fluid mechanics, specifically shock-wave theory. The core idea involves simplifying the neural network's parameter space by accounting for inherent symmetries and then applying a coarse-graining process. Under these transformations, the effective learning dynamics are shown to obey a viscous Hamilton-Jacobi equation. Furthermore, the gradient of the coarse-grained loss function can be described by a Burgers-type equation, which is known to lead to shock formation. This theoretical framework is applied to various neural network architectures, including MLPs, CNNs, and Transformers, demonstrating its broad applicability. The authors conjecture that this approach could provide practical diagnostics for deep learning, offering a more principled way to monitor and control training phase transitions, especially in architectures where raw parameter norms can be misleading due to symmetry redundancy.