Criticality Governs Learning Dynamics in Deep Neural Networks

This study delves into the fundamental principles governing learning in deep neural networks, specifically focusing on how initial network configurations influence information and gradient propagation. It combines mean-field theory and random matrix theory to reveal a direct connection between how correlations spread through the network and the Neural Tangent Kernel (NTK), which dictates learning behavior in very wide and deep networks. The key finding is that efficient correlation propagation to infinite depth is only possible at a unique "critical point" within the weight-bias variance plane. At this critical point, the end-to-end Jacobian, which measures how output changes with respect to input, vanishes algebraically with depth. This vanishing Jacobian leads to the crucial insight that the NTK becomes precisely proportional to the output correlation at infinite depth, establishing an equivalence between information flow and learning dynamics that was previously unrecognized. The research further demonstrates that orthogonal initialization, compared to Gaussian initialization, suppresses finite-size corrections, thereby providing a clearer path to understanding these asymptotic dynamics. These theoretical predictions are quantitatively validated on finite-width and finite-depth networks, underscoring the central role of orthogonal initialization at criticality in controlling deep learning behavior.