New Theory Quantifies Gaussian-Process Limits of Neural Networks.

Understanding the behavior of neural networks as their width approaches infinity, particularly their convergence to Gaussian Processes (GPs), is a critical area of theoretical deep learning. This paper delves into this phenomenon through the lens of tensor programs, offering a quantitative convergence theory. The core contribution is the derivation of explicit finite-width error bounds. These bounds demonstrate that the difference between the execution of a finite-width network and its infinite-width Gaussian Process limit decreases at an order inversely proportional to the square-root of the network's widths. This provides a concrete measure of how closely a real-world, finite network approximates its theoretical GP counterpart. Crucially, this framework is designed to be architecture-agnostic. It applies not only to standard feed-forward models but also extends to more complex architectures that incorporate weight-sharing schemes, such as recurrent neural networks and transformer-type models. This broad applicability makes the theory highly relevant for a wide range of modern deep learning systems.