Deep Learning Foundations: Algorithmic Complexity and Universal Approximation

The expressiveness of feedforward neural networks (NNs) is typically understood through their ability to approximate functions based on their regularity, often by emulating optimal basis-expansion schemes. However, this perspective is incomplete, as it fails to differentiate between functions of comparable regularity but vastly different intuitive complexity, such as a simple square-root function versus a complex Brownian path. This research proposes a new foundational view: neural networks should be considered not merely as flexible function approximators, but fundamentally as models of computation. The core message is that if a function can be computed by a real-valued circuit using a defined set of elementary gates, then a neural network can approximate it with comparable accuracy. The network's depth, width, and non-zero parameters are directly controlled by the circuit's characteristics, implying that NN complexity is governed by algorithmic complexity, not just regularity. The paper further establishes that any definable NN model satisfying a natural parallelization condition—even those with multivariate non-linearities like attention or layer normalization—achieves universal approximation if and only if it includes at least one non-affine nonlinearity. This comprehensive theory is illustrated by deriving universal approximation guarantees for continuous functions, minimax-optimal guarantees for Besov classes, and demonstrating NNs' capacity to emulate numerical algorithms like Newton-Raphson without architecture-specific arguments, showcasing exponential improvements in parameter efficiency for tasks like shortest-path computation.