Fixed-Architecture Neural Networks Achieve Super-Expressive Approximation

Previous research has shown that neural networks of a fixed size can achieve "super-expressive" approximation capabilities, meaning they can approximate complex functions very accurately. However, these studies often lacked a clear, quantitative understanding of how the magnitude of the network's parameters relates to the approximation error. This new work addresses that gap by providing explicit characterizations of parameter bounds for fixed-architecture neural networks using elementary activation functions. The key innovation in this research is the use of the Chinese Remainder Theorem as a constructive encoding mechanism. This mathematical tool allows for the precise construction of neural networks with specific properties. For Lipschitz continuous functions defined on a D-dimensional unit hypercube, the researchers constructed a network with a width of `max{D,4}` and a depth of 5, providing explicit trade-offs between parameter magnitude and approximation error. Furthermore, for H\"older-smooth functions, a fixed network with a width of `max{2D, D+5N+1}` and a depth of `r + 9` was shown to achieve a parameter magnitude `P` where `log2 P` scales as `O(epsilon^(-2D/(r+gamma)) * log(1/epsilon))`. This result is significant because it offers a dual perspective to existing paradigms that focus on parameter-bounded but architecture-unbounded networks, providing a concrete understanding of how fixed-size networks can achieve high expressivity with controlled parameter sizes.