Deep ReLU Networks Show Quadratic Depth Dependence in Regression.

Understanding the theoretical properties of deep neural networks, particularly how depth influences their learning capacity, remains a crucial area of research. This paper investigates Gaussian regression using the explicit vector-valued Parhi-Nowak deep-RBV^2 architecture, characterized by its depth (L), width (w), and variation budget. Previous theoretical bounds for this architecture showed a difference of one factor of depth, leaving an open question about the precise dependence. The researchers construct a novel local packing, a method used to establish lower bounds on statistical estimation problems, which explicitly demonstrates that the minimax risk has a quadratic dependence on depth (L^2). This quadratic dependence is shown to be intrinsic under a specific sample-size-dependent radius condition. Key to their findings are a bias-corrected approximation theorem and a balanced amplification technique that allows for controlled coefficient growth across layers. The results establish that the minimax risk is at least of order L^2 w^2 log(w) R^2/n, aligning with an upper bound and thus confirming the quadratic polynomial dependence on depth, up to logarithmic factors. This work provides a deeper theoretical understanding of why depth is so powerful in deep learning.