First-Order Criterion Quantifies Optimal Depth in Residual Networks

This research addresses a fundamental question in neural network design: how to determine if a trained residual neural network (ResNet) has sufficient depth. The authors introduce a first-order criterion for "residual depth saturation," which is defined as the point where adding more layers, following a specific function-preserving growth protocol, no longer yields a strict local decrease in the network's performance. This criterion provides a principled way to quantify when a network is "deep enough." The study rigorously proves that additional depth offers first-order value precisely when the conditional activation gradients possess a non-zero projection onto at least one admissible residual tangent space. This non-degeneracy condition is both necessary and sufficient for the benefit of increased depth. Furthermore, the research shows that under conditions of residual-signal realizability, the vanishing of raw activation-gradient magnitude directly certifies saturation. Empirical observations across various models, including ResNets, GPT-2 style models, and Pythia checkpoints, support these theoretical findings, showing that the maximum activation-gradient norm decreases with increasing depth, moving towards a low-signal regime. The study also notes that function-preserving growth can achieve performance comparable to training from scratch, reinforcing the idea that activation-gradient magnitude serves as a conservative diagnostic for the remaining first-order value of residual depth.