New Principle Guarantees Stable Residual Neural Network Training

This paper proposes a fundamental principle for designing stable deep residual neural networks, known as the "sublinear-growth principle." This principle defines a critical stability threshold based on the input-magnitude exponent of each residual block's velocity field. Specifically, it states that for stable training and inference, this exponent must be less than or equal to one. The authors provide two independent theoretical arguments for this threshold: classical ODE theory, which shows global forward flow for exponents less than or equal to one and divergence for greater than one, and optimal-control analysis, which reveals that optimal training solutions blow up when the exponent exceeds one. This criterion clarifies why certain architectural choices, like layer normalization, contribute to stability. The research also introduces an "arithmetic" for input-magnitude exponents, allowing efficient certification of architectural primitives. A practical demonstration shows how a supercritical Mamba block, initially with an exponent of five, can be modified to meet the stable threshold of one without needing layer normalization, confirming the principle's utility in designing robust architectures.