New Robustness Law for Two-Layer Neural Networks Proven.

A recent theoretical breakthrough confirms a long-standing conjecture regarding the robustness of two-layer neural networks. The study demonstrates that for networks fitting noisy data, their Lipschitz constant—a measure of sensitivity to input changes—must be at least proportional to the square root of the ratio between the number of data points and the number of neurons. This holds true even when the network's weights are unbounded. The proof, which applies to various continuous piecewise-linear activation functions like ReLU, addresses a challenge previously faced by bounded-parameter assumptions. It introduces a novel function-space covering argument instead of the traditional parameter-space covering, which is unsuitable for unbounded weights. This "law of robustness" provides fundamental insights into the generalization and stability properties of deep learning models, particularly highlighting how fitting noisy data inherently increases a network's sensitivity.