New Early Stopping Rule Uses Rademacher Complexity for Neural Networks.

Training neural networks effectively involves a delicate balance between fitting the training data and ensuring strong performance on unseen data, known as generalizability. This generalizability is determined by the difference between the empirical risk on the training set and the expected risk over the data distribution. Traditional methods for estimating generalization error often rely on numerical approaches, necessitating gradient descent training and an early stopping strategy. This new work presents an analytical framework that can estimate the optimal early stopping time without the need for actual training. Unlike other analytical estimations that often depend on random matrix theory and make assumptions about data or eigenvalue distributions, this approach is based on Rademacher complexity (RC). The authors argue for expressing RC with the L1-norm, rather than the L2-norm, for both theoretical and numerical reasons. While the focus is on linear models and linear regression, the results can be extended to nonlinear neural networks through the "linear probing" method, as demonstrated with a classification example on MNIST. This offers a potentially more efficient way to determine when to stop training, saving computational resources and improving model generalization.