Benign Misfitting: Good Generalization Despite High Training Error.

New research explores a counter-intuitive phenomenon in machine learning called "benign misfitting," or the "fourth quadrant." This refers to scenarios where a model, specifically linear regression with squared-error, can achieve good generalization performance on unseen test data even when it fits the training data worse than a trivial zero predictor. This challenges the conventional wisdom that lower training error always correlates with better generalization. The study uses a stylized single-spike model to demonstrate that within a specific range of training-set sizes, the best-performing linear predictors intentionally "misfit" the training data, overshooting the labels. Surprisingly, one-pass Stochastic Gradient Descent (SGD) with a large learning rate is shown to effectively navigate this regime, achieving low test error despite exhibiting high empirical training error. This suggests that in certain high-dimensional settings, interpolation is not always the optimal path to generalization, and some degree of training misfit can be beneficial. The research also links this unavoidable nuisance component to the predictor's adversarial sensitivity.