Fisher Rank Inflation Signals Memorization in Noisy Deep Networks

Deep neural networks, when trained with noisy labels, often first learn the underlying clean data structure before they begin to memorize the corrupted labels. This transition, from learning generalizable features to overfitting noise, leaves a distinct spectral signature in the centered scatter of per-example last-layer gradients, a phenomenon termed "Fisher Rank Inflation."During memorization, the effective rank of these gradients transiently expands, then contracts once the corrupted labels are fully fit. This inflation occurs because corrupted labels inject spectral mass into previously low-energy or unused eigendirections, increasing the entropy of the gradient spectrum. The research provides a first-order attribution formula, explaining why corrupted examples contribute more strongly to this rank increase.Experiments across various datasets (CIFAR-10, CIFAR-100, CIFAR-10N) and models (SmallCNN, ResNet18, Vision Transformers) consistently show this inflation-collapse trajectory aligned with memorization. At peak rank, corrupted examples are highly enriched among the top rank-contributing samples. This spectral signature can even precede observable test set degradation, offering a potential early warning system for overfitting to noise.