Grokking Metrics Overstate Compression, Lag Generalization

New research critically examines the measurement of "grokking," a phenomenon where neural networks generalize long after achieving zero training error. The study finds that metrics like effective rank, often used to quantify representation compression during grokking, are misleading. Specifically, they overstate the converged compression by a factor of 3-5x in MLPs and 1.3-1.5x in transformers when measured at the grokking transition. Furthermore, the research demonstrates that representation compression lags behind the accuracy transition by at least 10,000 steps, rather than coinciding with it. The presence of LayerNorm, for instance, significantly impacts this lag. To address these measurement issues, the authors propose a robust audit framework designed to separate the onset of generalization from actual compression, flag censoring, exclude non-generalizing cells, and ensure reference floors have plateaued.