New Method Accelerates Neural Network Grokking Generalization.

A new study explores "grokking," a phenomenon where neural networks achieve delayed generalization long after memorizing training data, often wasting significant training time. The researchers propose a novel approach, CvAdamW, which is a variant of the AdamW optimizer designed to accelerate this generalization process.The core idea behind CvAdamW is to treat the variance of Transformer attention logits as a "specific heat" (Cv) in a thermodynamic system. They found that a peak in this Cv reliably precedes the generalization transition. By monitoring Cv online, CvAdamW dynamically adjusts weight decay, effectively injecting "thermal energy" into the training process when a phase transition is detected.Through an iterative development process, the team identified and resolved several failure modes, leading to a robust method. On modular arithmetic tasks, CvAdamW enabled grokking within a fixed budget where the baseline failed to generalize. A scale-invariant z-score reformulation further improved its applicability, demonstrating a significant reduction in mean grokking latency across multiple seeds. This research suggests that neural networks exhibit detectable precursors to generalization, which can be leveraged for more efficient training.