Weight Norm Dictates Grokking Timescale in Neural Networks

This research addresses the debate surrounding the role of weight norm in "grokking," a phenomenon where neural networks achieve generalization long after perfectly fitting their training data. Previous studies offered conflicting views on whether a critical weight norm value directly causes this delay. The authors settled this dispute by actively manipulating the weight norm during training, rather than merely observing it. They discovered that under free training with weight decay, networks grok when their weight norm reaches a specific critical value, which remains consistent across different training conditions. Crucially, when the weight norm was clamped to a fixed multiple of this critical value, the grokking delay followed an exponential law. This demonstrates a causal link, showing that the weight norm directly dictates the timescale of grokking. The study also noted that LayerNorm can decouple weight scale from network function, removing this dependence.