Weight Norm Directly Influences Neural Network Grokking Timescale

This study investigates "grokking," a phenomenon where neural networks achieve generalization long after initially fitting their training data. The research specifically addresses the debate surrounding whether the weight norm of a neural network is a causal factor in this delayed generalization. By actively manipulating the weight norm during training, rather than merely observing it, the researchers established a causal link. They found that when the norm is clamped to a fixed value, the grokking delay follows an exponential law, with the delay increasing significantly as the clamped norm increases. This exponential relationship holds across various network configurations and learning rates, indicating a fundamental mechanism. The findings suggest that the weight norm is a primary driver of the grokking timescale, with a LayerNorm layer capable of decoupling this dependence. This work provides a clearer understanding of the underlying dynamics of generalization in deep learning models.