Logit Scale, Not Weight Norm, Controls Grokking in AI Models

Grokking, a phenomenon where neural networks first memorize training data and then suddenly generalize to unseen data after prolonged training, has often been linked to the weight norm, with smaller norms typically leading to earlier generalization. This study delves into the precise mechanism by which the weight norm influences grokking. The researchers found that by fixing the weight norm and only varying an output temperature, they could manipulate the grokking delay across its entire range under cross-entropy loss. They observed that matching the effective logit scale back to the baseline recovered a significant portion of the delay. Across various norms and temperatures, the grokking delay collapsed primarily onto the logit scale, with the weight norm contributing only a minor additional effect. This effect is also loss-dependent; under mean-squared error, the logit scale remains fixed, and the weight norm operates through a different pathway. The findings, supported by various controls and experiments, indicate that the logit scale and the resulting softmax saturation are the direct variables controlling grokking, with the weight norm serving merely as an indirect, upstream control.