Algebraic Representability Explains Grokking's Limiting Behavior.

The phenomenon of "grokking" in neural networks, where a delayed transition from memorization to generalization occurs, is known to depend on model capacity. This study investigates the extreme end of this spectrum by examining two-layer networks with a specific holomorphic monomial activation function, trained on modular arithmetic tasks. In this setup, the network's output is algebraically confined to a finite-dimensional subspace. The researchers found that a task is representable by such a network only if its discrete Fourier support meets a specific algebraic criterion. Crucially, if a target is not algebraically representable, the network cannot even memorize the training set, leading to a guaranteed positive lower bound on training loss. This results in a binary outcome: either instant success or complete failure, with no memorization phase or grokking observed. This suggests that when the expressible function class is severely constrained, the question shifts from *when* a network will grok to *whether* it can represent the target at all.