LLMs Learn Transient Semantic Structure Despite One-Hot Training

Neural Collapse theory suggests that balanced one-hot classification pushes model representations into a symmetric configuration, where they are equally distant from each other, ignoring semantic similarities in inputs. This presents a puzzle for next-token prediction language models, which are predominantly trained with one-hot labels but clearly learn rich latent semantic structures. This study investigates how gradient descent manages to find such categorical semantic structure when co-occurrence statistics, under a one-hot regime, would seem to eliminate shared next-tokens among different contexts. Using synthetic controlled settings, researchers found that semantic geometry emerges early in the training process, causing representations to cluster by shared attributes even without explicit supervision for this. However, this emergent structure is transient. Given sufficient model capacity and training time, the representations eventually converge to the symmetric state predicted by Neural Collapse, where all representations are equally separated. The research analyzes this phase transition using Gram matrix analysis and proposes a preliminary modification to the unconstrained features model to better capture this transient semantic geometry.