LLM Latent State Mixtures Collapse, Limiting Reasoning

Large Language Models (LLMs) sometimes use continuous latent states, such as weighted mixtures of token embeddings, to represent multiple possible reasoning paths simultaneously, rather than committing to a single discrete token. However, these continuous mixtures often fail to persist and instead "collapse," losing their intended diversity. This paper delves into the underlying reasons for this phenomenon. The researchers identify three independent causes for this mixture collapse. Firstly, the inherent architecture of transformers distorts the geometry of these mixtures, an effect that is significantly amplified during the training process. Secondly, even if the model perfectly linearly transports mixtures, the combination of the softmax readout and autoregressive feedback creates a dynamic system. This system either amplifies small differences until one component dominates, or contracts different mixtures until they become indistinguishable. Empirical investigations confirm these theoretical predictions, showing that pretrained models predominantly operate on the amplifying side of this dynamic system. Finally, the study generalizes to mixtures with many components, concluding that preserving these complex mixtures accurately often requires context-dependent correction, and the dimensionality of this correction can increase with the number of components.