LLM Reasoning Improved by Modeling as Attractor Dynamics

Traditional views often characterize Large Language Models (LLMs) as autoregressive generators, predicting the next token in a sequence. However, this research posits an alternative perspective: LLMs function as high-dimensional Dense Associative Memories, storing complex reasoning patterns as latent attractors within their internal states. From this viewpoint, correct reasoning chains are seen as deep, wide, and stable attractor basins—analogous to "flat minima" in the model's output distribution—while hallucinations or incorrect reasoning manifest as sharp, unstable local minima. To leverage this understanding of the model's internal geometry, the study introduces a novel retrieval mechanism. This mechanism is based on a Gibbs measure of the trajectory's spectral entropy. By sampling multiple potential reasoning paths and weighting them according to their inverse energy, the system approximates the equilibrium distribution of the associative memory. This process effectively allows the model to "relax" into a more robust and correct solution, rather than relying solely on greedy next-token prediction. Empirical validation demonstrated the effectiveness of this physics-inspired mechanism. When applied to Microsoft Phi-3.5, it improved performance on the GSM8K mathematical reasoning benchmark by a significant 5.38%, increasing accuracy from 84.7% to 90.1%. This suggests that modeling LLM inference as a dynamic settling process into an attractor basin offers a more powerful approach than conventional greedy decoding.