Laguerre Geometry Offers New Interpretability for Large Language Models.

A new research paper proposes Laguerre Geometry as a powerful framework for interpreting the internal workings of Large Language Models (LLMs). Unlike previous approaches that simplify concepts to points or directions, this method defines concepts as specific regions within a geometric space, specifically Laguerre-Voronoi cells. This allows for a more precise and nuanced understanding of how concepts are represented and separated within an LLM. The framework also naturally uncovers intricate concept structures, such as hierarchical relationships and inclusions, by analyzing Laguerre weights. By decomposing the transformer architecture into piecewise-linear operations, the researchers show that a token's hidden trajectory is governed by a combination of static internal flow and dynamic shifts induced by cross-token attention. This geometric understanding leads to the development of "Geometric Lens," a novel, training-free, and hyperparameter-free technique capable of accurately reading out the exact concept encoded by a hidden vector at any layer of an LLM. Additionally, the "Laguerre Autoencoder" provides a 2D visualization tool to render both decision geometry and the model's complete reasoning path. The work moves beyond mere explanation, demonstrating actionable interpretability by showing Geometric Lens can recover correct factual tokens even when the model faces in-context interference.