Graph Signal Processing Reveals LLM Numerical Inference Mechanisms

Large Language Models (LLMs) have shown remarkable abilities in performing numerical inference through in-context learning (ICL) when sequences are presented as text. While previous studies have focused on evaluating the output accuracy of these numerical tasks, the internal mechanisms of how LLMs represent and process numerical information remain largely unclear. This paper introduces a novel approach using graph signal processing to delve into these internal representations. By viewing attention as a mechanism that creates a weighted graph among tokens and considering token hidden states as signals on these graph nodes, the researchers uncover systematic patterns. They found that as the context length increases, the internal representations become more distinctly differentiated based on the dynamical complexity of the input. Simpler numerical inputs lead to token graphs with stronger global connectivity and smoother, spectrally concentrated hidden-state signals. Conversely, more complex inputs result in localized graphs and hidden-state signals with broader spectral support and higher high-frequency energy. These findings highlight consistent, context-dependent internal signatures across different LLM families during numerical ICL.