LLMs Show Cross-Language Reasoning Invariance with New Autoencoder Method

This research explores how multilingual large language models (LLMs) process information when solving the same problem in different languages. The core question is whether these models rely on a universal, shared understanding or if they perform distinct, language-specific computations that merely yield similar results. To investigate this, the study utilized the Multilingual Grade School Math dataset across six languages and five different LLMs. A key innovation in this work is the introduction of the Geometry-Invariant Sparse Autoencoder (GI-SAE). This method aims to identify and enhance cross-language feature alignment within the LLMs' internal representations. By comparing GI-SAE with a baseline, the researchers observed that GI-SAE generally improved geometric similarity across languages. However, the study also revealed that increased geometric similarity does not consistently translate to greater functional interchangeability of features between languages. The extent of cross-language feature sharing proved to be highly dependent on the specific LLM and its architecture, appearing at varying depths within different models. GI-SAE primarily strengthens existing cross-language structures rather than creating new ones, with effects differing significantly across models like Qwen, Gemma, Llama, and Phi.