Research Explores Cross-Lingual Effects and Separability in LLM Representations

This study investigates the internal workings of multilingual large language models (LLMs) to understand how different languages are represented and interact within these complex systems. Researchers used a causal-geometric analysis framework, examining 28 bilingual contrasts across three distinct LLMs. The goal was to determine when language concepts behave as independent factors and when structured dependencies persist. The findings indicate that language concepts possess stable linear representations that are mostly separable when using a covariance-adjusted inner product. However, structured deviations are observed, which reflect the inherent linguistic similarities between languages. For instance, languages belonging to the same family, such as Germanic or Romance, display a simplex-like geometric structure, suggesting a hierarchical organization within the model's representations. These results extend the applicability of causal-geometric interpretability to multilingual contexts, offering valuable insights into how separability and similarity manifest in LLM representations. This understanding is crucial for anticipating and diagnosing potential cross-lingual effects, particularly when models are monitored or modified, thereby contributing to more trustworthy deployment of multilingual AI systems.