New Unsupervised Method Extracts Reasoning Features from LLMs.

A new research paper presents Mining via Activation Geometry (MAG), an unsupervised framework designed to uncover the internal reasoning features within large language models (LLMs). Unlike traditional interpretability methods that often rely on pre-labeled examples, MAG operates by observing how an LLM's internal activation space changes when a specific natural language instruction, defining a reasoning feature, is prepended to an input. The framework measures the difference in activation patterns with and without the instruction, allowing researchers to identify and approximate these reasoning features as activation directions. The study explored eight different MAGs, finding that these extracted features accurately predict the model's own understanding and judgments. Importantly, these linear representations can be used for "activation steering," enabling researchers to influence LLM decisions by injecting specific reasoning features. Furthermore, MAG proves effective in selecting optimal training datasets for prompt-injection classifier probes, achieving high accuracy by leveraging RFD-based similarity. This work provides a powerful tool for both understanding and controlling the internal mechanisms of LLMs.