Sparse Autoencoder Features: Causal Necessity and Stability

Sparse Autoencoder (SAE) features are increasingly used to interpret and control large language models (LLMs), but their causal importance and consistency across different SAE implementations remain unclear. This study focuses on "single-token features," which activate for a single vocabulary item, providing a clear case for direct comparison. Analyzing millions of features across various models and three SAE families, the researchers found that single-token features are more tightly clustered in the decoder space and tend to appear in earlier layers of the LLM. Ablating these features led to statistically significant reductions in logit scores, with the depth of the layer influencing whether the damage cascaded or directly impacted the output. Crucially, the causal differences observed between different SAE families were more pronounced than those due to scale effects within a family. For instance, while GemmaScope and BatchTopK features maintained causal anchoring, LlamaScope features showed local redundancy. The study concludes that claims about cross-family interpretability are highly sensitive to the specific training methodology used for the SAE, rather than just the activation function or model scale.