Sparse Autoencoder Features Audited for Causal Inertness.

Sparse autoencoders (SAEs) are widely used to break down complex neural representations into more interpretable features. Current evaluation methods primarily rely on correlational metrics, such as cosine similarity, to assess how well SAEs recover these features. However, this research argues that such metrics conflate two distinct aspects: the alignment of the decoder's geometry with ground-truth directions and the actual behavior of the encoder's activations. The study reproduces known phenomena like the superposition phase diagram and the effects of L1 regularization, but its core contribution is a causal audit. By systematically ablating and steering individual features, the researchers discovered that a substantial percentage of features, even those with high correlational recovery scores, are "causally inert." This means they do not fire when the corresponding feature is present or have no causal impact on the model's output, despite appearing to be well-matched. The new sae-causal-audit method, which is model-agnostic, identifies different types of inertness, including structural inertness (due to antipodal-pair geometry) and competitive inertness (a pathology of degraded SAEs). It also differentiates between read- and write-inertness, showing that some features can be steered without being monitored. The findings underscore the limitations of purely correlational evaluations and emphasize the need for causal validation to truly understand and trust the interpretability of SAE-derived features, while also noting the inherent difficulties in achieving byte-exact reproducibility in such complex systems.