New Research Identifies "Computational Scaffold" in LLM Activations.

Sparse autoencoders (SAEs) are typically trained to reconstruct the entire residual stream of large language models (LLMs), assuming all activation content is amenable to sparse, monosemantic decomposition. This research questions that assumption, hypothesizing that activations contain a crucial low-rank, dense component that is computationally important but inherently unsuitable for sparse representation. This dense component is identified as a major source of the persistent dense latents observed in trained SAEs. To test this hypothesis, the researchers introduced a small rank-$r$ linear bottleneck in parallel with standard SAEs (BatchTopK and Matryoshka). This bottleneck allows the dense structure to be absorbed before sparse reconstruction. On Gemma-2-2B layer 12, a rank-24 bottleneck dramatically reduced dense latent count by up to 84% while simultaneously improving sparse probing and targeted probe perturbation. The absorbed component, termed a "computational scaffold," was found to be structurally identifiable as top principal components and outlier dimensions, causally necessary (its removal significantly increased next-token cross-entropy), and redundantly encoded by sparse dictionaries. These findings suggest a need to re-examine the scope and methods of sparsity-based interpretability, advocating for a hybrid approach that handles both sparse and dense components effectively.