Expander Sparse Autoencoders Boost Interpretability with Efficiency

Sparse autoencoders (SAEs) are a key tool for mechanistic interpretability, helping to decompose internal neural network activations into sparse combinations of learned features. They achieve this by fitting an overcomplete dictionary and inferring a sparse code. However, traditional SAEs with dense decoders require a large number of learned parameters, making them costly for large feature counts. This research proposes Expander SAEs, which are TopK SAEs where the decoder and tied encoder are constrained by a left-d-regular expander mask. This innovative structure drastically reduces the number of learned decoder values while keeping the sparse-coding problem fixed. For instance, in experiments with models like Qwen2.5-3B, an Expander SAE with d=7 used 293 times fewer learned decoder values than a full dense decoder, yet retained 84% of the dense model's CE-loss recovered. The study shows that this improved storage-fidelity tradeoff stems from the sparse, diverse decoder support structure, and provides theoretical conditions for noiseless k-sparse code identifiability.