Qwen3-Instruct SAEs Uncover Millions of Interpretable LLM Features

Researchers have developed and released Qwen3-Instruct SAE, a collection of Sparse Autoencoders (SAEs) specifically designed for the Qwen3 family of instruction-tuned language models. These SAEs are crucial for breaking down the complex, "superposed" internal representations of large language models into more granular, sparse, and human-interpretable features. The project involved training layer-wise SAEs across various activation sites—residual streams, MLP outputs, and attention outputs—for different Qwen3 model sizes. A systematic evaluation confirmed varying trade-offs between sparsity and fidelity depending on the layer and component analyzed. A practical application of these SAEs was demonstrated through a case study on refusal-steering. By manipulating specific SAE features, the researchers were able to causally influence Qwen3 models to exhibit refusal behavior, highlighting the potential for fine-grained control and understanding of LLM mechanisms. This release offers a valuable resource for further research into sparse representations and behavioral interventions in LLMs.