Sparse Transformer Weights Show Higher Interpretability

Understanding the internal workings of neural networks, particularly large language models, is a key challenge in mechanistic interpretability. Traditional approaches often focus on specific behaviors or sub-distributions, but individual components can have varied functions depending on the input. This study explores whether individual weights can be understood globally. The researchers developed an automated LLM pipeline to characterize when a specific weight matters by observing how its ablation affects model predictions. This pipeline generates a concise, human-readable description for each weight and then verifies its generalization on unseen text. The findings indicate that sparse transformers exhibit a higher fraction of interpretable weights compared to dense transformers. Between 12% and 31% of weights in sparse models could be reliably described by a single, short explanation of their function. This suggests that sparsity not only improves efficiency but also enhances the transparency of model components.