Transformers' FFNs Exhibit Sparse Inter-Layer Dependencies, Aiding Interpretability

This research investigates the internal structure of Feedforward Network (FFN) blocks within Transformer architectures, which are notoriously difficult to interpret despite their significant parameter count. The study introduces a novel, training-free attribution method to quantify the influence of upstream neuron activations and attention outputs on a target FFN neuron. Empirical findings across various models and layers indicate that only a small subset of preceding activations and attention outputs is needed to accurately preserve FFN neuron activations. This "effective sparsity" becomes even more pronounced when considering the inherent sparsity of upstream layers. Crucially, applying these neuron-specific masks across all layers simultaneously maintains model perplexity at moderate sparsity levels, suggesting that FFNs possess sparse and structured inter-layer dependencies despite their dense parameterization. This method offers a scalable tool for circuit-level interpretability and points towards potential pathways for more efficient inference.