New ParityTransformer Improves Interpretable AI at Scale

Traditional large language models often exhibit "superposition," where many features are compressed into fewer dimensions, making them hard to interpret. While sparse autoencoders (SAEs) can extract these features post-hoc, building interpretability directly into models has been challenging due to the high computational and memory costs of per-layer bottlenecks. A new architecture, the ParityTransformer, addresses this by integrating Deep Parity Bottleneck (DPB) layers. These layers replace learned over-complete bases with a parameter-free algebraic dictionary, ensuring efficient and sparse intermediate representations. This design eliminates the memory overhead that previously hindered scalable interpretable models. The ParityTransformer performs comparably to post-hoc SAEs on sparse probing tasks and excels in feature absorption, steering, and causal interventions. Its features are inherent to the model's forward pass, offering a significant step towards training large language models that are interpretable by construction rather than requiring post-analysis.