Attention-Only Transformers Match Standard Models with More Depth.

This research conducts a controlled study to investigate the necessity of feed-forward networks (FFNs) in transformer architectures, which typically account for two-thirds of non-embedding parameters. By comparing "Simple Attention Networks" (SANs), which are attention-only decoder transformers, against standard transformers matched for parameters, compute, and depth, the study reveals interesting trade-offs. Initially, removing FFNs and keeping depth constant leads to a performance deficit. However, by reallocating the freed computational budget into increased attention depth, SANs can largely close this gap. The study found that at matched parameters, the performance difference was negligible, shrinking further with larger training budgets. The remaining slight deficit in attention-only models was localized to parametric recall, meaning they performed worse on tasks requiring knowledge stored in weights rather than context-grounded answers. The research also highlights that QK-normalization, not FFNs or residual gating, is crucial for training deep attention-only stacks.