Reproducibility Study Clarifies Transformer Residual Connection Roles.

Residual connections are a cornerstone of modern transformer architectures, yet the distinct contributions of their attention and feed-forward pathways remain incompletely understood. This paper presents a rigorous reproducibility study focusing on partial residual ablations in Pre-Layer Normalization (Pre-LN) GPT-style transformers, examining models at two different scales: 10 million and 124 million parameters. The study systematically compared four architectural configurations by selectively removing either the attention residual connection, the feed-forward residual connection, or both. A key finding was that removing the attention residual (resulting in an "FFNOnly" configuration) consistently led to a deterministic collapse of the model's performance, reaching the baseline of a transformer without any residual connections. In contrast, removing the feed-forward residual (an "AttnOnly" configuration) exhibited a reproducible recovery effect at the 10 million parameter scale, though its behavior at the larger 124 million scale was less clear due to significant seed variance. During the investigation, the author identified and corrected an experimental measurement confound related to runtime gain scaling, meticulously documenting both the initial failed reproduction and the subsequent successful controlled replication. Based on these empirical results, a cross-position routing hypothesis is proposed to explain the observed asymmetry in the roles of the two residual pathways, distinguishing confirmed findings from areas requiring further research. The complete source code, configurations, checkpoints, and logs are released to support future reproducibility.