New Residual Scaling Improves Looped Transformer Stability and Transferability

Looped Transformers, which reuse a single residual block multiple times to increase effective depth without adding parameters, present unique challenges for stability. While traditional depth-scaling analyses for residual networks suggest a scaling factor of `1/√L` for a depth-L network, this research demonstrates that this is insufficient for looped architectures. The study reveals that weight sharing in looped Transformers causes residual updates to be correlated across iterations, necessitating a stronger scaling factor of `1/N`, where `N` is the number of loops. For multi-layer blocks, a new factored parameterization `λ/(N√L)` is derived. This parameterization effectively separates the two sources of growth: `1/N` for within-layer loop correlation and `1/√L` for across-layer variance. A significant implication of this finding is that the optimal learning rate for these models depends solely on the number of unique layers `L`, not on the loop count `N`. This allows for direct transfer of hyperparameters from smaller to larger `N` configurations without requiring extensive retuning. Experimental results on looped Transformers confirm that the `1/N` scaling significantly improves trainability and leads to better loss compared to `1/√N` scaling across various loop counts.