DeepLoop Improves Depth Scaling for Looped Transformers.

Looped Transformers offer a way to scale sequential computation by repeatedly applying a compact set of physical blocks, effectively increasing the unrolled depth without increasing the total number of stored parameters. However, this parameter reuse fundamentally alters the residual-scaling problem compared to untied Transformers, where each layer has unique parameters. The research formalizes this "tied-depth effect" through a first-order perturbation bound, which is influenced by a visit-alignment coefficient. This bound indicates that for stable recurrent depth, the residual scaling exponent needs to increase from 1/4 to 1/2 as the loop count grows, assuming a fixed physical depth. Based on this insight, the authors developed DeepLoop, which maintains the Post-LN DeepNorm architecture but sets specific alpha and beta scaling parameters for unrolled depth N. Experiments with GPT-style looped language models, at GPT-2 small and medium scales, show that DeepLoop is neutral when no physical block is revisited but significantly improves validation loss and downstream accuracy once recurrent depth is activated, confirming the importance of accounting for parameter visits in residual scaling.