Looped Transformers: When Recurrence Becomes an Algorithm

This paper investigates the conditions under which a weight-tied looped transformer, essentially a single block applied multiple times, begins to behave like a true algorithm. The study reveals a "budget law," where the computational speed (positions solved per loop) is directly proportional to the training budget. This implies that the model learns a linear computational frontier, and SGD selects a frontier matching the minimum required by the training contract. A key finding is that the architectural prior, rather than just the model's expressivity, determines which algorithm is learned; weight tying, for instance, favors serial computation over parallel scans. The research also challenges conventional wisdom about computational complexity, showing that certain "hard" problems are easily generalized while others, seemingly simpler, pose significant challenges unless an operator-first curriculum is used. Finally, the paper introduces a novel "convergence-time scaling" instrument, which can predict a model's performance on out-of-distribution data, a capability that traditional metrics lack.