New Looped LLM Architecture Optimizes Recurrent Compute for Better Reasoning

Large language models often use recurrent computation to enhance reasoning by repeatedly applying shared layers. Traditional methods typically loop an entire stack of layers, which can be inefficient as different components like mixers and feed-forward networks have varying computational demands. This paper investigates which specific parts of a model should be looped for optimal performance. The researchers propose MixerLoop, an architecture that applies recurrence only to the Gated DeltaNet mixer components, while the dense feed-forward network (FFN) is executed just once. They introduce Iterative Transport Rank (ITR) to quantify the cumulative influence of these repeated applications. Evaluations at 15M and 110M parameters show that MixerLoop outperforms full-block recurrence on aggregate reasoning tasks, retaining significant performance gains while reducing recurrent backbone FLOPs by nearly 46%. This demonstrates that the benefits of recurrent depth can be achieved without the overhead of repeatedly executing the dense FFN.