Looped State-Space Models Enhance LLM Reasoning Depth

Recent studies suggest that increasing computational depth, rather than merely adding more independent parameters, can significantly improve language models' reasoning abilities. While this principle has been primarily explored with Transformer architectures, this paper investigates its application in state-space language models, introducing Looped Mamba and Looped Hybrid Mamba-Transformer designs. These architectures repeatedly apply a shared Mamba or hybrid block, thereby creating explicit finite-depth recurrent computation. On controlled reasoning tasks such as modular arithmetic manipulation and p-hop induction, Looped Mamba consistently outperformed non-looped baselines with comparable parameters and, in several settings, matched or exceeded non-looped models of equivalent effective depth. During language model pre-training, looped models remained competitive on downstream benchmarks using substantially fewer distinct parameters. The research also adapted Ouro's two-stage exit gate to Looped Mamba, demonstrating that adaptive exit-state selection improves downstream performance at intermediate depths, although achieving actual inference-time savings still requires additional state-handling mechanisms.