Diagnosing Adaptive Depth in Looped Transformers

This study delves into adaptive depth mechanisms in looped Transformers, which repeatedly apply a shared recurrent block to adjust computation based on input complexity. The research highlights a critical issue: current learned halting objectives typically use a single exit distribution for both deciding when to stop inference and weighting per-depth losses during training. This entanglement means the halting gate not only selects the final recurrent state but also dictates how strongly each intermediate state is supervised, potentially leading to suboptimal adaptive compute performance. The researchers analyze adaptive depth through a "trajectory-readout" lens, using synthetic tasks and large-scale Ouro checkpoints. They discover that fixed-prior depth supervision, which shapes the model's internal trajectory without an input-dependent halting policy, often yields difficulty-aware trajectories with useful stopping signals in their intermediate states. Furthermore, simple post-hoc confidence readouts frequently perform as well as, or better than, more complex learned linear or MLP gates. The findings suggest that the primary failure point often lies in the trajectory induced by joint gate training, rather than a lack of expressivity in the gates themselves. This pattern holds for Ouro evaluations, where pretrained ponder gates are competitive but not uniformly optimal, and latency measurements confirm practical inference-time savings from reduced average exit depth. The study reframes adaptive depth as a joint problem of trajectory formation and exit readout, emphasizing a distinction often overlooked in prior work.