Depth-Recurrent Transformers Show Per-Token Fixed-Point Convergence.

Depth-recurrent transformers, which apply a weight-tied core multiple times, have been shown to work across various inference depths. New research delves into what these models compute at a per-token level, revealing a fascinating convergence behavior. The study found that the recurrent state of these transformers converges to a distinct fixed point for each token. While the median token converges quickly, about 10% of tokens continue updating for longer, with convergence depth varying by token type (e.g., whitespace converges shallower than content words). This non-uniform convergence is a key finding. Crucially, this per-token variation can be directly observed and leveraged. A simple, training-free rule that halts each token once its output stabilizes achieves the quality of a uniform depth-8 model but with a 38% reduction in average inference depth. This suggests significant potential for optimizing inference efficiency in depth-recurrent transformer architectures.