E$^2$-TTT Balances Expressivity and Efficiency in Test-Time Training.

This research introduces E$^2$-TTT, a novel approach to Test-Time Training (TTT) that aims to resolve the long-standing trade-off between model expressivity and hardware efficiency. TTT involves continuously updating model weights during inference, which is crucial for processing long contexts. However, existing methods struggle to balance the detailed, per-token update dynamics with the computational benefits of processing data in larger chunks. E$^2$-TTT addresses this by deriving a closed-form state transition. This mathematical formulation precisely replicates the fast-weight and momentum states that would result from a per-token recurrence, even when gradients are taken only at the start of each chunk. This innovation enables fully parallelized chunk-level training, significantly boosting throughput, while critically preserving the temporal structure of the update rule that prior chunk-wise methods often discard. The method was validated by training models up to 1.3 billion parameters, showing performance on par with previous TTT and hybrid attention baselines in language modeling, and outperforming them in in-context retrieval, especially for length extrapolation tasks like "Needle in a Haystack."