Modular TTT Framework Simplifies Test-Time Training Design

Test-time training (TTT) treats sequence modeling as an online learning process where model weights are rapidly updated. Current TTT approaches are often hard-coded, making it difficult to develop new methods or understand the impact of individual components. A new framework, Modular TTT, addresses this by representing the internal learning mechanism as a directed acyclic graph. This framework explicitly defines design elements such as the fast-weight network, loss function, learning rate, weight decay, and normalization. Modular TTT automates the composition of these elements into a complete TTT computation, including fast-weight state transitions. Through systematic experimentation, the researchers found that small learning-rate initialization, weight decay, and a single-layer nonlinearity significantly improve performance, while deeper fast-weight networks and normalization can be detrimental due to excessive activations.