Context-Ready Transformer Boosts Inference Speed, Performance.

This paper introduces a novel recurrent neural network architecture called the "context-ready transformer," which fundamentally rethinks how tokens are processed within a transformer block. Instead of feeding raw embeddings, this architecture pre-contextualizes each token before it enters the D-layer transformer block. During left-to-right generation, a correction network combines the previous position's block output—acting as a cached summary of past context—with the current token embedding, ensuring the token is already contextually aware upon entering the block. This correction chain transforms the architecture into a recurrent neural network for sequential inference. For training, the correction process is unrolled multiple times over the full sequence, allowing parallel processing of all positions at each step. A key advantage is the ability to convert a pretrained transformer into a context-ready model by simply adding a zero-initialized correction FFN and fine-tuning. Evaluations across various configurations and datasets show significant improvements: a D=5 context-ready model outperformed a 12-layer standard transformer while generating 1.7 times faster on an A100 GPU. Even a single-layer (D=1) model, with K=10 unrolling, surpassed a 6-layer transformer with a 2.6x inference speedup, demonstrating its efficiency and effectiveness, particularly for wide representations and long contexts.