Looped Latent Attention Compresses Transformer K/V Caches

Looped Transformers, which reuse a single block to reduce parameters, typically store a separate K/V cache for each recurrence step during decoding. This paper identifies that these loop-indexed cache vectors exhibit a highly structured, low-rank trajectory across loops. To address the memory overhead, the researchers propose Looped Latent Attention (LLA), a post-training codec that stores compact K and V latents. LLA reconstructs the full K/V vectors only when required by the attention mechanism. This method achieves significant compression, outperforming other techniques like head-axis MLA and KV quantization. The approach has been shown to increase batch capacity substantially, for instance, boosting Ouro-1.4B batch capacity from 32 to 768 sequences at 4k context with 21.3x compression on an H200 GPU.