New Multi-Head Memory Boosts LLM Long-Context Retention

Large Language Models (LLMs) often struggle with maintaining context over extremely long sequences, despite recurrent memory agents designed to consolidate input. This research identifies memory retention as the primary bottleneck, where existing monolithic memory blocks risk overwriting crucial past information with every update. To counter this, the paper proposes Multi-Head Recurrent Memory (MHM), an innovative, training-free architectural framework. MHM divides the LLM's memory into several independent "heads." At each processing step, only one head is selected for updating, while the others are structurally protected from being overwritten. This design shifts the burden of memory retention from the model's behavior to its architecture. A specific instantiation, MHM-LRU (Least-Recently-Updated), ensures uniform head utilization without adding token overhead. Extensive experiments demonstrate that MHM-LRU dramatically enhances both memory retention and overall accuracy across context lengths ranging from 100,000 to 1 million tokens, where traditional baselines typically degrade sharply. For instance, on the RULER-HQA benchmark at 896K tokens, MHM-LRU boosted memory retention from under 30% to nearly 74%. These improvements are consistent across various model families, scales, and task types, highlighting architectural optimization as a practical and cost-effective solution for reliable long-context recurrent memory.