MoNe Enables Efficient Long Context Inference for Transformers

MoNe, a novel modular neural memory, has been developed to enhance the ability of pre-trained Transformer models to handle extremely long contexts. This system integrates with existing frozen Transformers, eliminating the need for extensive retraining. It operates by processing context in fixed-size segments and learning fast-weight neural memory networks during inference. The design of MoNe is innovative because it decouples the inference cost from the context length. It achieves efficient preprocessing and query costs, with GPU memory usage remaining constant regardless of context size. This approach offers substantial reductions in both computational demands and peak GPU memory, making long-context applications more feasible. Tests show MoNe can handle contexts far beyond a backbone model's native window, performing strongly on benchmarks like needle-in-a-haystack and word extraction. It achieves approximately 80% reduction in compute and memory for 128K tokens, with only a small parameter overhead, where traditional in-context learning methods struggle.