New Linear Attention Method Improves Long-Context Retrieval

Linear attention mechanisms are designed to maintain a fixed-size state, which is efficient but can lead to interference when processing very long contexts, as many stored items compete for this limited state. Previous delta-rule models, such as Gated DeltaNet-2 (GDN-2), derive their erase vector solely from the key of the current token. However, the interference that degrades retrieval is often measured through the query, and a key-derived erase step cannot directly address the query-orthogonal part of the read. To overcome this limitation, a new approach called the Query-derived Erase Direction (QED) has been developed. QED introduces a second erase direction that is derived from the query and is orthogonal to the key. In the context of fast-weight networks, a key-directed delta edit is unable to modify the part of a read that is orthogonal to the key. QED utilizes this editable portion to effectively cancel out content from the old state, specifically targeting interference measured along the query. This innovation leads to substantial improvements in retrieval performance, particularly at context lengths extending beyond the original training window. Experiments demonstrate that QED approximately doubles the usable context length on the S-NIAH-1 benchmark. By providing a more targeted forgetting mechanism, QED enhances the ability of linear attention models to manage long-range dependencies and reduce interference, making them more effective for tasks requiring extensive contextual understanding.