New Fourier Delta Attention Improves Long-Context Memory.

This paper introduces Semidirect Fourier Delta Attention (SFDA), a significant advancement in linear attention mechanisms. Unlike traditional linear attention which compresses the KV cache into a fixed recurrent state, limiting long-context memory, SFDA employs a phase-controlled generalization of Kimi Delta Attention. This involves replacing real diagonal decay with block-rotational Fourier control, allowing for more nuanced state tracking. A key contribution is a constructive chunk-WY factorization method for products, which enables an exact affine chunk transfer and provides formal stability and complexity bounds. This factorization offers a compact way to characterize phase-plus-low-rank memory, addressing previous limitations in maintaining long-term dependencies. Numerical verification and toy state-tracking experiments demonstrate SFDA's ability to learn cyclic memory, where phase-disabled baselines struggle. This indicates a promising direction for developing models with more robust and extensive long-context understanding, though large-scale language model comparisons are reserved for future work.