New Cache Architecture Improves Long-Range Associative Recall.

Sequence models face a trade-off in memory and recall: fixed-state models compress the past into a bounded state, limiting associative recall, while attention mechanisms store every token, leading to quadratic compute and memory growth. This paper proposes a middle ground: a sparse cache that allocates memory slots only for novel inputs, effectively tracking distinct items rather than all tokens. The core of this innovation is a Dirichlet-Process (DP) cache, which uses a DP-means clustering rule as its key-value memory operator. This cache comes in static and surprise-adaptive variants, with the latter adjusting its concentration based on the recent novelty rate. The system is designed to be learnable end-to-end, with a two-parameter novelty-threshold gate trained on task loss alone. On controlled associative-recall benchmarks, the DP cache demonstrated recall performance comparable to full attention while requiring memory proportional only to the number of distinct items. It also outperformed fixed-budget eviction caches on the recall-versus-size frontier and achieved the lowest memory footprint for both recall and long-range aggregation tasks when integrated with a state-space backbone. The research highlights that the inductive bias of the DP-means rule is the operative ingredient, not just increased capacity.