Nexus Sampling Improves LLM KV-Cache Eviction for Long Contexts

Large Language Models (LLMs) handling long contexts or agentic tasks frequently exceed fixed memory budgets for their Key-Value (KV) cache, necessitating continuous token eviction. Existing eviction strategies typically rely on a direct attention score followed by deterministic top-K selection, which can inadvertently discard subtly important tokens. This new research introduces Nexus Sampling to address this limitation. Nexus Sampling combines two key innovations: Nexus scoring, an iterative process that identifies "bridge tokens" by walking over direct attention scores, and weighted reservoir sampling, which assigns inclusion probabilities to tokens instead of making irreversible deterministic decisions. Theoretically, Nexus Sampling is shown to improve the long-term survival of crucial tokens. Empirically, it matches dense attention performance on LongBench tasks while significantly outperforming top-K baselines on retrieval-heavy tasks, demonstrating up to a 10x reduction in per-sequence cache memory.