New EpiKV Method Boosts LLM Context Length by 16x

The increasing length of reasoning chains in large language models (LLMs) has made the Key-Value (KV) cache a significant bottleneck in deployment. Traditional cache eviction strategies often rely on attention weights to determine token importance, which can be noisy for extended reasoning traces and necessitates materializing the attention matrix, hindering fused kernel usage. A new method, EpiKV, introduces an "epiphany score" to address these issues. This score measures the change in a model's internal representation, directly read from the forward pass without requiring the attention matrix or significant extra state. This innovation allows EpiKV to be integrated directly into existing FlashAttention inference stacks without modifications, offering a 16x increase in feasible context length compared to attention-based scoring. EpiKV requires no training, classifiers, or custom kernels. It achieves competitive performance on complex reasoning benchmarks like MATH-500 and AIME-2024, matching or exceeding strong attention-based baselines while offering up to 2.8x faster processing. This advancement promises more efficient and scalable deployment of LLMs for long-context applications.