KV Cache Becomes Editable and Composable for Faster LLM Inference

Current prefix caching in large language models (LLMs) is limited, requiring full recomputation if even a single field in the input prefix changes. This is because the KV cache is typically seen as a static storage of input tokens. However, new research reveals that during the prefill phase, LLMs effectively "take notes" within their KV cache, storing conclusions conditioned on specific input fields rather than just raw key/value vectors. This understanding fundamentally changes how the KV cache can be utilized. It implies that the cache is not merely a memory but a dynamic "notebook" of memoized decisions. This allows for two powerful new capabilities: editability and composability. The KV cache can now be directly edited to amend specific "notes," for instance, correcting an erratum. With chain-of-thought prompting, editing a single field in the cache can recover the original decision with minimal computation. Furthermore, these "notes" are position-portable, meaning precompiled skills or segments of computation can be repositioned and spliced into any context within the cache. This composability allows for significant speedups, achieving decision-identical results to full recomputation but at O(L) rather than O(L^2) time-to-first-token complexity. An integrated edit-and-compose agent can reduce latency by up to 14.9x, and when combined with production prefix caching, it can cut p90 time-to-first-token by 53-398x, making LLM inference much more efficient across various model types and attention mechanisms.