Memory Compaction in LLMs: A Rate-Distortion Perspective

Large Language Models (LLMs) and the agents built upon them increasingly dedicate significant computational and memory resources to "remembering" information, including KV caches, long prompts, and recurrent states. This paper argues that diverse memory compaction techniques across different research communities—such as KV cache eviction, prompt pruning, and agent memory consolidation—are fundamentally instances of a single problem. The authors propose a rate-distortion framework to precisely define this problem: deciding what context-derived information to retain or discard, at what fidelity, given a resource budget, to preserve downstream task utility. They introduce a seven-axis taxonomy to classify existing methods uniformly and highlight common patterns, such as the reliance on attention magnitude or recency for retention decisions, which often fail by discarding information prematurely. The paper concludes by proposing a new benchmark and design principles to address the challenges of repeated memory compaction in agents, an area currently under-measured.