HARD-KV Boosts LLM Throughput with Adaptive KV Compression.

Large Language Models (LLMs) processing long contexts face a fundamental challenge: while advanced compression algorithms like Top-p nucleus sampling dynamically adjust memory usage for better accuracy, high-performance inference engines such as vLLM require fixed, predictable memory patterns for efficiency. This creates a "Static-Dynamic" mismatch that limits performance. HARD-KV is a novel framework designed to bridge this gap. It manages the token lifecycle through a multi-tiered cascade cache, intelligently moving data between dense, sparse, and condensed storage. A key innovation is its Logits Calibration mechanism, which standardizes diverse importance metrics into a single probability space, allowing consistent Top-p budgeting across different attention heads. Furthermore, HARD-KV includes a system-level solution that converts fragmented, dynamic memory indices into contiguous physical layouts, making them compatible with high-performance inference engines. Experiments on math-reasoning benchmarks show HARD-KV can double throughput compared to static baselines, all while maintaining high-fidelity generation for contexts exceeding 10,000 tokens.