PersistentKV Boosts Long-Context LLM Serving on Commodity GPUs

Serving large language models (LLMs), especially those with long contexts, is increasingly bottlenecked by the movement of key-value (KV) cache data rather than the core matrix multiplication operations. While modern paged-attention systems and optimized kernels like FlashInfer have improved KV-cache management, the optimal single-kernel implementation doesn't always translate to the best overall serving schedule, particularly with varied sequence lengths. This paper introduces PersistentKV, a novel native block-table decode attention engine specifically designed for grouped-query attention (GQA). PersistentKV features work mapping by KV-head group, efficient K,V tile reuse across grouped query heads, and support for native page tables. Crucially, it incorporates a compact workqueue schedule that executes only non-empty tasks, addressing the inefficiencies of low-active long-context decode on commodity GPUs. Evaluated on an RTX 3060, PersistentKV's adaptive policy dynamically selects between FlashInfer for small active batches and PersistentKV's sequence splitting or workqueue scheduling for long-context steps. This adaptive approach yielded significant throughput improvements, ranging from 1.063x to 1.265x on various bimodal, uniform, and Zipf-like workloads, and a 1.399x improvement on a B1 bucketed trace. These results underscore that intelligent work assignment and page-aware scheduling are critical variables for optimizing LLM serving systems.