Cache-Aware Prompt Compression Slashes LLM API Costs by 90%

Deploying Large Language Models (LLMs) in production involves managing costs, typically through prompt caching and compression. Existing compression methods, often "query-aware," generate unique compressed prefixes for each query, which inadvertently invalidates the cache and negates caching benefits. This study empirically characterized the caching behavior of Anthropic's Sonnet 4.6 API, revealing a two-tier cache architecture with a critical threshold around 3,500 tokens. The research found that below this threshold, the cache hit rate plateaus at approximately 83% over 30-call sessions, indicating that caching is not as ideal as often assumed. A new cost model was developed, predicting that query-aware compression only outperforms naive caching at very high compression ratios (r>=6) under realistic cache hit rates. To address these issues, the authors proposed Cache-Aware Prompt Compression (CAPC). CAPC pairs query-agnostic compression with explicit cache control and a tier-preserving ratio bound, preventing over-compression from pushing cached prefixes into a less efficient "hot tier." CAPC proved to be the most cost-effective strategy across all tested configurations on LongBench-v2, delivering mean savings of 49% over cache-only, 64% over query-aware compression, and a remarkable 90% over vanilla usage, all while maintaining output quality within 0.05 of the uncompressed baseline. It was also validated on production workloads, showing substantial cost reductions.