Cache-Aware Prompt Compression Reduces LLM API Costs

Large Language Model (LLM) deployments in production environments often rely on two primary strategies to manage costs: prompt caching, which offers discounted rates for reusing token prefixes, and prompt compression, which reduces the number of tokens sent. However, existing query-aware compression methods, which generate unique compressed prefixes for each query, inadvertently invalidate the prefix-strict caches, negating the benefits of caching. This research identifies a "question-first paradox" in VLM prompting. The study reveals that LLM API caches, specifically Anthropic's Sonnet 4.6, operate with a two-tier architecture, where a sharp threshold around 3,500 tokens dictates cache hit rates. Query-aware compression, while reducing tokens, often pushes prefixes into the "cold" tier or bypasses the cache entirely, leading to higher overall costs despite fewer tokens. This insight led to the development of Cache-Aware Prompt Compression (CAPC). CAPC combines query-agnostic compression with explicit cache control and a tier-preserving ratio bound. This ensures that compressed prefixes are more likely to hit the "hot" cache tier, maximizing cost savings. In extensive testing across various production workloads, CAPC consistently delivered substantial cost reductions—up to 90% compared to vanilla prompting and 64% over query-aware compression—while maintaining output quality within acceptable limits. This demonstrates a superior strategy for optimizing LLM API usage.