MiniCache Boosts LLM Inference Efficiency with Program Caching

Large Language Models are increasingly integral to applications like program-aided reasoning, agentic decision-making, and structured task execution. However, the high inference costs associated with these applications pose a significant challenge to their widespread and economical deployment. To address this, researchers have developed MiniCache, a novel reusable program caching framework. MiniCache works by converting Program-of-Thought (PoT) programs into parameterized cache objects, enabling the reuse of computations across requests that are structurally similar. This approach significantly optimizes LLM inference by reducing the need for repeated, costly invocations of the large target LLM. The framework leverages a smaller, more efficient model for two key functions: semantic variable extraction on cache-hit requests and speculative drafting during the target-LLM's generation process. This dual-role for the small model allows MiniCache to preserve task quality while achieving substantial cost reductions. Experiments on various shopping-style request datasets, including WebShop, Formula, and CodeTAT-QA, demonstrate that MiniCache improves the trade-off between inference latency, cache reuse, and accuracy, achieving up to 3.1x lower latency and 2.8x higher throughput under parallel serving. These results highlight the effectiveness of small models as lightweight interface models that facilitate reliable and efficient reusable program caching, rather than as direct replacements for larger models.