Tiered KV Cache Boosts Large LLM Inference on SageMaker HyperPod

Deploying large language models (LLMs) for inference at scale presents a significant challenge related to the Key-Value (KV) cache. Traditional approaches often force a difficult choice: either provision oversized and expensive GPU instances to accommodate the cache or accept slower response times, particularly for the initial token generation. This dilemma impacts both performance and cost efficiency. A novel solution involves implementing a tiered KV cache architecture on Amazon SageMaker HyperPod, integrated with Curvine. This approach extends the KV cache beyond the GPU's local memory into a shared, distributed NVMe storage pool. By doing so, multiple LLM replicas can efficiently reuse cached data at speeds comparable to local disk access, enabling the use of more cost-effective GPU instances without compromising on performance.