KV Compression Outperforms Tensor Parallelism for LLM Serving Cost

When serving large language models, memory limitations, particularly for the KV cache, present a significant challenge. Two primary solutions are commonly employed: tensor parallelism, which shards weights and KV cache across multiple GPUs, and KV compression, which reduces the cache size in place. This research provides a direct cost-normalized comparison of these two strategies across various models, GPU types, and memory relief levels. The study concludes that KV compression consistently offers a more cost-effective solution, being 1.2x to 2.0x cheaper per million tokens than tensor parallelism. While tensor parallelism improves latency, compression significantly multiplies capacity per dollar. The choice between strategies largely depends on the model size relative to device memory; for models below approximately 36B parameters on an 80GB card, compression dominates. For larger models like Llama-2-70B, tensor parallelism becomes a necessity as the weights themselves exceed single-device memory.