Power-of-Two Quantization Enables Deterministic LLM Inference

A new study investigates the challenges of achieving deterministic inference in large language models, particularly when using quantized GPU kernels. The research highlights that current tolerance-based conformance testing methods are often insufficient to detect subtle errors in quantized General Matrix Multiply (GEMM) operations, which can lead to non-deterministic outputs. The core finding is that by adjusting every weight scale in the quantization process to its nearest power of two, bitwise identical results can be achieved across different GPU kernel implementations, specifically CUTLASS and Triton. This intervention resulted in byte-identical generated token sequences for various LLM sizes (1.7B, 8B, 14B), a significant step towards reliable and reproducible inference. Crucially, this power-of-two requantization had only a minor impact on model perplexity, with observed changes within a small percentage range. The study also clarified that previously reported large perplexity increases from this intervention were due to a weight-scale mismatch, not the power-of-two constraint itself, demonstrating the practical viability of this approach for deterministic LLM deployment.