INT8 Quantization Kernels Show Unexpected Divergence in LLMs

This research uncovers a significant issue in the deployment of INT8-quantized Large Language Models (LLMs), specifically concerning the interchangeability of GPU kernels. Despite the common assumption that different kernels implementing the same scaled INT8 General Matrix Multiply (GEMM) interface should yield identical results, experiments show otherwise. When swapping only the INT8 linear kernel within vLLM (between CUTLASS and Triton), no end-to-end sequence agreement was found, even with fixed checkpoints, prompts, and hardware. The study pinpoints the source of this divergence. While the core INT32 dot product remains exact and order-independent, the differences emerge during the subsequent scale application and output rounding steps. Under power-of-two scales, bit-identical outputs were observed, but with real checkpoint scales, minor differences (at most one bfloat16 spacing) appeared. This indicates that the exact integer arithmetic is not the problem, but rather how floating-point scaling and rounding are handled post-accumulation. The researchers propose a conformance procedure to check kernel interchangeability and highlight that these subtle differences, though small at the layer level, accumulate to cause complete divergence in generated sequences. This has critical implications for reproducibility and reliability in quantized LLM inference.