Quantization Alters LLM Behavior Despite Preserved Accuracy, Study Finds

Deploying large language models (LLMs) in environments with limited resources often involves post-training quantization, a technique to reduce model size and computational demands. While standard evaluation metrics like accuracy and perplexity are commonly used to assess the impact of quantization, new research suggests these metrics may not fully capture the changes induced. The study introduces a novel metric called "correctness agreement," which measures the overlap in correct predictions between an original model and its quantized versions, independent of overall accuracy. Through experiments across various LLMs and quantization schemes (from 8-bit down to 2-bit), the researchers observed that significant behavioral differences emerge even when conventional performance metrics indicate no degradation. To understand this phenomenon, they analyzed quantization's structural impact on attention weights, quantifying distortions at each layer. Their findings highlight non-linear breakpoints at lower bit-widths and show that query and key projections are more sensitive to quantization than value and output projections. This work uncovers an "illusion of equivalence," emphasizing the need for more comprehensive behavioral evaluations beyond just accuracy.