LLM Quantization Damage Multiplies Decision Margins, Not Adds Noise.

Large language models are often deployed using quantization, a process that compresses them but can degrade performance below four bits. Previous assumptions suggested quantization added a fixed amount of noise, implying that highly confident decisions would remain robust. However, new findings indicate that quantization actually multiplies the decision margin—the difference between a model's chosen option score and its best alternative's score—causing it to shrink significantly with decreasing bit-width. This margin shrinkage means that the protection afforded by a large initial margin diminishes, making the model's inherent biases more influential in determining failure. For instance, at three bits, the decision to use a tool might collapse towards inaction, while the choice of *which* tool remains unaffected. Statistical comparisons show that additive-noise models consistently fail to explain these observed damages in tool and safety decisions, whereas the multiplicative model accurately predicts flip rates. The study found this multiplicative relationship holds across various models, families, and quantization methods, though the specific constants are unique to each model. A small, paired margin set can estimate which decisions will break without extensive generative evaluation, suggesting that increasing bit-width is the most cost-effective way to mitigate damage.