Quantization Masks LLM Agent Failures, Benchmark Error Budgets Hide Damage

Post-training quantization, particularly to 4-bit weights, is widely believed to be nearly lossless for Large Language Models (LLMs). However, new research challenges this assumption, especially when applied to multi-turn, tool-calling LLM agents. While standard evaluation metrics might show a flat score, suggesting no performance degradation, a deeper analysis reveals a different story. The study, conducted on various open-weight models and domains, found that quantization doesn't introduce many new types of failures. Instead, it significantly amplifies the failure modes that the full-precision model already exhibits, such as tool-name hallucination or entity errors. This amplification can be substantial, increasing failure volume by up to 2.5 times. The reason standard scores remain flat is often due to generous "error budgets" in benchmarks, which can absorb these additional failures without reflecting a score drop. By shrinking the error budget or analyzing per-channel error rates, the true damage becomes apparent. The research suggests that benchmarks should report per-channel error rates and success under shrinking budgets alongside task rewards to provide a more accurate picture of quantized model performance.