MixQuant Boosts LLM Accuracy with Adaptive Quantization

A novel adaptive mixed-precision quantization framework, named MixQuant, has been introduced to enhance the accuracy of large language models (LLMs). Unlike previous methods that optimize for a single, fixed memory budget, MixQuant offers a single offline calibration process that can serve various deployment memory constraints. This is achieved by addressing a critical limitation in existing adaptive quantization techniques: the failure to account for how a layer's sensitivity to quantization depends on the bitwidths of its upstream layers. MixQuant tackles this by marginalizing each layer's distortion over random upstream quantized configurations, generating budget-agnostic sensitivity scores. It then calibrates the quantizer's parameters using allocation plans it produces and penalizes allocations that assign the lowest bitwidths to critical layers. This allows for a single greedy pass at deployment to serve any budget. Evaluations across models like Llama-3.2-3B, Llama-2-7B, and Mistral-7B, using AWQ and GPTQ, show MixQuant consistently outperforms adaptive and mixed-precision baselines, achieving significant accuracy improvements and perplexity reductions, even matching ILP solvers at minimal deployment cost.