KronQ Improves LLM Quantization with Kronecker-Factored Hessian

A new post-training quantization (PTQ) framework called KronQ has been developed to enhance the compression of large language models (LLMs) without requiring retraining. Unlike previous second-order PTQ methods, such as GPTQ, which rely solely on input activation statistics, KronQ integrates gradient covariance into the quantization process by leveraging the Kronecker-factored Hessian approximation. This approach recognizes that the quantization loss is jointly dependent on both activation and gradient covariances. KronQ applies this insight at two levels. First, it introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using gradient covariance, thereby reducing weight magnitude variance across both input and output dimensions. Second, it derives a new sensitivity metric for inter-layer mixed-precision allocation, which is driven by the Hessian traces of both gradients and activations. The effectiveness of KronQ was demonstrated with 2-bit weight-only quantization on LLaMA-3-70B. While methods like GPTQ and GPTAQ failed or produced highly degraded quantizations (over 2000 perplexity), KronQ achieved a perplexity of 7.93, showcasing a substantial improvement in maintaining model quality at extreme compression levels.