Jacobian-Guided Noise Injection Improves LLM Quantization Robustness.

Quantization is a critical technique for deploying Large Language Models (LLMs) efficiently, but it often faces challenges due to the sensitivity of the self-attention mechanism to discretization errors. This research identifies the softmax operator within the self-attention mechanism as a key bottleneck for quantization stability, primarily due to its sensitivity to outliers and its state-dependent Jacobian. The theoretical foundation established in the paper shows that suppressing the norm of this Jacobian helps to bound the performance degradation induced by quantization. Based on this insight, the researchers introduce Jacobian-Guided Noise Injection. This training strategy involves injecting zero-mean Gaussian noise into the pre-attention logits. Crucially, the variance of this injected noise is directly derived from the Frobenius norm of the softmax Jacobian. Unlike previous heuristic approaches or direct Jacobian penalties, this method provides an optimal noise variance based on the local attention sensitivity. Empirical evaluations on state-of-the-art LLM architectures demonstrate significant improvements in robustness over popular Post-Training Quantization (PTQ) methods. The proposed method yields up to +37% relative gains in Top-1 accuracy on ImageNet-1K for SigLIP and improves relative perplexity by up to 40% on WikiText for language models in low-bit quantization settings, confirming its efficacy.