QUASAR Lowers QAT Loss Floor for Low-Bit LLMs

Researchers have developed QUASAR, a novel quantization-aware training (QAT) method designed to improve the quality of low-precision large language models (LLMs). As LLM inference increasingly relies on lower precision, traditional post-training quantization (PTQ) becomes brittle, making QAT essential. However, existing QAT methods suffer from a mismatch: they compute loss and surrogate gradients using a lossy reconstruction of full-precision weights, but apply updates to the latent full-precision weights themselves, leading to suboptimal training and a higher loss floor. QUASAR addresses this by continuously performing lightweight, loss-aware reconstruction within the training loop. At each step, it uses exponential moving averages of squared gradients for online saliency estimates, searches for optimal clipping ranges, and fits affine dequantizers via saliency-weighted least squares. This approach directly minimizes the loss-aware reconstruction error, which is shown to be a critical factor in the QAT convergence bound. QUASAR modifies only the training procedure, supports standard deployment formats, and achieves significant reductions in KL divergence and improvements in average accuracy across various LLMs and tasks at 2, 3, and 4 bits.