New Quantization Method Offers Flexible Multi-Precision LLM Deployment.

Researchers have introduced Recurrent Residual Quantization (RRQ), a novel post-training quantization (PTQ) framework designed to optimize the deployment of large language models (LLMs). Current quantization methods typically require generating a separate model checkpoint for each desired bit-width, which can be inefficient. RRQ addresses this by allowing a single checkpoint to support multiple effective precisions. The core idea behind RRQ is to represent model weights as a low-bit quantized base, to which a sequence of lightweight, quantized residual corrections is progressively added. This allows for the construction of 4-bit, 6-bit, and 8-bit representations starting from a 2-bit model. The method is calibration-free and avoids complex joint multi-bit optimization. Experiments on six recent LLMs demonstrate competitive accuracy at higher bit-widths and significantly faster construction times compared to existing methods like MatGPTQ.