RRQ Enables Flexible Multi-Precision Quantization for LLMs

Deploying large language models (LLMs) often requires balancing accuracy, memory footprint, and throughput, which can vary significantly based on deployment constraints. Traditional quantization methods typically necessitate creating a separate model checkpoint for each desired bit-width, leading to increased storage and management overhead. This research introduces Recurrent Residual Quantization (RRQ) to address this challenge. RRQ is a post-training quantization (PTQ) framework that provides a flexible multi-precision representation from a single checkpoint. It works by representing model weights as a low-bit quantized base, such as 2-bit, and then progressively adding a sequence of lightweight 2-bit quantized residual corrections. These residuals are generated via a simple round-to-nearest (RTN) approach, allowing the construction of 4-bit, 6-bit, and 8-bit representations from the same base. A significant advantage of RRQ is that it is calibration-free and avoids complex joint multi-bit optimization. For instance, in a Qwen3-8B setup, the entire 2-/4-/6-/8-bit package can be constructed in just 1,293 seconds, which is over three times faster than MatGPTQ. Experiments across six recent LLMs demonstrate competitive accuracy at 6 and 8 bits, with performance at 4 bits showing model-dependent behavior. The code for RRQ will be made publicly available.