GaugeQuant Improves LLM Quantization by Leveraging Internal Symmetries

Large Language Models (LLMs) based on the Transformer architecture possess inherent continuous symmetries that keep their outputs invariant even as quantization parameters change. A novel technique called GaugeQuant exploits these symmetries during the training process. It introduces a LogSumExp term into the loss function, which effectively breaks these symmetries and guides the model to select a basis that minimizes outliers in activations. A key aspect of GaugeQuant is its use of a stop-gradient operator, ensuring that only the rotation matrices are updated. This design choice means the primary language modeling objective remains completely unaltered. The method is highly efficient, requiring no specific calibration data, no quantization simulation, and incurring negligible additional training overhead. When applied to the LLaMA-2 7B model with W4A4 quantization and a group size of 128, GaugeQuant dramatically reduced perplexity from 8.22 to 6.73, achieving performance comparable to post-training methods that typically demand frozen models and dedicated calibration datasets. For W4A16 quantization, perplexity dropped even further from 11.16 to 5.45.