CubicQuant Boosts LLM Inference with Parametric Non-Uniform Quantization

This paper introduces CubicQuant, a novel quantization technique designed to optimize large language model (LLM) inference by enabling high-throughput execution with 1-8-bit weights. Unlike traditional uniform or fixed floating-point formats, CubicQuant employs a parametric non-uniform scalar format. This allows for adaptive reconstruction levels within each weight group while preserving a dense integer code stream, crucial for efficient GPU processing. The core of CubicQuant is a monotonic cubic curve, defined by two shape parameters and one scale, which maps uniformly spaced magnitude codes to non-uniform levels. This flexible family of formats spans 1-8-bit weight payloads and includes symmetric uniform integer quantization as a special case. The research demonstrates significant reductions in reconstruction RMSE compared to optimally clipped uniform integer quantization and even best enumerated floating-point formats across various distributions.