Variable Bit-width Quantization Shrinks Language Models, Boosts Efficiency

Quantization is a common technique to reduce the size of language models, but it typically applies a uniform bit-width across all weights. This new research proposes Variable Bit-width Quantization (VBQ), an innovative training-time approach where each small group of 64 weights dynamically learns its own optimal precision level from a set of options (1, 2, 4, or 8 bits). This learning process is integrated into an alternating optimization scheme, providing a clear, task-aligned signal for precision allocation. VBQ reveals a highly heterogeneous precision distribution within models, with some groups collapsing to 1 bit while others retain higher precision. This non-uniform allocation allows for a "bigger-but-smaller" paradigm: a 131M model with an average of 1.82 bits can outperform a 55M FP16 model with 3.8 times less storage. Furthermore, this learned precision recipe directly translates to faster inference, with custom kernels showing significant speedups, especially for larger models. The study also uncovers a mechanism where deeper layers effectively self-heal quantization errors from earlier layers, demonstrating the power of this from-scratch, train-time optimization.