New Quantization Method Boosts LLM Efficiency and Accuracy

Deploying large language models (LLMs) efficiently often relies on quantization techniques, which reduce precision to save computational resources. Block Quantization (BQ) is a promising approach that quantizes both weights and activations, offering better hardware efficiency than weight-only methods. However, traditional BQ faces a trade-off: increasing block size improves hardware efficiency but degrades accuracy. To overcome this, researchers developed Hierarchical Block Quantization (HBQ). This method leverages large blocks for maximum efficiency and introduces a novel, low-overhead significand (SIG) scaling for a second level of quantization. This hierarchical approach effectively compensates for the errors typically associated with large blocks, leading to superior accuracy. HBQ-A (accurate) achieves high accuracy with significantly reduced bit-widths (e.g., W4A5 for W4A16-level accuracy), while HBQ-E (efficient) further cuts hardware costs. Implemented on a 28nm ASIC accelerator, HBQ demonstrates 2.3x to 4.6x higher area/energy efficiency and 1.5x to 3.0x speedup compared to prior methods, making LLM inference more practical and cost-effective.