FAMPWQ Quantizes LLMs for Efficient Inference on Commodity GPUs

Large Language Models (LLMs) have achieved remarkable success across various domains, but their substantial resource demands often impede deployment on devices with limited computational power. Model quantization is a key strategy to address this, yet traditional methods frequently lead to performance degradation due to uniform bit-width application or simplistic sensitivity assessments. A novel approach, FAMPWQ (Fisher Information-based Adaptive Mixed Precision Weight Quantization), has been introduced to enable effective LLM inference on commodity GPUs. This method performs layer-adaptive weight quantization, leveraging a system model with a Fisher information metric to precisely gauge each layer's sensitivity to quantization. FAMPWQ incorporates a reinforcement learning-based bit-width allocator that dynamically generates an adaptive bit-width strategy based on these sensitivity metrics. Extensive experiments across seven models and five benchmarks demonstrate FAMPWQ's superior performance, achieving up to 3.39 smaller perplexity, 6.87% higher accuracy, and a 76% win rate in LLM-as-a-judge comparisons compared to seven baseline methods.