PolyQ Optimizes LLM Inference on Edge CPUs

Large Language Models (LLMs) are increasingly deployed on edge CPUs, but current low-bit quantization methods struggle to balance fine-grained precision with efficient execution on these platforms. PolyQ addresses this by introducing a novel co-design approach that integrates a compiler with a quantization framework specifically optimized for CPUs. PolyQ's core innovation lies in its ability to assign per-channel bit-widths (from 2 to 16 bits) based on activation characteristics, all while adhering to a user-defined average-bit budget. The accompanying compiler then intelligently permutes and clusters channels into bit-homogeneous blocks, generates optimized SIMD and LUT-compatible kernels, and merges compatible permutations to eliminate runtime layout regularization overhead. Evaluations across various LLMs (Falcon-H1-3B, Llama2-13B, Qwen3-32B) and CPU types (workstation, laptop, mobile) demonstrate PolyQ's effectiveness. It significantly improves perplexity (2.4-32.1%) over prior methods at a 3-bit target, reduces activation reorder traffic by up to 70.8%, and ensures prefill latency and decode throughput scale proportionally with the bit budget, maintaining energy overhead below 2%. This makes fractional-bit CPU deployment practical and efficient for edge devices.