QTEA Achieves Efficient Ternary Quantization for LLMs with High Accuracy

Serving large language models (LLMs) at scale is computationally intensive, and post-training quantization (PTQ) is a key strategy to reduce this burden. However, existing PTQ methods often struggle to maintain accuracy when quantizing below 2 bits, and many rely on unstructured sparsity, which sacrifices hardware efficiency. A new framework, QTEA, aims to overcome these limitations. QTEA quantizes LLM weights into ternary values (e.g., -1, 0, 1) and uses "salient weights" as residual error compensators. To preserve hardware efficiency, these residuals are applied to selected columns with a semi-structured 1:4 sparsity. The framework also introduces a column-wise rescale refinement to GPTQ-style quantization, iteratively updating per-column scales and ternary assignments to minimize reconstruction error. Furthermore, QTEA addresses order-dependent error propagation in GPTQ by incorporating an error decay mechanism. Empirical results on Qwen3-14B show QTEA compresses weights to an effective 1.7 bits per weight, achieving a 16.7% accuracy gain over the strongest ternary PTQ baseline and significantly lower perplexity. Similar improvements were observed on Llama3-8B. A custom lookup-table based kernel developed for QTEA delivers 7.2x faster per-token generation compared to an FP16 baseline, demonstrating substantial efficiency gains.