SharQ Boosts LLM Inference with FP4 Quantization.

Modern AI accelerators increasingly support low-bit floating-point formats and semi-structured sparsity, but effectively combining these for Large Language Model (LLM) activation compression remains a significant challenge. Activations often contain input-dependent outliers that complicate FP4 quantization, and direct application of sparsity masks can lead to information loss. This research introduces SharQ, a novel training-free inference method designed to bridge this gap. SharQ employs an online sparse-dense decomposition for each activation tensor. It generates an input-adaptive N:M mask to extract an outlier-dominated sparse backbone, which is then quantized to FP4. Crucially, a dense residual is defined relative to this quantized sparse backbone, compensating for both mask-induced activation loss and sparse-path quantization error. Both paths share a single FP4 weight payload with path-specific scale views, and a fused kernel integrates mask generation, residual construction, and layer normalization into one efficient operation. This method requires no calibration data, retraining, or model-specific tuning. Evaluated on several Llama and Qwen models, SharQ recovers 43-63% of the accuracy gap between NVFP4 and FP16 across various tasks. On an RTX 5090, it delivers 2.2-2.4x latency reduction over FP16 and 1.2-1.4x throughput improvement over FP8 in language model serving, demonstrating significant efficiency gains.