New Method Accelerates LLM Inference on GPUs with Moderate Sparsity

The increasing deployment of large language models (LLMs) has made their inference cost a major challenge. While pruning techniques can introduce sparsity into weight matrices to accelerate inference, maintaining model quality often limits this sparsity to moderate levels, typically around 50%. At these moderate sparsity levels, existing GPU kernels for sparse matrix multiplication (SpMM) have struggled to outperform their dense counterparts, hindering practical acceleration. This research introduces an innovative GPU inference method specifically designed for LLMs with moderate sparsity. The core of the method is a novel three-layer matrix storage format. This format leverages sparse tensor cores for acceleration, employs a slot-filling layer for efficient compression and on-chip decoding, and includes a lightweight residual layer to ensure accurate computation. By combining this storage format with a custom SpMM kernel that jointly utilizes sparse tensor cores and CUDA cores, the method achieves an efficient execution pipeline with overlapped computation and memory access. Evaluations demonstrate that this approach is the first to surpass dense matrix multiplication performance on modern GPUs with high-bandwidth memory, showing up to 1.64x kernel-level speedup and 1.41x end-to-end speedup compared to state-of-the-art sparse inference methods.