New Method Accelerates LLM Inference on GPUs with Sparsity.

The rising deployment costs of large language models (LLMs) are a significant challenge, particularly concerning inference. Pruning techniques, which introduce sparsity into weight matrices, can accelerate inference, but maintaining model quality often restricts pruning to moderate unstructured sparsity levels, typically around 50%. At these moderate sparsity levels, existing GPU kernels for sparse matrix multiplication (SpMM) have struggled to outperform their dense counterparts. This paper introduces a new GPU inference method specifically designed for LLMs with moderate sparsity, addressing this performance gap. The method utilizes a three-layer matrix storage format and a custom SpMM kernel that combines sparse tensor cores and CUDA cores. This design enables an efficient execution pipeline and overlaps computation with memory access. The research demonstrates that this approach is the first to achieve kernel-level speedups of up to 1.64x over state-of-the-art sparse methods and end-to-end speedups of up to 1.41x over leading dense LLM inference frameworks on modern GPUs.