Prox Boosts LLM Inference Speed with Training-Free FFN Sparsity

This paper introduces Prox, an innovative training-free framework designed to enhance the efficiency of large language model (LLM) inference by introducing sparsity in Feed-Forward Networks (FFNs). FFNs are a major bottleneck in LLM computation and memory usage, making them a prime target for optimization. Existing training-free sparsity methods often compromise model quality at high sparsity levels due to ineffective channel selection. Prox addresses this by recognizing that the SwiGLU intermediate state provides a strong signal for channel selection, even if its exact values are computationally expensive to obtain. The framework operates in two stages: first, it uses input sparsity and quantized proxy weights to construct a shared channel mask. Second, it computes only the selected channels precisely, enabling sparse execution across all three projections within the FFN. Evaluated across ten LLMs from six different families, Prox consistently outperformed other training-free baselines across all sparsity levels. It achieved up to a 1.99x end-to-end decoding speedup at 70% FFN sparsity, demonstrating compatibility with other optimization techniques like quantization and sparse attention.