New Methods Boost LLM Inference Efficiency with Activation Sparsification.

Efficient inference is a critical challenge for deploying Large Language Models (LLMs) at scale. A key area for optimization involves reducing computational load while maintaining model quality. This research explores two complementary strategies: multilayer perceptron (MLP) activation sparsification and token-level conditional routing. First, the paper proposes Sensitivity-Aware Thresholding for Sparsity (SATS). Unlike traditional methods that rely on activation percentiles, SATS calibrates layer-wise gate thresholds using a local MLP output sensitivity proxy. This allows for a more intelligent selection of which activations to sparsify, aiming to minimize impact on model performance. Second, a lightweight token routing framework is introduced. This framework enables dynamic selection between a base computational path and a modified, potentially more efficient, path on a per-token basis. This contrasts with static activation modifications applied uniformly to all tokens. Evaluations on several open-weight LLMs demonstrate that SATS outperforms percentile-based sparsification at equivalent sparsity levels. Furthermore, token routing yields a more favorable quality-throughput trade-off compared to static activation modification baselines. These results suggest that smarter threshold calibration and dynamic routing are effective strategies for enhancing LLM inference efficiency.