New Pruning Methods Improve LLM Sparsity by Preserving Output Differences.

Pruning is a critical technique for reducing the inference costs of large language models (LLMs). However, current pruning criteria primarily focus on preserving large activations or reconstructing layer outputs, potentially overlooking the nuanced computations performed by neurons sensitive to sparsity, especially in MLP up and gate projections. These neurons are crucial for separating similar inputs into dissimilar outputs. This research introduces "The Sparsity Whisperer," a new family of pruning methods designed to preserve output differences more broadly. Wisp, a first-order method, scores weights using input-difference norms, while Wisp+ refines this neuron-wise. Whisper, a second-order method, uses a lightly regularized difference Hessian as its reconstruction objective. Across various Llama 2 and 3.1 models (7B to 405B parameters), the second-order Whisper consistently outperforms strong reconstruction-based baselines. The update-free Wisp variants also improve upon activation-aware baselines, particularly in constrained settings. These improvements extend to structured sparsity, downstream evaluations, and other model families, and can further augment existing techniques like RIA and ALPS, pushing the accuracy-runtime frontier outward with minimal additional cost.