TriSP Pruning Method Boosts LLM Efficiency with Minimal Performance Loss

Deploying large language models (LLMs) is often hindered by their substantial memory and computational requirements. Structured pruning offers a solution by removing entire components like attention heads or MLP neurons, resulting in smaller, more efficient models. However, existing pruning techniques either demand excessive memory for gradient-based importance estimation or rely on statistical proxies that don't directly reflect the impact on model loss. This research introduces TriSP, a novel pruning metric that integrates three signals: weight magnitude scaled by activation norm, and first-order gradient sensitivity. By combining these signals geometrically, TriSP generates a channel-level score that captures both structural importance and sensitivity to loss. When paired with adaptive per-layer budget allocation and low-rank adaptation (LoRA) for recovery, TriSP achieves superior perplexity and zero-shot accuracy. For instance, it improved LLaMA-7B's inference throughput by 82% at 50% pruning while maintaining strong performance.