LLM Pruning Needs Group-Robust Metrics Beyond Simple Compression Scores

The paper highlights a critical flaw in current Large Language Model (LLM) pruning strategies: relying solely on compression statistics can lead to suboptimal outcomes, even when those statistics appear reliable. The authors show that a high-reliability compression score can still select a pruning candidate that performs significantly worse than controls. This discrepancy is attributed to "information interfaces" that limit what distinctions each statistic can truly support. For scenarios involving equal-weight groups, the research provides a conic law to quantify the pooling price for positive linear fixed-candidate damage. It also introduces methods to characterize what pooled and group-local moments leave unresolved. The study demonstrates that a group-resolved diagonal metric can recover broad damage order effectively, and a coarse depth allocation strategy can reduce worst-group perplexity inflation by 12.6-20.9% across various dense LLMs. Furthermore, model-specific complete-mask endpoint selection improves performance by 2.7-8.0% over references, and local measurements can construct better candidates. This work emphasizes the need for more nuanced metrics beyond simple compression for robust LLM pruning.