Pruning LLM Attention Layers Degrades Explainability and Calibration.

Pruning Large Language Models (LLMs) is a common strategy to reduce their memory footprint and inference costs by removing redundant parts of the network, often with minimal impact on accuracy. Given that attention layers are particularly resource-intensive, pruning them is a promising compression technique. Previous research has shown that significant portions of attention layers can be removed without substantial accuracy loss. However, the broader implications of such pruning on model interpretability, specifically explanation faithfulness and confidence calibration, have remained largely unexplored. This research addresses that gap by systematically studying how pruning attention layers affects these critical interpretability metrics across five different LLMs and eight datasets. The findings reveal a crucial insight: even when pruned models maintain high levels of accuracy, their explanation faithfulness and confidence calibration frequently degrade. This degradation can fluctuate significantly, indicating a disconnect between a model's predictive performance, its confidence in those predictions, and the reliability of its explanations. The study concludes that evaluating pruned models solely on accuracy and efficiency is insufficient; interpretability and calibration metrics must also be included to fully understand the impact of compression strategies.