COEC Improves LLM Pruning Accuracy with Orthogonal Compensation

Structured pruning is a technique used to reduce the size and inference cost of Large Language Models (LLMs) by removing weight columns. However, this process often leads to a degradation in model accuracy due to output error. Existing compensation methods, which typically add a bias or apply a single orthogonal rotation, are limited because they don't modify the input singular frame of the retained weights. A new training-free compensation framework called COEC (Calibrated Orthogonal-Equivalence Compensation) addresses this limitation. COEC applies alternating left and right orthogonal rotations to the retained weight, with the right rotation optimized on a reduced Stiefel manifold. It also rescales singular values using generalized cross-validation to determine the regularization strength for each layer. COEC further enhances compensation by tempering the calibration Gram matrix to reduce the influence of high-energy activation directions and introduces an alignment penalty to preserve the geometric relationship between adjacent attention projections. All these components rely on second-order statistics from a small calibration set, requiring no backpropagation or retraining. Experiments on Llama-3, Llama-3.1, and Qwen2.5 models showed that COEC consistently improves perplexity and zero-shot accuracy over other compensation methods, especially at higher sparsity levels, demonstrating its effectiveness in recovering performance after pruning.