LACE-SVD Improves LLM Compression with Loss-Aware Error Correction.

The increasing scale of large language models (LLMs) necessitates efficient compression techniques to reduce memory footprint and computational costs. Low-rank compression, particularly SVD-based methods, is a popular hardware-agnostic approach. However, current SVD methods often focus on local reconstruction, neglecting two critical issues: they don't explicitly account for how sensitive each layer is to compression-induced loss, and they fail to address the accumulation of local approximation errors as they propagate through the model's residual stream, leading to amplified global deviations. To overcome these challenges, researchers propose LACE-SVD (Loss-Aware SVD with Cumulative Error correction). This framework first estimates the increase in calibration negative-log-likelihood caused by different layer-wise compression ratios. It then solves a budget-constrained allocation problem to optimally assign rank budgets across layers, ensuring that compression is applied where it has the least detrimental impact on overall model performance. LACE-SVD further refines the compressed model using closed-form local updates and introduces a propagation-aware correction mechanism for residual-stream output modules. This correction actively reduces discrepancies in layer outputs, serving as a proxy for mitigating cumulative error propagation. Experimental results on LLaMA-7B demonstrate LACE-SVD's superior performance: at a high compression ratio of 0.6, it achieved a WikiText-2 PPL of 32.57, significantly better than Dobi-SVD's 46.18, highlighting its effectiveness in preserving model quality during aggressive compression.