New Method Optimizes LLM Compression with Neuron Importance

Large language models (LLMs) are powerful but demand substantial memory, limiting their deployment in resource-constrained environments. Existing compression methods often rely on Singular Value Decomposition (SVD) for matrix compression, focusing either on parameter importance or functional equivalence per layer. This paper introduces a unified approach that integrates both perspectives. The proposed method combines the concept of neuron importance with data-aware low-rank approximation within a single objective function. This allows for a more nuanced understanding of which parts of the model are most critical to preserve during compression. Furthermore, the research addresses the challenge of distributing compression rates across different layers and parameters. Unlike previous works that used uniform distribution or computationally expensive heuristics, this paper presents an enhanced, computationally efficient algorithm for dynamic compression rate allocation. Experimental results demonstrate that this combined approach performs comparably or significantly better than prior state-of-the-art methods, particularly when high compression ratios are required.