New Pruning Method Boosts Transformer Efficiency During Finetuning.

This paper introduces REP-LIE, a novel technique designed to make pruning Transformer-based language models more resource-efficient, particularly during the finetuning phase. Large pre-trained models often incur high computational and memory costs, which hinder their deployment in environments with limited resources. Traditional pruning methods typically require full gradient-based importance estimation and often necessitate prior finetuning, leading to substantial resource consumption. REP-LIE addresses these challenges by leveraging the gradients of LoRA (Low-Rank Adaptation) low-rank matrices to estimate the importance of model weights. This eliminates the need for computationally expensive full gradient computations. To enhance the reliability of importance estimation, the method incorporates a stability score, which guides an iterative pruning process for less important parameters. After pruning, the model undergoes further lightweight finetuning, avoiding the need for full-parameter optimization. Extensive experiments on both medium-scale encoder models and large generative models like LLaMA-7B and Mistral-7B demonstrate that REP-LIE achieves performance comparable to existing pruning approaches while being significantly more efficient in terms of resources.