Pruning Pretrained LLMs Outperforms Training Small Models from Scratch

The study investigates two primary approaches for developing efficient, smaller language models: pruning a larger, pre-trained model or training a new small model from scratch. Using Llama-3.1-8B, researchers applied various pruning methods across different granularities and compared the resulting models under controlled training conditions. The findings indicate that when the training token budget is limited, initializing a small model by pruning a larger parent model consistently yields better performance than training a model of similar size from random initialization. This suggests that the pre-trained model provides a robust knowledge base that is difficult to replicate with a fresh start and limited resources. However, if the training budget is not a constraint, training from scratch can become competitive, particularly for coarser pruning granularities. The research concludes that pruning is generally advantageous when a large pre-trained model is available and training resources are constrained, as it effectively transfers knowledge that additional training tokens alone cannot fully recover, especially at finer pruning granularities.