Compound Sparsity Improves LLM Compression and Performance

Large language models (LLMs) are notoriously resource-intensive, prompting extensive research into compression techniques. Traditionally, these methods focus on either statically reducing parameters or dynamically skipping computations for individual tokens. This paper investigates whether combining these two approaches, termed "compound sparsity," can overcome the limitations of single-dimensional compression. The proposed framework first applies low-rank approximation and channel pruning to create a compact model backbone. Subsequently, lightweight routers are introduced to enable dynamic layer skipping on a per-token basis. This dual-pronged strategy allows for independent control over both parameter and computation sparsity. Experimental results across various language benchmarks indicate that compound sparsity consistently outperforms single-mechanism compression at equivalent total sparsity levels. It effectively delays the point at which performance begins to degrade, particularly in language understanding tasks, and maintains stronger overall modeling performance. The findings also highlight the importance of a balanced allocation between parameter pruning and token skipping for optimal results.