Compression Trinity Boosts LLM Efficiency and Performance.

The high computational and environmental demands of Large Language Models (LLMs) hinder their widespread and scalable deployment. Traditional compression techniques, such as sparsity, quantization, and low-rank approximations, are typically applied in isolation, each reaching a limit in balancing accuracy and efficiency. This research proposes the "Compression Trinity," a novel framework that integrates these three pillars synergistically. The Trinity applies sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover lost accuracy. For pretraining, the framework introduces methods like MKOR for optimizer acceleration and SLoPe for faster training via double-pruned backward passes and lazy adapters. For post-training, OPTIMA stabilizes static masks, and PATCH dynamically learns hybrid sparsity ratios. The SLiM method, which embodies the full Compression Trinity, uses mathematically derived low-rank adapters to compensate for information loss from quantization and sparsity, achieving significant accuracy improvements and outperforming uncompressed models at equivalent parameter budgets.