Squeeze-Release Pruning Achieves Significant Model Compression.

Unstructured pruning typically results in sparse weight tensors, but often fails to reduce the actual deployed model size because tensor shapes remain unchanged. This research presents an exact structural rewrite, termed minimization, which converts a masked network into a smaller, dense network that preserves the original forward function, accounting for floating-point rounding. The core of the proposed method is the "Squeeze-Release" cycle. This cycle iteratively applies pruning and minimization. An intermediate "release" step re-enables the exact-zero positions within the compacted tensors as small, calibrated noise, effectively turning previously wasted capacity back into trainable parameters. Successive cycles leverage this re-enabled capacity to discover structural redundancies that a single pruning pass might miss. The researchers also introduce CompensatedLayerNorm, a function-preserving replacement for LayerNorm, which extends minimization to channel reduction across LayerNorm-equipped residual streams. Squeeze-Release demonstrates impressive compression, reducing deployable networks by up to 39 times on fully-connected models and 14.8 times on modern CNNs (ConvNeXt-Tiny) with comparable accuracy, and is proven extendable to transformer architectures.