Quantization Error Additivity Explains Mixed-Precision Model Performance

The study delves into mixed-precision quantization, a technique crucial for optimizing large language models, particularly at the 4-bit precision levels now being adopted. A key finding is that the loss incurred from quantizing a set of layers is largely attributable to the individual effects of each layer, accounting for 85-93% of the total variance. This challenges the common assumption that loss from quantizing multiple layers is a complex, non-additive interaction. To better understand and predict this behavior, the researchers introduce a "coverage model" and an "additive model." These models effectively reproduce the observed variance profile and accurately rank quantization configurations. The additive model, in particular, serves as an optimal first-order predictor, with its mean-squared error directly quantifying the variance unexplained by per-layer effects. These new models lead to improved allocation strategies for memory-constrained scenarios. When applied to models ranging from 30B to 355B parameters, they achieve the lowest KL divergence compared to other allocators. Crucially, at precisions below four bits, these allocations enable models to continue solving complex tasks, even when methods based on gradient sensitivities fail to produce functional outputs.