DAMP Quantization Boosts LLM Recurrent State Efficiency

This paper introduces DAMP, a novel mixed-precision quantization technique specifically designed for the recurrent states found in modern language models like those using Gated DeltaNet (GDN) or Kimi Delta Attention (KDA). These models utilize fixed-size recurrent states to manage memory growth with sequence length, but these states often consume substantial GPU memory and contribute to decoding latency when stored in FP32. The research identifies that uniform quantization methods perform poorly, causing significant accuracy degradation even at INT8. DAMP addresses this by observing that most quantization error energy is concentrated in a small subset of channels and that the decay strength of these channels remains stable. DAMP leverages these insights to identify "high-risk" channels during calibration, storing them at higher precision while quantizing the rest to INT8. This approach allows DAMP to achieve near FP32 accuracy at 9.9 bits per state value, reducing recurrent-state storage by 69.1% and accelerating updates by up to 2.01x, leading to a 10.9% reduction in full-model TPOT (Tokens Per Second).