Quantization Challenges in Recursive AI Models Addressed.

Recursive reasoning models, which solve complex problems by repeatedly applying compact, weight-tied blocks, face unique challenges when quantized. Quantization, the process of reducing the precision of model weights and activations, introduces errors at each step of the recursion, leading to an accumulation of bias. While 8-bit quantization maintains accuracy, a shift to per-tensor 4-bit formats can cause a severe degradation in performance, such as a complete collapse in Sudoku solving accuracy. The research identifies that this performance collapse is not primarily due to the reduced bit-width or number format itself, but rather the granularity of activation scaling. By implementing per-block scaling instead of per-tensor, the systematic bias accumulation is mitigated, and accuracy is fully restored. The study successfully applies MXInt4, a blockwise integer activation format, to these models, showing it performs comparably to blockwise float formats. It also highlights that deeper recursive architectures are more sensitive to quantization, but blockwise scaling effectively overcomes this vulnerability, maintaining robustness across different models and even transferring to open-ended benchmarks like ARC-AGI.