HiFloat4 Enables Efficient 4-bit RL Post-Training for Large Language Models

Quantizing large language models (LLMs) to 4-bit precision for reinforcement learning (RL) post-training offers significant computational and memory benefits, but often comes with a substantial accuracy penalty. Previous attempts to use 4-bit precision for both rollout and training phases have faced challenges, primarily due to activation quantization errors during the rollout phase, where outliers cause many values to underflow to zero. This research identifies that the mismatch between rollout and training precision, rather than just training-side quantization, is a principal failure mode. To overcome this, the authors propose Rollout Residual Quantization (Rollout-ResQ), a lightweight correction applied only to the 4-bit rollout matrix multiplication. When combined with the HiFloat4 (HiF4) format, which uses a hierarchical scaling to preserve resolution, this method dramatically closes the accuracy gap to BF16 precision (from 4.9% to 1.1% on Qwen2.5-3B). This breakthrough establishes HiF4 as a key enabler for practical, end-to-end 4-bit RL post-training of LLMs.