QUADS Stabilizes NVFP4 Reinforcement Learning for MoE Models

Researchers have identified a significant challenge in accelerating Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models using low-precision formats like NVFP4. While NVFP4 offers higher throughput than FP8 due to its fine-grained scaling and native W4A4 FP4 GEMMs, direct application leads to rapid training collapse and growing log-probability gaps between rollout and trainer. Through detailed error analysis, the team pinpointed activation error, rather than weight error, as the primary culprit for this instability. Weights can be synchronized, but activations are recomputed online, and their errors are amplified by the coarse E2M1 grid of FP4. To counter this, the paper proposes QUADS (QUantization-error Alignment across Dual Sides). QUADS introduces two main components: Asymmetric Quantization-Aware Training (QAT) on the trainer side, which fake-quantizes weights while keeping activations unquantized for better alignment; and Residual Activation Compensation (RAC) on the rollout side, which corrects high-error activation channels while preserving the efficiency of native W4A4 GEMMs. Empirical evaluations on several MoE RL benchmarks demonstrate that QUADS achieves BF16-level accuracy, improving average pass@1 by 21.49 points over naive NVFP4 RL, and delivers approximately 16% higher rollout throughput than FP8.