GAPO Improves RL with Adaptive Clipping for Harder Problems

A new research paper introduces Group Adaptive Clipping Policy Optimization (GAPO), an enhancement for Group Relative Policy Optimization (GRPO) methods used in reinforcement learning with verifiable rewards (RLVR). The core issue addressed is that traditional fixed clipping boundaries in GRPO can disproportionately suppress valuable learning signals from rare, correct rollouts on challenging problems, while treating abundant correct rollouts on easier problems similarly. GAPO tackles this by adapting the importance-sampling (IS) ratio clipping boundary dynamically, based on the rollout advantage. This approach is motivated by a reverse-KL trust-region perspective, which suggests that rollouts carrying stronger gradient signals, typically those from low-success groups, should be granted greater update headroom. Evaluated on math reasoning and coding benchmarks using Qwen and Llama models, GAPO consistently demonstrated improved Pass@1 and Pass@k metrics. Its benefits were most pronounced in scenarios where the base model's pass rates were initially low, indicating its effectiveness in accelerating learning on harder problems without requiring reward shaping or altering the standard PPO/GSPO surrogate.