New Critic-Free RL Method Improves Data Efficiency and Stability

Traditional critic-free reinforcement learning (RL) approaches often rely on generating multiple rollouts for a single query to establish value baselines. This method, however, can be inefficient with data, introduce synchronization challenges, and lack flexibility when dealing with structured rollouts. A new study re-examines the fundamental purpose of these "groups," concluding that their primary role is to prevent incorrect penalties on negative samples rather than solely estimating baselines. Based on this insight, researchers propose "negative token filtering." This filtering technique allows for stable training using just a single rollout, significantly improving data efficiency. When applied to existing batch-level advantage methods, it demonstrates performance on par with or superior to group-based RL techniques in reasoning and agentic tasks.