OM-GRPO Improves Label-Free LLM Reasoning with RLVR.

This paper introduces Outcome-Masked Group Relative Policy Optimization (OM-GRPO), a novel framework for label-free Reinforcement Learning with Verifiable Rewards (RLVR). While RLVR is effective for improving LLM reasoning, it typically relies on ground-truth answers, which limits its scalability. Existing voting-based label-free methods, which use answer-level consensus for rewards, often suffer from "collapse" where the model reinforces answer tokens directly rather than improving its underlying reasoning process. OM-GRPO addresses this critical issue by decoupling reward estimation from policy optimization. It achieves this by masking gradients on the answer span during policy updates, while still utilizing a soft consensus signal for answer-level rewards. This strategic masking shifts the optimization pressure away from merely reproducing answers and towards enhancing the reasoning steps that lead to those answers. The framework also incorporates Contrast-Augmented Reward, a low-cost method to refine reward estimation through pairwise comparisons of existing trajectories, avoiding additional rollouts. Across various reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms other label-free RLVR methods and achieves performance comparable to supervised ground-truth reward training, demonstrating stable optimization, especially in test-time training settings.