RRPO Enhances Reinforcement Learning with Reference-Relative Optimization

Reinforcement learning often benefits from verifiable feedback, where sampled actions can be compared against a correctness signal. However, many complex tasks lack a single, clear correctness criterion, making traditional group-relative optimization challenging. This research presents Reference-Relative Policy Optimization (RRPO), a novel framework that extends these methods to non-verifiable settings. RRPO generalizes Group Relative Policy Optimization (GRPO) by replacing direct correctness-based advantage construction with a system of reference-relative contrastive comparisons. It achieves this by first generating "stratified conditional rollouts" to create distinct positive and negative anchor sets. A metric projection head is then trained using a set-contrastive objective to compare candidate rollouts against these anchors, yielding alignment scores. These scores directly define contrastive advantages, which are then centered within each rollout group for policy optimization while the projection head remains frozen. RRPO demonstrates competitive performance with verifier-based optimization and significant improvements over weakly supervised baselines across diverse tasks, including reasoning, open-ended generation, and post-supervised fine-tuning.