New RL Method Improves Correctness in Verifiable Reward Learning

A new research paper introduces Contrastive Policy Optimization (CPO), an advanced method for reinforcement learning, particularly in scenarios where rewards need to be verifiable. Traditional approaches often rely on entropy for guiding the learning process, but entropy struggles to differentiate between beneficial uncertainty and outright errors. CPO overcomes this by employing a token-level contrastive disagreement mechanism. This mechanism compares outputs from a reference-guided generation with a standard generation, effectively identifying and signaling token-level correctness. The paper demonstrates that this disagreement is a reliable indicator of accuracy. The CPO framework also addresses the "zero-advantage problem" and shows that on-policy distillation can be seen as a specific instance of CPO. Experimental results across various benchmarks, including both in-domain and out-of-domain tasks, indicate that CPO significantly outperforms existing entropy-based methods in verifiable reinforcement learning, while also maintaining strong generalization capabilities. The findings suggest that balancing exploration (through correct responses) and exploitation (through incorrect responses) is crucial for optimal performance.