RLVR Can Invert Pass@k Performance in Language Models

New research identifies a critical issue in Reinforcement Learning with Verifiable Rewards (RLVR) where, despite improving one-sample accuracy, the model's overall ability to solve distinct problems under repeated sampling (known as pass@k) can actually worsen. This phenomenon, termed "pass@k inversion," is attributed to RLVR's tendency to shrink the "reasoning boundary" by inadvertently removing rare but correct solution trajectories that are present in the base model. These trajectories are often too sparse to be reliably sampled and reinforced during RLVR training. The study frames this as an "absence-of-evidence" failure, particularly affecting "boundary prompts" where the base model holds recoverable correct solutions. To mitigate this, a proof-of-concept method called Per-Problem Base Anchoring (PBA) was developed. PBA aims to preserve these rare verifier-positive trajectories by sharpening prompts with sufficient frozen-base correct evidence and anchoring risky prompts to the base distribution. Experiments on mathematical reasoning benchmarks showed PBA improving both single-sample accuracy and high-budget coverage compared to standard GRPO, confirming its ability to preserve valuable, rare solutions.