Verifier-Induced Support Reshaping Impacts On-Policy RL.

On-policy reinforcement learning with verifiable rewards (RLVR) aims to improve specific objectives, but this research reveals an unintended consequence: "verifier-induced support reshaping." This phenomenon can make successful behaviors for *later* or *different* objectives too rare to sample and reinforce, effectively narrowing the model's effective rewardable support. The study demonstrates this across mathematical reasoning and constrained instruction following tasks, showing that while RLVR might boost average success for one task, it can significantly reduce the overall number of prompts yielding *any* successful response under repeated sampling. The changes primarily occur in the first few response tokens, indicating that RLVR mainly reranks existing openings rather than generating new ones. This initial selection causally impacts the searchability for subsequent tasks. The paper concludes that simple reference-policy constraints or on-policy distillation only partially preserve cross-task support, meaning endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization.