New Method Addresses Diversity Collapse in LLM Reinforcement Learning

Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent technique for enhancing the reasoning capabilities of large language models (LLMs). However, a significant challenge known as "diversity collapse" often arises. This phenomenon is characterized by improvements in Pass@1 (single-shot success) while Pass@k for higher k values degrades, suggesting a narrowing of the model's overall reasoning boundary. This paper formalizes diversity collapse through the lens of overtraining. It posits that once a problem's contribution to the reference metric has effectively saturated, further updates, especially with limited rollouts per problem, no longer expand the model's ability to solve new problems. Instead, these updates merely concentrate probability mass on trajectories already favored by on-policy sampling, thus structurally biasing against high-k Pass@k. To mitigate this, the researchers propose Bayesian Boundary Gating (BBG). This method redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Empirical results show that restricting updates to problems with zero observed success can lift Pass@256 above the base model on difficult benchmarks. BBG, building on these insights, significantly improves average Pass@k across a wide range of k values on multiple reasoning benchmarks, demonstrating its effectiveness in fostering greater reasoning diversity.