RLVR Narrows AI Solution Space Early in Reasoning Trajectories

Reinforcement Learning with Verifiable Rewards (RLVR) has been shown to improve the single-sample accuracy of AI models, particularly in reasoning tasks. However, this improvement comes at a cost: RLVR can significantly contract the policy's solution space, diminishing the benefits of test-time scaling. This research investigates precisely where this loss of breadth occurs within a reasoning trajectory. Using the Countdown task, which allows for exhaustive enumeration of solution families, the study found that solution coverage decreased by up to 67% under RLVR-trained policies. Crucially, this contraction was overwhelmingly concentrated at the initial steps of the reasoning process, specifically prior to the first arithmetic operation. Per-token likelihood shifts were 11-16 times larger at this "entrance" phase compared to later stages of reasoning. The findings indicate that alternative solutions remain executable if initiated, but the policy fails to access or initiate them early on. Interventions targeting these early steps, such as late-layer parameter interpolation with early checkpoints, successfully increased solution coverage without sacrificing accuracy. This "early-step entropy collapse" was observed across multiple math benchmarks and models, suggesting it's a common byproduct of reasoning optimization, though SFT baselines and staged training pipelines can mitigate it.