SARA Boosts RLVR Efficiency with Adaptive Rollout Allocation.

Reinforcement Learning with Verifiable Rewards (RLVR) is often bottlenecked by the high computational cost of generating rollouts, especially since many sampled prompts yield "saturated groups" where all responses are either correct or incorrect, providing no useful policy-gradient signal. Existing solutions either oversample and discard, incurring extra costs, or predict difficulty, which can be unreliable. This research introduces SARA (Sequential Adaptive Rollout Allocation), a novel approach that addresses this inefficiency. SARA operates on the observation that a prompt group's effectiveness is typically determined within its first few rollouts. Instead of completing full groups, SARA frames per-step rollout collection as a budget-constrained sequential allocation problem. It maintains a Beta posterior for each prompt's success rate, uses a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule. This rule allows SARA to commit to effective groups, abandon saturated ones early after a brief probe, and reallocate the freed budget to new, more promising prompts without needing additional prediction rollouts. The authors provide theoretical proofs for SARA's abandonment reliability, expected rollout savings, and fixed-budget yield dominance. Empirically, on mathematical reasoning and planning tasks with 1.5B/3B models, SARA matches the accuracy of dynamic sampling (DPS) while using 22% fewer rollouts than a standard baseline. When SARA is combined with DPS, it achieves the best accuracy, slightly exceeding the baseline, with a remarkable 67% reduction in rollouts, demonstrating near-uniform cost efficiency.