New RL Method Improves Credit Assignment for Rare Solutions

In reinforcement learning with verifiable rewards (RLVR), particularly with methods like GRPO, a common issue arises where correct solutions are treated uniformly. This leads to a "multiplicity-induced structure-level credit concentration," meaning frequently sampled correct solution forms accumulate more positive credit, while rarer but equally valid solutions receive insufficient recognition. This skews the learning process towards common patterns and away from potentially novel or more efficient rare ones. To counteract this, the researchers propose a partition-conditioned rule that redistributes positive advantages according to the rarity of solution clusters. Cue-GRPO implements this rule efficiently by using deterministic Strategy Cues to create local partitions of verified-correct traces during rollouts, avoiding the need for auxiliary model inference. Evaluations on Qwen2.5-Math-7B and Llama-3.1-8B-Instruct models demonstrate that Cue-GRPO significantly improves AIME repeated-sampling performance, with the most notable gains observed at higher sampling budgets. The method adds only a minimal 6% wall-clock training overhead, making it a practical design axis for RLVR, particularly in domains like competition mathematics where diverse solution strategies are valuable.