Language Model Training Methods Share Core Disagreement Metric

Three widely used methods for training language models to reason—Group Relative Policy Optimization (GRPO), GRPO Done Right (Dr. GRPO), and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)—appear distinct but are, in fact, manipulating a single underlying metric. This research demonstrates that all three methods are essentially adjusting the standard deviation of a prompt's sampled answers, which quantifies the level of disagreement among those answers. When a language model is trained, it typically generates multiple responses to a given problem, and an automated checker evaluates each response as right or wrong. The standard deviation of these right/wrong marks serves as a direct measure of disagreement: it is highest when answers are evenly split between correct and incorrect, and zero when all answers agree. The paper proves a "group-standard-deviation identity," showing that this disagreement metric precisely dictates the magnitude of the training update. This identity implies that groups of answers with high disagreement provide the most valuable learning signals, while groups where all answers are unanimous (zero standard deviation) contribute nothing to the learning process and are effectively silenced. This finding not only clarifies the common mechanism behind these seemingly disparate training tricks but also indicates which problems should receive the most weight and how many attempts are needed for effective learning. The intuition is confirmed through experiments on a large real-world dataset (Big-Math) and controlled training runs.