New Model Predicts GRPO Training Dynamics for LLMs

A new theoretical model has been developed to mechanistically describe the training dynamics of Group Relative Policy Optimization (GRPO), a crucial technique for enhancing the reasoning abilities of large language models (LLMs). Previously, GRPO dynamics were understood primarily through empirical observations and curve fitting, lacking a fundamental explanation for hyperparameter choices. This new first-principles, reduced-order model provides a closed-form solution that explains these dynamics. The model subsumes existing empirical laws, such as single-exponential saturation, by recasting fitted parameters into mechanistic terms like fixed points and stiffness. It also introduces an inertial term, accounting for the "slow-start" phase that previous empirical models couldn't represent. Crucially, the model yields predictions tied to independently measurable quantities, including group-size invariance, a sharp stability threshold in the refresh interval, and an overdamped-to-oscillatory transition. Furthermore, this framework offers valuable diagnostics that can distinguish between various failure modes, such as reward hacking or dynamical instability, which might otherwise be conflated by simple reward curves. Validated across multiple models and group sizes, the closed-form trajectory accurately fits training reward and demonstrates predicted group-size invariance, offering a significant step forward in understanding and optimizing GRPO.