New Method Stabilizes LLM Reinforcement Learning Post-Training.

On-policy rollout methods, like GRPO, are crucial for fine-tuning large language models after initial training. However, these methods often face challenges such as unstable training, reduced performance on diverse tasks, and overly long responses. This paper investigates the underlying geometric variations in how models are updated during these processes. The researchers identified that transient spikes in the "Principal-Subspace Overlap" — a measure of how individual updates align with the model's dominant learned features — often precede performance degradation. To counter this, they developed Geometrically Constrained Policy Optimization (GCPO). GCPO actively restricts model updates to specific complementary subspaces, preventing these destabilizing excursions. Evaluations on models like Qwen3-8B and GLM4-9B across tasks such as mathematical reasoning, code generation, and tool use showed GCPO consistently outperforming existing methods. It improved base models and leading baselines by up to 27.69 and 2.37 points respectively, while also preserving general capabilities and preventing response inflation.