BV-Blend Stabilizes Critic-Free Reinforcement Learning for LLMs

This paper introduces BV-Blend, a novel framework designed to stabilize critic-free reinforcement learning (RL) for large language models (LLMs), particularly in scenarios involving verifiable rewards. Critic-free methods, like Group Relative Policy Optimization (GRPO), aim to reduce computational overhead by avoiding a value function, but can suffer from instability when local reward statistics are uniform, leading to zero advantage estimates and stalled learning. BV-Blend tackles this by combining immediate, prompt-local on-policy statistics with historical reward moments, which are tracked for semantic clusters. It uses a confidence weight, derived from a standard error of the mean proxy, to intelligently blend these historical and local baselines and variance statistics. This blended approach yields a more robust, standardized advantage for PPO-style updates. Experiments on verifiable reasoning benchmarks demonstrate that BV-Blend significantly improves training stability and overall performance, proving resilient where other group-normalized methods might fail.