Robust Reinforcement Learning for Small Language Models Achieved

The application of reinforcement learning (RL) to small language models (SLMs), typically in the 70-500 million parameter range, has often been considered unstable. This research systematically investigates the underlying reasons for these failures, identifying three key issues. These include silent freezing of LoRA parameters in standard PEFT/TRL pipelines, numerical overflow when using bfloat16 precision for importance ratios, and catastrophic policy collapse caused by errors in the reward model. To counter these problems, the researchers developed several solutions: a merge-and-reinitialize adapter technique for LoRA, using float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. They also propose a "capacity-headroom hypothesis," suggesting that PPO performance in SLMs relies more on a fluent supervised model and a discriminative reward signal than on the sheer number of model parameters. Their system achieved stable convergence across all experiments, outperforming instruction-tuned baselines and requiring less training data, particularly when a fluent prior and informative reward signal were present.