Larger Batches Can Accelerate LLM Reinforcement Learning.

The impact of larger batch sizes on the training speed of reinforcement learning for large language models (LLMs) is often debated, as statistical benefits from reduced gradient variance must be weighed against increased sample consumption and execution time. This study systematically explores this trade-off by dissecting algorithmic and systems-level effects. At the algorithmic level, researchers found that with appropriate learning rate scaling (e.g., square-root scaling with Adam), learning curves can be approximately batch-size-invariant over a bounded range of batch sizes when compared at equal cumulative sample counts. From a systems perspective, larger batches significantly improve generation throughput, particularly because autoregressive generation is often memory-bandwidth-bound at low concurrency. By combining these insights, the research concludes that larger batches reduce wall-clock time-to-target only if the throughput gain surpasses the penalty of more samples needed per target. Experiments with GRPO and PPO showed that combining higher throughput with learning-rate retuning can reduce time-to-target by up to 29%.