Optimizing Compute Allocation for RL Foundation Model Post-Training

As reinforcement learning (RL) is increasingly used to adapt large foundation models for tasks like reasoning and robotics, understanding how to best allocate limited post-training computational resources becomes crucial. This research addresses the challenge of optimizing a fixed FLOP budget, exploring whether compute should prioritize larger policies, longer training for smaller policies, more extensive rollout search, or stronger reward feedback mechanisms. The authors developed a FLOP-accounting framework specifically for GRPO post-training, breaking down compute into categories such as rollout/search, policy-update/learning, and reward/feedback model evaluation. Their findings, using LoRA-adapted Qwen2.5 policies, indicate that there's no single optimal allocation; the best strategy is conditional and depends on factors like the model's size, the total compute budget, the type of reward system employed, and the specific evaluation goals. Crucially, the study highlights that larger policies consume more compute per token, which directly impacts the number of updates or rollouts achievable within the same budget. Different reward systems also shift the compute balance; for instance, rule-based rewards primarily allocate compute to policy rollouts, while PRM-style feedback dedicates a notable portion to reward-model inference. The proposed RACE protocol serves as a diagnostic tool to identify these allocation regimes before costly validation runs, suggesting that future RL post-training research should detail compute distribution alongside total FLOPs.