Demystifying Reinforcement Learning Post-Training for Language Models

Reinforcement learning (RL) post-training has become a crucial technique for enhancing the capabilities of large language models (LLMs), enabling impressive advancements in reasoning, mathematics, and coding. Despite its power, the underlying principles of classical RL within this context often remain opaque to many researchers and practitioners. This research aims to demystify the RL post-training algorithm by meticulously investigating each step to reveal its inner workings. By isolating the mechanics of RL with verifiable rewards in a controlled, simplified environment, the study examines how various factors influence RL outcomes. These factors include the base model's prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and the overall model scale. The investigation uses the entropy of the policy's output distribution to compare the certainty distributions learned through pretraining, supervised fine-tuning (SFT), and RL post-training, illustrating how each stage shapes model confidence. It sheds light on how these choices interact to affect post-training success, demonstrating, for instance, that the impact of "spurious rewards" depends on the prompt distribution. The work also connects RL post-training success to the classical RL concept of exploration, emphasizing the need for the base model to already assign sufficient probability to desired behaviors. This primer serves as a valuable resource for the NLP community seeking to integrate RL into their toolkit.