RL Post-Training Builds New Compositional Reasoning Strategies in LLMs

A fundamental question in AI is whether reinforcement learning (RL) post-training merely enhances a base model's latent skills or if it can construct entirely new, higher-level strategies. A new study investigates this in a fully observable rewrite-grammar environment, where pretraining distributions are known and every rewrite can be audited. A Transformer model, initially pretrained on primitive symbol-rewrite chains, was then post-trained on a trace-based reasoning task using only a binary final-answer reward.The results indicate that RL successfully solves problems that the pretrained model rarely could, even with extensive sampling. Trace analysis revealed that RL reorganizes primitive competence through a phased compositional mechanism. It first strengthens primitive reductions, then discovers and consolidates valid composed procedures. These include sequential compositions, which condense ordered chains of primitive contractions, and parallel compositions, which combine independent contractions in a single step. These composed procedures are not isolated instances but are reused and integrated into a stable repertoire.Comparing RL with rejection fine-tuning (RFT) showed that RL's advantage lies in its selectivity, concentrating exploration on valid, reusable structures, whereas RFT often produces many invalid shortcuts. Pretraining ablations further demonstrated that the emergence of compositional strategies depends not just on primitive exposure, but on whether pretraining organizes primitive competence into reduction procedures that RL can later compress. This suggests RL builds reliable higher-level strategies from weak procedural ingredients provided by the base model.