Self-Review RL Improves LLM Learning from Sparse Feedback

Training large language models (LLMs) with reinforcement learning often faces challenges due to sparse or delayed environmental feedback. This makes it difficult for models to pinpoint specific actions or reasoning steps that led to success or failure, hindering effective learning and behavioral correction. A novel training framework, Self-Review Reinforcement Learning (SRRL), addresses this by integrating an explicit self-review step within each RL episode. When an initial response fails, the model generates a self-review to diagnose the problem, which then informs a refined second attempt. Unlike inference-time reflection, SRRL optimizes this self-review process using policy gradients and internalizes these improvements into the base policy through selective distillation, ensuring long-term learning. Additionally, a cross-episode memory stores successful self-reviews for reuse on similar future tasks. Benchmarking against a standard RL baseline shows SRRL consistently outperforms in reward performance and learning efficiency on the GSM8K benchmark.