Continual Prompt Replay Boosts RLVR Performance

This research explores continual Reinforcement Learning with Verifiable Rewards (RLVR), a method where reasoning models are updated incrementally as new tasks emerge, rather than being retrained from scratch. To investigate this, the "Continual Reasoning Gym" was developed, an environment featuring text and visual reasoning tasks organized into five sequential task flows. The study revealed two key insights: firstly, sequential RLVR exhibits only minor forgetting of previously learned tasks. However, its overall performance still falls short of models trained jointly on all tasks simultaneously. The performance gap is not solely attributable to forgetting, suggesting other factors are at play. A crucial discovery was the existence of "shared reasoning," indicating that training on one task can positively influence performance on others. To leverage this, the researchers introduced Continual Prompt Replay (CPR). CPR improves learning on new and future tasks by replaying prompts from previous tasks and regenerating their responses using the current policy. On average, CPR successfully enabled continual RLVR to match the performance levels of jointly trained models.