Dream Rehearsal Prevents Catastrophic Forgetting in Model-Based RL

This study investigates catastrophic forgetting in model-based reinforcement learning agents, specifically those from the DreamerV3 family, when trained on sequential tasks. Contrary to assumptions, the research reveals that even with an unbounded replay buffer, the world model largely retains information about old tasks, including reward discrimination and value estimates. The primary component that suffers from forgetting is the actor, whose behavior collapses. The paper demonstrates this by showing that freezing the world model and performing reinforcement learning in imagination fails to recover lost skills. However, supervised self-imitation using the world model's own "graded dreams" successfully recovers these skills without any additional environment interaction. By interleaving this graded dream rehearsal during training, the authors achieve a task-label-free, parameter-constant continual learner that significantly outperforms plain replay, retaining skills across multiple task chains. The effectiveness of the dream-grading step is highlighted, with specific rules provided to prevent scoring failures.