Co-observation is Key to Continual Learning Generalization

Continual learning in deep neural networks faces persistent challenges, primarily attributed to catastrophic forgetting (losing old knowledge) and a decline in plasticity (inability to learn new information). However, this research argues that these two issues alone do not fully account for the performance gap between naive sequential training and ideal offline joint training. The paper introduces "data co-observation" as a crucial, distinct factor. It highlights the representational advantages gained when a model observes training data simultaneously, rather than sequentially. By carefully decoupling the constraints of separate data access from stability and plasticity, the study systematically investigates these benefits. Empirical results consistently show a performance difference between joint and separate training across both supervised and self-supervised learning paradigms, even in generic data-incremental scenarios where forgetting and plasticity are controlled. This indicates that the simultaneous observation of training data yields generalization benefits that extend beyond mere knowledge retention. The findings also recontextualize existing continual learning mechanisms, suggesting that while distillation helps retention, the success of memory replay largely stems from its ability to reintroduce the benefits of data co-observation into the learning process.