ComMem Enhances Vision-Language Model Adaptation with Dual Memory

This research introduces ComMem, a novel framework designed to improve the adaptability of vision-language models (VLMs) in dynamic, real-world environments during test time. Current test-time adaptation (TTA) methods often lack the ability to accumulate knowledge over time or fully leverage the multimodal nature of VLMs. ComMem draws inspiration from the brain's complementary memory systems, specifically the hippocampus and neocortex, to address these limitations. ComMem operates with two distinct yet cooperative memory components: a fast-adapting "detailed memory" that functions like a dynamic visual cache, learning from high-confidence test samples; and a "slow-integrating abstract memory" that continuously refines global textual prototypes. For each new test instance, ComMem jointly optimizes both memory systems to ensure strong cross-modal consistency. Extensive evaluations across 15 benchmark datasets demonstrate that ComMem significantly surpasses existing state-of-the-art TTA methods, proving its effectiveness against natural distribution shifts and in cross-dataset generalization.