Multimodal LLMs Suffer Hidden Forgetting, Losing Evidence Grounding

Multimodal large language models (MLLMs) are designed to adapt continuously to new tasks and domains. However, current evaluation metrics for continual learning primarily focus on whether the model's answers remain correct, often overlooking the stability of how these models ground their responses in multimodal evidence. New research identifies a critical issue termed "hidden evidence-use forgetting." This phenomenon occurs when an MLLM, despite maintaining high answer accuracy, subtly changes its reliance on visual, textual, or other forms of evidence. Essentially, the model might still get the right answer but for the wrong or less grounded reasons, indicating a loss of robust understanding. To address this, the researchers propose Reliance-Constrained Continual Learning (RCL), a replay-free framework. RCL works by freezing a previous model checkpoint as a behavioral reference and then jointly optimizing for task learning, prediction preservation, and crucially, reliance preservation. This method significantly improves performance and reduces evidence reliance drift across various multimodal benchmarks without adding any inference-time overhead.