SiGMA Improves Multimodal Continual Instruction Tuning

Multimodal Continual Instruction Tuning (MCIT) is a vital process for adapting Multimodal Large Language Models (MLLMs) to a continuous stream of evolving downstream tasks. Existing methods, often relying on Mixture of Experts or expansion merge approaches, primarily focus on preventing catastrophic forgetting. However, these methods frequently suffer from "negative interference," where newly acquired knowledge inadvertently overwrites or degrades previously learned, useful information, leading to a decline in overall performance. To tackle this challenge, researchers have introduced SiGMA (Sign-Guided Merging and Adaptation), a straightforward yet highly effective framework. SiGMA incorporates two core components. During the training phase, "sign-guided adaptive tuning" is employed to minimize collisions with past knowledge, allowing the model to learn new tasks with minimal drift and thereby mitigating severe forgetting. At inference time, "sign-guided merging" further enhances knowledge consolidation. This mechanism selectively scales salient parameters, ensuring that valuable task-specific knowledge is preserved and amplified. Experiments conducted on standard benchmarks like UCIT and DCL demonstrate that SiGMA significantly reduces negative interference and outperforms state-of-the-art MCIT methods, marking a notable advancement in the field of continual learning for MLLMs.