MAG Boosts Multi-modal In-Context Learning with Unlabeled Data.

Multi-modal large language models (MLLMs) can adapt to new tasks with few-shot in-context learning (ICL), but their performance heavily depends on the quality of selected demonstrations. The challenge lies in effectively using the vast amounts of unlabeled multi-modal data available. This research proposes MAG (MAnifold-Guided semi-supervised in-context demonstration selection), a novel framework designed to exploit this unlabeled data. MAG formulates demonstration selection as a semi-supervised propagation problem on a multi-modal graph, employing a two-stage strategy. First, it uses relevance score propagation to pinpoint a compact set of high-impact unlabeled samples for efficient pseudo-labeling, thereby reducing MLLM inference costs. Second, it utilizes multi-modal relevance to make the final demonstration selections. The study highlights that textual representations are more effective for the initial relevance propagation, while both visual and textual modalities are crucial for selecting high-quality demonstrations. Experiments across eight multi-modal benchmarks demonstrate that MAG consistently outperforms strong baselines, especially in scenarios where labeled data is scarce, achieving significant performance gains with limited pseudo-labeling budgets.