ColGraphRAG Enhances Multimodal QA with Late-Interaction Image Retrieval.

Multimodal question answering systems that use graph-structured evidence often struggle with accurately ranking visual assets. Traditional single-vector bi-encoder similarity for images can lose crucial fine-grained details needed for precise alignment, impacting overall accuracy. This research proposes ColGraphRAG to address this limitation. ColGraphRAG modifies the visual candidate-ranking component within a GraphRAG system. Instead of single-vector similarity, it employs a late-interaction, multi-vector scoring approach, inspired by ColBERT/ColPali models, for images linked within the evidence graph. Other components, such as graph construction, text/table retrieval, and downstream reasoning, remain unchanged. Evaluations on the MultimodalQA dataset show that this late-interaction method improves retrieval-stage point estimates for image candidates and boosts downstream QA performance. The gains are particularly noticeable in questions where visual evidence is highly relevant, suggesting a more effective inclusion of graph-linked visual information.