MKG-RAG-Bench: New Benchmark for Multimodal Knowledge Graph Retrieval.

A new benchmark, MKG-RAG-Bench, has been developed to address a critical gap in evaluating Retrieval-Augmented Generation (RAG) systems that utilize multimodal knowledge graphs (MKG-RAG). Existing benchmarks often overlook the complexities of retrieval in these systems, where heterogeneous multimodal data presents significant alignment challenges for retrievers. MKG-RAG-Bench is constructed from two distinct multimodal knowledge graphs, covering both general and medical domains. It includes carefully aligned question-answering datasets that allow for controlled assessment of both retrieval accuracy and the quality of downstream generation. The benchmark's creation involved an LLM-based curation pipeline to ensure high-utility knowledge and structurally grounded queries. Extensive experiments conducted using MKG-RAG-Bench confirm that effective multimodal retrieval is not only challenging but also paramount for the overall performance of end-to-end MKG-RAG systems. The quality of the retrieved information directly and significantly impacts the quality of the generated output, underscoring retrieval as a primary bottleneck.