Triple-Filtered Multimodal RAG Boosts PDF Reasoning

This research introduces Multimodal CoLRAG-TF, an advanced retrieval-augmented generation (RAG) system designed to tackle the challenges of processing complex, heterogeneous PDF documents. Traditional RAG systems struggle with multimodal content, specialized terminology, and the need for multi-hop reasoning across dispersed information. CoLRAG-TF addresses these by integrating a four-axis fusion architecture. The system builds a multimodal index from PDF blocks, leveraging a hybrid OCR pipeline and LLM-based caption generation. It enhances compositional reasoning by extracting and indexing OpenIE triples into a FAISS database, enabling rapid lookup and hierarchical propagation of relevance signals. A coarse-to-fine retriever further refines the search space. Bayesian optimization revealed that the knowledge-graph triple filtering axis is crucial, dominating the fusion weights to overcome lexical bias and maintain multi-hop retrieval quality. Evaluated on a benchmark of Japanese disaster lesson PDFs, Multimodal CoLRAG-TF achieved a high retrieval recall and a significant improvement in multi-hop answer similarity compared to single-hop queries. The approach also demonstrated applicability to visual inputs via an image-to-lesson pipeline using a vision LLM. These results highlight that triple-filtered multimodal fusion is essential for robust structured reasoning over noisy and diverse PDF collections, with potential applications beyond the disaster domain.