MM-R2 Enhances Multi-modal RAG with Agentic Retrieval Planning.

Multimodal Retrieval-Augmented Generation (mRAG) systems aim to answer image-text queries using external knowledge. However, most current systems retrieve directly from raw multimodal input over a flat evidence space, often struggling because retrieval targets are underspecified and search spaces are weakly structured. This leads to semantically distinct evidence competing in a single global ranking. Researchers propose MM-R2, a multimodal agentic retrieval framework that addresses these issues by incorporating reasoning *before* retrieval. MM-R2 explicitly models both what information to retrieve and where to search for it. The framework first builds an intent-grounded retrieval state from the image-question pair, capturing the information need, visual referent, and retrieval constraints. It then performs retrieval over a structured KnowledgeMap, allowing the agent to select relevant retrieval units before issuing grounded queries within them. Trained using a large-scale trajectory dataset (MM-R2-Traj) and a two-stage post-training strategy, MM-R2 significantly outperforms strong baselines on answer accuracy on datasets like Infoseek and Encyclopedic VQA, while also providing more interpretable retrieval trajectories.