DocHop Benchmarks Multi-Hop Reasoning in Documents

Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in understanding structured visual information, particularly in tasks like chart and document question answering. However, existing evaluation benchmarks typically assess these domains in isolation, overlooking a crucial ability: how well models can leverage textual context to guide the selection, interpretation, and aggregation of data presented in charts. To address this gap, researchers have introduced DocHop, a novel benchmark specifically created for integrated chart-context reasoning within document-style images. In DocHop, a document's narrative provides multi-step compositional constraints, while embedded charts supply the necessary data values. Questions are grounded in semantic reference labels defined within the narrative, compelling models to first resolve target entities from the context before aggregating evidence across multiple charts. DocHop was constructed using a stochastic logic-first generation pipeline, allowing for controlled reasoning depth and visual density. It comprises 2,074 examples across six task categories. Initial experiments with various proprietary and open-source MLLMs reveal a substantial performance gap compared to human annotators, who achieve over 90% accuracy, while the best model only reaches 62.83%. Models enhanced with reasoning capabilities show improvements, but their performance degrades as reasoning complexity increases, indicating a significant challenge for current MLLMs in complex multi-hop document reasoning.