MLLMs and Visualization: Distinguishing Chart-Supported from Model-Supplied Claims

Multimodal Large Language Models (MLLMs) possess the capability to connect visual patterns in charts with external knowledge, inferring causes and consequences. However, the underlying evidential basis for these interpretations often remains ambiguous. This exploratory study investigates how MLLMs generate claims when presented with visualizations, categorizing these claims into DIRECT (supported by the chart), DERIVED (inferred from the chart), and SPECULATIVE (model-supplied interpretation). The research analyzed 1,224 descriptions generated by three MLLMs (Gemini, GPT, and another unnamed model) across 102 visualizations, under four different input conditions. These conditions varied the MLLMs' access to the image, source-specific accessible chart context, and a "withheld-context" framing. A key finding was that providing accessible chart context significantly influenced Gemini and GPT, leading them to produce more DIRECT claims and improving numeric agreement in some cases. Interestingly, simply adding the image to the full context did not consistently improve numeric agreement, and prompts designed to encourage cautious language in "withheld-context" scenarios did not reliably increase it. The "Real-World Significance" section, as defined in the prompt, remained predominantly SPECULATIVE. These results underscore the critical need for accessible description systems that clearly distinguish between claims directly supported by the provided visual evidence and those that are purely model-supplied interpretations or external knowledge.