VLMs Vulnerable to Chart Deception; New Benchmark and Mitigation Proposed.

Information visualizations are crucial for communicating data, but deceptive design choices—like truncated axes or distorted aspect ratios—can mislead interpretation even when the underlying data is accurate. As Vision-Language Models (VLMs) are increasingly used for chart understanding and analytical reasoning, their susceptibility to such deceptions is a critical concern for trustworthy data analysis. Researchers have introduced VisDeception, the first controlled benchmark specifically designed to evaluate VLM robustness against misleading chart designs. This benchmark comprises 1,600 pairs of faithful and misleading charts, covering eight common deceptive tactics, all generated from the same data. A new metric, the Deception Score, quantifies how much misleading visuals shift model responses from the correct interpretation. Experiments with ten state-of-the-art VLMs using VisDeception revealed that even advanced models are highly vulnerable to these visual manipulations. To address this, an inference-time multi-agent mitigation framework is proposed. This framework grounds VLM reasoning in structured chart metadata extracted from the visualization before generating an answer, effectively reducing the impact of deceptive visual cues without requiring explicit user input. These findings highlight significant reliability gaps and point towards benchmark-driven evaluation and structured reasoning as key directions for developing more trustworthy VLMs for visual analytics.