New Benchmark Evaluates MLLMs for Chart Annotation

While Multimodal Large Language Models (MLLMs) have shown impressive progress in understanding, generating, and editing charts, their capability to annotate existing charts has remained largely unexplored. Chart annotation is a complex communicative task that requires models to infer the intended message, interpret chart semantics, and appropriately place textual or graphical elements. To address this gap, a new benchmark called ChartAnno has been developed. ChartAnno comprises 1,200 real-world charts, each paired with code and annotation instructions across three levels of specificity. The benchmark was used to evaluate ten representative MLLMs under two main input settings: using chart code alone, and using both chart code and the chart image. An ablation study with image-only input was also conducted. The evaluation results indicate that proprietary MLLMs generally achieve stronger overall performance, though large-scale open-source models are narrowing the gap. The study also found that providing more specific instructions significantly improves annotation quality, while the task of inferring abstract intent from charts remains the most difficult challenge for current MLLMs. Interestingly, providing chart images offered only limited overall gains, with improvements primarily observed in design-related metrics. These findings underscore that chart annotation generation is a demanding task requiring deep semantic grounding and effective design capabilities.