Visual Prompts Challenge Multimodal LLMs, New Grounding Helps.

Multimodal large language models (MLLMs) are increasingly used to reason over visual inputs like screenshots and documents. However, most benchmarks present task questions as plain text, leaving a gap in understanding how well these models process instructions embedded directly within the image itself. This study introduces Visualized Task Semantics (VTS), a controlled experimental setup where the task question is moved into the image while the underlying problem and answer remain constant. Across six MLLMs and four benchmarks, the research consistently shows a significant drop in accuracy, averaging 17.8 points, when the question is presented visually. Models often correctly transcribe the visual question but fail to effectively use it as an instruction, indicating a "semantic channel gap" beyond simple Optical Character Recognition (OCR) limitations. To mitigate this issue, the paper proposes "prompt-region grounding." This core design aligns the visual region containing the question with typed semantics and reconstructs a clean representation from a masked view. This method, achieved at matched training cost, boosts VTS accuracy from 58.0% to 66.3% while maintaining performance on traditional text-based interfaces. Crucially, it requires no OCR or region metadata during inference, highlighting that reading task-bearing text and grounding it as an instruction are distinct and challenging capabilities for MLLMs.