LVLMs Struggle with Interactive Visual Grounding, Study Finds

This research introduces a novel evaluation framework designed to assess the interactive visual grounding capabilities of large vision-language models (LVLMs). Unlike traditional one-shot evaluations, this framework focuses on scenarios where target information is incomplete or ambiguous, necessitating dialogue and interaction to clarify. The study varied the amount of upfront information and the need for dialogue across different visual contexts and interaction protocols. The findings indicate that current LVLMs perform substantially below human benchmarks in interactive visual grounding tasks. While interaction can improve performance when refining or repairing initial descriptions, models struggle most when they must proactively acquire target information through questions. Furthermore, LVLMs exhibit poor calibration, often overestimating their accuracy. These results highlight that interactive visual grounding remains a significant challenge for LVLMs, requiring not only visual matching but also sophisticated information seeking and synthesis abilities. The research suggests that future development should focus on improving these interactive and proactive reasoning skills in LVLMs.