ATS Framework Assesses Trust in VLM Physical Reasoning Answers

This research addresses a critical challenge in deploying Vision-Language Models (VLMs): determining the trustworthiness of individual predictions, especially for quantitative physical reasoning tasks where ground truth is unavailable. While VLMs can estimate physical quantities, existing benchmarks primarily focus on overall model performance. The paper proposes Answer-Level Trust Selection (ATS), a post-hoc and model-agnostic framework designed to accept or reject individual VLM predictions. ATS operates without fine-tuning, auxiliary verifiers, or access to the model's internal logits. Instead, it compiles eight interpretable behavioral diagnostic scores, derived from repeated queries and controlled interventions, into a single trust score. Extensive evaluation on Qwen2.5-VL-7B and 20 other VLM backbones demonstrates that intervention-based diagnostics are crucial for identifying predictions that are consistently wrong ("stable-but-wrong") or overly reliant on textual priors rather than visual evidence ("prior-tracking"), which simple self-consistency might miss. However, improved rejection of failure cases can sometimes lead to lower retention of correct predictions, highlighting a trade-off. ATS thus complements model-level capability assessments with answer-level reliability.