VLM Chain-of-Thought Relies on Hidden State, Not Constant Visual Access.

Chain-of-Thought (CoT) prompting is a widely adopted strategy to enhance Vision-Language Models (VLMs) at test time, but the exact mechanism by which it extends reasoning traces has been unclear. Specifically, it was unknown whether CoT requires continuous access to image tokens throughout the reasoning process or if it primarily operates on visual information already processed and embedded earlier in the forward pass. To address this, researchers developed "Visual Access Sweep," a causal intervention that masks attention from generated-token queries to image-token keys at varying layer depths and generation times. This allowed them to define the "Visual Access Boundary" (VAB) as the minimal region of visual access necessary to maintain task accuracy. Across several VLM configurations (Qwen2.5-VL and InternVL3), both direct answering (no-CoT) and CoT prompting exhibited finite VABs. Notably, for larger models, the VAB layer for CoT differed by at most two layers from the no-CoT boundary, even with significantly longer generations. This suggests that CoT's performance gains do not primarily stem from extended direct image-token access throughout the reasoning trace. Instead, CoT appears to improve performance by extending language-side computation over visual information that has already been derived and encoded into the model's hidden states. The analysis further indicates that CoT gains are constrained by the initial perceptual readout, helping when visual attributes are reliably extracted but not when the readout itself is unreliable.