New Theory Reveals Computable Blind Spots in VLM Reliability

Label-free reliability for vision-language models (VLMs) often relies on the principle of invariance: if an input is perturbed, a faithful model's answer should remain unchanged. However, this approach has a known limitation: systematic misreadings can persist through perturbations, leading to incorrect certifications. This research demonstrates that this "blind spot" is not just a phenomenon but is computable. An error becomes invisible to an edit if the error and the edit commute, meaning the errors a test suite cannot detect form its "joint centralizer." The study then shifts focus to equivariance, where editing a figure's underlying data should result in a predictably changed correct answer. They show that two matched edits are provably complete for affine reading errors, and while no suite of swap edits is complete for label permutations, cyclic relabeling significantly closes this gap. This theory is instantiated as the Equivariance-Consistency Score, a label-free and training-free detector, and accompanied by REND-EQUIV, a dataset pairing invariance and equivariance sets. The findings confirm the predicted ordering across multiple models and a hand-labeled population. A second invariance-family method further validates that the blind spot is inherent to the relationship between edits and errors, not just a specific implementation. The research also explains a previously reported inversion in classifier metamorphic testing, emphasizing that detectability is a joint property of the relation and the fault class.