CARGO-VL Improves Vision-Language Model Reliability with Counterfactual Arbitration.

Vision-language models often struggle when visual and textual information conflict or are individually insufficient to support an answer. Current training methods typically evaluate instances in isolation, failing to ensure consistent model behavior when evidence changes counterfactually. A new framework, CARGO-VL, addresses this by optimizing model responses across a bundle of evidence states, including aligned, image-correct, text-correct, and both-wrong scenarios. This approach introduces a group-relative objective that couples condition-wise correctness with transition rewards, promoting answer invariance, source equivariance, and appropriate switching to abstention. A primal-dual controller manages the balance between providing unsafe answers and excessive deferral. The framework, along with a new conflict training resource called XMC, significantly improves conflict handling, reduces unsupported answers, and enhances modality balance compared to existing baselines.