New Attack Method Evades Vision Language Models by Targeting Vision Encoder

Vision Language Models (VLMs) are becoming integral to multimodal AI systems, enabling joint reasoning over visual and textual inputs in various applications, including safety-critical ones. Despite their growing deployment, the robustness of VLMs against adversarial attacks, particularly those targeting multimodal alignment, remains underexplored. This research investigates the susceptibility of VLMs to adversarial perturbations applied to visual inputs.The study explores two attack scenarios: untargeted attacks, aiming to disrupt the model's original image interpretation, and targeted attacks, designed to force the model to generate a specific, unrelated semantic description. To efficiently create these adversarial examples, a novel gradient-based attack method is proposed. This method optimizes perturbations exclusively on the VLM's vision encoder, rather than the entire multimodal architecture.This focused approach significantly reduces computational cost and resource requirements while maintaining high effectiveness. Evaluations on several open-source VLMs, including Qwen2.5-VL and Phi-3.5-Vision, demonstrated that small, human-imperceptible perturbations can drastically alter the textual interpretations produced by these models. These findings underscore a critical vulnerability in modern VLMs, emphasizing the urgent need for enhanced robustness and security mechanisms in multimodal AI systems.