Fine-Tuning Vision-Language Models for Visual Grounding Creates Controllable Interference

This research investigates the effects of fine-tuning vision-language models (VLMs) to produce dense coordinate lists, a technique used to improve visual grounding. The study found that this specific fine-tuning method creates an "interference surface" within the model, which alters how it generates and concludes structured outputs. A key observation was the emergence of repeated output tails, where the model duplicates parts of its generated coordinate lists. For instance, in Gemma 4 12B, high-capacity LoRA significantly improved visual grounding but also led to a notable duplicate rate. Crucially, the study demonstrates that this interference is controllable. By implementing object-level repeat-stop mechanisms, researchers were able to eliminate duplicate records while maintaining or even slightly improving the model's F1 scores for visual grounding. This suggests that the introduced behavior is a structure-bound, cross-family phenomenon that can be managed.