Learnijoy
ResearchAI ResearchAI Engineering & DevTools

Vision-Language Fine-Tuning Damages LLM Text Capabilities

Minsik Choi, Geewook Kim, Young Geun Kim· September 2, 2026 View original

Key takeaways

  • Fine-tuning LLMs into VLMs can degrade their text capabilities, especially for rule-based tasks.
  • "Attention-sink corruption" is identified as a key cause of this degradation.
  • "Sink Strength" is a new diagnostic tool to predict text capability loss before VLM training.
  • Current mitigation strategies like post-pretraining QK-RMSNorm or weight merging are largely ineffective.

Who benefits

AI DevelopmentRoboticsAutonomous VehiclesHealthcareE-commerce

Summary

Fine-tuning large language models (LLMs) into vision-language models (VLMs) often degrades their original text capabilities, particularly for tasks requiring strict output rules. This degradation is linked to "attention-sink corruption," where VL fine-tuning perturbs early attention-sink positions crucial for text processing.

Research reveals a significant challenge in adapting large language models (LLMs) into vision-language models (VLMs): the process of fine-tuning often compromises the LLM's foundational text capabilities. This degradation is particularly evident in tasks that demand precise adherence to output rules, such as instruction following, chain-of-thought reasoning with strict parsing, and other evaluations requiring exact answers. The study attributes this capability loss to "attention-sink corruption," where the vision-language fine-tuning process disrupts the early attention-sink positions within the model's architecture. These sinks are critical for anchoring a large proportion of attention probability and are essential for robust text processing. To diagnose this issue, researchers introduced "Sink Strength," a scalar metric computable on the base LLM in seconds, which accurately predicts the extent of post-VL degradation without requiring any actual VL training. This diagnostic tool consistently tracks relative degradation across various VLM-LLM pairs and format-sensitive tasks. The study also explored potential mitigation strategies, finding that post-pretraining QK-RMSNorm injection does not replicate the protective effects of native QK-RMSNorm, and several off-the-shelf weight-merging techniques fail to restore the lost text capabilities after VL training. These findings emphasize the importance of pre-screening LLM backbones with Sink Strength before embarking on costly VL training and suggest a need for more targeted, head-selective protection during adaptation.

Why it matters

Professionals developing VLMs need to be aware that adapting LLMs can compromise their text reasoning abilities, necessitating careful selection of base models and potentially new fine-tuning strategies to preserve critical functionalities.

How to implement this in your domain

  1. 1Before fine-tuning an LLM into a VLM, use "Sink Strength" to assess the base model's susceptibility to text capability loss.
  2. 2Prioritize LLM backbones that demonstrate higher Sink Strength for VLM adaptation to minimize degradation.
  3. 3Develop or explore fine-tuning strategies that specifically protect attention-sink positions during vision-language adaptation.
  4. 4Implement rigorous evaluation metrics for both vision-language and pure text capabilities post-adaptation to ensure balanced performance.

Original post by Minsik Choi, Geewook Kim, Young Geun Kim

"arXiv:2609.00746v1 Announce Type: new Abstract: Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thoug…"

View on X

Originally posted by Minsik Choi, Geewook Kim, Young Geun Kim on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Research

AI Engineering & DevToolsAI Research

Subspace Levenberg-Marquardt Algorithms Boost Neural Network Training

This research evaluates subspace Levenberg-Marquardt (LM) algorithms, such as KSLM and HSLM, for training neural networks on regression and classification tasks. These methods address the high computational and memory costs of classical LM, offering more efficient second-order optimization compared to first-order methods like SGD and Adam.

M. Duc HoangSep 2, 2026
AI ResearchAI Engineering & DevTools

Neural Networks Show Varied Conceptual Separation Internally

A study examined "conceptual separation" in CNNs and LLMs, analyzing how internal activations represent concepts. It found that CNNs form coherent representations for familiar concepts, while LLMs show clear separation for distinct domains but collapse distinctions for ambiguous topics.

Jaee Ponde, Roshni Agarwal, Subhashis BanerjeeSep 2, 2026
AI Engineering & DevToolsAI Research

FCCA Boosts Low-Resource LLM Adaptation with Fewer Parameters

Researchers introduced Frozen-Core Cross-Covariance Adaptation (FCCA), a new parameter-efficient fine-tuning method that significantly improves LLM performance with a minimal trainable parameter budget. FCCA achieves results comparable to LoRA and DoRA while optimizing orders of magnitude fewer parameters by carefully selecting fixed core coordinates.

Wentao Ye, Zhanming Shen, Zhiqing Xiao, Yao Ding, Haobo Wang, Gang ChenSep 2, 2026