LLMs Distill Conceptual Knowledge to Vision Models.

Large Language Models (LLMs) possess a vast repository of conceptual knowledge derived from extensive text pretraining, yet their potential to guide models in other modalities, particularly vision, remains largely untapped. This research introduces LaViD (Language-to-Visual Knowledge Distillation), a straightforward and effective framework designed to transfer sophisticated semantic knowledge from a language-only LLM teacher to a vision-only student model. Crucially, LaViD achieves this without requiring paired multimodal data, which simplifies the knowledge transfer process. LaViD operates by prompting an LLM to generate multiple-choice questions (MCQs) that specifically probe the semantic distinctions between various visual classes. Each visual class is then mapped to a soft label distribution over these MCQs, creating a rich "conceptual signature." This signature subsequently guides the vision student model through an auxiliary distillation loss. Remarkably, LaViD consistently surpasses recent methods that rely on vision-language models for distillation and achieves competitive or superior performance compared to state-of-the-art visual distillation techniques, with further improvements when combined with logit standardization. The framework also significantly enhances worst-group accuracy on datasets like Waterbirds, indicating improved robustness against spurious correlations.