CARPRT Enhances Zero-Shot VLM Classification with Class-Aware Prompt Reweighting

Pre-trained Vision-Language Models (VLMs) enable zero-shot image classification by comparing an image to textual descriptions, often generated by inserting class labels into prompts. The effectiveness of this process is highly sensitive to the chosen prompt. Current methods typically use a single weighting vector for prompts across all classes, assuming prompts are conditionally independent of classes, which is often inaccurate. To address this limitation, researchers introduce CARPRT (Class-Aware Zero-Shot Prompt Reweighting). This novel scoring scheme dynamically adjusts the weighting vector for each class label, capturing the class-specific relevance of different prompts without requiring additional training. CARPRT quantifies this relevance by averaging image-text scores for images predicted to a specific class under a given prompt, then normalizes these estimates to derive class-specific weights. Evaluations demonstrate that CARPRT significantly outperforms existing class-independent reweighting methods, confirming the importance of modeling prompt-class dependencies for effective zero-shot prediction and broader VLM applications.