New Benchmark Evaluates Vision-Language Models on Rare Remote Sensing Images

While Vision-Language Models (VLMs) have shown strong performance in general remote sensing tasks, their capabilities for interpreting rare or specialized scenes remain largely unexplored. Existing benchmarks primarily feature common urban and rural imagery, creating a gap in understanding VLM robustness in niche applications. To address this, a new benchmark called RRS-10K has been developed. RRS-10K comprises 10,738 military-related remote sensing images, accompanied by diverse question-answer pairs. These images are sourced firsthand and categorized into three capability dimensions, six sub-dimensions, and 20 specific tasks, covering perception, reasoning, and robustness. A novel similarity-based distractor filtering strategy (SDFS) was employed during construction to enhance the quality of multiple-choice questions. Evaluations of 52 representative VLMs on RRS-10K revealed that current models achieve only moderate zero-shot performance. They exhibit particular weaknesses in tasks requiring visual grounding, referring segmentation, and complex semantic reasoning. This benchmark offers a systematic way to analyze failure modes in long-tail remote sensing interpretation, guiding the development of more reliable and robust VLMs for specialized applications.