LMMs Struggle with Spatial Modality Transfer for GIS

The development of autonomous Geographic Information System (GIS) agents requires AI models to effectively process and understand spatial information across different modalities, much like humans do. While current AI models are improving in spatial reasoning, most research has focused on text-based inputs and outputs, contrasting with human GIS workflows that blend visual and textual information. This study proposes a "modality transfer" task to evaluate Large Multimodal Models (LMMs) on their ability to convert spatial information between images and text. In this task, an LMM describes an image of colored squares, and a second LMM then attempts to regenerate the original image from that textual description. The results indicate that even recent LMMs, such as those from OpenAI, struggle significantly with this transfer, highlighting a critical need for more robust multi-modal alignment to achieve strong geospatial understanding and truly autonomous GIS agents.