Multimodal AI Struggles with Abstract Perceptual Reasoning

A new challenge, "The Unwritten Benchmark," has been introduced to test the abstract perceptual reasoning capabilities of multimodal machine learning models. Unlike tasks involving static visual or auditory content, this benchmark requires models to infer unseen information from dynamic, generative processes. The core task is acousto-kinematic word inference: models must decipher words being written in three different styles, using only the audio of pen scratches and video of hand movements, with no visible ink trace. The evaluation revealed a substantial performance gap between humans and machines. Human participants achieved over 80% ordered letter accuracy, while leading multimodal models, including GPT-4o and Gemini 2.5-Pro, struggled significantly, failing to surpass 10%. This stark difference highlights a critical limitation in current AI's ability to perform abstract perceptual and cognitive tasks. Furthermore, the study identified a paradoxical "fusion effect" where providing both audio and video modalities often led to degraded performance rather than improvement. This suggests a fundamental breakdown in the models' capacity to synthesize complementary perceptual cues for this complex cognitive task, indicating limitations in cross-modal causal reasoning and understanding micro-kinematics.