New Benchmark Exposes AI Model "Blind Spots" Beyond Standard Tests

Current AI benchmarks often fail to capture subtle yet significant limitations in advanced models, particularly when it comes to tasks humans find simple. To address this, a new benchmark called Blind-Spots-Bench has been developed. This benchmark focuses on exposing "blind spots" in multimodal AI systems by presenting them with tasks that are trivial for humans but challenging for AI. The benchmark utilizes questions sourced from AI students, which are then cleaned, annotated, and categorized into a task taxonomy. An automated grading system evaluates various models, including both open-weight and closed-source language, vision-language, and image-generation models. Initial findings indicate that closed-source frontier models, while generally superior, still exhibit a performance gap of approximately 10% compared to open-weight models, even when they perform similarly on traditional benchmarks. Further analysis reveals that no single model excels across all task types, and certain challenges remain difficult for all evaluated systems. This highlights Blind-Spots-Bench's utility as a diagnostic tool for pinpointing specific areas where modern AI models need improvement.