New Benchmark Reveals AI Models Struggle with Geometry Diagrams

While large language models (LLMs) like GPT and Claude have shown impressive proficiency in solving Olympiad-level mathematics, a new benchmark highlights a distinct limitation: their inability to accurately construct geometric diagrams. Solving a geometry problem often relies on a faithful diagram with correct auxiliary constructions and incidences, a skill not measured by existing benchmarks. To address this gap, researchers introduced an open-source benchmark comprising 954 self-contained Olympiad geometry problems, including a hard subset of 297 problems. Each problem is paired with its solution and a human-authored, high-fidelity diagram rendered in Asymptote code. Evaluating current foundation models against this benchmark revealed a pronounced disparity. Despite strong mathematical reasoning capabilities, these models produced diagrams that were markedly less faithful, achieving an average compile success rate of only 36.14%. This indicates that advanced mathematical reasoning does not automatically translate into the ability to construct accurate visual representations.