SciDraw-Bench Evaluates AI Scientific Figure Generation

A new benchmark, SciDraw-Bench, has been developed to specifically evaluate how well text-to-image and multimodal generative AI models can create scientific figures. Unlike existing benchmarks that focus on natural images and photorealism, SciDraw-Bench addresses the unique requirements of scientific diagrams, such as mechanism schematics, experimental designs, and graphical abstracts. The benchmark comprises 32 structured tasks across eight figure types and ten scientific disciplines. Each task includes a natural-language prompt paired with a machine-checkable specification for required labels, entity relations, diagrammatic structure, and adherence to disciplinary drawing conventions. This rigorous setup allows for a comprehensive evaluation. A four-dimensional evaluation protocol is proposed, measuring Text Fidelity (via OCR), Semantic Correctness (using vision-language models), Structural Quality, and Convention Adherence. Preliminary results show that a domain-specific system, SciDraw AI, significantly outperforms general-purpose text-to-image models across all dimensions and figure types, particularly in semantic correctness and convention adherence, though text fidelity remains a challenge for all systems.