IMCBench: New Benchmark for Multimodal Medical LLMs

A new benchmark, IMCBench, has been developed to rigorously evaluate multimodal large language models (LLMs) in the context of medical conversations that involve both text and images. Existing medical AI benchmarks often fall short by either lacking image input for multi-turn dialogues or focusing only on single-turn questions with images. IMCBench addresses this by simulating realistic patient-clinician interactions, pairing publicly available clinical images with synthetic patient profiles to create multi-turn scenarios. The evaluation protocol is comprehensive, assessing models across three critical clinical dimensions: safety, accuracy, and the appropriate use of uncertainty in diagnoses. The study benchmarked eight frontier multimodal models, including those from Claude, GPT, Nova, and Llama families. Results indicate that Claude Opus 4.6 achieved the highest overall score, but no single model dominated all evaluation criteria. A significant finding was the degradation of safety performance for both malignant and rare conditions. Ablation studies further highlighted the importance of both visual input and electronic health record (EHR) context for safe guidance, with stronger models demonstrating better utilization of visual features. This research underscores that merely providing accurate clinical descriptions does not guarantee safe patient guidance, emphasizing the need for multi-dimensional evaluation in medical AI.