T2D-Bench Evaluates LLM Clinical Advice for Type 2 Diabetes.

This paper introduces T2D-Bench, a novel benchmark and evaluation framework specifically designed to rigorously test the clinical accuracy and evidence-based reasoning of large language models (LLMs) when providing recommendations for Type 2 Diabetes. The framework is built upon a sophisticated multi-layer knowledge graph that integrates biomedical information, computable ADA Standards of Care rules, and lifestyle knowledge linked to glycemic effects. This allows T2D-Bench to verify if LLM outputs adhere to explicit, verifiable evidence requirements. The evaluation framework employs an "evidence gate" that detects unsupported omissions and facilitates constrained revisions to bring LLM outputs into compliance. Initial tests on 100 structured clinical vignettes showed that models like GPT-4o-mini and GPT-4o failed evidence-path checks in approximately one-third of cases. These findings highlight the critical need for robust verification mechanisms to ensure the reliability of LLMs in sensitive domains like healthcare, demonstrating that computable evidence constraints can make clinical errors explicit and correctable.