LLMs Unsafe for Autonomous Clinical Decision Support, Lacking Real-World Safety

A recent perspective paper raises significant concerns about the safety of deploying large language models (LLMs) for autonomous clinical decision support, especially in scenarios involving the triage of self-presenting patients without direct clinician oversight. While LLMs demonstrate impressive medical knowledge and can perform well in curated diagnostic cases, their fundamental design for generating the most probable text output conflicts with the critical safety requirements of clinical care. The authors highlight that safe triage prioritizes identifying rare but catastrophic diagnoses, even if they are improbable, over simply selecting the most likely condition. LLMs, optimized for probability, struggle with this asymmetric cost function and often fail to actively seek missing "red flag" information or broaden their differential diagnoses under uncertainty. Furthermore, current evaluation benchmarks, which often rely on complete and well-curated simulations, may not accurately reflect real-world clinical challenges where information is incomplete. The inherent "assistant-like" behaviors of LLMs, such as credulity and agreeableness, can also amplify these risks if not constrained by robust clinical triage logic.