EarlyDx Benchmark Evaluates LLM Diagnostic Accuracy at Hospital Admission

A new research paper introduces EarlyDx, a large-scale benchmark designed to evaluate the ability of large language models (LLMs) to generate open-ended, evidence-supported diagnoses at the critical point of hospital admission. Unlike previous benchmarks that rely on closed code sets or discharge diagnoses, EarlyDx uses 154,834 emergency department encounters from MIMIC-IV, restricting evidence to what's available at admission and supervising with diagnoses recorded during the ED encounter. A key feature of EarlyDx is its LLM auditor, which verifies every free-text label for support from the admission-time evidence, scoring only fully supported diagnoses. Using a semantic LLM-as-judge protocol, the evaluation revealed that no tested system—whether general, medical-specialized, or in-domain post-trained—could reliably synthesize admission-time evidence. Zero-shot models primarily extracted information, recovering only a small fraction of diagnoses requiring inference. While post-training improved inference-dependent recall, a substantial gap remains, and no system achieved a clinician's balance of sensitivity and precision for time-critical conditions. The full construction and evaluation pipeline are publicly released.