New Medical Benchmark Tests LLM Long-Context Clinical Reasoning

Existing medical Question Answering (QA) benchmarks for Large Language Models (LLMs) primarily focus on short-context knowledge or single-document grounding, leaving a gap in evaluating their ability to process and connect information across extended patient histories. To address this, researchers have introduced MedLoCoMo, a Medical Long-Context Memory benchmark. This benchmark is specifically designed to test LLMs on patient-specific clinical reasoning over multi-admission medical dialogues. MedLoCoMo is constructed from deidentified MIMIC-IV and MIMIC-IV-Note records. It synthesizes grounded doctor-patient conversations and generates evidence-linked QA items across various settings, including single-admission, cross-admission, and adversarial unanswerable questions. The benchmark features 100 patient timelines, each averaging nearly 1,700 turns, 30 sessions, and over 74,000 tokens per conversation, representing a truly long-context challenge. Evaluations using MedLoCoMo consistently show that cross-admission reasoning is significantly more difficult for LLMs than using localized evidence. This challenge persists even when models have large context windows or employ external memory and retrieval methods. The benchmark and its code are publicly available, encouraging further research into improving LLM capabilities for complex medical reasoning.