LLM Medical Consultations Need Earlier Evaluation for Vague Patient Concerns.

This paper identifies a critical flaw in how large language models (LLMs) are currently assessed for medical consultation roles. Evaluations typically focus on scenarios where the patient's medical issue is already well-defined, overlooking the initial, often ambiguous, stages of a real-world consultation. Patients frequently present with vague, minimized, or even misframed symptoms, a phase the authors term the "preformulation gap." The study tested three API models using physician-authored vignettes and simulated patient interactions. It found that without specific instructions, LLMs frequently offered self-care advice prematurely, before gathering sufficient patient information. Conversely, with targeted instructions, models were more likely to produce structured handoff summaries, indicating improved documentation. However, even with instructions, the models did not consistently elicit all decisive facts. The research concludes that evaluating LLMs for medical consultation should directly assess their behavior during the initial patient contact, rather than solely relying on diagnostic accuracy or the quality of final answers. This shift in evaluation methodology is crucial for developing truly effective and safe AI in healthcare.