MedEvoEval: Evaluating Doctor Agents' Continual Evolution

This paper introduces MedEvoEval, a novel and executable longitudinal evaluation framework designed to assess the continual evolution of "doctor agents" in simulated clinical environments. Moving beyond single-turn question-answering, this framework focuses on outpatient episodes where agents must acquire evidence, utilize examination and consultation resources, and make decisions regarding diagnosis and management plans. Crucially, MedEvoEval also supports the evaluation of how agent behavior changes over time through mechanisms like memory, retrieval, and reflection across multiple episodes. The framework converts source clinical cases into role-specific patient, examination, and manager views, revealing evidence only through valid actions. Each episode generates a structured trace, detailing observations, actions, final outputs, scores, and optional experience write-back. Experiments using 700 processed episodes demonstrate that MedEvoEval can uncover process costs often hidden by final-answer scoring, illustrate resource reallocation during multi-disciplinary team-style consultations, and facilitate longitudinal analyses of memory development, transfer learning, and retention. This framework provides a concrete foundation for evaluating whether doctor agents genuinely improve with experience and maintain capabilities over time.