MentalHospital: Virtual Environment for Psychiatric AI Evaluation

While large language models (LLMs) have shown promise in isolated psychiatric tasks like dialogue and diagnosis, comprehensive evaluation of their performance in full clinical encounters has been lacking. To address this, researchers have developed MentalHospital, a virtual environment designed to simulate and evaluate LLM-based psychiatric clinical encounters. This environment follows the standard S.O.A.P. (Subjective Interviewing, Objective Examination, Diagnostic Assessment, Treatment Planning) workflow. MentalHospital utilizes skill-augmented standardized patients, which are constructed from a large dataset of 1,193 de-identified psychiatric electronic health record (EHR) cases, covering a wide range of ICD-11 categories and 76 disorders. Each simulated encounter is assessed through a dual-track protocol, combining objective comparisons against EHR-derived references with subjective evaluations of clinical process quality. To scale expert judgment, a suite of five domain-specific evaluators, called MentalEval, was developed, covering empathy, professionalism, note quality, diagnostic rigor, and treatment appropriateness. These evaluators were trained with expert-guided methods and achieved strong alignment with human clinicians. Benchmarking results within MentalHospital indicate that even the most advanced LLMs currently trail human clinicians by a substantial 37.28 percentage points in overall objective psychiatric competence. A key bottleneck identified is the mental status assessment. This highlights the significant gap still present between AI and human expertise in complex clinical settings.