New Benchmark Challenges AI Agents in Realistic Healthcare Tasks.

A new benchmark, HealthAgentBench, has been developed to provide a rigorous and holistic evaluation for advanced AI agents in healthcare settings. This suite comprises 54 distinct agentic tasks, categorized into seven areas, each simulating a unique real-world healthcare environment. The tasks cover a broad spectrum of the patient journey and incorporate various data modalities, requiring agents to perform multi-step solutions beyond simple prompting, from exploring raw data to operating within complex clinical workflows. Initial evaluations of frontier AI agents on HealthAgentBench reveal that overall task success rates remain low, with the strongest and most cost-effective agent, Codex GPT-5.5, achieving only approximately 42%. The benchmark effectively uncovers specific strengths and weaknesses; for instance, agents show promise in developing research modeling pipelines for EHR data, but medical imaging tasks, especially for Claude Code models, prove particularly difficult. Tasks demanding extensive search spaces combined with compositional reasoning also pose significant challenges for all current agents, indicating substantial room for future AI progress in healthcare.