Specialized Clinical AI Outperforms General LLMs on Real Queries

This study presents a comprehensive, blinded evaluation of AI tools designed for clinical use, focusing on real-world point-of-care queries rather than hypothetical scenarios. Over 149 practicing physicians, matched by specialty, compared answers from three leading general-purpose large language models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5) against a specialized clinical tool, OpenEvidence (OE). The evaluation encompassed 620 real-world queries submitted by physicians across 30 specialties, alongside 187 questions from HealthBench. Physicians graded the tools across five critical dimensions for clinical decision support: accuracy, clinical utility, source quality, verifiability, and completeness. The specialized clinical tool consistently received higher scores on all axes, demonstrating win differences ranging from 25 to 39 percentage points over the general LLMs. The findings remained robust across various sensitivity analyses. The research highlights two key conclusions: the necessity of evaluating AI tools with real-world query distributions and expert judges reflecting medical specialization, and that targeted engineering and customization can yield substantial performance gains for specialized AI applications, even if general models show promise. The Real-POCQi benchmark is now publicly available.