Pythia AI System Detects Clinical Symptoms Without Fine-Tuning.

Clinical notes contain a wealth of patient symptom information that is often unstructured and difficult to extract for analysis. Traditional methods, like rule-based systems, can produce many false positives, while supervised models require extensive and costly fine-tuning. This paper introduces Pythia, an innovative multi-agent system designed to overcome these limitations. Pythia operates by autonomously writing and optimizing extraction prompts for clinical concepts, eliminating the need for manual prompt engineering or fine-tuning. The system runs on a locally hosted open-weights model, ensuring that sensitive clinical notes remain on local infrastructure, addressing critical privacy concerns. Pythia selects optimal prompts based on development-set sensitivity and specificity. In validation tests across 72 signs and symptoms from 400 clinical notes, Pythia achieved a mean sensitivity of 0.76 and specificity of 0.95. This performance often matched or exceeded curated lexicons and significantly outperformed a BERT classifier fine-tuned on the same data, particularly for low-prevalence concepts where BERT's sensitivity collapsed. The findings suggest that autonomous, fine-tuning-free prompt optimization can effectively generalize for symptom extraction while maintaining local deployment capabilities.