Improving Rare Disease Diagnosis with Selective AI Prediction.

Diagnostic AI systems typically rank potential diseases and then decide whether to endorse their top prediction or defer it for human review. This decision is often based on a simple threshold of the highest prediction score. However, this study reveals significant limitations of this approach, especially when dealing with rare diseases. It demonstrates that even small, open-weight Large Language Models (LLMs) achieve very low recall on ultra-rare conditions, indicating that even perfect confidence ranking cannot achieve high selective accuracy in these challenging scenarios. The research emphasizes that the confidence signal used for decision-making must align with the specific decision being made. For systems that provide a fixed list of candidate diseases, the margin between the top two scores proves to be a more effective confidence metric than the top score alone. This margin helps filter out cases where the model is less certain among its top choices. However, the study also proves that relying solely on unlabelled scores cannot definitively determine when switching to the margin will be beneficial, suggesting a nuanced application is required. The findings underscore the need for more sophisticated selective prediction strategies in high-stakes domains like healthcare, particularly for conditions with low prevalence. It suggests that simply improving model accuracy might not be enough; the confidence mechanisms also need to be rethought to ensure reliable and safe AI-assisted diagnosis.