LLMs Struggle with Self-Awareness on Clinical Data, New Method Improves Reliability

Large Language Models are increasingly being deployed for tasks involving structured clinical data, yet their ability to discern the boundaries of their own knowledge in such critical applications remains largely unexamined. This study investigates this crucial question by analyzing "epistemic blind spots" in LLMs, specifically comparing Qwen 2.5 7B with XGBoost on a clinical prediction task using cross-model attribution divergence. The findings highlight several key issues. Firstly, LLMs' verbalized confidence is shown to be unreliable, remaining consistently high even when prediction accuracy is low, indicating it tracks prompt format rather than actual prediction quality. Secondly, LLMs exhibit an inverse difficulty effect, performing worse on tasks where XGBoost is highly accurate but matching XGBoost's performance on moderately uncertain tasks. The research also demonstrates that combining few-shot examples with SHAP-derived feature evidence acts as a powerful, super-additive intervention, drastically reducing attribution disagreement and improving accuracy without requiring model training. Finally, a novel cross-model calibrator, which uses attribution divergence signals, significantly reduces expected calibration error, providing patient-specific reliability estimates that are far more informative than the LLM's inherent verbalized confidence. This work frames these challenges as a "cold start problem" for LLMs on structured data and outlines a path towards achieving genuine epistemic self-awareness.