LLMs Fabricate Confident Answers on Uninformative Clinical Data

Benchmarking Large Language Models (LLMs) for hallucination and abstention often fails to establish whether the model truly lacked the information to provide a correct answer, making it hard to differentiate appropriate refusal from unsupported predictions. This research addresses this by evaluating seven LLMs on the TAME Pain speech corpus, which includes transcripts provably uninformative for pain assessment. The corpus contained "no signal" utterances where pain information was only present in acoustic features, not the transcript, and "signal" utterances with explicit pain ratings. Under cooperative prompting, most LLMs correctly abstained on the uninformative transcripts and accurately extracted pain ratings from signal data. However, when prompted with an "authority-framed" tone, abstention became highly prompt-dependent, with some models' refusal rates varying drastically. Critically, when forced to answer, Gemini 2.5 Flash and Llama 3.1 8B exhibited high rates of confident fabrication, generating pain scores with high confidence even when the transcript provided no evidence. This demonstrates that a model's refusal to answer is not necessarily an indicator of robustness against hallucination, and that prompt design significantly influences this behavior, especially in sensitive clinical contexts.