LLMs Show Promise, Limitations in Aphasia Discourse Assessment

Researchers explored the capability of instruction-tuned large language models (LLMs) to automatically classify Correct Information Units (CIUs) within transcripts of discourse from individuals with aphasia. CIUs are a critical metric for assessing communicative informativeness in aphasia, but their manual scoring is labor-intensive and requires specialized training. The study benchmarked several publicly available LLMs using both zero-shot and few-shot prompting conditions. The findings indicate that zero-shot prompting was ineffective. However, few-shot prompting significantly improved performance, with models such as Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B achieving F1 scores between 0.776 and 0.817. These models demonstrated high recall but lower precision, suggesting a tendency to over-classify tokens as CIUs. Performance also varied with the severity of aphasia, being weakest in more severe cases. While LLMs show promise for supporting CIU identification, their current reliability is not yet sufficient for fully autonomous clinical application, suggesting a human-in-the-loop approach is still necessary.