LLMs Benchmark Against Official Crash Data Coding

This research evaluates the capability of six advanced large language models (LLMs) to extract and code crash attributes from police narratives, comparing their output against official, structured crash databases. The study linked over 5,500 fatal crash narratives from Arkansas with nearly 6,000 structured records, focusing on attributes like crash manner, intersection type, and light conditions.The LLMs were tested using a zero-shot prompt, and their performance was assessed using various metrics including agreement, F1 score, and Cohen's kappa. Results showed that GPT-5.5 High achieved the highest agreement among the LLMs. However, simple baselines, such as always-majority or keyword-rule approaches, often matched or even surpassed the LLMs in raw agreement or F1 scores.The study highlighted that performance varied more significantly across different crash attributes than across the LLM models themselves. Attributes like non-motorist relation and crash manner were coded with higher agreement, while light condition and roadway surface condition proved more challenging. This suggests that LLM deployment for such tasks requires attribute-specific evaluation and human oversight.