LLMs Show Promise, Limits in Coding Humanitarian Data.

Humanitarian organizations face significant challenges in timely and consistent interpretation of qualitative data from affected populations, often lacking the resources for large-scale analysis. Large Language Models (LLMs) offer a potential solution, but their reliability in coding such nuanced data has been largely unproven. This study directly addresses this by benchmarking 46 LLMs against human expert adjudication. The research utilized 150 high-fidelity synthetic humanitarian transcripts, evaluating LLMs using inter-rater reliability (Krippendorff's alpha), discrepancy analysis, and qualitative assessment against humanitarian-specific criteria. The findings indicate that several LLMs can perform deductive coding at reliability levels similar to experienced human coders, especially when guided by structured prompts and reasoning-enabled configurations. However, the study also revealed significant limitations. LLMs struggled with recognizing needs expressed indirectly, identifying needs outside predefined categories, and discerning protection-relevant concerns like physical safety and discrimination. This suggests that while LLMs can augment analytical capacity, they are not substitutes for human judgment. Effective deployment requires structured codebooks, reasoning-enabled models, careful attention to theme-specific performance, and tiered human oversight, particularly for sensitive data where miscoding could have severe programmatic consequences. Open-weight models on self-hosted infrastructure are suggested for better data governance.