LLMs Outperform Traditional ML in Open-Ended Survey Analysis

This research investigates the effectiveness of large language models (LLMs) in analyzing open-ended survey data, a task traditionally challenging to scale. Building on prior work using conventional machine learning, the study compares leading LLMs—including OpenAI's GPT series, Twitter-roBERTa-base, and Meta's LLaMA—against older ML models for tasks such as sentiment analysis and thematic classification of student survey responses. The findings reveal that current LLMs consistently surpass traditional machine learning models in classification accuracy, particularly in discerning nuanced mood and thematic patterns. However, the study also highlights a significant trade-off: while LLMs offer superior predictive power, they vary considerably in the explicitness and consistency of their reasoning and how they apply category boundaries. This suggests that while LLMs provide enhanced automation for qualitative research, researchers must carefully consider the balance between predictive strength and the need for interpretive rigor and explainability.