LLMs Simulate Survey Respondents with High Accuracy

A new evaluation framework called "cross-survey transfer" has been proposed to assess the effectiveness of large language models (LLMs) in simulating human survey respondents. Unlike previous methods that focused on distributional comparisons, this framework tests an LLM's ability to predict answers to entirely new questions within the same survey, given its responses to an initial set. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, the study found that zero-shot LLMs achieved 52% accuracy on genuinely unseen items, approaching within 6 percentage points of a supervised random forest model trained on the same population. The research also provided clearer insights into common LLM limitations, showing that "variance collapse" can also affect supervised models and "safety alignment" effects vary significantly across different model families.