New Method Improves LLM Social Simulation Fidelity

Large Language Models (LLMs) are increasingly used for social simulations, offering a promising alternative or complement to traditional research methods like surveys. A key challenge in this domain is accurately evaluating how well LLMs mimic human behavior and optimizing them to do so. Current practices often rely on accuracy, checking if an LLM reproduces a single observed human response. However, human behavior is inherently subjective; a person might reasonably act differently in the same situation, meaning a single observed response is just one possibility from a broader distribution. This makes accuracy-based evaluation misleading and hard-label training ineffective. To address this, researchers propose the "subjectivity coefficient," an entropy-based metric that quantifies the degree of subjectivity in a task, distinguishing it from objective tasks. Based on this, they introduce Subjectivity-Adaptive soft-Label Training (SALT). SALT pools observed outputs from semantically similar inputs to create soft distributional labels, with the aggregation radius dynamically adjusted based on the estimated subjectivity of each input. This approach naturally reverts to standard single-label training for objective tasks. To support this new evaluation paradigm, a new benchmark dataset called SUBJSIM was created, featuring 19,300 contexts across 193 annotators and 100 subjective questions, providing full response distributions. Experiments using SUBJSIM demonstrate that SALT significantly improves LLM performance in social simulations, even when trained with only single observed outputs, proving its feasibility in realistic data settings.