LLMs Cannot Sample Distributions They Describe

A new study reveals a significant limitation in instruction-tuned language models: their inability to effectively sample from distributions they can accurately describe. When used as proxies for human survey respondents, these models frequently produce the same answer for a given persona and question, indicating a "collapse to a single output" rather than diverse sampling. This issue is exacerbated by instruction tuning, with base models performing better in this regard. The research identifies this discrepancy as the "KNOWS/DOES split," where a model can articulate a response distribution precisely in one call but fails to generate varied samples from it. This failure is linked to a degenerate sampling primitive observed in the models' internal logits, induced by alignment training. Leveraging this split, the study found that explicitly asking the model to describe the response distribution in a single call significantly reduces error against human survey data compared to aggregating persona-based outputs. For applications requiring per-persona outputs, a new method called Prompt-Perturbed Argyle (PPA) is proposed, which reduces error by 21% without additional cost.