LLM Psychological Profiles Found to Be Measurement Artifacts, Not Intrinsic Traits.

Psychological instruments designed for humans are increasingly being used to assign stable psychological profiles to Large Language Models (LLMs). These profiles are then used to assess usability, safety, and even to employ LLMs as proxies for human participants in research. However, a new study, using a formal psychometric framework, suggests that these apparent profiles are largely an artifact of the measurement process itself. The researchers administered a battery of personality and risk-preference instruments, including self-reports and behavioral tasks, to 56 instruction-tuned LLMs and compared their responses to large human reference samples. They found that differences between models were primarily driven by a "directional response bias"—a tendency for the LLM to respond towards one end of a scale or a particular labeled option, irrespective of the item's content. This bias accounted for a substantial 81-90% of the variation observed between models, significantly higher than the 9-16% seen in humans. Further findings revealed that while this bias decreases with increased model capability, it is not entirely eliminated. The study also introduced the concept of "response orthogonality," which predicts an instrument's apparent reliability based on the proportion of items where trait and bias point in opposite directions. Crucially, the psychological profile an LLM appears to have can shift depending on the specific items used and can even be manufactured through item selection. These results strongly suggest that the observed psychological profiles of LLMs are products of the instruments used, rather than inherent properties of the models themselves, calling for dedicated, LLM-specific assessment methods.