LLM Preference Judgments Lack Self-Consistency, Challenging Utility Models

A growing trend in AI involves using Large Language Models (LLMs) to interpret human preferences by querying them for numerical judgments, such as willingness-to-pay. These judgments are then often used to estimate a utility function, which guides agent actions. However, this entire pipeline rests on the assumption that the LLM-derived judgments are largely self-consistent, meaning they can be explained by a single underlying utility function. This paper investigates that critical assumption, measuring the self-consistency of cardinal LLM preference judgments. For instance, the stated difference in willingness-to-pay between two items should logically align with the payment that makes a person indifferent to exchanging them. The researchers developed statistical tests and interpretable metrics to quantify deviations from self-consistency. Experiments conducted across various scenarios (flights, apartments, hotels) and six different LLMs consistently revealed significant and persistent inconsistencies. This strong evidence suggests that LLM-derived preference judgments cannot be accurately summarized by a single utility function, posing a fundamental challenge to current methods of preference elicitation and agent decision-making.