LLMs Struggle with Indeterminate Preference Reasoning.

As large language models (LLMs) increasingly take on decision-making roles, their ability to reason about preferences becomes crucial for alignment, coordination, and collective intelligence. However, unlike many standard benchmarks that assume clear-cut solutions, real-world preference reasoning often involves indeterminacy, where information is incomplete or valid solutions simply do not exist. This inherent ambiguity poses a significant challenge for AI. Researchers formalized this challenge along two dimensions: epistemic indeterminacy, which arises from incomplete or partial preferences, and structural indeterminacy, which occurs when standard social choice concepts yield no solutions. Across a range of tasks, the study found that even state-of-the-art LLMs consistently struggle to differentiate between situations where a clear preference can be determined and those where it cannot. This systematic failure results in miscalibrated reasoning, even when the models are asked to verify existing solutions. The findings suggest that current LLMs lack a robust understanding of when a definitive answer is possible versus when the problem itself is ill-defined or lacks a unique solution, underscoring indeterminacy as a fundamental hurdle for advanced AI reasoning.