New Framework Quantifies LLM Logical Reasoning Consistency

Large Language Models (LLMs) often produce correct answers through unstable or contradictory reasoning paths, particularly in multi-step deductive tasks. Traditional methods for assessing reliability primarily focus on output dispersion, which measures how much sampled answers vary. Researchers have introduced "structural uncertainty," a novel framework that evaluates consistency by analyzing the stability of an LLM's self-preferences among its own generated reasoning candidates. This involves generating multiple solutions for a query, then asking the model to pairwise rank its own outputs. The framework aggregates these self-preferences into ranking distributions and decomposes the signal into two entropy-based components: across-trial ranking instability and within-trial candidate ambiguity. Experiments across various LLMs and benchmarks show that structural signals offer complementary information to answer dispersion, especially for logical and mathematical reasoning tasks, improving the identification of unreliable instances.