New Framework Audits AI Chain-of-Thought Validity

While Chain-of-Thought (CoT) reasoning is a powerful technique for AI, prior research indicates that it can often be unfaithful, meaning the model's stated reasoning doesn't reliably reflect how it arrived at its output. Detecting this unfaithfulness typically requires controlled experimental interventions, which are not feasible for post-hoc analysis of evaluation transcripts. To address this, a new method called Reasoning Consistency Scanning has been introduced. This framework focuses on a more tractable question: whether the stated reasoning is logically consistent with the accompanying answer, a property that can be assessed solely from a transcript. This approach is distinct from faithfulness, which concerns the internal process. The framework formalizes reasoning consistency, establishing a six-subtype taxonomy of inconsistencies. Researchers built a validated benchmark of 60 transcripts, adapted from InstrumentalEval outputs, and implemented a working scanner for InspectScout. Results across four generator models and three evaluations from inspect_evals demonstrate that reasoning inconsistency is present, detectable, and varies systematically depending on the model and task type.