New Score Detects Silent Reasoning Failures in LLM Math.

Evaluating large language models (LLMs) on mathematical chain-of-thought (CoT) tasks typically focuses solely on whether the final answer is correct. This approach overlooks "silent reasoning failures," where an invalid derivation might accidentally lead to the right answer, or a valid calculation is marred by a transcription error. This discrepancy is termed the reasoning-answer consistency gap. To address this, a new framework introduces the Reasoning Answer Faithfulness Score (RAFS). RAFS is a reference-free, instance-level diagnostic designed to assess the local credibility of an LLM's mathematical trace, its support for the final answer, and its stability under resampling and targeted counterfactual interventions. It combines metrics like step validity, reasoning-to-answer entailment, counterfactual sensitivity, answer consensus, and conditional reasoning stability. The score focuses on the agreement at the trace level, not the model's internal computation or external factual correctness. A preregistered confirmatory study on GSM8K and MATH datasets is underway, with a feasibility pilot already conducted to verify execution and intervention coverage. RAFS aims to complement traditional answer accuracy with an auditable warning signal for subtle reasoning flaws and extraction errors.