CRiT-QA Dataset Challenges LLM Multi-hop Reasoning

A new dataset named CRiT-QA (Counterfactual Reasoning with Traps) has been developed to provide a more stringent evaluation of large language models' multi-hop reasoning abilities. Existing datasets often allow models to succeed by relying on internal parametric knowledge or by exploiting superficial shortcuts, rather than genuinely aggregating evidence across multiple documents. CRiT-QA addresses these limitations by introducing two key features. Firstly, it transforms factual reasoning chains with counterfactual entities, forcing models to strictly adhere to the provided context rather than their memorized knowledge. Secondly, it incorporates "distractor traps"—plausible but incorrect reasoning paths that diverge at various points—requiring models to follow the entire logical process to avoid errors. Experiments with CRiT-QA reveal a substantial drop in LLM performance compared to standard benchmarks, highlighting their current weaknesses in handling counterfactual conditions and complex distractor information, thus serving as a critical diagnostic tool for developing more robust, evidence-grounded LLMs.