New Benchmark QMFOL Evaluates LLM Deductive Reasoning with Precision

Evaluating the deductive reasoning abilities of Large Language Models (LLMs) is crucial, especially for high-stakes applications. However, current benchmarks often lack the ability to precisely control logical complexity or maintain a balance between semantic diversity and logical consistency. This makes it difficult to accurately assess how LLMs perform under varying reasoning demands. To address these challenges, researchers have developed QMFOL, an automated framework designed to generate monadic first-order logic reasoning tasks. QMFOL allows for fine-grained control over various aspects of complexity, including reasoning depth, width, label types, and the presence of distractors. It constructs formal logical structures and then translates them into natural language, ensuring logical consistency through a round-trip verification process. Using this framework, QMFOLBench was created, comprising 2880 instances across diverse logical and semantic dimensions. Evaluations on several large reasoning models and LLMs revealed that performance declines and computational overhead increases with rising logical complexity. Models also showed better performance on 'True' labeled tasks compared to 'False' or 'Unknown' ones, and exhibited sensitivity to semantic variations. QMFOL offers a scalable and reliable method for constructing benchmarks that enable more precise evaluation of LLM reasoning.