LLMs Detect System Changes but Struggle with Abductive Reasoning.

Large Language Models (LLMs) are highly proficient in pattern recognition and generating text, yet their capability for abductive inference—the process of inferring the most plausible explanation for an observation—remains largely unexplored. This research introduces "Elenchos," a novel generative evaluation framework designed to assess this specific type of reasoning. Elenchos frames abductive reasoning as a structural inverse problem. Given a reference formal system and a potentially modified version, LLM agents are tasked with determining if a mutation has occurred and then inferring the specific rule changes responsible for any observed behavioral differences. Evaluations of various LLMs, from frontier to mid-tier, consistently showed a "detection-attribution dissociation." Models frequently recognized that a system had been altered but struggled significantly to pinpoint the exact latent mutations causing the discrepancies. This performance degradation was particularly pronounced when multiple mutations interacted, with models often only recovering a partial set of the underlying changes. Preliminary findings also suggest that increasing inference-time reasoning budgets yielded only modest improvements, indicating a potential ceiling on current LLMs' abductive capabilities.