CausalDS Benchmarks AI Agents in Data Science Causal Reasoning

As large language models increasingly function as integrated data-science agents, capable of abstract reasoning and advanced tool use, there's a growing need for benchmarks that accurately assess their causal reasoning abilities. Existing benchmarks often separate symbolic causal reasoning from realistic data analysis, or lack principled causal data-generating structures. To bridge this gap, CausalDS has been introduced: a new benchmark specifically for evaluating causal reasoning within agentic data-science workflows. Each instance in CausalDS consists of a sampled structural causal model (SCM), generated observational data, and a synthetic natural-language story grounded in a realistic domain. This setup helps reduce the "causal parrot" risk by generating novel synthetic causal structures. The benchmark includes tasks across all three of Pearl's rungs of causation, from prediction to counterfactuals, often requiring data science coding and tool use due to imperfect observations. A key feature is the explicit scoring of an agent's ability to recognize when a question has no warranted answer and to abstain, jointly evaluating symbolic causal reasoning, data science skills, uncertainty quantification, and tool use.