Paper deep dive
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
Ayush Sawarni, Jiyuan Tan, Vasilis Syrgkanis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 2:30:16 PM
Summary
The paper introduces CausalReasoningBenchmark, a dataset of 173 queries across 132 real-world datasets designed to disentangle the evaluation of causal identification (research design formulation) from causal estimation (numerical implementation). It highlights that while LLMs can often identify high-level strategies, they struggle with the nuanced details of research design specifications, as evidenced by a drop in correctness from 79% (high-level strategy) to 34% (full specification).
Entities (9)
Relation Signals (9)
CausalReasoningBenchmark â contains â 132 real-world datasets
confidence 95% ¡ We introduce CausalReasoningBenchmark, a benchmark of 173 queries across 132 real-world datasets
CausalReasoningBenchmark â contains â 173 queries
confidence 95% ¡ We introduce CausalReasoningBenchmark, a benchmark of 173 queries across 132 real-world datasets
LLM â achievesaccuracyon â high-level strategy identification
confidence 90% ¡ while the model correctly identifies the high-level strategy in 79% of cases
LLM â achievesaccuracyon â full identification-specification correctness
confidence 90% ¡ full identification-specification correctness drops to only 34%
CausalReasoningBenchmark â curatedfrom â 79 peer-reviewed research papers
confidence 90% ¡ curated from 79 peer-reviewed research papers and three widely-used causal-inference textbooks
CausalReasoningBenchmark â evaluates â causal estimation
confidence 90% ¡ disentangled evaluation of causal identification and estimation
CausalReasoningBenchmark â evaluates â causal identification
confidence 90% ¡ disentangled evaluation of causal identification and estimation
CausalReasoningBenchmark â hostedon â Hugging Face
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification - formulating a valid research design under stated assumptions - and estimation - implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 173 queries across 132 real-world datasets, curated from 79 peer-reviewed research papers and three widely-used causal-inference textbooks. For each query a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state of the art LLM show that, while the model correctly identifies the high-level strategy in 79% of cases, full identification-specification correctness drops to only 34%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.
Tags
Links
- Source: https://arxiv.org/abs/2602.20571v2
- Canonical: https://arxiv.org/abs/2602.20571v2
Trouble viewing inline? Open PDF directly â
Full Text
65,307 characters extracted from source content.
Expand or collapse full text
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation â Ayush Sawarni, Jiyuan Tan, and Vasilis Syrgkanis Stanford University ayushsaw,jiyuantan,vsyrgk@stanford.edu Abstract Many benchmarks for automated causal inference evaluate a systemâs performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identificationâformulating a valid research design under stated assumptionsâand estimationâimplementing that design numerically on finite data. We introduce CausalReasoning- Benchmark, a benchmark of 173 queries across 132 real-world datasets, curated from 79 peer-reviewed research papers and three widely-used causal-inference textbooks. For each query a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (i) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 79% of cases, full identification-specification correctness drops to only 34%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark 1 is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems. 1 Introduction LLMs and LLM-based agents are increasingly used for causal reasoning over observational data. However, the evaluation of these systems often falls short of the rigor required for real-world applications. A common practice is to assess a modelâs performance based solely on a single numerical outputâtypically an effect estimate such as the Average Treatment Effect (ATE). This single-output evaluation is limited because it conflates two distinct steps that are central to any empirical causal analysis. The first step is identification: a conceptual exercise in which the analyst determines whether a causal quantity of interest is recoverable from the available data, given a set of assumptions about the data-generating process. This requires specifying a valid research designâoften called an identification strategy âsuch as an Instrumental Variable (IV) design, a Regression Discontinuity Design (RDD), or a Difference-in-Differences (DiD) design, and defining all of its necessary components (e.g., the instrument, the running variable and cutoff, or the time and group indices). The second step is estimation: a numerical exercise in which the identified strategy is implemented on a finite data sample to compute a point estimate of the causal effect and to quantify the uncertainty around that estimate. Existing benchmarks typically collapse these two steps into a single score, making it impossible to diagnose the source of errors. Did the model fail because it chose an invalid identification strategy, or did it implement a valid strategy incorrectly? Furthermore, many benchmarks rely on synthetic or simplified data, which may not reflect the complexities of real-world empirical research: missing data, ambiguous variable definitions, and study-specific design choices. â The authors used ChatGPT and Manus as research and writing assistants in preparing this manuscript. All interpretations, conclusions, and any errors remain solely the responsibility of the authors. 1 https://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark 1 arXiv:2602.20571v2 [cs.AI] 14 May 2026 To address these gaps, we introduce CausalReasoningBenchmark, a benchmark for evaluating auto- mated causal reasoning systems. Our main contributions are: 1.A curated real-world benchmark. We curate 173 queries over 132 unique datasets from 79 peer-reviewed research papers and three causal-inference textbooks. 2.A structured identification schema. The benchmark requires agents to specify not only the design family, but also the estimand, treatment, outcome, controls, and design-specific fields required for IV, RDD, DiD, conditional-exogeneity, and RCT designs. 3. Disentangled evaluation of identification and estimation We provide gold identification specifica- tions, reference estimation scripts, and an evaluator that separately scores research-design correctness and numerical estimation accuracy. 4. Standardized estimation scripts. We provide gold-standard estimation code for every query, allowing failures in identification to be isolated from failures in implementation. The rest of this paper is organized as follows. Section 2 reviews related benchmarks and agents. Section 3 motivates the separation of identification and estimation. Section 4 describes the CausalReasoningBenchmark dataset. Section 5 provides a formal description of the identification strategies covered. Section 6 defines the evaluation task and metrics. Section 7 presents baseline results, including a qualitative analysis of identification errors. Section 8 walks through a concrete example. Section 9 discusses hosting and maintenance. Section 10 addresses limitations and future work. Section 11 concludes. 2 Related Work We situate CausalReasoningBenchmark relative to two lines of work: benchmarks for causal reasoning and LLM-based causal-inference agents. Benchmarks for Causal Reasoning. Liu et al.[65]introduce QRData, a benchmark of quantitative reasoning tasks over spreadsheet-style data, including some causal estimation problems. While QRData tests a broad range of data-analysis skills, it does not focus on the specific identification strategies used in observational studies and does not evaluate identification separately from estimation. Zhou et al.[91]present CausalBench, which covers causal graph identification, counterfactual reasoning, and statistical estimation from text and tables. CausalBench is valuable for testing general causal reasoning, but it does not require systems to produce a full identification specification for quasi-experimental designs. Lee et al.[61]build a benchmark by extracting validated, but not quantitative, causeâeffect relations from economics and policy papers. Their dataset includes common designs such as IV, DiD, and RDD, butâlike the othersâit primarily evaluates the final effect estimate, making it difficult to distinguish between identification and estimation errors. LLM-Based Causal-Inference Agents. Several agent-based systems have been developed to automate parts of the causal inference workflow. CATE-B [2] is an LLM co-pilot that constructs directed acyclic graphs (DAGs), selects adjustment sets, and suggests estimators. ORCA [12] connects LLMs to causal inference libraries (e.g.,DoWhy) to load data, fit models, and summarize results. In the biomedical domain, MRAgent [89] automates Mendelian randomization by selecting instruments from the literature and analyzing GWAS datasets. These systems are typically evaluated on internal or synthetic tasks. A separate line of work focuses on causal reasoning over graphical models. Jin et al.[52]introduce CLadder, a benchmark for formal causal reasoning on synthetic graphs, testing aspects like identifying confounding bias. Jin et al.[51] propose Corr2Cause, which tasks models with inferring causal relationships from correlational statements. Sheth et al.[83]present CausalGraph2LLM, a large-scale benchmark with over 700k queries on diverse causal graphs. While these benchmarks are crucial for evaluating graph-based and counterfactual reasoning, they do not focus on the quasi-experimental designs common in applied empirical research. CausalReasoningBenchmark provides a challenging, external evaluation suite derived from peer-reviewed research, with a focus on disentangling identification from estimation. 2 Table 1: Comparison of CausalReasoningBenchmark with related benchmarks. âID evalâ indicates whether identification is evaluated separately from estimation. âReal dataâ indicates whether the benchmark uses real-world (non-synthetic) datasets. âQuant. evalâ means evaluating causal effect estimation from data (e.g., ATE / ATT / LATE / CATE). âDesign-specificâ indicates whether the benchmark requires specification of design-specific elements (e.g., instruments, running variables). Benchmark # Queries Real data ID eval Quant. eval Design- specific Designs covered QRData [65]899âPartialMixed CausalBench [91]495 PartialMixed CLadder [52]6.6kâGraph-based Corr2Cause [51]413kâGraph-based CausalGraph2LLM [83]700k+âGraph-based CausalReasoningBenchmark173âIV, RDD, DiD, CE, RCT Table 1 summarizes the key differences between CausalReasoningBenchmark and the most closely related benchmarks. 3 Why Separate Identification from Estimation? Separating identification from estimation mirrors the structure of empirical causal analysis. In applied research, identification is where the core intellectual contribution resides: it requires understanding the data-generating process, articulating the assumptions under which a causal quantity is recoverable, and specifying all the components of a valid research design. Estimation, by contrast, is largely a technical exercise: given a correctly specified design, the choice of estimator (e.g., two-stage least squares for IV, local polynomial regression for RDD, or a two-way fixed-effects model for DiD) is often well-understood and can even be automated. This distinction has practical consequences for evaluation. Consider a model that correctly identifies an IV design and names the right instrument, treatment, and outcome, but makes a coding error in the two-stage least squares implementation. Under a single-score evaluation, this model would receive the same score as one that misidentifies the entire research design. By scoring identification and estimation separately, CausalReasoningBenchmark can distinguish between these two very different failure modes. Moreover, the identification specification itself is a rich, structured object that can be evaluated along multiple dimensions. For example, a model might correctly identify the strategy (IV) and the instrument, but fail to exclude a post-treatment variable from the control set, a critical error that would bias the estimate. Our evaluation framework captures this level of detail, as described in Section 6. 4 The CausalReasoningBenchmark Dataset CausalReasoningBenchmark is designed to evaluate an agentâs ability to correctly specify and execute a causal analysis. It focuses on the canonical research designs used in observational studies: Instrumental Variables (IV), Regression Discontinuity (RD), Difference-in-Differences (DiD), Conditional Exogeneity (selection on observables), and Randomized Controlled Trials (RCT). The benchmark consists of 173 queries over 132 datasets (see Tables 2â4). Each query includes: ⢠A natural-language causal question. ⢠A dataset in CSV format. ⢠A metadata file describing the variables and providing study context. ⢠A gold-standard solution, including a detailed identification specification (as a JSON object) and a reference estimation script (in Python or R). The dataset is sourced from two main categories, described below. 3 Table 2: CausalReasoningBenchmark queries and datasets by source group. Source group#queries#datasets Research papers12079 Textbook5353 Total173132 Table 3: CausalReasoningBenchmark composition by identification strategy. Identification strategy#queries#datasets Difference-in-Differences6737 Regression Discontinuity4439 Instrumental Variable2222 Conditional Exogeneity3939 RCT11 Research Papers. We curated 120 queries from 79 papers, drawing from three large-scale reanalysis studies in political science: ⢠IV: Lal et al. [58] provide a replication of 67 instrumental variable studies. ⢠RDD: Stommes et al. [85] re-evaluate 44 regression discontinuity designs. ⢠DiD: Chiu et al. [9] conduct a reanalysis of 62 difference-in-differences studies. We selected cases from these corpora where the original paper presented a clear and defensible identification strategy, and where the documentation was sufficient to reconstruct the analysis. The research-paper subset spans three top political science journals, providing a diverse set of real-world causal problems. Textbook and Instructional Collections. To include classic and pedagogical examples, we added 53 queries from three popular causal inference textbooks: ⢠Causal Inference: The Mixtape [20] ⢠The Effect: An Introduction to Research Design and Causality [49] ⢠Causal Inference: What If [45] Several of these examples also appeared in causaldata R-package, [50] and QR Dataset [65] which served as a starting point for us. The textbook subset complements the research-paper subset by providing well- documented examples with clear pedagogical intent, and by adding coverage of Conditional Exogeneity designs (39 queries) that are not represented in the research-paper subset (Table 4). Dataset Composition. Tables 2â4 provide a detailed breakdown of the benchmarkâs composition. Table 2 shows the split between research papers and textbooks. Table 3 shows the distribution across identification strategies: DiD is the most common (67 queries), followed by RDD (44), Conditional Exogeneity (39), IV (22), and RCT (1). Table 4 reveals that the research-paper subset is dominated by DiD, RDD, and IV, while the textbook subset provides the bulk of the Conditional Exogeneity examples. The research-paper subset spans three top political science journalsâThe Journal of Politics, American Journal of Political Science, and American Political Science Review âwhich account for the vast majority of the research-paper queries. 5 Identification Strategies A key feature of CausalReasoningBenchmark is that it requires systems to produce a structured identification specification for each query. This section provides a brief formal description of each identification strategy covered by the benchmark, along with the specific fields that the system must specify. 4 Table 4: Query counts by source group and identification strategy. Source groupDiDRDDIVCond. Exog.RCT Research papers62391900 Textbook553391 5.1 Instrumental Variables (IV) An instrumental variable design exploits an exogenous source of variation (the instrument,Z) that affects the treatment (D) but has no direct effect on the outcome (Y) except throughD. The key assumptions are: (i) relevance:Zis correlated withD; (i) the exclusion restriction:ZaffectsYonly throughD; and (i) independence: Z is independent of the potential outcomes (unobserved confounders). In many applications, these assumptions may only be plausible after conditioning on a set of pre- treatment covariates X. The IV assumptions are thus relaxed to hold conditionally: (i) conditional relevance: Cov(D,Z |X)̸= 0; (i) conditional exclusion:Zis independent of potential outcomesY(d) conditional onDand X; and (i) conditional independence:Zis independent of potential outcomes conditional on X (Z ⼠Y(d)|X). When these conditions hold, the LATE can be estimated using methods like two-stage least squares with controls. Under the unconditional assumptions, the LATE for compliers is identified as: LATE = Cov(Y,Z) Cov(D,Z) .(1) Required fields:strategy=Instrumental Variable;instrument(column name(s) of the instrument); isencouragementdesign(whether the instrument is a randomized binary encouragement);treatments; outcomes; controls; causalquantity (typically LATE). 5.2 Regression Discontinuity (RDD) A regression discontinuity design exploits a known threshold (the cutoff ) on a continuous running variable (X) that determines treatment assignment. Units just above and just below the cutoff are assumed to be comparable, so the causal effect is identified as the discontinuity in the conditional expectation of the outcome at the cutoff: Ď RDD = lim xâc E[Y | X = x]â lim xâc E[Y | X = x],(2) wherecis the cutoff value. In a sharp design, treatment is a deterministic function of the running variable; in a fuzzy design, the probability of treatment changes discontinuously at the cutoff, and the design is analogous to an IV with the threshold indicator as the instrument. Required fields:strategy=Regression Discontinuity;runningvariable(column name);cutoff (numeric threshold); treatments; outcomes; controls; causal quantity. 5.3 Difference-in-Differences (DiD) A difference-in-differences design compares the change in outcomes over time between a treated group and a control group. The key assumption is parallel trends: in the absence of treatment, the average outcomes for the treated and control groups would have followed parallel paths over time. This assumption can be relaxed to a conditional parallel trends assumption, which posits that parallel trends hold after conditioning on a set of pre-treatment covariates X. This allows the baseline trends to differ, as long as they are parallel within strata defined by X. Formally, the assumption isE[Y(0) post â Y(0) pre | D = 1,X] =E[Y(0) post â Y(0) pre | D= 0,X]. Under the unconditional assumption, the Average Treatment Effect on the Treated (ATT) is identified as: Ď DiD = E[Y post | D = 1]â E[Y pre | D = 1] â E[Y post | D = 0]â E[Y pre | D = 0] .(3) Required fields:strategy=Difference-in-Differences;timevariable(column name of the time index);groupvariable(column name of the unit/group identifier);treatments;outcomes;controls; causalquantity (typically ATT). 5 5.4 Conditional Exogeneity (Selection on Observables) Under conditional exogeneity, treatment assignment is assumed to be independent of potential outcomes after conditioning on a set of observed covariates X: (Y (0),Y (1))⼠D | X.(4) This assumption, also known as unconfoundedness or selection on observables, allows identification of the ATE (or ATT) via regression adjustment, inverse probability weighting, or matching. Required fields:strategy=Conditional Exogeneity;treatments;outcomes;controls(the condi- tioning set X, which must include a minimal sufficient adjustment set); causalquantity. 5.5 Randomized Controlled Trials (RCT) In a randomized controlled trial, treatment is assigned randomly, so identification is straightforward: the ATE is simply the difference in mean outcomes between the treated and control groups. While RCTs are the gold standard for causal inference, they are included in CausalReasoningBenchmark primarily for completeness (1 query). Required fields: strategy = RCT; treatments; outcomes; controls; causalquantity. 6 Evaluation Task and Metrics 6.1 Task Definition For each query in the benchmark, an agent is provided with the following inputs: ⢠Question: A causal query in natural language. ⢠Dataset: A CSV file containing the data. ⢠Metadata: A text file with column descriptions and study context. The agent must produce two outputs: 1.Identification Specification: A structured JSON object that adheres to the schema described in Section 5, detailing the chosen identification strategy and all its components. 2.Estimation Output: A point estimate of the causal effect (effectestimate) and its standard error (standarderror). 6.2 Identification Metrics We evaluate the identification specification by comparing it field-by-field with the gold standard. The evaluator checks the following conditions: ⢠Strategy: Exact match of the identification strategy label (e.g., Instrumental Variable). ⢠Causal quantity: Exact match of the estimand label (e.g., ATE, LATE). â˘Treatments and outcomes: Exact set match of the variable names specified by the agent against the gold standard. ⢠Controls: We check two conditions: (1) the agentâs specified controls must be a superset of the gold-standard minimal sufficient adjustment set âthe smallest set of pre-treatment covariates needed for identification (e.g., to satisfy conditional exogeneity or conditional parallel trends); and (2) the agentâs controls must not include any variables from the gold-standard bad controls list, which includes post-treatment variables, mediators, and colliders whose inclusion would bias the estimate. 6 â˘Strategy-specific fields: Depending on the strategy, we require correct specification of all compulsory fields: for IV, theinstrumentlist andisencouragementdesignflag; for RDD, therunningvariable and cutoff; for DiD, the timevariable and groupvariable. â˘Overall identification correctness: A binary indicator that istrueonly if all of the above checks pass. This is the strictest metric and captures whether the model has fully specified a valid research design. 6.3 Estimation Metrics Given a predicted effectbĎand standard error c SE , and gold-standard valuesĎ â andSE â , we form 95% Wald confidence intervalsCI pred = [bĎ â1.96 c SE, bĎ+ 1.96 c SE] andCI gold = [Ď â â1.96SE â , Ď â + 1.96SE â ], and compute: â˘Point-estimate error: Absolute error|bĎ â Ď â |, signed errorbĎ â Ď â , and (whenĎ â ̸= 0) relative absolute error |bĎâĎ â | |Ď â | Ă 100%. ⢠Estimate within gold CI: Whether bĎ â CI gold . â˘Null-hypothesis agreement: Whether both intervals lead to the same reject/fail-to-reject decision for H 0 : Ď = 0. â˘Opposite-direction flag: Whether both intervals rejectH 0 but imply opposite effect signsâa particularly dangerous type of error. â˘Interval overlap (Jaccard): We measure the overlap between the two confidence intervals using the Jaccard index: J (CI pred , CI gold ) = |CI pred ⊠CI gold | |CI pred ⪠CI gold | ,(5) which equals 0 when the intervals are disjoint and 1 when they coincide. â˘CI Overlap: A binary indicator of whether the predicted and gold-standard confidence intervals overlap, i.e., 1[CI pred ⊠CI gold ̸=â ]. ⢠Standard-error gap: | c SEâ SE â | and the relative gap | c SEâSE â | SE â . Auto-rescaling. A common source of spurious estimation error is a unit mismatch (e.g., an effect reported in percentage points vs. proportions). For example, if the gold-standard effect is 0.05 (a 5 percentage point increase) and the model predicts 5.0, a naive error calculation would be enormous. To mitigate this, the evaluator can optionally rescale the predicted effect and standard error by a multiplicative factor from a fixed candidate set (e.g.,0.01,0.1,10,100). The evaluator selects the factor that minimizes the absolute error. In the example above, multiplying the prediction of 5.0 by 0.01 yields 0.05, which perfectly matches the gold standard. The evaluation would proceed with this rescaled value. This ensures that trivial unit-conversion errors do not dominate the estimation metrics. 7 LLM Baseline Evaluation To demonstrate the utility of our benchmark, we evaluated a simple LLM-agent baseline. The baseline was run withgpt-5.3with reasoning. For each query, the runner supplied the model with the natural-language causal question, an inline metadata description, and the corresponding CSV dataset. For the primary execution path, the CSV file was uploaded as a file attachment and made available to the APIâs Python code-interpreter environment. The model was instructed to inspect the data, write and execute Python code for the estimate and standard error, and return a single JSON object containing both a schema-conformant identification specification and the estimation code. The returned identification payload was validated against our output schema before being scored. The full prompt template is shown below. 7 LLM prompt template SYSTEM: You are a meticulous causal inference research assistant. You can read structured metadata , inspect CSV datasets , craft valid identification strategies , and run Python code to obtain causal effect estimates. USER: You are provided with a causal question , an inline metadata description , and a CSV dataset (available as a file in your Python environment). CAUSAL QUESTION: ... METADATA ABOUT THE DATASET: ... TASKS: 1) Read metadata to understand variables/context. 2) Load and analyze the CSV via Python to estimate the causal effect; report point estimate and standard error. 3) Fill out a JSON object with the identification specification , adhering to the provided schema. 4) Return the full Python script executed to produce the numbers. FORMAT: Output a single JSON object with keys: "identification_output ": ..., "estimation_code ": "language ": "python", "code": "..." , "explanation ": "..." 7.1 Aggregate Results Table 5 shows the aggregate performance of the baseline across all 173 queries. The model correctly identifies the high-level strategy in 79.2% of cases and the outcome variables in 93.6% of cases. However, performance drops sharply on more nuanced aspects of identification: causal quantity is correct in only 73.4% of cases, and the overall identification specification is fully correct in only 34.1% of cases. This gap between high-level strategy recognition and full specification correctness is the central finding of our baseline evaluation, and it validates the design of CausalReasoningBenchmark: a single-score evaluation based on the final estimate would have obscured this important distinction. 7.2 Per-Strategy Breakdown Table 6 provides a per-strategy breakdown of the baseline results. The model performs best on Regression Discontinuity queries, where the strategy is correct in 95.5% of cases, though overall identification correctness is significantly lower. Second, Difference-in-Differences queries prove more challenging, particularly in specifying the correct time and group variables in panel data. Third, Instrumental Variable queries show the largest gap between strategy-level correctness and full specification correctness, suggesting that the model struggles with the nuances of IV designs (e.g., identifying the correct instrument). 7.3 Analysis The baseline results reveal several important insights. First, the large gap between strategy-level correctness (79.2%) and full identification correctness (34.1%) confirms that the bottleneck in automated causal reasoning lies not in recognizing the broad category of research design, but in specifying its detailed components. A single final-estimate score would not distinguish these failure modes. Second, the estimation errors (median 18.1% relative error, median Jaccard overlap of 0.57) are non-trivial but secondary to the identification errors. In many cases, the model produces a reasonable estimate even when the identification specification is incorrect, because it may still use a plausible (but not gold-standard) approach. This further underscores the importance of evaluating identification separately. Third, the per-strategy breakdown reveals that different designs pose different challenges. RDD queries are relatively easy to identify but may still have estimation errors due to bandwidth selection. DiD queries 8 Table 5: Aggregate evaluation of the GPT-5.3 baseline on all 173 queries. Identification metrics are exact- match or set-based checks against the gold specification; estimation metrics compare the returned effect and uncertainty to the gold solution. Values in brackets denote the interquartile range. MetricValue Identification Metrics Strategy correct79.2% (137/173) Causal quantity correct73.4% (127/173) Treatments correct86.1% (149/173) Outcomes correct93.6% (162/173) Minimal controlling set included77.5% (134/173) Post-treatment set excluded89.6% (155/173) Controls correct67.6% (117/173) Strategy-specific fields correct89.0% (154/173) Identification spec correct (all checks)34.1% (59/173) Estimation Metrics Median absolute error |bĎ âĎ â |0.070 [0.010, 0.421] Median percentage error18.1% Median CI Jaccard overlap0.57 CI Overlap88 % Estimate Within gold CI82 % Null Hypothesis Agreement80 % Opposite Direction Flag1.73 % Table 6: Per-strategy breakdown of the GPT-5.3 baseline. âStrategyâ = fraction with correct strategy label; âFull IDâ = fraction with fully correct identification specification; âMed. %Errâ = median percentage error on the effect estimate. Design#QueriesStrategy (%)Full ID (%)Med. %Err Difference-in-Differences6792.552.227.0 Regression Discontinuity4495.511.420.0 Instrumental Variable2277.331.848.5 Conditional Exogeneity3938.528.24.0 RCT1100.0100.00.0 Overall17379.234.118.1 require understanding temporal structure and group assignments. IV queries demand the identification of a valid instrumentâa task that requires deep domain knowledge. To understand why the model fails, we categorize incorrect identification across the 114 failed cases. Table 7 summarizes the error types. Figure 1 further decomposes these errors by identification strategy, revealing that each design family has a distinct error profile. Control variable errors. Control specification is the most error-prone component overall (39 cases with missing necessary controls, 18 with bad or post-treatment controls). A representative bad-control error occurs in a DiD study of parliamentary gender quotas and public-health spending [14], where the model includes the quota-induced change in womenâs legislative representation as a covariate. Because this variable is a post-treatment mediator that lies on the causal path from quota adoption to health spending, conditioning on it blocks part of the treatment effect and biases the estimate. The correct specification controls only for pre-treatment country characteristics, yet the model fails to distinguish a downstream consequence of treatment from a confounder. IV instrument misidentification. Among the 22 IV queries, 10 cases exhibit errors in design-specific fields. An example comes from a study of peasant unrest and zemstvo representation in Imperial Russia 9 Table 7: Taxonomy of identification errors across all 114 incorrect cases. A single case may contribute to multiple categories. Counts are ordered by frequency. Error FamilyError TypeCount Control variablesMissing required controls39 Included bad / post-treatment controls18 EstimandCATE â LATE29 CLATE â LATE7 Other estimand errors10 StrategyCE â RCT24 DiD â CE or IV â CE10 Other strategy errors2 Variable specificationWrong treatment variable24 Wrong outcome variable11 Design-specific fieldsWrong IV-specific fields10 Wrong RDD / DiD-specific fields9 [24], which uses historical religious polarization as an instrument for the frequency of peasant disturbances. The model selects a first-stage fitted value as the instrument rather than the actual instrument (historical religious polarization), suggesting it confuses the mechanics of two-stage estimation with the conceptual identification argument. Strategy confusion: CE misidentified as RCT. The single most striking pattern is the modelâs tendency to classify observational studies requiring covariate adjustment as Randomized Controlled Trials (24 cases, all from the IHDP textbook subset studying the effect of specialist home visits on childrenâs cognitive outcomes). The metadata explicitly lists 25 baseline covariatesâincluding birth weight, gestational age, maternal education, and neonatal health indicatorsâand a binary treatment indicator, but does not state that treatment was randomly assigned. The model appears misled by the word âassignmentâ in the query text, predicting RCT with an empty control set when the correct strategy is Conditional Exogeneity with all 25 covariates in the adjustment set. Estimand confusion: CATEâLATE in RDD. In 29 of 44 RDD queries (66%), the model correctly identifies the strategy but mislabels the estimand as LATE instead of CATE. For instance, in a study of incumbency advantage in Japanese lower-house elections [1], the estimand is the treatment effect at the vote-margin cutoffâa conditional average treatment effect (CATE)âbut the model labels it as LATE, conflating the RDD estimand with the complier effect from an IV design. 8 Sample Query To illustrate the task format, we present an example from Coppock and Green [17]. Example Causal Query âWhat is the causal effect of voting in the November 2007 municipal election on the probability of voting in the January 2008 primary, estimated for the subgroup of individuals whose November 2007 turnout is changed by the studyâs experimental encouragement?â This query, along with the dataset and detailed metadata (excerpts below), is provided to the agent. 10 Strategy error Estimand error Control error Variable error Design-specific field error 0 20 40 60 80 100 % of Incorrect Cases Affected 16% 6% 72% 38% 12% 5% 97% 31% 13% 33%33% 80% 7% 67% 86% 4% 71% 11% Error Family Prevalence by Strategy DiD (n=32) RDD (n=39) IV (n=15) CE (n=28) Figure 1: Percentage of incorrect cases affected by each error family, grouped by gold strategy. Categories are non-exclusive: a single case may exhibit multiple error types simultaneously (49% of incorrect cases do). Each bar shows the fraction of that strategyâs incorrect cases containing the given error family. RDD is dominated by estimand errors (97%), CE by strategy errors (86%), and DiD, IV, and CE all show high rates of control errors (72%, 80%, 71%). First Few Rows of Provided CSV (excerpt): yob1 ,id ,precinct ,..., treatmen ,h,lastelec ,voted ,avhh0 ,color ,hhsize ,... 571129 ,00000026 ,506800006 ,... , control ,489243,,,,,1,... 160626 ,00000030 ,705804003 ,... , control ,934562,,,,,2,... 421017 ,00000066 ,638800041 ,... , control ,880582,,,,,1,... Partial Metadata: Variable Description treatmenMailing-group label (e.g., âcontrolâ, âmail-arm Aâ) votedBinary indicator for voting (0/1) hhsizeHousehold size (registered individuals) ...... Dataset Context: This dataset combines official voter-registration and voter-history records with household- level mailing assignments from a large mail program around municipal elections in 2007. . . Gold-Standard Identification. The correct identification for this query is an Instrumental Variable design. The treatment isvoted(voting in the November 2007 election), the outcome is voting in the January 2008 primary, and the instrument istreatmen(the randomized mailing assignment). The causal quantity isLATE, because the IV design identifies the effect for compliersâthose whose November 2007 turnout was 11 changed by the mailing encouragement. This example illustrates the level of reasoning required: the agent must recognize that the mailing assignment is a randomized encouragement (instrument) for the endogenous treatment (voting), and that the estimand is a LATE rather than an ATE. 9 Hosting and Maintenance CausalReasoningBenchmark is publicly available on Hugging Face Datasets. 2 All associated code for evaluation, including the evaluator script and the gold-standard estimation scripts, is available in the same repository. Licensing and Access. The benchmark metadata, evaluation code, and gold-standard identification specifications are released under the MIT license. The underlying datasets are redistributed under the terms of their original licenses; we provide attribution and licensing information for each dataset. Maintenance Plan. We plan to update the benchmark periodically: adding queries as new reanalysis studies appear, incorporating additional designs (synthetic control, event studies), and revising metrics in response to feedback. We welcome contributions from the community. 10 Limitations and Future Work CausalReasoningBenchmark has several limitations that we plan to address in future work. Domain Coverage. The research-paper subset is drawn entirely from political science, reflecting the availability of large-scale reanalysis studies in that field. While the textbook subset provides some cross- domain coverage, the benchmark would benefit from the inclusion of datasets from economics, epidemiology, and other fields where causal inference is central. Strategy Coverage. The current benchmark focuses on five identification strategies. Important designs such as synthetic control methods, event studies, and regression kink designs are not yet covered. We plan to expand the strategy coverage in future releases. Single Gold Standard. For each query, we provide a single gold-standard identification specification. In practice, there may be multiple defensible identification strategies for a given dataset and question. Future work could explore evaluation frameworks that accommodate multiple valid specifications. Estimation Sensitivity. The gold-standard estimates are produced by specific estimation scripts. Different but equally valid estimation choices (e.g., different bandwidth selectors for RDD, different standard error clustering for DiD) could produce different estimates. Scale. With 173 queries, CausalReasoningBenchmark is smaller than some existing benchmarks. However, we prioritize quality and depth of evaluation over quantity: each query requires a full identification specification, a python script to produce the causal estimate. We plan to expand the benchmark over time. 11 Conclusion CausalReasoningBenchmark provides a new, challenging, and realistic benchmark for evaluating automated causal reasoning systems. By separating the evaluation of identification and estimation, it offers a more nuanced view of model capabilities than existing benchmarks. Our baseline results demonstrate that state- of-the-art LLMs struggle with the detailed specification of causal research designs, even when they can correctly identify the broad design family. This finding highlights the need for more sophisticated reasoning capabilities in automated causal inference systems. We hope that CausalReasoningBenchmark will help with 2 https://huggingface.co/datasets/syrgkanislab/CausalReasoningBenchmark 12 the development of more robust and reliable AI systems for causal inference, and we welcome contributions from the research community. 12 Acknowledgement Vasilis Syrgkanis, Ayush Sawarni and Jiyuan Tan were supported by NSF Award IIS-2337916. References [1] Kenichi Ariga. Incumbency disadvantage under electoral rules with intraparty competition: Evidence from japan. The Journal of Politics, 2015. doi: 10.1086/681718. URL https://doi.org/10.1086/681718. [2]Jeroen Berrevoets, Julianna Piskorz, Robert Davis, Harry Amad, Jim Weatherall, and Mihaela van der Schaar. Technical report: Facilitating the adoption of causal inference methods through LLM-empowered co-pilot. arXiv preprint arXiv:2508.10581, 2025. doi: 10.48550/arXiv.2508.10581. [3] Graeme Blair, Darin Christensen, and Valerie Wirtschafter. How does armed conflict shape investment? evidence from the mining sector. The Journal of Politics, 2022. doi: 10.1086/715255. URLhttps: //doi.org/10.1086/715255. [4]Taylor C. Boas, F. Daniel Hidalgo, and Neal P. Richardson. The spoils of victory: Campaign donations and government contracts in brazil. The Journal of Politics, 2014. doi: 10.1017/s002238161300145x. URL https://doi.org/10.1017/s002238161300145x. [5]David E. Broockman and Timothy J. Ryan. Preaching to the choir: Americans prefer communicating to copartisan elected officials. American Journal of Political Science, 2015. doi: 10.1111/ajps.12228. URL https://doi.org/10.1111/ajps.12228. [6] Allison Carnegie and Nikolay Marinov. Foreign aid, human rights, and democracy promotion: Evidence from a natural experiment. American Journal of Political Science, 2017. doi: 10.1111/ajps.12289. URL https://doi.org/10.1111/ajps.12289. [7]Jamie L. Carson and Joel Sievert. Congressional candidates in the era of party ballots. The Journal of Politics, 2017. doi: 10.1086/688077. URL https://doi.org/10.1086/688077. [8]Devin Caughey, Christopher Warshaw, and Yiqing Xu. Incremental democracy: The policy effects of partisan control of state government. The Journal of Politics, 2017. doi: 10.1086/692669. URL https://doi.org/10.1086/692669. [9]Albert Chiu, Xingchen Lan, Ziyi Liu, and Yiqing Xu. Causal panel analysis under parallel trends: Lessons from a large reanalysis study. American Political Science Review, 120(1):245â266, 2026. doi: 10.1017/S0003055425000243. [10] Alberto Chong, Gianmarco Le Ěon-Ciliotta, Vivian Roza, Mart ĚÄąn Valdivia, and Gabriela Vega. Urbanization patterns, information diffusion, and female voting in rural paraguay. American Journal of Political Science, 2018. doi: 10.1111/ajps.12404. URL https://doi.org/10.1111/ajps.12404. [11] Darin Christensen and Francisco Garfias. The politics of property taxation: Fiscal infrastructure and electoral incentives in brazil. The Journal of Politics, 2021. doi: 10.1086/711902. URLhttps: //doi.org/10.1086/711902. [12]Joanie Hayoun Chung, Chaemyung Lim, Sumin Lee, Songseong Kim, and Sungbin Lim. ORCA: ORchestrating causal agent. arXiv preprint arXiv:2508.21304, 2025. doi: 10.48550/arXiv.2508.21304. [13]Andrew J. Clarke. Party sub-brands and american party factions. American Journal of Political Science, 2020. doi: 10.1111/ajps.12504. URL https://doi.org/10.1111/ajps.12504. 13 [14]Amanda Clayton and P Ěar Zetterberg. Quota shocks: Electoral gender quotas and government spending priorities worldwide. The Journal of Politics, 2018. doi: 10.1086/697251. URLhttps://doi.org/10. 1086/697251. [15]Alexander Coppock and Donald P. Green. Is voting habit forming? new evidence from experiments and regression discontinuities. American Journal of Political Science, 2015. doi: 10.1111/ajps.12210. URL https://doi.org/10.1111/ajps.12210. [16] Alexander Coppock and Donald P. Green. Is voting habit forming? new evidence from experiments and regression discontinuities. American Journal of Political Science, 2015. doi: 10.1111/ajps.12210. URL https://doi.org/10.1111/ajps.12210. [17] Alexander Coppock and Donald P. Green. Is voting habit forming? new evidence from experiments and regression discontinuities. American Journal of Political Science, 60(4):1044â1062, 2016. doi: 10.1111/ajps.12210. [18]Benjamin Hans Creutzfeldt. China y e. u. en latinoam Ěerica. Revista Cient ĚÄąfica General Jos Ěe Mar ĚÄąa C Ěordova, 2016. doi: 10.21830/19006586.1. URL https://doi.org/10.21830/19006586.1. [19]Kevin Croke, Guy Grossman, Horacio A. Larreguy, and John Marshall. Deliberate disengagement: How ed- ucation can decrease political participation in electoral authoritarian regimes. American Political Science Review, 2016. doi: 10.1017/s0003055416000253. URLhttps://doi.org/10.1017/s0003055416000253. [20]Scott Cunningham. Causal Inference: The Mixtape. Yale University Press, London, 2021. ISBN 9780300251685. URL https://mixtape.scunning.com/. [21]Carl Dahlstr Ěom and Mikael Holmgren. Loyal leaders, affluent agencies: The budgetary implications of political appointments in the executive branch. The Journal of Politics, 2023. doi: 10.1086/717756. URL https://doi.org/10.1086/717756. [22]Justin de Benedictis-Kessner. Off-cycle and out of office: Election timing and the incumbency advantage. The Journal of Politics, 2018. doi: 10.1086/694396. URL https://doi.org/10.1086/694396. [23]Greg Distelhorst and Richard M. Locke. Does compliance pay? social standards and firm-level trade, 2018. URL https://doi.org/10.31235/osf.io/tcrhq. [24]Paul Casta Ěneda Dower, Evgeny Finkel, Scott Gehlbach, and Steven Nafziger. Collective action and representation in autocracies: Evidence from russiaâs great reforms. American Political Science Review, 112(1):125â147, 2018. [25]Laurel Eckhouse. Metrics management and bureaucratic accountability: Evidence from policing. American Journal of Political Science, 2021. doi: 10.1111/ajps.12661. URLhttps://doi.org/10.1111/ajps. 12661. [26]Andrew C. Eggers and Jens Hainmueller. Mps for sale? returns to office in postwar british politics. American Political Science Review, 2009. doi: 10.1017/s0003055409990190. URLhttps://doi.org/10. 1017/s0003055409990190. [27] Andrew C. Eggers and Arthur Spirling. Incumbency effects and the strength of party preferences: Evidence from multiparty elections in the united kingdom. The Journal of Politics, 2017. doi: 10.1086/690617. URL https://doi.org/10.1086/690617. [28] Robert S. Erikson, Olle Folke, and James M. Snyder. A gubernatorial helping hand? how governors affect presidential elections. The Journal of Politics, 2015. doi: 10.1086/680186. URLhttps://doi. org/10.1086/680186. [29] Jane Esberg and Alexandra A. Siegel. How exile shapes online opposition: Evidence from venezuela. American Political Science Review, 2022. doi: 10.1017/s0003055422001290. URLhttps://doi.org/10. 1017/s0003055422001290. 14 [30]Jeremy Ferwerda and Nicholas L. Miller. Political devolution and resistance to foreign rule: A natural experiment. American Political Science Review, 2014. doi: 10.1017/s0003055414000240. URLhttps: //doi.org/10.1017/s0003055414000240. [31]Olle Folke and James M. Snyder. Gubernatorial midterm slumps. American Journal of Political Science, 2012. doi: 10.1111/j.1540-5907.2012.00599.x. URLhttps://doi.org/10.1111/j.1540-5907.2012. 00599.x. [32]Alexander Fouirnaies and Andrew B. Hall. The financial incumbency advantage: Causes and consequences. The Journal of Politics, 2014. doi: 10.1017/s0022381614000139. URLhttps://doi.org/10.1017/ s0022381614000139. [33]Adriane Fresh. The effect of the voting rights act on enfranchisement: Evidence from north carolina. The Journal of Politics, 2018. doi: 10.1086/697592. URL https://doi.org/10.1086/697592. [34] Francisco Garfias. Elite coalitions, limited government, and fiscal capacity development: Evidence from bourbon mexico. The Journal of Politics, 2019. doi: 10.1086/700105. URLhttps://doi.org/10.1086/ 700105. [35]Alan S. Gerber, Gregory A. Huber, and Ebonya Washington. Party affiliation, partisanship, and political beliefs: A field experiment. American Political Science Review, 2010. doi: 10.1017/s0003055410000407. URL https://doi.org/10.1017/s0003055410000407. [36]Jacob M. Grumbach. Laboratories of democratic backsliding. American Political Science Review, 2022. doi: 10.1017/s0003055422000934. URL https://doi.org/10.1017/s0003055422000934. [37]Jacob M. Grumbach and Charlotte Hill. Rock the registration: Same day registration increases turnout of young voters. The Journal of Politics, 2022. doi: 10.1086/714776. URLhttps://doi.org/10.1086/ 714776. [38]Jacob M. Grumbach and Alexander Sahn. Race and representation in campaign finance. American Political Science Review, 2019. doi: 10.1017/s0003055419000637. URLhttps://doi.org/10.1017/ s0003055419000637. [39] Laurenz Guenther. Correcting misperceptions can increase anti-immigration attitudes, 2024. URL https://doi.org/10.2139/ssrn.5001788. [40] Jens Hainmueller and Dominik Hangartner. Does direct democracy hurt immigrant minorities? evidence from naturalization decisions in switzerland. SSRN Electronic Journal, 2014. doi: 10.2139/ssrn.2503141. URL https://doi.org/10.2139/ssrn.2503141. [41]Andrew B. Hall. What happens when extremists win primaries? American Political Science Review, 2015. doi: 10.1017/s0003055414000641. URL https://doi.org/10.1017/s0003055414000641. [42]Andrew B. Hall and Daniel M. Thompson. Who punishes extremist nominees? candidate ideology and turning out the base in us elections. American Political Science Review, 2018. doi: 10.1017/ s0003055418000023. URL https://doi.org/10.1017/s0003055418000023. [43] Michael Hankinson and Asya Magazinnik. The supply-equity trade-off: The effect of spatial representation on the local housing supply. The Journal of Politics, 2023. doi: 10.1086/723818. URLhttps: //doi.org/10.1086/723818. [44]Andrew Healy and Neil Malhotra. Childhood socialization and political attitudes: Evidence from a natural experiment. The Journal of Politics, 2013. doi: 10.1017/s0022381613000996. URLhttps: //doi.org/10.1017/s0022381613000996. [45] Miguel A. Hern Ěan and James M. Robins. Causal Inference: What If. Chapman & Hall/CRC, Boca Raton, 2020. URL https://miguelhernan.org/whatifbook. 15 [46]F. Daniel Hidalgo and Simeon Nichter. Voter buying: Shaping the electorate through clientelism. American Journal of Political Science, 2015. doi: 10.1111/ajps.12214. URLhttps://doi.org/10.1111/ ajps.12214. [47]Shigeo Hirano, Jaclyn Kaslovsky, Michael P. Olson, and James M. Snyder. The growth of campaign advertising in the united states, 1880â1930. The Journal of Politics, 2022. doi: 10.1086/719008. URL https://doi.org/10.1086/719008. [48] John B. Holbein and D. Sunshine Hillygus. Making young voters: The impact of preregistration on youth turnout. American Journal of Political Science, 2015. doi: 10.1111/ajps.12177. URL https://doi.org/10.1111/ajps.12177. [49] Nick Huntington-Klein. The Effect: An Introduction to Research Design and Causality. CRC Press, Taylor & Francis Group, Boca Raton, 2022. ISBN 9781032125787. [50] Nick Huntington-Klein and Malcolm Barrett. causaldata: Example data sets for causal inference textbooks, 2021. URL https://github. com/nickch-k/causaldata. R package version 0.1, 4. [51] Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Sch Ěolkopf. Can large language models infer causation from correlation? arXiv preprint arXiv:2306.05836, 2023. doi: 10.48550/arXiv.2306.05836. [52] Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Sch Ěolkopf. CLadder: Assessing causal reasoning in language models. In Advances in Neural Information Processing Systems, volume 36, pages 31038â31065, 2023. [53]Mitchell Kilborn and Arjun Vishwanath. Public money talks too: How public campaign financing degrades representation. American Journal of Political Science, 2021. doi: 10.1111/ajps.12625. URL https://doi.org/10.1111/ajps.12625. [54]Jeong Hyun Kim. Direct democracy and womenâs political engagement. American Journal of Political Science, 63(3):594â610, 2019. [55]Marko KlaËsnja and Roc ĚÄąo Titiunik. The incumbency curse: Weak parties, term limits, and unfulfilled accountability. American Political Science Review, 2017. doi: 10.1017/s0003055416000575. URL https://doi.org/10.1017/s0003055416000575. [56] Mary Kroeger and Maria Silfa. Motivated corporate political action: Evidence from an sec experiment. The Journal of Politics, 2023. doi: 10.1086/723998. URL https://doi.org/10.1086/723998. [57]Nicholas Kuipers and Alexander Sahn. The representational consequences of municipal civil service reform. American Political Science Review, 2022. doi: 10.1017/s0003055422000521. URLhttps: //doi.org/10.1017/s0003055422000521. [58] Apoorva Lal, Mackenzie Lockhart, Yiqing Xu, and Ziwen Zu. How much should we trust instrumental variable estimates in political science? practical advice based on 67 replicated studies. Political Analysis, 32(4):521â540, 2024. doi: 10.1017/pan.2024.2. [59] Alan Lambert, Fade Eadeh, and Emily Hanson. Anger and its consequences for judgment and behavior: Recent developments in social and political psychology, 2018. URLhttps://doi.org/10.31234/osf. io/svcux_v1. [60]Audrey Latura and Ana Catalano Weeks. Corporate board quotas and gender equality policies in the workplace. American Journal of Political Science, 2022. doi: 10.1111/ajps.12709. URLhttps: //doi.org/10.1111/ajps.12709. [61] Donggyu Lee, Sungwon Park, Yerin Hwang, Hyoshin Kim, Hyunwoo Oh, Jungwon Kim, Meeyoung Cha, Sangyoon Park, and Jihee Kim. Benchmarking LLM causal reasoning with scientifically validated relationships. arXiv preprint arXiv:2510.07231, 2025. doi: 10.48550/arXiv.2510.07231. 16 [62]Yphtach Lelkes, Gaurav Sood, and Shanto Iyengar. The hostile audience: The effect of access to broadband internet on partisan affect. American Journal of Political Science, 2015. doi: 10.1111/ajps.12237. URL https://doi.org/10.1111/ajps.12237. [63]Amy E. Lerman and Katherine T. McCabe. Personal experience and public opinion: A theory and test of conditional policy feedback. The Journal of Politics, 2017. doi: 10.1086/689286. URLhttps: //doi.org/10.1086/689286. [64]Steven Liao. The effect of firm lobbying on high-skilled visa adjudication. The Journal of Politics, 2023. doi: 10.1086/723984. URL https://doi.org/10.1086/723984. [65]Zeqi Liu, Ke Li, Yu Cheng, Lichao Xue, Xuhui Fan, Yue Chen, Aobo Yang, Kun Ma, Zhiyuan Zhao, Peng Jiang, Yuxiang Zhou, Hao Wang, Jianxing Yu, Qian Zhang, Yang Liu, and Yangfeng Ji. Are LLMs capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9215â9235, 2024. doi: 10.18653/v1/2024.findings-acl.548. [66]Beatriz Magaloni, Edgar Franco-Vivanco, and Vanessa Melo. Killing in the slums: Social order, criminal governance, and police violence in rio de janeiro. American Political Science Review, 2020. doi: 10.1017/s0003055419000856. URL https://doi.org/10.1017/s0003055419000856. [67]Wayde Z. C. Marsh. Trauma and turnout: The political consequences of traumatic events. American Political Science Review, 2022. doi: 10.1017/s0003055422001010. URLhttps://doi.org/10.1017/ s0003055422001010. [68] Gwyneth H. McClendon. Social esteem and participation in contentious politics: A field experiment at an lgbt pride rally. American Journal of Political Science, 2013. doi: 10.1111/ajps.12076. URL https://doi.org/10.1111/ajps.12076. [69]Michael McDevitt and Steven Chaffee. From top-down to trickle-up influence: Revisiting assump- tions about the family in political socialization. Political Communication, 2002. doi: 10.1080/ 01957470290055501. URL https://doi.org/10.1080/01957470290055501. [70]Marc Meredith. Exploiting friends-and-neighbors to estimate coattail effects. American Political Science Review, 2013. doi: 10.1017/s0003055413000439. URLhttps://doi.org/10.1017/s0003055413000439. [71]Gareth Nellis and Niloufer Siddiqui. Secular party rule and religious violence in pakistan. American Political Science Review, 2017. doi: 10.1017/s0003055417000491. URLhttps://doi.org/10.1017/ s0003055417000491. [72] Lucas M. Novaes. Disloyal brokers and weak parties. American Journal of Political Science, 2017. doi: 10.1111/ajps.12331. URL https://doi.org/10.1111/ajps.12331. [73]Ana L. De La O. Do conditional cash transfers affect electoral behavior? evidence from a randomized experiment in mexico. American Journal of Political Science, 2012. doi: 10.1111/j.1540-5907.2012.00617.x. URL https://doi.org/10.1111/j.1540-5907.2012.00617.x. [74] Agustina S. Paglayan. Education or indoctrination? the violent origins of public school systems in an era of state-building. American Political Science Review, 2022. doi: 10.1017/s0003055422000247. URL https://doi.org/10.1017/s0003055422000247. [75]Maxwell Palmer and Benjamin Schneer. Capitol gains: The returns to elected office from corporate board directorships. The Journal of Politics, 2016. doi: 10.1086/683206. URLhttps://doi.org/10. 1086/683206. [76]Julia A. Payson. The partisan logic of city mobilization: Evidence from state lobbying disclosures. American Political Science Review, 2020. doi: 10.1017/s0003055420000118. URLhttps://doi.org/10. 1017/s0003055420000118. 17 [77]Jan H. Pierskalla and Audrey Sacks. Unpaved road ahead: The consequences of election cycles for capital expenditures. The Journal of Politics, 2018. doi: 10.1086/694547. URLhttps://doi.org/10. 1086/694547. [78]Nico Ravanilla, Renard Sexton, and Dotan Haim. Deadly populism: How local political outsiders drive duterteâs war on drugs in the philippines. The Journal of Politics, 2022. doi: 10.1086/715257. URL https://doi.org/10.1086/715257. [79] Miguel R. Rueda. Small aggregates, big manipulation: Vote buying enforcement and collective monitoring. American Journal of Political Science, 2016. doi: 10.1111/ajps.12260. URLhttps://doi.org/10.1111/ ajps.12260. [80]Jerome Schafer, Enrico Cantoni, Giorgio Bellettini, and Carlotta Berti Ceroni. Making unequal democracy work? the effects of income on voter turnout in northern italy. American Journal of Political Science, 66 (3):745â761, 2022. [81]Eric Schickler, Kathryn Pearson, and Brian D. Feinstein. Congressional parties and civil rights politics from 1933 to 1972. The Journal of Politics, 2010. doi: 10.1017/s0022381610000095. URLhttps: //doi.org/10.1017/s0022381610000095. [82] Sophie Schuit and Jon C Rogowski. Race, representation, and the voting rights act. American Journal of Political Science, 61(3):513â526, 2017. [83]Ishaan Sheth, Zheng Yuan, Keyi Fu, et al. CausalGraph2LLM: Evaluating LLMs for causal queries. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2076â2098, 2025. doi: 10.18653/v1/2025.findings-naacl.110. [84]Leah C. Stokes. Electoral backlash against climate policy: A natural experiment on retrospective voting and local resistance to public policy. American Journal of Political Science, 2015. doi: 10.1111/ajps.12220. URL https://doi.org/10.1111/ajps.12220. [85]Drew Stommes, Peter M. Aronow, and Fredrik S Ěavje. On the reliability of published findings using the regression discontinuity design in political science. Research & Politics, 10(2):20531680231166457, 2023. doi: 10.1177/20531680231166457. [86]David Szakonyi. Businesspeople in elected office: Identifying private benefits from firm-level returns. SSRN Electronic Journal, 2016. doi: 10.2139/ssrn.2844901. URL https://doi.org/10.2139/ssrn.2844901. [87]Anjali Thomas. Targeting ordinary voters or political elites? why pork is distributed along partisan lines in india. American Journal of Political Science, 2018. doi: 10.1111/ajps.12374. URLhttps: //doi.org/10.1111/ajps.12374. [88] Jessica Trounstine. The geography of inequality: How land use regulation produces segregation. American Political Science Review, 2020. doi: 10.1017/s0003055419000844. URLhttps://doi.org/10.1017/ s0003055419000844. [89]Wei Xu, Yuncheng Zhang, Rui Guo, Xiaowei Wang, Qian Liu, Xiaoxiao Li, et al. MRAgent: an LLM-based automated agent for causal knowledge discovery in disease via mendelian randomization. Briefings in Bioinformatics, 26(2):bbaf140, 2025. doi: 10.1093/bib/bbaf140. [90] Qi Zhang, Dong Zhang, Mingxing Liu, and Victor Shih. Elite cleavage and the rise of capitalism under authoritarianism: A tale of two provinces in china. The Journal of Politics, 2021. doi: 10.1086/711131. URL https://doi.org/10.1086/711131. [91] Yuan Zhou, Zifan Wang, Chenyang Gao, Xiaocheng Li, Junfeng Lou, Bo Li, and Jian Tang. Causalbench: A comprehensive benchmark for causal learning capability of LLMs. arXiv preprint arXiv:2404.06349, 2024. doi: 10.48550/arXiv.2404.06349. 18 A Paper list and citations Table 8: Research papers included in the benchmark. Each row corre- sponds to one paper-sourced dataset; some papers contribute two queries. PaperDesignTitle#queries [60]DiDCorporate Board Quotas and Gender Equality Policies in the Workplace 2 [78]DiDDeadly Populism: How Local Political Outsiders Drive Duterteâs War on Drugs in the Philippines 2 [23]DiDDoes Compliance Pay? Social Standards and Firm-level Trade2 [40]DiDDoes Direct Democracy Hurt Immigrant Minorities? Evidence from Naturalization Decisions in Switzerland 2 [74]DiD Education or Indoctrination? The Violent Origins of Public School Systems in an Era of State-Building 2 [90]DiD Elite Cleavage and the Rise of Capitalism under Authoritari- anism: A Tale of Two Provinces in China 2 [34]DiDElite Coalitions, Limited Government, and Fiscal Capacity Development: Evidence from Bourbon Mexico 2 [3]DiD How Does Armed Conflict Shape Investment? Evidence from the Mining Sector 2 [29]DiDHow Exile Shapes Online Opposition: Evidence from Venezuela2 [8]DiDIncremental Democracy: The Policy Effects of Partisan Control of State Government 2 [66]DiDKilling in the Slums: Social Order, Criminal Governance, and Police Violence in Rio de Janeiro 2 [36]DiDLaboratories of Democratic Backsliding2 [21]DiDLoyal Leaders, Affluent Agencies: The Budgetary Implications of Political Appointments in the Executive Branch 2 [80]DiDMaking unequal democracy work? The effects of income on voter turnout in Northern Italy 2 [25]DiDMetrics Management and Bureaucratic Accountability: Evi- dence from Policing 2 [56]DiDMotivated Corporate Political Action: Evidence from an SEC Experiment 2 [13]DiDParty Sub-Brands and American Party Factions1 [53]DiDPublic Money Talks Too: How Public Campaign Financing Degrades Representation 2 [14]DiDQuota Shocks: Electoral Gender Quotas and Government Spending Priorities Worldwide 2 [38]DiDRace and Representation in Campaign Finance2 [82]DiDRace, representation, and the voting rights act2 [37]DiDRock the Registration: Same Day Registration Increases Turnout of Young Voters 2 [64]DiDThe Effect of Firm Lobbying on High-Skilled Visa Adjudication2 [33]DiDThe Effect of the Voting Rights Act on Enfranchisement: Evi- dence from North Carolina 2 [88]DiDThe Geography of Inequality: How Land Use Regulation Pro- duces Segregation 2 [47]DiDThe Growth of Campaign Advertising in the United States, 1880â1930 2 [76]DiDThe Partisan Logic of City Mobilization: Evidence from State Lobbying Disclosures 2 [11]DiD The Politics of Property Taxation: Fiscal Infrastructure and Electoral Incentives in Brazil 2 [57]DiDThe Representational Consequences of Municipal Civil Service Reform 2 19 PaperDesignTitle#queries [43]DiDThe Supply-Equity Trade-Off: The Effect of Spatial Represen- tation on the Local Housing Supply 2 [67]DiD Trauma and Turnout: The Political Consequences of Traumatic Events 1 [77]DiD Unpaved Road Ahead: The Consequences of Election Cycles for Capital Expenditures 2 [28]RDDA Gubernatorial Helping Hand? How Governors Affect Presi- dential Elections 2 [86]RDDBusinesspeople in Elected Office: Identifying Private Benefits from Firm-Level Returns 1 [75]RDDCapitol Gains: The Returns to Elected Office from Corporate Board Directorships 2 [7]RDDCongressional Candidates in the Era of Party Ballots1 [81]RDDCongressional Parties and Civil Rights Politics from 1933 to 1972 4 [39]RDD Correcting Misperceptions Can Increase Anti-Immigration At- titudes 2 [54]RDDDirect democracy and womenâs political engagement1 [72]RDDDisloyal Brokers and Weak Parties1 [69]RDD From Top-Down to Trickle-Up Influence: Revisiting Assump- tions About the Family in Political Socialization 4 [31]RDDGubernatorial Midterm Slumps1 [8]RDDIncremental Democracy: The Policy Effects of Partisan Control of State Government 1 [1]RDDIncumbency Disadvantage under Electoral Rules with Intra- party Competition: Evidence from Japan 2 [27]RDDIncumbency Effects and the Strength of Party Preferences: Evidence from Multiparty Elections in the United Kingdom 1 [16]RDDIs Voting Habit Forming? New Evidence from Experiments and Regression Discontinuities 1 [48]RDD Making Young Voters: The Impact of Preregistration on Youth Turnout 1 [26]RDDMPs for Sale? Returns to Office in Postwar British Politics1 [22]RDDOff-Cycle and Out of Office: Election Timing and the Incum- bency Advantage 2 [30]RDDPolitical Devolution and Resistance to Foreign Rule: A Natural Experiment 1 [5]RDDPreaching to the Choir: Americans Prefer Communicating to Copartisan Elected Officials 1 [87]RDDTargeting Ordinary Voters or Political Elites? Why Pork Is Distributed Along Partisan Lines in India 1 [32]RDDThe Financial Incumbency Advantage: Causes and Conse- quences 2 [55]RDD The Incumbency Curse: Weak Parties, Term Limits, and Unfulfilled Accountability 1 [4]RDDThe Spoils of Victory: Campaign Donations and Government Contracts in Brazil 1 [46]RDDVoter Buying: Shaping the Electorate through Clientelism1 [41]RDDWhat Happens When Extremists Win Primaries?2 [42]RDDWho Punishes Extremist Nominees? Candidate Ideology and Turning Out the Base in US Elections 1 [59]IVAnger and its Consequences for Judgment and Behavior: Re- cent Developments in Social and Political Psychology 1 [44]IV Childhood Socialization and Political Attitudes: Evidence from a Natural Experiment 1 [18]IVChina y E. U. en Latinoam Ěerica1 20 PaperDesignTitle#queries [24]IVCollective action and representation in autocracies: Evidence from Russiaâs great reforms 1 [24]IV Collective action and representation in autocracies: Evidence from Russiaâs great reforms 1 [19]IV Deliberate Disengagement: How Education Can Decrease Po- litical Participation in Electoral Authoritarian Regimes 1 [73]IVDo Conditional Cash Transfers Affect Electoral Behavior? Evidence from a Randomized Experiment in Mexico 1 [84]IVElectoral Backlash against Climate Policy: A Natural Experi- ment on Retrospective Voting and Local Resistance to Public Policy 1 [70]IVExploiting Friends-and-Neighbors to Estimate Coattail Effects1 [6]IV Foreign Aid, Human Rights, and Democracy Promotion: Evi- dence from a Natural Experiment 1 [15]IVIs Voting Habit Forming? New Evidence from Experiments and Regression Discontinuities 1 [35]IV Party Affiliation, Partisanship, and Political Beliefs: A Field Experiment 1 [63]IV Personal Experience and Public Opinion: A Theory and Test of Conditional Policy Feedback 1 [71]IVSecular Party Rule and Religious Violence in Pakistan1 [79]IVSmall Aggregates, Big Manipulation: Vote Buying Enforce- ment and Collective Monitoring 1 [68]IVSocial Esteem and Participation in Contentious Politics: A Field Experiment at an LGBT Pride Rally 1 [62]IVThe Hostile Audience: The Effect of Access to Broadband Internet on Partisan Affect 2 [57]IVThe Representational Consequences of Municipal Civil Service Reform 1 [10]IV Urbanization Patterns, Information Diffusion, and Female Voting in Rural Paraguay 1 21