Paper deep dive
Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS
Spandan Ghose Chowdhury
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 9/24/2026, 5:05:38 AM
Summary
This paper investigates the gap between planned and achieved statistical power in quasi-experimental designs (Difference-in-Differences and Instrumental Variables) used in Information Systems research. Through a large-scale Monte Carlo simulation (9,837 parameter conditions), the authors find that while serial correlation effects can be recovered using AR(1)-aware calculators, other factors like panel attrition, staggered adoption bias, and parallel-trends pretesting are not captured by closed-form formulas. Exogenous attrition significantly reduces power (8-11 percentage points), while treatment-correlated attrition induces bias. For IV, identification relies on instrument strength rather than sample size.
Entities (15)
Relation Signals (9)
Panel Attrition ā causes ā Power Loss
confidence 92% Ā· exogenous attrition alone costs approx 8 to 11 percentage points at the few-hundred-to-thousand sample sizes IS studies use.
Treatment-Correlated Attrition ā induces ā Bias
confidence 90% Ā· Treatment-correlated, outcome-dependent attrition instead induces bias, not just power loss.
Serial Correlation ā isrecoverableby ā AR(1)
confidence 90% Ā· The serial-correlation component is recoverable by an AR(1)-aware calculator when rho is known
Sample Size (N) ā doesnotimprove ā IV Power
confidence 88% Ā· For IV, holding first-stage F fixed, larger N neither raises power nor curbs exclusion bias
Staggered Adoption ā causes ā Bias
confidence 87% Ā· panel attrition, staggered-adoption bias, and parallel-trends pretesting are captured by no closed-form formula
Parallel-Trends Pretesting ā causes ā Inference Issues
confidence 85% Ā· parallel-trends pretesting are captured by no closed-form formula
Instrument Strength ā determines ā IV Identification
confidence 85% Ā· identification rests on instrument strength, not sample size.
Two-Way Fixed Effects ā isbenchmarkedagainst ā Sun and Abraham
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Information systems (IS) researchers increasingly use quasi-experimental methods such as difference-in-differences (DiD) and instrumental variables (IV) to recover causal effects from observational panel data. Power calculations that justify these designs assume i.i.d. errors, but the deeper problem is what even a cluster-robust calculator cannot see. We report a Monte Carlo study over 9837 parameter conditions (approx 9.8 million datasets) and decompose the planned-versus-achieved power gap. The serial-correlation component is recoverable by an AR(1)-aware calculator when rho is known, and partially when rho must be estimated from short pre-periods, but panel attrition, staggered-adoption bias, and parallel-trends pretesting are captured by no closed-form formula; exogenous attrition alone costs approx 8 to 11 percentage points at the few-hundred-to-thousand sample sizes IS studies use. Treatment-correlated, outcome-dependent attrition instead induces bias, not just power loss. For IV, holding first-stage F fixed, larger N neither raises power nor curbs exclusion bias, though with a fixed instrument more data does sharpen the first stage, so identification rests on instrument strength, not sample size.
Tags
Links
- Source: https://arxiv.org/abs/2609.27299v1
- Canonical: https://arxiv.org/abs/2609.27299v1
Trouble viewing inline? Open PDF directly ā
Full Text
49,702 characters extracted from source content.
Expand or collapse full text
Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS Spandan Ghose Chowdhury Affiliation: Georgia Institute of Technology, GA, USA Email: spandan_gc@gatech.edu Abstract Information systems (IS) researchers increasingly use quasi-experimental methods such as difference-in-differences (DiD) and instrumental variables (IV) to recover causal effects from observational panel data. Power calculations that justify these designs assume i.i.d. errors, but the deeper problem is what even a cluster-robust calculator cannot see. We report a Monte Carlo study over 9,837 parameter conditions (ā 9.8 million datasets) and decompose the planned-versus-achieved power gap. The serial-correlation component is recoverable by an AR(1)-aware calculator when Ļ is known, and partially when Ļ must be estimated from short pre-periods; but panel attrition, staggered-adoption bias, and parallel-trends pretesting are captured by no closed-form formula; exogenous attrition alone costs ā 8ā11 percentage points at the few-hundred-to-thousand sample sizes IS studies use. Treatment-correlated, outcome-dependent attrition instead induces bias, not just power loss. For IV, holding first-stage F fixed, larger N neither raises power nor curbs exclusion biasāthough with a fixed instrument more data does sharpen the first stageāso identification rests on instrument strength, not sample size. We provide a unified DiD+IV benchmark calibrated to IS effect sizes, a companion lookup tool, and design guidance for observational causal work in IS: https://github.com/spandan1305/is-power-lookup Keywords: Power analysis, difference-in-differences, instrumental variables, Monte Carlo simulation, research methodology. 1 Introduction Causal designs now occupy a central position in Information Systems (IS) empirical research (Tafti and Shmueli, 2025), particularly through difference-in-differences (DiD) and, where endogeneity concerns are stronger, instrumental variables (IV). Yet ex ante design justification often remains anchored in power calculations derived from i.i.d. settingsāpoorly aligned with IS panels, where outcomes are serially correlated, attrition unbalances the panel, treatment timing is staggered, and effects are heterogeneous. Our aim is not to re-establish that i.i.d. calculators are optimisticāthat much is well known. We ask instead how much of the planned-versus-achieved power gap survives a competent, cluster-robust, AR(1)-aware calculator, and which frictions remain invisible to any closed-form formula. First, once the recoverable serial-correlation component is removed, how much power loss persists from panel attrition and staggered timing? Second, what is the inferential consequence of conditioning DiD analysis on pretrend-test non-rejection (Roth, 2022)? Third, how do weak-instrument and exclusion-restriction failures in IV differ as sample size increases? Several methodological studies provide partial answers. Bertrand et al. (2004) document serial-correlation distortions in DiD inference; Roth (2022) formalizes the conditional-inference problem in pretrend testing; and Goodman-Bacon (2021), de Chaisemartin and DāHaultfÅuille (2020), and Sun and Abraham (2021) characterize TWFE bias under staggered timing. What is missing is a joint quantification that treats panel frictions, estimator choice, conditional inference from pretesting, and IV identification failures as components of a single design problem, evaluated over an IS-calibrated parameter space with consistent metrics. The individual mechanisms we study are established in econometricsāserial-correlation distortion (Bertrand et al., 2004), pretest selection (Roth, 2022), TWFE bias under staggered timing (Goodman-Bacon, 2021), and the weak-versus-exclusion IV trade-off. Our contribution is to quantify their joint magnitude in IS-typical panels and package that evidence for practice. First, we provide a unified Monte Carlo benchmark covering both DiD and IV under one IS-realistic data-generating process (DGP), reporting power, bias, root-mean-squared error, and confidence interval coverage over 9,837 parameter conditions. Second, we calibrate the effect-size grid to documented IS effect magnitudes (Table 1), validate the simulation against canonical open datasets, and release a companion lookup tool over the full grid; the tool matches categorical inputs exactly, snaps numeric inputs to the nearest simulated grid value, and reports the matched design so users can judge how far the returned cell sits from their target. Third, we document an asymmetry with direct bearing on IV practice: weak-instrument bias attenuates with N, while exclusion bias is not cured by sample size, so identification rests on instrument strength rather than N. We translate these results into conditional design guidance rather than a universal checklist. 2 Background In contemporary IS empirical work, two-way fixed effects (TWFE) DiD remains the default framework for policy rollouts, platform interventions, and staged feature launches. In the canonical model Yiāt=Ī±i+Ī»t+Ī“āDiāt+ĪµiātY_it= _i+ _t+Ī“ D_it+ _it, identification of Ī“ rests on parallel trends, typically assessed with event-study plots and joint tests on pre-treatment coefficientsādiagnostics that are themselves finite-sample objects and noisy in observational panels. Once timing is staggered and effects are heterogeneous, TWFE no longer has a straightforward ATT interpretation: it combines multiple 2x2 comparisons, including already-treated vs. not-yet-treated contrasts that can receive negative weight and induce bias (Goodman-Bacon, 2021; de Chaisemartin and DāHaultfÅuille, 2020). We therefore benchmark TWFE against alternatives that re-center identification at the group-time level, notably Sun and Abraham (2021) and Callaway and SantāAnna (2021); for a recent systematic comparison see Ulloa-PĆ©rez et al. (2025). Pretrend testing introduces a second inferential issue: Roth (2022) shows that conditioning interpretation on jointly insignificant pre-treatment coefficients induces selection bias, because samples that pass are those in which sampling noise masks true trend differences. Conditional error rates can then exceed unconditional ones, especially when pre-periods are short and noisy ā conditions common in IS panels. IV designs face a related dual-identification challenge: relevance and exclusion. In the just-identified case, Ī²^IāV=(Zā²āX)ā1āZā²āY Ī²_IV=(Z X)^-1Z Y, relevance is often summarized through first-stage diagnostics such as Stock and Yogo (2005), while exclusion requires that Z affect Y only through X. IS instruments are often geographic, temporal, or peer-exposure based and plausibly vulnerable to contextual spillovers (Semadeni et al., 2014), making first-stage strength and exclusion plausibility distinct empirical concerns. Because serially dependent, attrition-thinned panels reduce effective information relative to nominal row counts (Bertrand et al., 2004), our simulation treats serial dependence, attrition, and clustering as first-order design inputs rather than implementation details. Beyond SunāAbraham and CallawayāSantāAnna, the DiD literature has expanded to include imputation-based approaches (Borusyak et al., 2024), two-stage DiD (Gardner, 2022), panel counterfactual methods rooted in matrix completion and synthetic-control logic (Athey et al., 2021; Abadie, 2021), and causal machine learning (Chernozhukov et al., 2018; Wager and Athey, 2018; von Zahn et al., 2026). We do not implement these classes; the present grid is a benchmark against which they can be compared in follow-up work. Finally, we extend simulation-based design diagnostics in management and IS (Semadeni et al., 2014) by jointly analyzing DiD and IV on one panel grid that models serial correlation and attrition and quantifies both estimator performance and inference calibration. 3 Simulation Design 3.1 Data-Generating Process The DiD DGP generates Yiāt=Ī±i+Ī»t+Ī“āDiāt+ĪµiātY_it= _i+ _t+Ī“ D_it+ _it with Īµiāt=ĻāĪµi,tā1+uiāt _it=Ļ _i,t-1+u_it, uiātā¼ā”(0,Ļ2ā(1āĻ2))u_it (0,Ļ^2(1-Ļ^2)), Ī±iā¼ā”(0,1) _i (0,1), and Ī»t=0.5āt/T _t=0.5\,t/T. To span the design space most frequently encountered in IS panels, we vary nine inputs: sample size (n_units ā50,100,200,500,1000,5000ā\50,100,200,500,1000,5000\), panel length (n_periods ā6,10,16,24ā\6,10,16,24\), pre-period share (pre_frac ā0.33,0.50ā\0.33,0.50\), standardized effect magnitude (effect_size ā0.1,0.2,0.5ā\0.1,0.2,0.5\ SD), serial persistence (autocorr Ļā0,0.3,0.5,0.7Ļā\0,0.3,0.5,0.7\), adoption timing (staggered āFalse,Trueā\False,True\), treatment-effect heterogeneity (het_effects, with early cohorts receiving 2āĪ“2Ī“ and later cohorts Ī“), exogenous attrition (churn_rate ā0,0.3ā\0,0.3\), and covariance structure for inference (cluster_type āunit,two-wayā\unit,two-way\). After removing infeasible cells, this yields 6,9126,912 DiD conditions. (het_effects is defined only under staggered adoption, which removes one quarter of the 9,2169,216 nominal cells.) We additionally run a 45-cell pretrend sub-study and a 2,8802,880-cell IV grid that varies first-stage strength F, endogeneity Ļvāu _vu, sample size, instrument count, and a direct ZāYZā Y channel. Crucially, the grid fixes the target first-stage F rather than the first-stage coefficient, so the instrument strength is set to Ļ=F/(N+F)Ļ= F/(N+F), which declines as Nā1/2N^-1/2; this normalization drives the IV scaling result below. 3.2 Estimators For DiD, we estimate (i) TWFE under unit-clustered or two-way clustered covariance, (i) the interaction-weighted event-study estimator of Sun and Abraham (2021), and (i) cohort-specific ATT aggregation following Callaway and SantāAnna (2021). Pretrend diagnostics use a joint Wald test over all pre-treatment event-time coefficients in a two-way FE event-study regression. For IV, we estimate just-identified and over-identified (three-instrument) 2SLS with heteroskedasticity-robust standard errors. To calibrate the standardized effect-size grid to effects plausibly encountered in IS research, Table 1 maps each range to a representative study type. The anchors come from a structured (though not systematic) review of published IS quasi-experiments and large-scale digital experiments, combined with domain judgment: massive online field experiments report standardized effects at the low end (Aral and Walker, 2011), platform- and policy-scale interventions on short persistent panels fall in the middle ranges (Aziz et al., 2023; Huang et al., 2017; Seamans and Zhu, 2014), and effects above 0.50.5 SD are rare in credible causal designs (Gelman and Carlin, 2014). Researchers with better priors should query the lookup tool at those values instead. Effect (SD) IS example 0.05ā0.10 UX A/B test on conversion (button, copy) 0.10ā0.20 Recommender change on click-through 0.20ā0.35 Platform policy change (pricing, matching) 0.35ā0.50 Major ERP/system implementation on firm productivity >0.50>0.50 Rarely observed in credible causal designs Table 1: IS effect-size anchors: illustrative calibration points from a structured review plus domain judgment (see text), not meta-analytic estimates. 3.3 Performance Metrics For each design cell, we run Nsim=1,000N_sim=1,000 independent Monte Carlo replications (500 in the dedicated pretrend exercise) and report empirical power (P^(reject H0:Ī²=0) P(reject H_0:Ī²=0)), bias (ā”[Ī²^āĪ²]E[ Ī²-Ī²]), RMSE, and 95% confidence-interval coverage. In the pretrend module we additionally compute the unconditional versus conditional rejection gap emphasized by Roth (2022). The full experiment covers 9,8379,837 parameter conditions (6,9126,912 DiD ++ 4545 pretrend ++ 2,8802,880 IV cells) and approximately 9.8 million simulated datasets. Three supplementary sub-studiesāa 48-cell attrition grid (Section 4.2), a 500-pilot-per-cell estimated-Ļ analysis (Table 4), and a ten-cell reference-estimator validation (Section 4.1)ālie outside this count and are described where used. 3.4 Validation Before launching the full grid, we validate implementation fidelity using five unit tests summarized in Table 2. All pass within their pre-registered tolerances. Test Target Achieved Tol. High-power recovery ā„0.90ā„ 0.90 0.94 Ā±0.03Ā± 0.03 Type I (Ī²=0Ī²=0) ā0.05ā 0.05 0.047 Ā±0.01Ā± 0.01 95% CI coverage ā0.95ā 0.95 0.955 Ā±0.01Ā± 0.01 Churn symmetry diff <0.02<0.02 0.008 ā AR(1) Ļ recovery |Ļ^āĻ0|<0.05| Ļ- _0|<0.05 0.023 ā Table 2: Pre-flight validation results against five implementation-fidelity targets. Each test uses 1,000 replications. 4 DiD Results 4.1 Sample-Size Scaling Under Serial Correlation and Staggered Adoption Two features of panel data drive a wedge between the power classical formulas predict and the power TWFE delivers: within-unit serial correlation, and the interaction between staggered timing and heterogeneous treatment effects. Panels (a) and (b) of Figure 1 summarize both. Panel (a) reports TWFE power over N and residual autocorrelation Ļ, holding effect_size =0.2=0.2 SD, a balanced panel, non-staggered timing, and a 50% pre-period. At Ļ=0Ļ=0 ā the i.i.d. benchmark underlying most closed-form power calculators ā N=500N=500 reaches 92% power, consistent with the analytical formula. Raising Ļ to 0.5, a value representative of weekly platform outcome series, reduces power at the same N to 76%; at Ļ=0.7Ļ=0.7 it falls to 70%, and N=200N=200 delivers only 34%. The decay is steep and non-linear, with the largest marginal losses between Ļ=0Ļ=0 and Ļ=0.5Ļ=0.5. Table 3 summarizes the nominalāachieved gap across representative IS panel conditions. In the small-to-moderate-N regime where IS studies most commonly operate (Nā[100,1,000]Nā[100,1,000]), the i.i.d.-versus-achieved gap ranges from ā6ā 6 p (at N=1,000N=1,000) to ā33ā 33 p (at N=500N=500 with attrition); the next paragraph decomposes how much of this is recoverable. A natural objection is that this gap is measured against a naive i.i.d. calculator, whereas competent applied work already clusters and anticipates serial correlation. We therefore decompose it against a stronger baseline: an AR(1)-aware, cluster-robust calculator, operationalized as the same simulation under the cellās Ļ but a balanced panel with correct unit clustering. This baseline is an oracleāit uses the true Ļ and DGPāso the recoverability claims below are statements about that best-case upper bound. For balanced designs without attrition the baseline recovers essentially the entire gap: the serial-correlation loss is recoverable in principle by modeling the dependence. What survives is attrition: at effect_size =0.2=0.2 SD and Ļ=0.5Ļ=0.5, an AR(1)-aware design still overstates achievable power by roughly 99 p at N=200N=200 and 1111 p at N=500N=500 once 30% of unit-periods churn out (Table 3; Monte Carlo SE ā2ā 2 p). The staggered-timing bias and pretest selection documented below are likewise invisible to any closed-form formula. The headline: serial correlation is recoverable; attrition, staggered timing, and pretesting are not. How much of this recoverability survives when Ļ must be estimated rather than known? Table 4 quantifies the question for the same design cells. With the five pre-periods these designs afford, within-unit demeaning and the short estimation window jointly depress the per-unit lag-1 correlation well below its true value: the naive plug-in yields Ļ^āā0.12 Ļā-0.12 when the true Ļ is 0.30.3, so a naive plug-in planner behaves almost like the i.i.d. calculator it was meant to replace, and the residual gap grows from 00ā1414 p (oracle) to 77ā3535 p. Adding the first-order Nickell-type correction 1/(Tpreā1)1/(T_pre-1) moves Ļ Ļ to ā0.13ā 0.13āabout half of the way backāand correspondingly recovers roughly half of the shortfall (44ā2626 p); the remainder reflects small-sample bias in the within-unit correlation itself, which the first-order term does not capture. Realizing the recovery on short pre-periods thus requires bias-aware estimation of Ļ (or longer pre-periods); plugging a raw demeaned-residual autocorrelation into a calculatorāor into our lookup toolās autocorr inputāsystematically overstates achievable power. N Condition i.i.d. AR(1) Ach. Resid. 100 Ļ=0.3Ļ=0.3, no churn 0.29 0.20 0.20 0 200 Ļ=0.5Ļ=0.5, churn 0.3 0.58 0.37 0.28 9 500 Ļ=0.5Ļ=0.5, churn 0.3 0.92 0.70 0.59 11 500 Ļ=0.3Ļ=0.3, no churn 0.94 0.80 0.80 0 1,000 Ļ=0.7Ļ=0.7, no churn 1.00 0.94 0.94 0 Resid. == AR(1)-aware planned power minus achieved power, in percentage points: the part of the gap that survives a correctly specified AR(1)/cluster-robust calculation. Table 3: TWFE power across representative IS panel conditions (effect_size =0.2=0.2 SD, T=10T=10, 50% pre-period, unit clustering) under a naive i.i.d. calculator, an AR(1)-aware cluster-robust calculator (same Ļ, balanced panel), and achieved (simulation). Serial correlation is recoverable (residual ā0ā 0 absent attrition); the surviving loss is attrition. Monte Carlo SE ā1.5ā 1.5ā22 p. Figure 1(a) averages over Tā6,10,16,24Tā\6,10,16,24\, so its Ļ=0.5Ļ=0.5 value (0.76) differs from this tableās T=10T=10 AR(1) column (0.70). N Condition Oracle Plug-in Corr. Ach. Resid. 100 Ļ=0.3Ļ=0.3, no churn 0.23 0.33 0.28 0.23 0 / 10 / 5 200 Ļ=0.5Ļ=0.5, churn 0.3 0.35 0.56 0.46 0.27 8 / 29 / 19 500 Ļ=0.5Ļ=0.5, churn 0.3 0.72 0.93 0.84 0.58 14 / 35 / 26 500 Ļ=0.3Ļ=0.3, no churn 0.79 0.93 0.89 0.80 0 / 13 / 9 1,000 Ļ=0.7Ļ=0.7, no churn 0.92 1.00 0.97 0.93 0 / 7 / 4 Resid. == planned minus achieved power (p) under each baseline, reported as oracle / plug-in / corrected. Table 4: Planned power when Ļ is estimated from five pre-periods (500 pilot draws per cell; same cells as Table 3; Monte Carlo SE ā2ā 2 p). Oracle plans at the true Ļ; Plug-in at the raw demeaned-residual Ļ Ļ; Corr. adds the first-order 1/(Tpreā1)1/(T_pre-1) demeaning-bias correction. Oracle and Ach. columns are re-simulated for this exercise and differ from Table 3 by Monte Carlo error. Panel (b) isolates the cost of staggered adoption with cohort-heterogeneous effects (early cohorts receive 2āĪ“2Ī“, later cohorts Ī“). Measured against the base effect Ī“, TWFE exhibits a persistent ā0.042ā 0.042 SD deviation that does not attenuate with N. Only part of this is the Goodman-Bacon (2021) distortion from negatively weighted āforbiddenā comparisons; the remainder is a benchmark mismatch that the reference-implementation check below isolates. Measured against the same base effect, our implementation of Callaway and SantāAnna (2021) shows a smaller deviation (ā0.033ā 0.033 SD) and our implementation of Sun and Abraham (2021) a larger one (ā0.054ā 0.054 SD), stable across sample sizes (Monte Carlo SE ā0.004ā 0.004 SD). We caution strongly against reading this ordering as an estimator ranking; two robustness checks show it is an artifact. First, re-analyzing the full grid, the apparent SA excess declines monotonically with pre-period length (from 0.0800.080 SD at two pre-periods to 0.0310.031 SD at twelve) but persists everywhereāpointing to something systematic. Second, we validated our simplified implementations on a ten-cell subset (200 replications per cell) against fixest::sunab (via its pyfixest port) and the differences implementation of CallawayāSantāAnna. Under this DGP the two reference estimators are numerically identical and approximately unbiased for the realized cohort-share ATTāwhich under heterogeneous effects is 0.230.23ā0.300.30 SD, not the base Ī“=0.2Ī“=0.2. Against that estimand, TWFE shows a modest āā0.02ā-0.02 SD attenuation (the GoodmanāBacon distortion proper), simplified SA tracks the reference, and simplified CS understates the ATT by ā0.03ā 0.03 SD via its equal-weight aggregation. The panel-(b) ordering thus stacks a benchmark mismatch (deviations from Ī“ rather than the estimand) on simplified aggregation weights: with reference implementations and the correct estimand, the cohort-aware estimators are unbiased and effectively tied, while TWFE retains a real but modest distortion. We retain the simplified implementations only for the power and coverage columns of the main grid, where they track the references at short panels (per-replication correlations 0.700.70ā0.830.83 at Tā¤10Tā¤ 10, falling to ā0.47ā 0.47 in the two T=24T=24 validation cells; T=16T=16 was not validated). 50502002001,0001,0005,0005,000000.50.511N (log scale)TWFE powerĻ=0.0Ļ=0.0Ļ=0.3Ļ=0.3Ļ=0.5Ļ=0.5Ļ=0.7Ļ=0.7 (a) TWFE power vs. N, Ļ. 50502002001,0001,0005,0005,0000.030.030.040.040.050.050.060.06N (log scale)|Ī“^āĪ“|| Ī“-Ī“| (SD)TWFESun & Abr.CS (b) Bias (staggered, het. effects). 50502002001,0001,0005,0005,000000.50.511N (log scale)TWFE powerchurn =0=0churn =0.3=0.3 (c) TWFE power loss from attrition. Figure 1: DiD precision under panel frictions. (a) TWFE power decays steeply with residual autocorrelation (effect size =0.2=0.2 SD, non-staggered, 50% pre-period). (b) Deviation from the base effect Ī“ (not the estimand) under staggered, heterogeneous effects for TWFE and our simplified SunāAbraham and CallawayāSantāAnna implementations (Ļ=0.3Ļ=0.3). The ordering shown is an artifact of measuring against Ī“ rather than the cohort-share ATT; see text. (c) Exogenous attrition (30% missing unit-periods) further erodes power at moderate N (Ļ=0.3Ļ=0.3). Dashed lines mark the 80% target; 1,000 replications per cell. 4.2 Panel Attrition and Effective Sample Size Panel (c) of Figure 1 isolates the effect of exogenous attrition on TWFE power at Ļ=0.3Ļ=0.3 (effect_size =0.2=0.2 SD, non-staggered, averaged over T and pre-period share). Moving from churn_rate=0=0 to 0.3 ā 30% of unit-period observations missing, drawn exogenously from both arms ā reduces power from 0.822 to 0.742 at N=500N=500, and from 0.491 to 0.411 at N=200N=200. In Table 3ās Ļ=0.5Ļ=0.5, T=10T=10 cells the contrast costs 99ā1111 p. The loss exceeds what a naive āeffective sample sizeā correction predicts: surviving observations remain within-unit correlated, so usable signal contracts faster than the observation count. The penalty is most pronounced at the sample sizes IS studies most commonly operate in (roughly Nā[200,1000]Nā[200,1000]) and shrinks as power saturates. The main grid draws attrition identically from both arms, and the 88ā1111 p penalty is specific to that mechanism. Because the more common real-world failure mode is attrition that differs by treatment statusāadopters who churn after a disappointing experience, controls who exit a comparison platformāwe ran a dedicated 48-cell sub-grid (Nā200,500Nā\200,500\, Ļā0.3,0.5Ļā\0.3,0.5\, staggered and non-staggered, 1,000 replications per cell) varying the mechanism while holding total attrition near 30%. Two findings qualify the headline. First, attrition that differs by treatment status but remains exogenous in timing (40% of treated units versus 20% of controls, or the mirror image) stays essentially unbiased (|bias|<0.01|bias|<0.01 SD) and costs only a few additional percentage points of power (0.72ā0.660.72ā 0.66 at N=500N=500, Ļ=0.3Ļ=0.3): it remains a power problem, and the lookup toolās single churn_rate input approximates it serviceably. Second, outcome-dependent attritionāelevated exit hazard after adverse outcome shocksāchanges the nature of the problem. Symmetric across arms, the selection cancels in the DiD contrast; concentrated in the treated arm, the estimate acquires an attenuation bias of āā0.05ā-0.05 SD, coverage drops to 0.920.92ā0.940.94, and power falls to 0.440.44 at N=500N=500, Ļ=0.3Ļ=0.3. No power calculation repairs this case: it is a bias problem requiring design or estimation remedies (tracking exiters, selection modeling, bounds); the cohort-specific Callaway and SantāAnna (2021) estimator is largely robust to it in our staggered cells because its short cohort-window comparisons limit exposure to post-exit selection. Attrition claims elsewhere in the paperāincluding the abstractās 88ā1111 p figureārefer to the exogenous case; outcome-dependent attrition is strictly worse and differently shaped. 4.3 Parallel-Trends Pretesting and Conditional Inference We next examine pretesting for parallel trends. Figure 2(a) reports the rejection rate of the joint Wald test on pre-period event-time coefficients: under a true null it rejects at 0.028ā0.037 across all sample sizes (approximately correct size), and its power rises monotonically in the violation magnitude and in N (at N=1,000N=1,000, violations of 0.05 SD/period are detected in 63% of replications). The substantive concern, as Roth (2022) emphasizes, is the inferential consequence of conditioning on having passed the pretest. Figure 2(b) uses a DGP with a true null treatment effect (Ī²=0Ī²=0) but non-zero pre-trend violations, so any rejection of H0:Ī²=0H_0:Ī²=0 is a false positive. At a violation of 0.10 SD/period, unconditional rejection is 0.717, while among replications that pass the pretest, conditional rejection remains 0.249ānearly five times the nominal 0.05 level. The mechanism is selection: pretest-passing replications are those in which sampling noise attenuates the estimated pre-trend, and the same noise propagates into a biased post-treatment estimate. A binary āpretest-gateā rule therefore confers far less Type I error protection than its nominal size suggests, and the gap widens for the small violations most likely to escape detection. 000.10.10.20.20.30.3000.50.511Violation (SD/period)Rejection rateN=100N=100N=500N=500N=1,000N=1,000 (a) Pretrend rejection. 000.10.10.20.20.30.3000.50.511Violation (SD/period)Rej. of H0:Ī²=0H_0:Ī²=0UnconditionalPretest-pass (b) Cond. Type I error. 50502002001,0001,0005,0005,0000.950.950.960.96N (log scale)CI coverageĻ=0.0Ļ=0.0Ļ=0.3Ļ=0.3Ļ=0.5Ļ=0.5Ļ=0.7Ļ=0.7 (c) TWFE 95% CI coverage. 50502002001,0001,0005,0005,0000.950.950.960.96N (log scale)CI coverageĻ=0.0Ļ=0.0Ļ=0.3Ļ=0.3Ļ=0.5Ļ=0.5Ļ=0.7Ļ=0.7 (d) SunāAbraham 95% CI coverage. Figure 2: Pretrend behavior and inferential calibration. (a) The pretrend test has approximately nominal size and gains power with N. (b) Conditioning on pretest passage inflates Type I error (Ī²=0Ī²=0 DGP; the conditional curve ends at 0.20 because the pretest rejects everywhere at larger violations). (c)ā(d) 95% CI coverage for TWFE and SunāAbraham (non-staggered, homogeneous, unit clustering). 4.4 Confidence-Interval Coverage Panels (c) and (d) of Figure 2 report empirical 95% CI coverage for TWFE and SunāAbraham across N and Ļ under the non-staggered, homogeneous benchmark. Coverage remains near nominal (0.951ā0.963) with no systematic drift, indicating that the power losses above reflect effective-sample-size contraction rather than under-coverage. For CallawayāSantāAnna, whose simplified across-cohort standard error is undefined in this single-cohort benchmark, a unit-level bootstrap yields coverage of 0.910.91ā0.970.97 (mean ā0.94ā 0.94) across Nā200,500Nā\200,500\ and Ļā0.3,0.5Ļā\0.3,0.5\ānear-nominal and comparable to the other estimators. The picture is different in the staggered cells, where the simplified C&S analytic intervalāthe across-cohort dispersion of per-cohort ATTsāundercovers systematically. Across all 2,3042,304 staggered homogeneous-effects cells its coverage averages 0.9060.906, never exceeds 0.9550.955, and one third of cells fall below 0.900.90. The undercoverage is flat in N and improves only with panel length (0.8560.856 at T=6T=6 rising to 0.9340.934 at T=24T=24): the signature of a standard error estimated from a handful of cohort-level ATTs, whose count grows with T but not N. The supplementās worked example (0.8760.876, a staggered heterogeneous cell with attrition) is part of this pattern, not an outlier. By comparison, TWFE coverage under unit clustering in the same cells is nominal (0.9570.957) and SunāAbraham close to it (0.9370.937); a separate tail arises under two-way clustering, where the truncated CGM variance estimator undercovers (mean 0.8650.865, worsening with T)āa distinct issue we flag but do not pursue. Practically: treat the simplified C&S analytic interval in staggered designs as anti-conservative by roughly 44ā99 coverage points, and prefer the reference multiplier-bootstrap inference of the did package whenever its intervals matter for a conclusion. 4.5 Supplementary DiD Robustness Modules Beyond the core grid, a treatment-process module allows non-absorbing and continuous-dose exposure. Binary TWFE degrades under continuous treatment (bias ā0.101ā 0.101, RMSE ā0.143ā 0.143, power ā0.491ā 0.491) while absorbing and non-absorbing designs stay near-unbiased with coverage ā0.95ā 0.95ā0.960.96: treatment-process misspecification can be first-order. We restrict supplementary estimators to specifications we can implement faithfully, leaving imputation, matrix-completion, and causal-ML comparisons to their reference implementations in follow-up work. 4.6 Empirical Anchor: Classic DiD Design To anchor the simulation evidence in observed data, we replicate a classic open DiD panel design using the firm-level job-training dataset (54 firms, 3 years; grant adoption in 1988ā1989; Wooldridge, 2010). The estimated TWFE effect is ā0.279-0.279 (SE 0.1980.198). Treating this magnitude as an ex ante design anchorāa posited target effect rather than observed power, which is an uninformative transform of the p-value (Hoenig and Heisey, 2001)āthe design-based i.i.d. power for this panel is 0.2910.291, the calculation most frequently reported in applied IS design sections. Mapping the design to the nearest cells in the simulation grid (staggered adoption, unit clustering, nearest N, T, pre-period share, autocorrelation, and effect size) yields materially lower achieved power: 0.0670.067 for TWFE, 0.0970.097 for SunāAbraham, and 0.1340.134 for CallawayāSantāAnna. Because the anchor is mapped to the nearest grid cell rather than simulated at its exact parameters, part of this gap reflects cell coarseness, and here that part cannot be tightly bounded: the panelās three periods lie below the gridās minimum T=6T=6, so the mapped cell is a coarse approximation. For the IV anchor below, an exact-parameter replication bounds the corresponding contribution at about one percentage point. The nominal-achieved gap for TWFE is about 22 percentage points, consistent with the main result that classical power heuristics overstate inferential strength in short, persistent panels. 5 IV Results IV identification rests on two conceptually independent requirements ā relevance and exclusion ā often evaluated through the single lens of first-stage diagnostics. Our IV grid separates them, varying 2SLS performance over first-stage strength, sample size, and direct ZāYZā Y violations of exclusion, with the true structural effect at Ī²=0.1Ī²=0.1 (panels (a)ā(b) of Figure 3) or Ī²=0Ī²=0 (panels (c)ā(d)). 5.1 Power as a Function of First-Stage Strength and Sample Size Panel (a) of Figure 3 plots IV power over the target first-stage F and N, conditional on a true structural effect of Ī²=0.1Ī²=0.1 (a āsmallā effect by the Table 1 anchors, and the range most common in IS platform and recommender studies). For this effect size, 80% power is infeasible within our grid: power never exceeds ā0.19ā 0.19, even at F=100F=100 and N=10,000N=10,000. At F=10F=10 ā the canonical Stock and Yogo (2005) reference point ā power is ā0.08ā 0.08ā0.100.10 regardless of N, and in the F=10F=10ā2020 range typical of IS instruments (geographic variation, peer exposure) even N=1,000N=1,000 delivers power below 0.11. The ceiling reflects the fixed-F normalization: holding Fā100Fā 100 pins the concentration parameter near 100 regardless of N, so SEā”(Ī²^)SE( Ī²) stays ā0.10ā 0.10 and |Ī²^|/SEā1| Ī²|/SEā 1 for Ī²=0.1Ī²=0.1, far short of conventional power. This is a statement about fixed F, not data collection: with a fixed instrument (constant first-stage coefficient), F grows with N, SEā”(Ī²^)SE( Ī²) falls as Nā1/2N^-1/2, and power rises in the usual way. The grid holds F fixed precisely to isolate its role, so the lever it identifies is instrument strength, not sample sizeāand the F>10F>10 convention, while informative about size, is an inadequate proxy for power when the target effect is small. 5510105050100100000.10.10.20.2First-stage FFIV powerN=100N=100N=300N=300N=500N=500N=1,000N=1,000N=3,000N=3,000N=10N=10k (a) IV power (Ī²=0.1Ī²=0.1). 1001005005003,0003,00010,00010,000000.020.020.040.040.060.060.080.080.10.1N (log scale)|Ī²^āĪ²|| Ī²-Ī²|F=5F=5F=10F=10F=20F=20F=50F=50F=100F=100 (b) Weak-IV bias. 1001005005003,0003,00010,00010,000000.50.5111.51.5N (log scale)Ī²^āĪ² Ī²-Ī²Ī³=0.00Ī³=0.00Ī³=0.05Ī³=0.05Ī³=0.10Ī³=0.10Ī³=0.20Ī³=0.20F (c) Exclusion bias (F=100F=100). 5510105050100100000.020.020.040.040.060.060.080.08First-stage FFType I errorĻvāu=0.3 _vu=0.3Ļvāu=0.5 _vu=0.5Ļvāu=0.7 _vu=0.7F (d) IV Type I error (Ī²=0Ī²=0). Figure 3: IV diagnostics across first-stage strength, sample size, and assumption failures. (a) Power never exceeds ā0.19ā 0.19āfar below the 0.80 targetāeven at N=10,000N=10,000 and F=100F=100 (Ī²=0.1Ī²=0.1, Ļvāu=0.5 _vu=0.5, single instrument). (b) Weak-instrument bias attenuates with F and N; fluctuations at Fā5,10Fā\5,10\, N=1,000N=1,000 are Monte Carlo noise. (c) With F=100F=100 but a direct ZāYZā Y channel Ī³, bias scales with N (true Ī²=0Ī²=0, Ļvāu=0.5 _vu=0.5). (d) Heteroskedasticity-robust 2SLS holds size near 0.05 (dashed) even at F=5F=5; distortion rises only at Ļvāu=0.7 _vu=0.7. 5.2 Two Failure Modes: Weak Instruments versus Exclusion Violations The two canonical IV assumption failures ā a weak first stage and a non-zero direct ZāYZā Y channel ā produce bias signatures that, under our fixed-F normalization, scale with N in opposite directions. Panel (b) of Figure 3 plots the absolute finite-sample bias of 2SLS as F ranges from 5 to 100. At Fā„50Fā„ 50 bias is in the 10ā310^-3 range across all sample sizes; at F=5F=5 it is visible but attenuates as N grows, consistent with weak-instrument bias being Oā”(Nā1)O(N^-1) conditional on identification (Angrist and Pischke, 2009). Panel (c) of Figure 3 fixes F at 100 and varies the direct-effect parameter Ī³ governing the ZāYZā Y channel outside the structural equation. Because the design holds the target F fixed, Ļ=F/(N+F)āNā1/2Ļ= F/(N+F) N^-1/2, and the just-identified 2SLS plim bias from a direct channel is Ī³/ĻĪ³/Ļ, so it grows as N N: for Ī³=0.05Ī³=0.05 it rises from 0.052 at N=100N=100 to 0.398 at N=10,000N=10,000 (7.7Ć7.7Ć observed versus 7.1Ć7.1Ć predicted). Under a fixed instrument coefficient instead, the plim bias Ī³/ĻĪ³/Ļ is constant in N while sampling variance shrinksāthe estimator becomes confidently wrong. The message shared across both normalizations is narrow but firm: sample size does not cure exclusion bias. The asymmetry with panel (b) has direct diagnostic implications: the standard toolkit (first-stage F, Hansen J) cannot separate the two regimes with a single-instrument design, so over-identification tests provide limited leverage in exactly the settings where the bias grows fastest. 5.3 Size Properties of Robust 2SLS Under Weak Instruments Panel (d) of Figure 3 sets Ī²=0Ī²=0 and reports the Type I error of heteroskedasticity-robust 2SLS. Even at F=5F=5, size never exceeds 0.069, and it lies above 0.05 only for strong endogeneity (Ļvāu=0.7 _vu=0.7); for Fā„20Fā„ 20 it is indistinguishable from nominal. Size distortion is thus more limited than classical Stock and Yogo (2005) warnings imply, consistent with Angrist and Pischke (2009) and Semadeni et al. (2014) on robust standard errors (see Londschien, 2025 for a recent overview). Weak instruments in our DGP damage designs primarily through low power and first-stage bias rather than inflated false-positive rates. 5.4 Empirical Anchor: Classic IV Design We also replicate a standard open IV design using the Card returns-to-schooling dataset (instrument: nearc4, N=3,010N=3,010; Card, 1995). The estimated 2SLS coefficient on schooling is 0.1190.119 (SE 0.0530.053), with first-stage F=14.49F=14.49. Treating 0.1190.119 as an ex ante design anchor (Hoenig and Heisey, 2001), the design-based i.i.d. power is 0.6140.614. Mapping this design to the nearest IV-grid cell (N=3,000N=3,000, F=20F=20, Ļvāu=0.3 _vu=0.3, Ī²=0.1Ī²=0.1) gives achieved power of 0.0600.060, a nominal-achieved gap of roughly 55 percentage points. The gap is not an artifact of nearest-cell coarseness: a targeted replication at Cardās exact parameters (Ī²=0.119Ī²=0.119, F=14.49F=14.49, N=3,010N=3,010) shifts achieved power by under one percentage point, bounding the cell-coarseness contribution for this anchor at about one point. The remainder reflects a weak first stage interacting with a small true effect: clearing conventional relevance heuristics does not guarantee adequate power at Cardās realized N and F. Because nearc4 is a fixed instrument, a larger-N version of the same design would have proportionally higher F and materially higher power; the lesson concerns instrument strength at the available sample, not the futility of collecting more data. 6 Discussion 6.1 Implications for Study Design We offer these as evidence-based guidance, not prescriptions. (1) Residual autocorrelation is a first-order but recoverable input to DiD power (Table 3). Where pre-period data exist, estimate Ļ from unit-demeaned residualsāwith the bias correction of Table 4 on short pre-periodsāand propagate it through a calibrated simulation; otherwise report a power curve over Ļā[0,0.7]Ļā[0,0.7]. (2) What is not recoverableāattrition and staggered timingāmust be built into the design directly, and is better read off the companion lookup toolās nine-dimensional cell than 2D slices. (3) Estimator choice matters, and so do implementation fidelity and the estimand: reference implementations of SunāAbraham and CallawayāSantāAnna are approximately unbiased for the cohort-share ATT while TWFE attenuates it by ā0.02ā 0.02 SD, but orderings among simplified reimplementations can invert that conclusion (Section 4). Report TWFE alongside a cohort-aware estimator from a reference package and state the estimand explicitly. (4) Binary pretesting inflates conditional Type I error (Roth, 2022); prefer reporting the unconditional estimate with a sensitivity analysis (Rambachan and Roth, 2023) over a pass/fail gate. (5) The F>10F>10 convention speaks to size, not power (at F=10F=10, N=500N=500, IV power <0.11<0.11); report achieved power explicitly. (6) Exclusion bias is not cured by sample size; defend exclusion on substantive or design grounds, not via larger panels. (7) CI coverage is a useful second-order diagnostic: deviations from 0.950.95 signal mis-specification such as inadequate clustering. (8) Advanced estimators (imputation, matrix completion, causal ML) rely on their reference inference machinery; validate coverage on a study-like DGP before trusting their default intervals. 6.2 Limitations Our DGP uses AR(1) residuals, unit and time effects, and a simple trend; multi-way error dependence in the DGP (we vary only the clustering used for inference), unit-specific trends, and seasonality are out of scope, so the irreducible gap we isolate is a conservative lower bound for messier panels. The attrition sub-grid covers treatment-differential and outcome-dependent exit, but not richer dynamic selection (anticipation, selection on effect size). Our simplified CallawayāSantāAnna analytic SE undercovers across the staggered grid (Section 4.4); porting the reference did multiplier-bootstrap SE across the full grid is future work. We do not evaluate parallel-trends sensitivity analyses (Rambachan and Roth, 2023); the main DiD grid uses effect_size >0>0 (DiD Type I error is taken from the pretest and IV modules); Monte Carlo precision is 1,0001,000 replications per cell; and, as noted at the point of each claim, the AR(1)-aware baseline is an oracle (Table 4 quantifies the estimated-Ļ case) and each empirical anchor is mapped to the nearest grid cell (coarseness bounded for IV by the exact-parameter Card replication; unbounded for the DiD anchor, whose T lies outside the grid). 6.3 Next Steps: Toward IS Research Methodology for the 2030s The gridāan empirically anchored, validated simulation released as a queryable artifactāis a prototype for how ex ante design evidence in IS could be produced and consumed. Four next steps: richer failure modes (informative missingness, dynamic selection, anticipationāmapping rather than assuming the power-versus-bias boundary); estimator fidelity as infrastructure, extending reference SunāAbraham, CallawayāSantāAnna (multiplier-bootstrap inference), imputation (Borusyak et al., 2024; Gardner, 2022), matrix-completion (Athey et al., 2021), and causal-ML (Chernozhukov et al., 2018; Wager and Athey, 2018) estimators across the full grid so the community compares estimators on equal footing rather than through per-paper reimplementations; design tooling that folds estimated-parameter uncertainty and parallel-trends sensitivity (Rambachan and Roth, 2023) into the lookup tool for registration-ready workflows; and a living, versioned benchmark open to community-contributed cells, as shared benchmarks did for machine learning. As IS panels grow larger and messier, we expect empirically calibrated simulation evidence to displace closed-form power formulas as the default justification standard. 6.4 Reproducibility and Computational Cost The pipeline is implemented in Python with per-cell hash-derived seeds, so any cell is independently reproducible. The full grids (ā¼ 9.8 million datasets) required roughly four days of wall-clock time on a 14-core consumer laptopāno cluster access was used or needed. This cost matters mainly for full replication: a single design cell reproduces in minutes and a targeted sub-grid within an hour on commodity hardware, and the lookup tool removes the need to re-simulate at all for designs inside the released grid. Acknowledgment LLMs were used strictly for language polishing (grammar and readability); we reviewed and verified all content and assume full responsibility for its accuracy and integrity. References Abadie (2021) A. Abadie Using synthetic controls: feasibility, data requirements, and methodological aspects. Journal of Economic Literature 59 (2), p. 391ā425. External Links: Document Cited by: Ā§2. Angrist and Pischke (2009) J. D. Angrist and J. Pischke Mostly harmless econometrics: an empiricistās companion. Princeton University Press. External Links: Document Cited by: Ā§5.2, Ā§5.3. Aral and Walker (2011) S. Aral and D. Walker Creating social contagion through viral product design: a randomized trial of peer influence in networks. Management Science 57 (9), p. 1623ā1639. External Links: Document Cited by: Ā§3.2. Athey et al. (2021) S. Athey, M. Bayati, N. Doudchenko, G. Imbens, and K. Khosravi Matrix completion methods for causal panel data models. Journal of the American Statistical Association 116 (536), p. 1716ā1730. External Links: Document Cited by: Ā§2, Ā§6.3. Aziz et al. (2023) A. Aziz, H. Li, and R. Telang The consequences of rating inflation on platforms: evidence from a quasi-experiment. Information Systems Research 34 (2), p. 590ā608. External Links: Document Cited by: Ā§3.2. Bertrand et al. (2004) M. Bertrand, E. Duflo, and S. Mullainathan How much should we trust differences-in-differences estimates?. Quarterly Journal of Economics 119 (1), p. 249ā275. External Links: Document Cited by: Ā§1, Ā§1, Ā§2. Borusyak et al. (2024) K. Borusyak, X. Jaravel, and J. Spiess Revisiting event-study designs: robust and efficient estimation. Review of Economic Studies 91 (6), p. 3253ā3285. External Links: Document Cited by: Ā§2, Ā§6.3. Callaway and SantāAnna (2021) B. Callaway and P. H. C. SantāAnna Difference-in-differences with multiple time periods. Journal of Econometrics 225 (2), p. 200ā230. External Links: Document Cited by: Ā§2, Ā§3.2, Ā§4.1, Ā§4.2. Card (1995) D. Card Using geographic variation in college proximity to estimate the return to schooling. In Aspects of Labour Market Behaviour: Essays in Honour of John Vanderkamp, p. 201ā222. Cited by: Ā§5.4. Chernozhukov et al. (2018) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 21 (1), p. C1āC68. External Links: Document Cited by: Ā§2, Ā§6.3. de Chaisemartin and DāHaultfÅuille (2020) C. de Chaisemartin and X. DāHaultfÅuille Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review 110 (9), p. 2964ā2996. External Links: Document Cited by: Ā§1, Ā§2. Gardner (2022) J. Gardner Two-stage differences in differences. arXiv preprint. External Links: Document Cited by: Ā§2, Ā§6.3. Gelman and Carlin (2014) A. Gelman and J. Carlin Beyond power calculations: assessing Type S (Sign) and Type M (Magnitude) errors. Perspectives on Psychological Science 9 (6), p. 641ā651. External Links: Document Cited by: Ā§3.2. Goodman-Bacon (2021) A. Goodman-Bacon Difference-in-differences with variation in treatment timing. Journal of Econometrics 225 (2), p. 254ā277. External Links: Document Cited by: Ā§1, Ā§1, Ā§2, Ā§4.1. Hoenig and Heisey (2001) J. M. Hoenig and D. M. Heisey The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician 55 (1), p. 19ā24. External Links: Document Cited by: Ā§4.6, Ā§5.4. Huang et al. (2017) N. Huang, Y. Hong, and G. Burtch Social network integration and user content generation: evidence from natural experiments. MIS Quarterly 41 (4), p. 1035ā1058. External Links: Document Cited by: Ā§3.2. Londschien (2025) M. LondschienA statisticianās guide to weak-instrument-robust inference in instrumental variables regression with illustrations in Python(Website) External Links: 2508.12474, Link Cited by: Ā§5.3. Rambachan and Roth (2023) A. Rambachan and J. Roth A more credible approach to parallel trends. Review of Economic Studies 90 (5), p. 2555ā2591. External Links: Document Cited by: Ā§6.1, Ā§6.2, Ā§6.3. Roth (2022) J. Roth Pretest with caution: event-study estimates after testing for parallel trends. American Economic Review: Insights 4 (3), p. 305ā322. External Links: Document Cited by: Ā§1, Ā§1, Ā§1, Ā§2, Ā§3.3, Ā§4.3, Ā§6.1. Seamans and Zhu (2014) R. Seamans and F. Zhu Responses to entry in multi-sided markets: the impact of craigslist on local newspapers. Management Science 60 (2), p. 476ā493. External Links: Document Cited by: Ā§3.2. Semadeni et al. (2014) M. Semadeni, M. C. Withers, and S. T. Certo The perils of endogeneity and instrumental variables in strategy research: understanding through simulations. Strategic Management Journal 35 (7), p. 1070ā1079. External Links: Document Cited by: Ā§2, Ā§2, Ā§5.3. Stock and Yogo (2005) J. H. Stock and M. Yogo Testing for weak instruments in linear IV regression. In Identification and Inference for Econometric Models: Essays in Honor of Thomas Rothenberg, D. W. K. Andrews and J. H. Stock (Eds.), p. 80ā108. Cited by: Ā§2, Ā§5.1, Ā§5.3. Sun and Abraham (2021) L. Sun and S. Abraham Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics 225 (2), p. 175ā199. External Links: Document Cited by: Ā§1, Ā§2, Ā§3.2, Ā§4.1. Tafti and Shmueli (2025) A. Tafti and G. Shmueli Causal inference grounded in causal diagrams: benefits, limitations, and opportunities for the Information Systems field. MIS Quarterly 49 (2), p. 409ā428. External Links: Document Cited by: Ā§1. Ulloa-PĆ©rez et al. (2025) E. Ulloa-PĆ©rez, E. F. Bair, A. S. Navathe, and K. A. LinnComparative evaluation of difference in differences methods for staggered adoption interventions(Website) External Links: 2508.14365, Link Cited by: Ā§2. von Zahn et al. (2026) M. von Zahn, A. GĆ¼ler, J. Pfeiffer, H. A. Reijers, and O. Hinz Causal machine learning in information systems research. Business & Information Systems Engineering 68, p. 235ā242. External Links: Document Cited by: Ā§2. Wager and Athey (2018) S. Wager and S. Athey Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association 113 (523), p. 1228ā1242. External Links: Document Cited by: Ā§2, Ā§6.3. Wooldridge (2010) J. M. Wooldridge Econometric analysis of cross section and panel data. 2 edition, MIT Press. External Links: ISBN 9780262232586 Cited by: Ā§4.6.