Paper deep dive
Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
Stefan Hut, Lorenzo Masoero
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a \emph{Simulated Randomized Controlled Trial} (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model---from a fine-tuned specialist to a general-purpose foundation model---can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ${\sim}77\times$; a within-subject design---where each agent is exposed to both arms---reduces standard errors by ${\sim}2.4\times$. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.
Tags
Links
- Source: https://arxiv.org/abs/2608.02345v1
- Canonical: https://arxiv.org/abs/2608.02345v1
Trouble viewing inline? Open PDF directly →
Full Text
34,058 characters extracted from source content.
Expand or collapse full text
Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation Stefan Hut hutstefa@amazon.com &Lorenzo Masoero11footnotemark: 1 masoerl@amazon.com Equal contribution. Abstract A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents—conditioned on behavioral profiles and contextual descriptions of the intervention—simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a Simulated Randomized Controlled Trial (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model—from a fine-tuned specialist to a general-purpose foundation model—can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.700.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ∼77× 77×; a within-subject design—where each agent is exposed to both arms—reduces standard errors by ∼2.4× 2.4×. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals. 1 Introduction Online controlled experimentation (A/B testing) is how technology services learn what works for customers, enabling safe, systematic iteration over product experiences at scale Kohavi et al. (2009; 2020). Yet each experiment requires engineering integration, weeks of wall-clock time to accumulate statistical power, and exposure of real customer traffic to the intervention under study. At the scale of a large organization running thousands of experiments per year, the aggregate cost is substantial: each experiment consumes engineering time, traffic allocation, and analysis overhead, while exposing customers to potentially underperforming treatments. Teams that iterate through dozens of candidate treatments before finding a winner bear this cost repeatedly. This affects applied scientists, product managers, and builders alike. A sufficiently accurate behavioral simulator could fundamentally change this dynamic: letting teams vet dozens of candidates cheaply and reserve live traffic for the most promising few, so that customers benefit from improvements sooner. The advent of large foundation models offers a path toward such a simulator: conditioning a general-purpose model on a behavioral profile and eliciting decisions directly, without the domain-specific engineering bottleneck of hand-crafted agent-based models Epstein and Axtell (1996); Tesfatsion (2006). Importantly, the simulation engine is a pluggable module: while we instantiate it here with an off-the-shelf foundation model for rapid iteration, the framework accommodates any behavioral model, including fine-tuned specialists and non-language-model architectures. We expect domain-specific models to eventually outperform general-purpose ones for this task. The idea of using language models as simulated economic agents dates to Horton (2023); recent work on agentic A/B testing Mansour et al. (2025); Castelo et al. (2026); Zhang et al. (2025) takes the next step, proposing to simulate user behavior with AI agents and estimate treatment effects from the simulated outcomes. Early validation found promising aggregate alignment with human baselines but systematic individual-level divergences Maier et al. (2025); Anthis et al. (2025); Liu et al. (2026); subsequent work showed that structured persona pipelines and rich contextual grounding narrow the gap Paglieri et al. (2026); Bougie et al. (2026); Meshi et al. (2026). The most recent wave deploys agents directly to simulate controlled experiments: AgentA/B Lu et al. (2025) sends interactive agents to browse two website variants; PAARS Mansour et al. (2025) conditions agents on structured personas; S-Researcher Wang et al. (2026) scales to 10510^5 concurrent agents. None of these systems reports an explicit decomposition of estimation error into independently controllable components, making it difficult to diagnose or reduce error systematically. A complementary line of work uses AI predictions not to replace experiments but to reduce variance within them: Arbour et al. (2026) show that including AI-generated predictions as covariates in standard regression adjustment yields efficiency gains with a “do no harm” guarantee. Our work takes the more ambitious step of simulating entire experiments, but connects to this literature through the calibration protocol (Section 4.2). We propose a framework for agentic A/B prediction, formalized as a Simulated Randomized Controlled Trial (S-RCT), and validate it against real historical outcomes. We develop a two-layer error decomposition that separates agent approximation error from subsampling error, and instantiate the framework with pre-period calibration and within-subject estimation. We validate on a benchmark of 67 historical marketing A/B tests. Our results surface both positive findings (calibration and within-subject estimation compress magnitude error substantially) and fundamental bottlenecks, including systematic agent over-responsiveness and the limits of persona completeness. We emphasize lessons learned and failure modes, which may be as informative for the field as the accuracy gains. Contributions. (1) We formalize agentic A/B prediction as an S-RCT and derive a two-layer error decomposition separating approximation error from subsampling error (Section 2). (2) We validate the framework on 67 historical A/B tests, documenting accuracy gains and fundamental bottlenecks (Section 3). (3) We propose calibration and within-subject techniques that address each error layer independently (Section 4). 2 Method 2.1 Framework and formalism Consider a universe of M possible users eligible for an experiment. An experiment-specific triggering mechanism determines which users are exposed to the intervention, defining a triggered set ⊂1,…,MP⊂\1,…,M\ with ||=Nexp≤M|P|=N_exp≤ M. Each triggered customer j∈j is randomly assigned to a treatment arm T or C and is characterized by two unit-level quantities: • a persona Πj∈ _j —a vector of observable profile features whose value is independent of treatment assignment; • an outcome Yjt∈ℝY_j^t , t∈C,Tt∈\C,T\, for a metric of interest. The quantity of interest is the average treatment effect (ATE), a population-level causal parameter that is a latent state of nature: it can be estimated from data but never directly observed (Neyman, 1990; Rubin, 1978; Imbens and Rubin, 2015). Formally, ATE:=Π∼Q[μT(Π)−μC(Π)],ATE\;:=\;E_ Q\! [μ^T( )-μ^C( ) ], (1) where μt(Π):=[Yt∣Π]μ^t( ):=E[Y^t ] is the conditional mean outcome under arm t∈T,Ct∈\T,C\ (treatment or control) and Q is a distribution over personas. The choice of Q determines which population the ATE summarizes: setting Q to the empirical distribution of the triggered set P yields the finite-population ATE (the effect for the users who actually entered the experiment); treating P as a sample from a broader superpopulation yields the superpopulation ATE (the effect one would expect for future users) (Imbens and Rubin, 2015). A Simulated Randomized Controlled Trial (S-RCT) replaces live traffic with computational surrogates: given a description of who the user is (persona Πj _j), what the user experiences (context), and what decision the user must make (task), a simulator produces a simulated outcome Y~j Y_j. Table 1 illustrates these inputs for a marketing creative test. The simulator is characterized by a parameter θ encoding an approximating response distribution P~θ(⋅∣Πj,t) P_θ(· _j,t). Table 1: Example S-RCT inputs for a marketing creative A/B test measuring click-through rate on a product page. Component Concrete example Persona Πj _j 35-year-old frequent shopper, electronics enthusiast, high brand affinity Context (control) Current banner: “Free shipping on orders $25+” Context (treatment) New banner: “Members save 20% today” Task Does this user click the banner? (yes/no) Outcome Y~j Y_j 11 (click) or 0 (no click) The S-RCT estimator in its simplest form is the difference in simulated means: ATE~^=Y~¯T−Y~¯C=1|T|∑j∈TY~j−1|C|∑j∈CY~j, ATE\;=\; Y_T- Y_C\;=\; 1|S_T| _j _T Y_j\;-\; 1|S_C| _j _C Y_j, (2) computed over Nagt:=|T|+|C|N_agt:=|S_T|+|S_C| simulated agents—computational units that map (persona, context, task) triples to decisions. Our framework is engine-agnostic: the simulator could be a rule-based model, a learned behavioral surrogate, or a foundation model. Persona completeness. S-RCTs introduce a representation bottleneck: the simulator receives only the persona Πj _j as its window into who user j is. Recall that X is the space of all observable profile features available to the simulator. If the true response depends on factors not captured in X (e.g., the user’s current intent, mood, or external context), even a perfect simulator incurs an irreducible gap. Assumption 1 (Persona completeness) Yjt⟂Zj|(Πj,t)Y_j^t\; \!\!\! \;Z_j\; |\;( _j,t) for all Zj∉Z_j . 1 is strong, and likely violated in practice. Violations contribute to the approximation error formalized next. Two-layer error decomposition. Define the simulator-population ATE as the treatment effect the simulator would produce if every triggered customer were evaluated: ATE~θ:=P~θ[1Nexp∑j∈(Y~jT−Y~jC)]. ATE_θ\;:=\;E_ P_θ\! [ 1N_exp _j ( Y_j^T- Y_j^C ) ]. (3) The S-RCT estimator’s total error admits: ATE~^−ATE=(ATE~θ−ATE)⏟approximation error+(ATE~^−ATE~θ)⏟subsampling error. ATE-ATE\;=\; ( ATE_θ-ATE )_approximation error\;+\; ( ATE- ATE_θ )_subsampling error. (4) The approximation error is the gap between the simulator’s population-level prediction and the true ATE—a function of the behavioral model, the persona rendering, and conditioning. The subsampling error is the finite-sample gap between the realized NagtN_agt-agent estimate and the simulator-population ATE. These two layers are independently addressable: calibration targets the first; principled agent selection targets the second (Section 4). 2.2 Instantiation We use an internal simulation platform as the engine, with three design choices. First, the simulator is a general-purpose foundation model prompted with the customer’s behavioral profile; we use this as a practical bootstrapping choice, though the engine-agnostic framework readily accommodates domain-specific models fine-tuned on behavioral data. Second, agents are constructed via one-to-one agentic-twin pairing: every agent is bound to a specific real customer who participated in the historical experiment (trigger-based sampling). This yields individual-level outcome pairs (Yj,Y~j)\(Y_j, Y_j)\ supporting per-customer error analysis. Third, because the simulator is a stateless function, each agent can be queried under both arms—and repeatedly—yielding paired individual treatment effects Δ~j=Y~jT−Y~jC _j= Y_j^T- Y_j^C with no analogue in real experiments. This full counterfactual access is exploited in Section 4.3. 3 Baseline Results We apply the S-RCT framework to a benchmark of I=67I=67 historical marketing A/B test treatment pairs run on a large e-commerce service. Each experiment is a marketing creative test measuring click-through rate (CTR) on a high-traffic product surface. The benchmark includes a variety of creative treatments spanning text, imagery, and layout variations. Historical percent impacts span a range typical of marketing creative tests, with most effects small in magnitude. The benchmark contains a roughly even split of launch, harmful, and inconclusive historical decisions. Each simulation uses Nagt=1,000N_agt=1,000 agents in a within-subject design where every agent is exposed to both treatment and control, with each agent mapping one-to-one to a real customer who participated in the historical experiment. We use a general-purpose foundation model with no hyperparameter tuning; agents are chosen uniformly at random from the triggered population. Table 2: Evaluation metrics. σ^i σ_i: standard error of ATE^i ATE_i; pi,qip_i,q_i: posterior probabilities of a positive effect; dec(x)=−1dec(x)=-1 if x<0.33x<0.33, 11 if x>0.66x>0.66, 0 otherwise. Corrected MSE Sign overlap Launch alignment I−1∑i[(ATE~^i−ATE^i)2−σ^i2]I^-1 _i [( ATE_i- ATE_i)^2- σ_i^2 ] I−1∑i(1−|pi−qi|)I^-1 _i(1-|p_i-q_i|) I−1∑idec(pi)=dec(qi)I^-1 _i1\dec(p_i)=dec(q_i)\ Results. Table 3 reports accuracy metrics across all 67 experiment-treatment pairs. Sign overlap is 0.700.70 (SE 0.030.03), indicating modest directional accuracy. Agentic impact estimates are systematically larger in magnitude than historical ATEs (MAE =0.0893=0.0893, several times the median historical percent impact). Launch alignment is 0.410.41—near the random floor of 0.330.33—driven by the magnitude gap pushing agentic posterior probabilities to extremes. In summary, the baseline simulator captures directional signal but is not yet useful for launch decisions without calibration, motivating the improvements in Section 4. Table 3: Baseline accuracy (within-subject design, Nagt=1,000N_agt=1,000, no calibration). Random floors: sign overlap 0.500.50, launch alignment 0.330.33. Metric Value SE Corrected MSE 0.02220.0222 0.01080.0108 Sign overlap 0.700.70 0.030.03 Sign overlap (BC) 0.800.80 0.030.03 Launch alignment 0.410.41 0.060.06 MAE 0.08930.0893 0.01780.0178 Sign accuracy 0.700.70 0.060.06 4 Improvements We address the two error layers in Equation 4 with three improvements: principled subsampling (Section 4.1), pre-period calibration (Section 4.2), and within-subject estimation (Section 4.3). 4.1 Principled subsampling Simulations are cheaper than real experiments, but they are not free. A large service where many teams run A/B tests in parallel, each involving large user populations, faces a clear scaling constraint: simulating every triggered user is infeasible. Smart sampling of which users to simulate is therefore critical for this approach to scale. The subsampling error in Equation 4 arises precisely because we simulate only Nagt≪NexpN_agt N_exp agents. We can always decompose the population response as a mixture of heterogeneous subgroups: P(Yt)=∑kπkP(Yt∣stratum k)P(Y^t)= _k _k\,P(Y^t k), where πk _k is the population share of stratum k. This decomposition is useful beyond variance reduction. Understanding which subpopulation drives the KPI of interest is valuable in its own right. Teams lacking deep experimental expertise often struggle to identify subgroups where a treatment helps or harms. If the simulation is sufficiently accurate, it offers a cheap way to pre-screen an intervention across subpopulations, identifying potentially harmed groups before live deployment (Wang et al., 2025) and advancing heterogeneous treatment effect estimation at scale (Figure 1c). For variance reduction specifically, classical survey sampling results (Imbens and Rubin, 2015) show that allocating agents proportionally to πkσk _k _k (where σk _k is the within-stratum standard deviation) minimizes the variance of the stratified estimator (Neyman allocation; Neyman, 1990): VaruniformVaroptimal=1+Varπ(σk)(π[σk])2. Var_uniformVar_optimal=1+ Var_π( _k)(E_π[ _k])^2. (5) The practical gain depends on how heterogeneous stratum variances are (Figure 1a). For binary outcomes with low base rates (as in our CTR benchmark), within-stratum standard deviations are compressed near p p, limiting the gain to a few percent. For continuous metrics such as revenue, where high-value segments can have 1010–100×100× the variance of low-value ones, the gain reaches 22–5×5× (Figure 1b). Figure 1: Principled subsampling: (a) efficiency gain from Neyman allocation grows with stratum variance heterogeneity; (b) for a stylized revenue metric, Neyman allocation achieves a given RMSE with fewer agents; (c) simulation pre-screens a treatment across subgroups, flagging the harmed segment before live deployment. The key takeaway is that the ratio Varπ(σk)/(π[σk])2Var_π( _k)/(E_π[ _k])^2 can be estimated from historical data before any simulation is run, giving practitioners a principled criterion for when smart subsampling is worth the added complexity. We note a tension here: the estimator we employ (Equation 2, difference in simulated means) is model-free and makes no assumptions about the response surface. One could in principle do better by imposing more structure, for instance modeling the conditional response μt(Π)μ^t( ) directly and using the simulation budget to refine uncertain regions. However, stronger modeling assumptions carry higher misspecification risk. The stratified difference-in-means estimator strikes a middle ground: it exploits population heterogeneity for efficiency while remaining valid regardless of the within-stratum response model. 4.2 Agent calibration The magnitude gap documented in Section 3, where agentic ATEs systematically overshoot historical effects, is a manifestation of the approximation-error term in Equation 4. To confidently rely on agent-based simulation, the behavioral model must be calibrated against real activity patterns. A natural solution is to rely on a foundation model trained specifically for this task, whether via post-training (e.g., supervised fine-tuning on historical behavioral data) or by building dedicated behavioral models from scratch. Here we investigate a third, much cheaper (and admittedly less powerful) alternative: a lightweight two-phase calibration protocol, related in spirit to prediction-powered inference (Angelopoulos et al., 2023), that uses pre-exposure user data as a free supervision signal. The protocol has two phases (Figure 2). In Phase 1 (calibration), we run the simulator on the pre-period: the pre-period outcome YjpreY_j^pre is held out of the feature set, and both arms render the control context (an A/A simulation with zero treatment effect by construction). The simulator produces Y~jpre Y_j^pre, and because the real YjpreY_j^pre is known, we fit a calibration function: f^=argminf∑jℓ(f(Y~jpre),Yjpre), f= _f _j \! (f( Y_j^pre),\;Y_j^pre ), (6) where ℓ is a suitable loss (e.g., log-loss for Platt scaling). Figure 3a illustrates this step: the raw simulator systematically overestimates user activity relative to real pre-period outcomes, and Platt scaling learns a monotone correction. In Phase 2 (prediction), we run the simulator with full features and real treatment assignment. Raw simulated outcomes are passed through f f to produce calibrated estimates. The calibration function f f is fit independently per experiment on Phase-1 data and applied to Phase-2 data; the train/test split is temporal (pre-period vs. treatment period). On 16 experiments from the benchmark using Platt scaling, calibration compresses the squared prediction error by ∼77× 77×, from systematic magnitude overshoot to estimates near the historical noise floor (Figure 3b). Phase 1LLM on X∖YpreX \Y^pre\both arms → controlY~jpre Y_j^prePhase 2LLM on full XXreal T/CT/C assignmentY~jt Y_j^tf^(Y~jt) f( Y_j^t)ATE~ ATEf^=argminf∑jℓ(f(Y~jpre),Yjpre) f= _f _j \! (f( Y_j^pre),\;Y_j^pre ) Figure 2: Two-phase calibration pipeline. In Phase 1 we fit the calibration function f f on pre-period data. In Phase 2 we apply f f to the raw outputs Y~jt Y_j^t to produce the calibrated estimate ATE~ ATE. Figure 3: Two-phase calibration. (a) Phase 1: the raw simulator overestimates user activity; Platt scaling learns a correction from pre-period data. (b) Phase 2 result: per-experiment ATE estimates before (red crosses) and after (blue circles) calibration, relative to the historical ATE (black bars). Calibration compresses the magnitude gap by ∼77× 77×. 4.3 Within-subject estimates With agents split across arms, a between-subject design is susceptible to random composition imbalances. For subtle effects—where the true ATE is small—the estimated difference can be dominated by chance differences in who was assigned to each arm rather than by the treatment itself. We address this with a within-subject design where every agent is exposed to both treatment and control. This eliminates between-agent composition differences entirely: each agent serves as its own control, and the estimator becomes the within-subject paired difference. The paired structure removes the persona-level variance component while targeting the same population ATE. To address ordering effects, we randomize the order in which treatment and control are presented to each agent. The within-subject design’s primary advantage is variance reduction: standard errors shrink by ∼2.4× 2.4× on average. Sign accuracy rises from 65%65\% (between-subject) to 0.700.70 (within-subject), consistent with the elimination of random arm-composition imbalance. Launch alignment improves from 0.330.33 to 0.410.41, though the magnitude gap continues to push posterior probabilities toward extremes. Combining within-subject estimates with calibration (Section 4.2) is the natural next step. 5 Discussion and Conclusion This paper presents a framework for agentic A/B prediction formalized as Simulated Randomized Controlled Trials (S-RCTs) with a two-layer error decomposition separating behavioral approximation error from subsampling error. Validation on 67 historical marketing A/B tests reveals modest directional accuracy and a systematic magnitude gap where agentic estimates overshoot historical effects. Within-subject estimation improves sign accuracy; calibration addresses magnitude error; principled subsampling targets the remaining variance. Behavioral implications. The systematic over-responsiveness of agent behavior—predicting larger effects than observed in real customers—suggests that current general-purpose foundation models exhibit a form of behavioral amplification: when conditioned on a persona and asked to choose, they respond more decisively to treatment differences than real customers do. Understanding and correcting this behavioral gap is, we believe, a core challenge for the agent behavior community: it reflects not just miscalibration but a fundamental mismatch between how these models represent decision-making and how humans actually behave in low-stakes product contexts. Bridging this gap may require behavioral fine-tuning on revealed preference data, structured constraints on agent response distributions, or hybrid architectures that combine language model reasoning with learned behavioral priors. Applications beyond ATE estimation. Estimating the average treatment effect is arguably the hardest target for agentic simulation: it requires both correct direction and accurate magnitude. However, the framework produces several secondary outputs that are useful even when magnitude accuracy is unreliable. Directional screening (flagging likely losers before committing live traffic) requires only sign accuracy, where we already achieve 0.700.70. Reasoning traces (qualitative explanations of why a simulated agent chose one option over another) offer experimenters a novel interpretive lens with no analogue in classical A/B testing. Flipper analysis (identifying which agent personas flip their decision between treatment and control) surfaces treatment-effect heterogeneity and can guide segment-level analysis in real experiments. Agentic priors for early decisions: even a noisy directional signal, when combined with early real-experiment data, can accelerate go/no-go decisions by serving as an informative prior in a Bayesian framework. These applications lower the accuracy bar required for practical value and represent the nearest-term path from the current system to practical deployment. Limitations. Near-term limitations include magnitude overshoot even with correct sign; a single-domain benchmark (marketing CTR); potential correlation in agent responses from a shared model that could understate variance; and retrospective-only evaluation against historical targets. The benchmark evaluates a prediction against a known historical outcome—in a prospective setting, the triggering population is unknown and no ground truth is available for calibration. Predicting the triggering population is the main missing piece for deployment; it is best understood as a thin upstream layer that can be developed independently of the behavioral simulator. Broader considerations. If the framework is used to pre-screen which experiments to run, treatments benefiting customer segments poorly captured by the behavioral model—low-frequency users, underrepresented demographics—may be filtered out before reaching real measurement. The two-phase calibration protocol partially mitigates this by surfacing per-segment bias on the pre-period, but practitioners must remain attentive to whose behavior the simulator represents faithfully and whose it does not. The goal is not to replace real experiments but to make the experimentation pipeline faster and more informed, so that customers benefit from product improvements sooner. References A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei (2023) Prediction-powered inference. Science 382 (6671), p. 669–674. External Links: Link Cited by: §4.2. J. R. Anthis, R. Liu, S. M. Richardson, A. C. Kozlowski, B. Koch, J. Evans, E. Brynjolfsson, and M. Bernstein (2025) LLM social simulations are a promising research method. arXiv preprint arXiv:2504.02234. External Links: Link Cited by: §1. D. Arbour, E. Ben-Michael, A. Feller, A. Lal, and L. Yuan (2026) AI-assisted variance reduction in randomized experiments. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), External Links: Link Cited by: §1. N. Bougie, G. M. Marconi, X. Ye, and N. Watanabe (2026) Beyond offline A/B testing: context-aware agent simulation for recommender system evaluation. arXiv preprint arXiv:2604.09549. External Links: Link Cited by: §1. A. Castelo, Z. Z. Foumani, A. Fan, K. Y. Koay, V. Malik, Y. Zhu, H. Li, M. Feghhi, R. Uliana, S. Xie, et al. (2026) SimGym: traffic-grounded browser agents for offline A/B testing in e-commerce. arXiv:2602.01443. External Links: Link Cited by: §1. J. M. Epstein and R. Axtell (1996) Growing artificial societies: social science from the bottom up. MIT Press. Cited by: §1. J. J. Horton (2023) Large language models as simulated economic agents: what can we learn from homo silicus?. Working Paper Technical Report 31122, National Bureau of Economic Research. External Links: Link Cited by: §1. G. W. Imbens and D. B. Rubin (2015) Causal inference for statistics, social, and biomedical sciences: an introduction. Cambridge University Press. External Links: Link Cited by: §2.1, §2.1, §4.1. R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne (2009) Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery 18, p. 140–181. External Links: Link Cited by: §1. R. Kohavi, D. Tang, and Y. Xu (2020) Trustworthy online controlled experiments: a practical guide to A/B testing. Cambridge University Press. External Links: Link Cited by: §1. X. Liu, H. Shang, Z. Liu, X. Liu, Y. Xiao, Y. Tu, and H. Jin (2026) HumanStudy-Bench: towards AI agent design for participant simulation. arXiv preprint arXiv:2602.00685. External Links: Link Cited by: §1. Y. Lu, T. Hsu, H. Gu, L. Cui, Y. Xie, W. Headden, B. Yao, A. Veeragouni, J. Liu, S. Nag, J. Wang, and D. Wang (2025) AgentA/B: automated and scalable web A/B testing with interactive LLM agents. arXiv preprint arXiv:2504.09723. External Links: Link Cited by: §1. B. F. Maier, U. Aslak, L. Fiaschi, N. Rismal, K. Fletcher, C. C. Luhmann, R. Dow, K. Pappas, and T. V. Wiecki (2025) LLMs reproduce human purchase intent via semantic similarity elicitation of Likert ratings. arXiv preprint arXiv:2510.08338. External Links: Link Cited by: §1. S. Mansour, L. Perelli, L. Mainetti, G. Davidson, and S. D’Amato (2025) PAARS: persona aligned agentic retail shoppers. In Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025), p. 143–159. Cited by: §1. O. Meshi, K. Balog, S. Goldman, A. Caciularu, G. Tennenholtz, J. Jeong, A. Globerson, and C. Boutilier (2026) ConvApparel: a benchmark dataset and validation framework for user simulators in conversational recommenders. arXiv preprint arXiv:2602.16938. External Links: Link Cited by: §1. J. Neyman (1990) On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science 5 (4), p. 465–472. Note: Translated and edited by D. M. Dabrowska and T. P. Speed from the 1923 Polish original External Links: Link Cited by: §2.1, §4.1. D. Paglieri, L. Cross, W. A. Cunningham, J. Z. Leibo, and A. S. Vezhnevets (2026) Persona generators: generating diverse synthetic personas at scale. arXiv preprint arXiv:2602.03545. External Links: Link Cited by: §1. D. B. Rubin (1978) Bayesian inference for causal effects: the role of randomization. The Annals of Statistics 6 (1), p. 34–58. External Links: Link Cited by: §2.1. L. Tesfatsion (2006) Agent-based computational economics: a constructive approach to economic theory. In Handbook of Computational Economics, Vol. 2, p. 831–880. Cited by: §1. A. Wang, J. Morgenstern, and J. P. Dickerson (2025) Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence 7 (3), p. 400–411. External Links: Link Cited by: §4.1. L. Wang, Y. Li, J. Wu, H. Gao, X. Bo, X. Chen, and J. Wen (2026) LLM agents as social scientists: a human-AI collaborative platform for social science automation. arXiv preprint arXiv:2604.01520. External Links: Link Cited by: §1. Y. Zhang, T. Wang, J. Gesi, Z. Wang, Y. Lu, J. Lin, S. Zhan, V. Gao, R. Jiao, J. Liu, et al. (2025) Shop-r1: rewarding LLMs to simulate human behavior in online shopping via reinforcement learning. arXiv:2507.17842. Cited by: §1. Appendix A Evaluation Metrics: Detailed Definitions Corrected MSE. The noise-corrected MSE removes the irreducible historical sampling variance from the squared error: CMSE=1I∑i=1I[(ATE~^i−ATE^i)2−σ^i2],CMSE= 1I _i=1^I [( ATE_i- ATE_i)^2- σ_i^2 ], where σ^i σ_i is the standard error of the historical ATE estimate for experiment i. Without this correction, MSE would be dominated by the historical noise floor rather than the simulator’s error. Sign overlap. Sign overlap measures the agreement between historical and agentic posterior probabilities of a positive effect: SO=1I∑i=1I(1−|pi−qi|),SO= 1I _i=1^I(1-|p_i-q_i|), where pip_i and qiq_i are the posterior probabilities of a positive effect under the historical and agentic estimates, respectively. A value of 1.01.0 indicates perfect calibration of directional confidence; the random floor is 0.500.50. Launch alignment. Launch alignment discretizes the posterior into three decisions (harmful: p<0.33p<0.33; inconclusive: 0.33≤p≤0.660.33≤ p≤ 0.66; launch: p>0.66p>0.66) and measures agreement: LA=1I∑i=1Idec(pi)=dec(qi).LA= 1I _i=1^I1\dec(p_i)=dec(q_i)\. The random floor is 0.330.33 (three equally likely categories). Appendix B Benchmark Details The benchmark comprises 67 treatment-control pairs from marketing creative tests on a high-traffic e-commerce product surface. All experiments measure click-through rate (CTR) as the primary metric. Historical percent impacts are small, consistent with the subtle nature of marketing creative changes. The distribution of historical decisions is approximately balanced across harmful, inconclusive, and launch outcomes. Each experiment draws from a large triggered population, representing a broad cross-section of customer segments.