Paper deep dive
Incentive Alignment in Online Experimentation
Ermis Soumalias, Richard Mudd, Abbas Zaidi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/7/2026, 8:30:15 AM
Summary
This paper addresses the principal-agent conflict in decentralized online experimentation, where experimenters are rewarded based on significance-filtered empirical effects, leading to misaligned incentives and potential erosion of platform value. The authors propose two mechanisms to align incentives: sample splitting, which perfectly aligns incentives at a bounded traffic cost, and shrinkage, which preserves the full budget while ensuring negative-effect interventions are unprofitable.
Entities (6)
Relation Signals (5)
Sample Splitting → solves → Principal-Agent Conflict
confidence 95% · Sample splitting aligns incentives perfectly... effectively bridge this gap.
Shrinkage → solves → Principal-Agent Conflict
confidence 95% · shrinkage... guarantees that interventions with negative expected effects are strictly unprofitable to field.
Platform → seekstomaximize → Ground-Truth Value
confidence 93% · platform seeks to maximize the ground-truth value of launched features
Experimenters → arerewardedby → Empirical Average Treatment Effects
confidence 92% · experimenters... are typically rewarded based on empirical average treatment effects
Significance-Filtered Estimates → causes → Winner's Curse
confidence 90% · significance-filtered estimates are inherently susceptible to the winner’s curse
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating the causal effect of new features is a central goal for online platforms. While recent literature addresses limited testing traffic via centralized portfolio optimization, this perspective abstracts away a critical institutional reality: experimentation is operationally decentralized. The experimenters who develop new features also dictate which hypotheses to test, and they are typically rewarded based on empirical average treatment effects that are prone to upward bias. Left unchecked, this principal-agent conflict can severely erode platform value, a structural failure that conventional centralized levers, such as significance thresholds and traffic budgets, cannot resolve. By reframing experimentation as an incentive design problem, we demonstrate that two practical mechanisms, sample splitting and shrinkage, can effectively bridge this gap. Sample splitting aligns incentives perfectly at a bounded traffic cost, while shrinkage consumes no additional traffic and guarantees that interventions with negative expected effects are strictly unprofitable to field.
Tags
Links
- Source: https://arxiv.org/abs/2610.05922v2
- Canonical: https://arxiv.org/abs/2610.05922v2
Trouble viewing inline? Open PDF directly →
Full Text
34,758 characters extracted from source content.
Expand or collapse full text
Incentive Alignment in Online Experimentation Ermis Soumalias Richard Mudd Abbas Zaidi Affiliation: Meta Platforms, Inc. Abstract Evaluating the causal effect of new features is a central goal for online platforms. While recent literature addresses limited testing traffic via centralized portfolio optimization, this perspective abstracts away a critical institutional reality: experimentation is operationally decentralized. The experimenters who develop new features also dictate which hypotheses to test, and they are typically rewarded based on empirical average treatment effects that are prone to upward bias. Left unchecked, this principal-agent conflict can severely erode platform value—a structural failure that conventional centralized levers, such as significance thresholds and traffic budgets, cannot resolve. By reframing experimentation as an incentive design problem, we demonstrate that two practical mechanisms, sample splitting and shrinkage, can effectively bridge this gap. Sample splitting aligns incentives perfectly at a bounded traffic cost, while shrinkage consumes no additional traffic and guarantees that interventions with negative expected effects are strictly unprofitable to field. 1 Introduction Evaluating the causal effects of candidate interventions via A/B testing is a cornerstone of decision-making for online platforms. As these platforms face an ever-growing pool of ideas competing for scarce testing traffic, recent literature has predominantly modeled this bottleneck as a portfolio optimization problem [14, 2, 17]. Under this paradigm, a centralized decision-maker maximizes global returns by strategically trading off a few well-powered tests against a high volume of noisier estimates. However, the theoretical appeal of this centralized approach obscures a practical institutional reality: in practice, resource allocation is highly decentralized. Individual experimenters—who possess the most informed local priors about their own features—typically dictate how testing budgets are spent [18, 3, 10]. Crucially, this decentralization introduces misaligned incentives. While the platform seeks to maximize the ground-truth value of launched features, experimenters are typically evaluated on the empirical average treatment effects of their successful tests. Because significance-filtered estimates [20] are inherently susceptible to the winner’s curse, experimenters face a rational incentive to favor testing strategies with high statistical upside, which can diverge significantly from genuine platform value. To formalize this misalignment, we construct a budget allocation game. We demonstrate that when experimenters are rewarded for empirical effects, the resulting principal-agent conflict cannot be resolved through standard statistical gating. Indeed, for any fixed significance threshold and budget limit, strategic experimenters can select an allocation yielding exactly zero platform value, even when a platform-optimal allocation would generate arbitrarily large returns. Fundamentally, statistical gatekeeping alone cannot rescue structural misalignment. To address this vulnerability, we reframe experimentation from a purely inferential exercise into a mechanism design problem. Specifically, we demonstrate how two practical mechanisms can successfully align experimenter behavior with platform value: • Sample Splitting: While traditionally viewed merely as an ex post statistical tool to recover unbiased estimates, we provide a novel ex ante economic rationale for its deployment. We prove that reserving a strictly independent evaluation sample perfectly aligns experimenter incentives, and that the resulting cost to overall platform value is strictly bounded. • Shrinkage: Alternatively, practitioners often utilize shrinkage to dampen marginal launches inflated by the winner’s curse. We extend this statistical concept directly into the experimenter’s economic incentive structure. By charging a variance penalty equal to the expected reward of a zero-effect intervention, we guarantee that negative-expected-value interventions become strictly unprofitable to test, all while preserving the full testing budget. Taken together, our findings demonstrate that treating experimentation solely as a centralized optimization problem misses the practical behavioral dynamics of decentralized teams. However, by viewing causal inference through a mechanism-design lens, platforms can safely decentralize experimentation and fully leverage the rapid pace of AI-driven feature generation without compromising the true value of their products. 2 Related Work A growing literature models online experimentation as a resource-constrained portfolio problem, balancing well-powered tests against a larger volume of noisier estimates [14, 2, 17, 6]. Crucially, what unites these treatments is the assumption of a single decision maker whose objective perfectly aligns with the platform. We retain the allocation framework but introduce strategic dynamics: the budget is spent by a self-interested experimenter maximizing an empirical proxy reward. Consequently, our work connects most closely to recent studies at the intersection of statistics and mechanism design [9, 15, 16], which model testing as a principal-agent game. Rather than deploying complex menus of statistical contracts, we demonstrate how to align incentives using practical mechanisms already familiar to practitioners: sample splitting and shrinkage. Our sample-splitting mechanism (Section 4.2) bridges the statistical literature on replication [5, 21] and selective inference [7, 19] with mechanism design. It is well established that conditioning on significance inflates reported effects [12, 4], an inferential hurdle formally characterized by Andrews et al. [1]. While we borrow the sample-splitting machinery directly from this literature, our core contribution is the economic rationale: we establish its ex ante utility for incentive alignment. Finally, shrinking significance-filtered estimates toward a prior serves as a practical remedy for selection bias [20, 8, 11, 13]. Our shrinkage mechanism (Section 4.3) applies this underlying intuition directly to the experimenter’s incentive structure. By correcting the payoff itself rather than just the point estimate, we preserve the upside of large true effects while rendering null interventions strictly unprofitable (Theorem 4). 3 Formal Model An experimenter allocates a budget of B experimental units across N candidate interventions (bi∈ℕ≥0b_i _≥ 0, ∑bi≤BΣ b_i≤ B). Intervention i∈[N]i∈[N] has a true expected treatment effect μi∈ℝ _i and a known unit-level variance σi2>0 _i^2>0. Testing intervention i with bi>0b_i>0 units yields an empirical effect μ^i(bi)∼(μi,σi2/bi) μ_i(b_i) \! ( _i, _i^2/b_i ). We assume the sampling variances σi2 _i^2 are known and that the experimenter’s prior belief matches the truth (μ~i=μi μ_i= _i). This best-case information setting neatly isolates structural incentive misalignment from estimation error. At significance level α (with critical value cα:=z1−α/2c_α:=z_1-α/2), the test is deemed significantly positive when Li(bi,α)=μ^i(bi)>cασibiL_i(b_i,α)= 1\! \ μ_i(b_i)>c_α _i b_i \ is equal to 11. The resulting launch probability is: pi(b)=Pr[Li(b,α)=1]=Φ¯(cα−μibσi)p_i(b)= \! [L_i(b,α)=1 ]= \! (c_α- _i b _i ) (1) Because the platform values launched interventions solely by true effects, launching based on Li(bi,α)L_i(b_i,α) yields a realized platform reward of ∑i=1NμiLi(bi,α) _i=1^N _iL_i(b_i,α). The Incumbent Mechanism. Under common industry practice, each intervention is launched upon a significantly positive result, and the experimenter’s reward is proportional to the empirical effects of their launched interventions: REinc(,α)=∑i=1Nμ^i(bi)Li(bi,α).R_E^inc(b,α)= _i=1^N μ_i(b_i)L_i(b_i,α). (2) By comparison, the platform’s reward is RPinc(,α)=∑i=1NμiLi(bi,α)R_P^inc(b,α)= _i=1^N _iL_i(b_i,α). For bi>0b_i>0, the experimenter’s expected reward from intervention i is the mean of a truncated Gaussian: rE,iinc(μi,bi)=[μ^i(bi)Li(bi,α)]=μipi(bi)+σibiϕ(cα−μibiσi),r_E,i^inc( _i,b_i)=E [ μ_i(b_i)L_i(b_i,α) ]= _ip_i(b_i)+ _i b_iφ (c_α- _i b_i _i ), (3) while the platform’s is rP,iinc(μi,bi)=μipi(bi)r_P,i^inc( _i,b_i)= _ip_i(b_i). We denote the platform- and experimenter-optimal allocations as Pinc∈argmax[RPinc(,α)]b_P^inc∈ _bE[R_P^inc(b,α)] and Einc∈argmax[REinc(,α)]b_E^inc∈ _bE[R_E^inc(b,α)], respectively. 4 Theoretical Results 4.1 Incentive Misalignment Can Completely Destroy Platform Value We first prove that no fixed statistical-significance threshold can eliminate this principal-agent conflict. Consequently, the resulting loss in platform value can be arbitrarily large. Theorem 1. For every α∈(0,1)α∈(0,1), B≥1B≥ 1, and V>0V>0, there exists an instance such that [RPinc(Einc,α)]≤0E[R_P^inc(b_E^inc,α)]≤ 0 while [RPinc(Pinc,α)]>VE[R_P^inc(b_P^inc,α)]>V. The formal proof of Theorem 1 is provided in Section A.1. 4.2 The Sample-Splitting Mechanism To resolve this structural flaw, the sample-splitting mechanism rewards the experimenter based on an independent, unbiased estimate of the launched interventions’ true effects. Sample-Splitting Mechanism. Fix a holdout fraction γ∈(0,1)γ∈(0,1). Partition the allocated bib_i units into a selection sample of size (1−γ)bi(1-γ)b_i and an independent evaluation sample of size γbiγ b_i. The launch decision Li=Li((1−γ)bi,α)L_i=L_i((1-γ)b_i,α) is made using only the selection sample. If launched, the evaluation sample generates a fresh estimate μ^iE μ_i^E. The experimenter receives REsplit(,α)=∑iμ^iELiR_E^split(b,α)= _i μ_i^EL_i, and the platform receives RPsplit(,α)=∑iμiLiR_P^split(b,α)= _i _iL_i. Theorem 2. For any budget allocation b and holdout fraction γ∈(0,1)γ∈(0,1), the expected platform and experimenter rewards under the sample-splitting mechanism satisfy: [REsplit(,α)]=[RPsplit(,α)]E[R_E^split(b,α)]=E[R_P^split(b,α)]. The formal proof is deferred to Section A.2. Consequently, the optimal budget allocations for the platform and experimenter coincide (Psplit=Esplitb_P^split=b_E^split). This alignment comes at a strictly bounded cost to overall statistical power. Theorem 3. For any holdout fraction γ∈(0,1)γ∈(0,1) and threshold cα≤2c_α≤ 2, the expected platform value under the sample-splitting mechanism satisfies [RPsplit(Esplit,α)]≥(1−γ)[RPinc(Pinc,α)]E[R_P^split(b_E^split,α)]≥(1-γ)E[R_P^inc(b_P^inc,α)]. The formal proof is provided in Section A.3. 4.3 The Shrinkage Mechanism While sample splitting achieves perfect alignment by sacrificing some statistical power, the shrinkage mechanism offers an alternative tradeoff. It preserves the full testing budget while partially aligning incentives, guaranteeing that non-positive expected-value interventions remain strictly unprofitable. Shrinkage Mechanism. For each intervention i receiving bi>0b_i>0 units, the experimenter is rewarded based on the conditional empirical effect minus a variance penalty equal to the expected reward of a zero-effect intervention, Ci(bi,α)=σibiϕ(cα)C_i(b_i,α)= _i b_iφ(c_α). The net reward is: REshrink(,α)=∑i:bi>0[μ^i(bi)Li(bi,α)−Ci(bi,α)].R_E^shrink(b,α)= _i:b_i>0 [ μ_i(b_i)L_i(b_i,α)-C_i(b_i,α) ]. Theorem 4 (Sign Alignment). For every intervention i and allocation bi>0b_i>0, the expected reward under the shrinkage mechanism satisfies [μ^i(bi)Li(bi,α)−Ci(bi,α)]>0,μi>0,=0,μi=0,<0,μi<0.E\! [ μ_i(b_i)L_i(b_i,α)-C_i(b_i,α) ] cases>0,& _i>0,\\ =0,& _i=0,\\ <0,& _i<0. cases Thus, under risk neutrality, negative-effect interventions are strictly unprofitable to test, while zero-effect interventions offer no expected gain from favorable experimental noise. The formal proof is deferred to Section A.4. Theorem 5 (Platform-Value Guarantee). For any fixed α∈(0,1)α∈(0,1), the platform’s expected reward evaluated at the experimenter-optimal allocation in the shrinkage mechanism is a constant-fraction approximation of the maximum platform value achievable under the incumbent mechanism. In particular, for standard significance α=0.05α=0.05, numerical evaluation gives an approximation factor of approximately 0.17720.1772. The formal proof can be found in Section A.5. 5 Empirical Evaluation (a) Instance 1: Perfect Priors (b) Instance 2: Noisy Priors Figure 1: Expected platform value vs. acceptance threshold under platform-optimal allocation, experimenter-optimal allocation (incumbent mechanism), and sample splitting (γ=0.2γ=0.2). We complement our theoretical findings with empirically calibrated simulations across two distinct instances (N=1600N=1600, B=200,000B=200,000, minimum experiment size 500500 units). Distributing true expected treatment effects along a Gaussian mixture (a bulk of marginally harmful interventions alongside a thin tail of high-value winners), we evaluate sample splitting with a holdout fraction of γ=0.2γ=0.2. Instance 1 assumes perfect experimenter priors, whereas Instance 2 injects noise (μ~i=μi+εi μ_i= _i+ _i, with εi∼(0,0.22) _i (0,0.2^2)) to rigorously test robustness against imperfect information. In Figure 1, we plot expected platform value against the acceptance threshold. Under perfect priors (Instance 1), a zero threshold is optimal for both the platform-optimal allocation and sample splitting: with aligned incentives, statistical filtering is entirely unnecessary. With noisy priors (Instance 2), thresholds become essential to screen out harmful interventions. Crucially, even when optimally tuned, the incumbent mechanism significantly underperforms the platform optimum, empirically reinforcing Theorem 1: statistical gatekeeping cannot rescue structural misalignment. Ultimately, these simulations highlight our core theoretical tradeoff. Sample splitting sacrifices a mere 1.0%1.0\% and 11.6%11.6\% of optimal platform value in Instances 1 and 2, respectively. In sharp contrast, strategic decentralization under the incumbent mechanism costs 7.4%7.4\% and a drastic 42.1%42.1\% of optimal platform value. Thus, sample splitting recovers the vast majority of value otherwise lost to decentralized decision-making. 6 Conclusion Experimentation is operationally decentralized, driving a structural misalignment between platforms maximizing product value and experimenters optimizing significance-filtered empirical estimates. As we have demonstrated, this conflict is severe enough to drive program value to zero, and standard levers like significance thresholds fail to systematically resolve it. To address this failure, we provided novel economic interpretations for sample splitting and shrinkage, establishing their capacity to realign incentives ex ante. Sample splitting achieves perfect alignment at a bounded cost to experimentation budget, whereas shrinkage fully preserves the budget while strictly deterring negative-expected-value interventions. Moving forward, platforms must navigate this fundamental tradeoff between budget efficiency and behavioral alignment to safely leverage decentralized, AI-driven feature generation. References [1] I. Andrews, T. Kitagawa, and A. McCloskey (2024) Inference on winners. The Quarterly Journal of Economics 139 (1), p. 305–358. External Links: Document Cited by: §2. [2] E. M. Azevedo, A. Deng, J. L. Montiel Olea, J. Rao, and E. G. Weyl (2020) A/b testing with fat tails. Journal of Political Economy 128 (12), p. 4614–000. Cited by: §1, §2. [3] E. Bakshy, D. Eckles, and M. S. Bernstein (2014) Designing and deploying online field experiments. In Proceedings of the 23rd International Conference on World Wide Web, p. 283–292. External Links: Document Cited by: §1. [4] R. Berman and C. Van den Bulte (2022) False discovery in A/B testing. Management Science 68 (9), p. 6762–6782. External Links: Document Cited by: §2. [5] D. R. Cox (1975) A note on data-splitting for the evaluation of significance levels. Biometrika 62 (2), p. 441–444. External Links: Document Cited by: §2. [6] E. M. Feit and R. Berman (2019) Test & Roll: profit-maximizing A/B tests. Marketing Science 38 (6), p. 1038–1058. External Links: Document Cited by: §2. [7] W. Fithian, D. Sun, and J. Taylor (2014) Optimal inference after model selection. arXiv preprint arXiv:1410.2597. Cited by: §2. [8] A. Gelman and J. Carlin (2014) Beyond power calculations: assessing type S (Sign) and type M (Magnitude) errors. Perspectives on Psychological Science 9 (6), p. 641–651. External Links: Document Cited by: §2. [9] S. Hossain, Y. Chen, and Y. Chen (2025) Strategic hypothesis testing. Note: arXiv:2508.03289 External Links: 2508.03289 Cited by: §2. [10] R. L. Kaufman, J. Pitchforth, and L. Vermeer (2017) Democratizing online controlled experiments at Booking.com. Note: arXiv:1710.08217 External Links: 1710.08217 Cited by: §1. [11] R. Kessler (2024) Overcoming the winner’s curse: leveraging Bayesian inference to improve estimates of the impact of features launched via A/B tests. Note: Presented at the 2024 Conference on Digital Experimentation @ MIT (CODE@MIT)Amazon Science External Links: Link Cited by: §2. [12] M. R. Lee and M. Shen (2018) Winner’s curse: bias estimation for total effects of features in online controlled experiments. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p. 491–499. External Links: Document Cited by: §2. [13] R. Mudd, A. Zaidi, R. Friedberg, I. Gorbachev, A. Choubey, and H. Nassif (2026) Breaking the winner’s curse with Bayesian hybrid shrinkage. Note: arXiv:2603.12867. Extended version; a shorter version appeared as arXiv:2511.06318 External Links: 2603.12867 Cited by: §2. [14] S. Schmit, V. Shah, and R. Johari (2019) Optimal testing in the experiment-rich regime. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, K. Chaudhuri and M. Sugiyama (Eds.), Proceedings of Machine Learning Research, Vol. 89, p. 626–633. Cited by: §1, §2. [15] F. C. Shi, S. Bates, and M. J. Wainwright (2024) Sharp results for hypothesis testing with risk-sensitive agents. Note: arXiv:2412.16452 External Links: 2412.16452 Cited by: §2. [16] F. C. Shi, M. J. Wainwright, and S. Bates (2025) Instance-adaptive hypothesis tests with heterogeneous agents. Note: arXiv:2510.21178 External Links: 2510.21178 Cited by: §2. [17] T. Sudijono, S. Ejdemyr, A. Lal, and M. Tingley (2025) Optimizing returns from experimentation programs. In Proceedings of the 26th ACM Conference on Economics and Computation, EC ’25, New York, NY, USA, p. 869. External Links: ISBN 9798400719431, Document Cited by: §1, §2. [18] D. Tang, A. Agarwal, D. O’Brien, and M. Meyer (2010) Overlapping experiment infrastructure: more, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 17–26. External Links: Document Cited by: §1. [19] X. Tian and J. Taylor (2018) Selective inference with a randomized response. The Annals of Statistics 46 (2), p. 679–710. External Links: Document Cited by: §2. [20] E. W. van Zwet and E. A. Cator (2021) The significance filter, the winner’s curse and the need to shrink. Statistica Neerlandica 75 (4), p. 437–452. External Links: Document Cited by: §1, §2. [21] L. Wasserman and K. Roeder (2009) High-dimensional variable selection. The Annals of Statistics 37 (5A), p. 2178–2201. External Links: Document Cited by: §2. Appendix A Supplementary Material A.1 Proof of Theorem 1 Proof. Fix α,B,Vα,B,V and consider B+1B+1 interventions: intervention 00 has (μ0,σ0)=(M,1)( _0, _0)=(M,1), while each j∈1,…,Bj∈\1,…,B\ has (μj,σj)=(0,S)( _j, _j)=(0,S). The platform’s expected reward for allocating budget b>0b>0 to any j∈1,…,Bj∈\1,…,B\ is zero, whereas for intervention 0, it is MΦ¯(cα−Mb)>0M (c_α-M b)>0, which is monotonically non-decreasing in b. Thus, the platform-optimal allocation Pincb_P^inc assigns the entire budget B to intervention 0, achieving an expected reward of MΦ¯(cα−MB)>VM (c_α-M B)>V for sufficiently large M. In contrast, for any zero-mean intervention j∈1,…,Bj∈\1,…,B\, the experimenter’s expected reward rE,jinc(0,b)=Sϕ(cα)/br_E,j^inc(0,b)=Sφ(c_α)/ b is strictly decreasing in b. Consequently, the experimenter maximizes reward across these interventions by allocating exactly one unit to each. By choosing S>max1≤k≤BrE,0inc(M,k)kϕ(cα)S> _1≤ k≤ B r_E,0^inc(M,k)kφ(c_α), allocating any k≥1k≥ 1 units to intervention 0 yields a total experimenter reward of at most rE,0inc(M,k)+(B−k)Sϕ(cα)<BSϕ(cα)r_E,0^inc(M,k)+(B-k)Sφ(c_α)<BSφ(c_α). Thus, the experimenter-optimal allocation is Einc=(0,1,…,1)b_E^inc=(0,1,…,1), which yields [RPinc(Einc,α)]=0E[R_P^inc(b_E^inc,α)]=0. ∎ A.2 Proof of Theorem 2 Proof. Fix any feasible budget allocation b, and let Li∈0,1L_i∈\0,1\ indicate whether intervention i is launched. By construction, LiL_i depends only on the selection sample, while μ^iE μ_i^E is computed from the independent evaluation sample. Thus, μ^iE μ_i^E and LiL_i are independent. Moreover, the evaluation-sample estimate is unbiased on i.i.d. samples, so [μ^iE]=μiE[ μ_i^E]= _i. The experimenter’s expected reward is therefore [REsplit(,α)]=∑i=1N[μ^iE][Li(bi(1−γ),α)]=∑i=1Nμipi(bi(1−γ))=[RPsplit(,α)]E[R_E^split(b,α)]= _i=1^NE[ μ_i^E]E[L_i(b_i(1-γ),α)]= _i=1^N _ip_i(b_i(1-γ))=E[R_P^split(b,α)]. Since their expected rewards coincide for every feasible allocation b, their sets of optimal allocations must also coincide under risk neutrality. ∎ A.3 Proof of Theorem 3 Proof. For any intervention with μi>0 _i>0, its launch probability given b>0b>0 units is pi(b)=Φ¯(cα−μib/σi)p_i(b)= (c_α- _i b/ _i), which becomes pi((1−γ)b)p_i((1-γ)b) under the sample-splitting mechanism. We first show that pi((1−γ)b)≥(1−γ)pi(b)p_i((1-γ)b)≥(1-γ)p_i(b). Let ai=μi/σi>0a_i= _i/ _i>0 and x=aibx=a_i b. Then, pi′(b)=ai44x3ϕ(cα−x)(cαx−x2−1)p_i (b)= a_i^44x^3φ(c_α-x)(c_αx-x^2-1). Thus, pi′(b)≤0p_i (b)≤ 0 if and only if x2−cαx+1≥0x^2-c_αx+1≥ 0. For cα≤2c_α≤ 2, we have x2−cαx+1=(x−cα/2)2+1−cα2/4≥0x^2-c_αx+1=(x-c_α/2)^2+1-c_α^2/4≥ 0, meaning pi(b)p_i(b) is concave. Since its continuous extension satisfies pi(0)=Φ¯(cα)≥0p_i(0)= (c_α)≥ 0, concavity implies pi((1−γ)b)≥(1−γ)pi(b)+γpi(0)≥(1−γ)pi(b)p_i((1-γ)b)≥(1-γ)p_i(b)+γ p_i(0)≥(1-γ)p_i(b). The full-budget optimum allocates zero budget to interventions with μi<0 _i<0, as they would yield negative expected value. Applying the concavity inequality to all positively valued interventions in Pincb_P^inc gives [RPsplit(Pinc,α)]≥(1−γ)[RPinc(Pinc,α)]E[R_P^split(b_P^inc,α)]≥(1-γ)E[R_P^inc(b_P^inc,α)]. Finally, by Theorem 2, Esplitb_E^split maximizes platform value under sample splitting, yielding [RPsplit(Esplit,α)] [R_P^split(b_E^split,α)] =[RPsplit(Psplit,α)] =E[R_P^split(b_P^split,α)] ≥[RPsplit(Pinc,α)] [R_P^split(b_P^inc,α)] ≥(1−γ)[RPinc(Pinc,α)]. ≥(1-γ)E[R_P^inc(b_P^inc,α)]. (4) ∎ A.4 Proof of Theorem 4 Proof. Fix an intervention i and budget bi>0b_i>0. Recall that its expected reward for the experimenter under the incumbent mechanism is rE,iinc(μi,bi)=μipi(bi)+σibiϕ(cα−μibiσi),r_E,i^inc( _i,b_i)= _ip_i(b_i)+ _i b_iφ (c_α- _i b_i _i ), where cα=z1−α/2c_α=z_1-α/2. The shrinkage charge is exactly the expected reward of a zero-effect intervention with the same variance as i: Ci(bi,α)=rE,iinc(0,bi)=σibiϕ(cα).C_i(b_i,α)=r_E,i^inc(0,b_i)= _i b_iφ(c_α). Hence, the experimenter’s expected reward for allocating bib_i experimental units to intervention i under the shrinkage mechanism is: rE,ishrink(μi,bi)=rE,iinc(μi,bi)−rE,iinc(0,bi)r_E,i^shrink( _i,b_i)=r_E,i^inc( _i,b_i)-r_E,i^inc(0,b_i). It immediately follows that rE,ishrink(0,bi)=0r_E,i^shrink(0,b_i)=0, meaning the experimenter has no incentive to launch experiments where the true effect is zero. Finally, differentiating rE,ishrink(μ,bi)r_E,i^shrink(μ,b_i) with respect to μ: ∂rE,ishrink(μ,bi)∂μ=∂rE,iinc(μ,bi)∂μ+0=pi(bi)+cαϕ(cα−μbiσi)>0, ∂ r_E,i^shrink(μ,b_i)∂μ= ∂ r_E,i^inc(μ,b_i)∂μ+0=p_i(b_i)+c_αφ (c_α- μ b_i _i )>0, since pi(bi)>0p_i(b_i)>0, cα>0c_α>0, and ϕ(⋅)>0φ(·)>0. Thus, rE,ishrink(μi,bi)>0,μi>0,=0,μi=0,<0,μi<0,r_E,i^shrink( _i,b_i) cases>0,& _i>0,\\ =0,& _i=0,\\ <0,& _i<0, cases which proves the claim. ∎ A.5 Proof of Theorem 5 Proof. Fix an intervention i with μi>0 _i>0 and an allocation bi>0b_i>0, and define its signal-to-noise ratio as λi=μibiσi>0. _i= _i b_i _i>0. Its launch probability is pi(bi)=Φ¯(cα−λi).p_i(b_i)= (c_α- _i). Hence, its expected contribution to the platform’s reward is rP,ishrink(μi,bi)=rP,iinc(μi,bi)=μipi(bi)=μiΦ¯(cα−λi).r_P,i^shrink( _i,b_i)=r_P,i^inc( _i,b_i)= _ip_i(b_i)= _i (c_α- _i). From Equation 3 under the incumbent mechanism, the experimenter’s expected reward from this intervention is rE,iinc(μi,bi)=μipi(bi)+σibiϕ(cα−λi).r_E,i^inc( _i,b_i)= _ip_i(b_i)+ _i b_iφ(c_α- _i). The shrinkage mechanism subtracts Ci(bi,α)=σibiϕ(cα).C_i(b_i,α)= _i b_iφ(c_α). Thus, the experimenter’s expected reward under the shrinkage mechanism is rE,ishrink(μi,bi)=μipi(bi)+σibi[ϕ(cα−λi)−ϕ(cα)].r_E,i^shrink( _i,b_i)= _ip_i(b_i)+ _i b_i [φ(c_α- _i)-φ(c_α) ]. (5) Since μi=λiσibi, _i= _i _i b_i, we can rewrite the platform’s expected reward as rP,ishrink(μi,bi)=μipi(bi)=σibiλiΦ¯(cα−λi).r_P,i^shrink( _i,b_i)= _ip_i(b_i)= _i b_i _i (c_α- _i). Thus, rE,ishrink(μi,bi)rP,ishrink(μi,bi) r_E,i^shrink( _i,b_i)r_P,i^shrink( _i,b_i) =1+σibi[ϕ(cα−λi)−ϕ(cα)]σibiλiΦ¯(cα−λi) =1+ _i b_i [φ(c_α- _i)-φ(c_α) ] _i b_i _i (c_α- _i) =1+ϕ(cα−λi)−ϕ(cα)λiΦ¯(cα−λi) =1+ φ(c_α- _i)-φ(c_α) _i (c_α- _i) =:qα(λi). =:q_α( _i). (6) We next demonstrate that qα(λ)∈ℝ>0q_α(λ) _>0 for all λ>0λ>0. Let mα=infλ>0qα(λ)m_α= _λ>0q_α(λ) and Mα=supλ>0qα(λ)M_α= _λ>0q_α(λ). First, by Theorem 4, rE,ishrink(μi,bi)>0r_E,i^shrink( _i,b_i)>0 whenever μi>0 _i>0. Since rP,ishrink(μi,bi)=μipi(bi)>0r_P,i^shrink( _i,b_i)= _ip_i(b_i)>0 as well, qα(λ)>0∀λ>0.q_α(λ)>0\;∀λ>0. To see that Mα<∞M_α<∞, first note that qα(λ)q_α(λ) has the finite limit limλ→0+qα(λ) _λ→ 0^+q_α(λ) =1+limλ→0+ϕ(cα−λ)−ϕ(cα)λΦ¯(cα−λ) =1+ _λ→ 0^+ φ(c_α-λ)-φ(c_α)λ (c_α-λ) =1+limλ→0+(cα−λ)ϕ(cα−λ)Φ¯(cα−λ)+λϕ(cα−λ) =1+ _λ→ 0^+ (c_α-λ)φ(c_α-λ) (c_α-λ)+λφ(c_α-λ) =1+cαϕ(cα)Φ¯(cα), =1+ c_αφ(c_α) (c_α), (7) where the second equality follows from L’Hôpital’s rule using ϕ′(x)=−xϕ(x)φ (x)=-xφ(x). Hence, by continuity, qα(⋅)q_α(·) is bounded on (0,δ](0,δ] for some δ>0δ>0. For λ≥δλ≥δ, λΦ¯(cα−λ)≥δΦ¯(cα−δ)>0,λ (c_α-λ)≥δ (c_α-δ)>0, while ϕφ is bounded. Thus qα(λ)q_α(λ) is also bounded on [δ,∞)[δ,∞), and therefore Mα<∞M_α<∞. Thus, 0<mα≤Mα<∞.0<m_α≤ M_α<∞. It follows that, for every positive-effect intervention i, mαrP,ishrink(μi,bi)≤rE,ishrink(μi,bi)≤MαrP,ishrink(μi,bi).m_αr_P,i^shrink( _i,b_i)≤ r_E,i^shrink( _i,b_i)≤ M_αr_P,i^shrink( _i,b_i). We now extend this intervention-level bound to complete budget allocations. By Theorem 4, interventions with negative and zero true effects yield strictly negative and zero expected reward for the experimenter under the shrinkage mechanism, respectively. Thus, an experimenter-optimal allocation will never allocate budget to a negative-effect intervention. Similarly, a platform-optimal allocation never benefits from allocating budget to any intervention with non-positive true effect. Thus, for any relevant allocation b: [REshrink(,α)]=∑i:μi>0rE,ishrink(μi,bi)≥mα∑i:μi>0rP,ishrink(μi,bi)=mα[RPshrink(,α)],E[R_E^shrink(b,α)]= _i: _i>0r_E,i^shrink( _i,b_i)≥ m_α _i: _i>0r_P,i^shrink( _i,b_i)=m_αE[R_P^shrink(b,α)], (8) and similarly, [REshrink(,α)]≤Mα[RPshrink(,α)].E[R_E^shrink(b,α)]≤ M_αE[R_P^shrink(b,α)]. Let Eshrinkb_E^shrink and Pshrinkb_P^shrink denote experimenter- and platform-optimal allocations under the shrinkage mechanism, respectively. By definition, [REshrink(Eshrink,α)]≥[REshrink(Pshrink,α)].E [R_E^shrink(b_E^shrink,α) ] [R_E^shrink(b_P^shrink,α) ]. Putting everything together, Mα[RPshrink(Eshrink,α)] M_αE [R_P^shrink(b_E^shrink,α) ] ≥[REshrink(Eshrink,α)] [R_E^shrink(b_E^shrink,α) ] ≥[REshrink(Pshrink,α)] [R_E^shrink(b_P^shrink,α) ] ≥mα[RPshrink(Pshrink,α)]. ≥ m_αE [R_P^shrink(b_P^shrink,α) ]. Since Mα>0M_α>0, [RPshrink(Eshrink,α)]≥mαMα[RPshrink(Pshrink,α)].E [R_P^shrink(b_E^shrink,α) ]≥ m_αM_αE [R_P^shrink(b_P^shrink,α) ]. Since shrinkage changes only the experimenter’s payoff, not the launch rule, RPshrink(,α)=RPinc(,α)R_P^shrink(b,α)=R_P^inc(b,α) for every allocation b, so Pshrink=Pincb_P^shrink=b_P^inc and the right-hand side equals mαMα[RPinc(Pinc,α)] m_αM_αE[R_P^inc(b_P^inc,α)]. For α=0.05α=0.05, numerical evaluation yields mα≈0.9891m_α≈ 0.9891 and Mα≈5.5820M_α≈ 5.5820, and therefore mαMα≈0.1772 m_αM_α≈ 0.1772, completing the proof. ∎