Paper deep dive
Optimizing Social Utility in Sequential Experiments
Ander Artola Velasco, Stratis Tsirtsis, Manuel Gomez-Rodriguez
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/8/2026, 10:56:32 AM
Summary
This paper proposes a statistical protocol for sequential randomized controlled trials where a regulator (principal) partially subsidizes a product developer (agent) to overcome high trial costs. Modeled as a belief Markov decision process, the agent's optimal strategy is found via dynamic programming, while the principal optimizes the subsidy level to maximize social utility using divide-and-conquer. Simulations on antibiotic development show a >35% increase in social utility compared to non-sequential protocols.
Entities (8)
Relation Signals (8)
Agent → conducts → SequentialExperimentation
confidence 90% · agent conducts a randomized controlled trial sequentially
SequentialExperimentation → increases → SocialUtility
confidence 90% · increase social utility by more than 35% relative to standard, non-sequential protocols
SubsidyLevel → influences → SocialUtility
confidence 90% · social utility is a piecewise linear and convex function over the subsidy level
SequentialExperimentation → modeledas → BeliefMDP
confidence 90% · By modeling the protocol using a belief Markov decision process
Principal → subsidizes → Agent
confidence 90% · regulator (the principal) partially subsidizes its cost
SocialUtility → optimizedby → DivideAndConquer
confidence 85% · socially optimal subsidy can also be found efficiently using divide-and-conquer
BeliefMDP → solvedby → DynamicProgramming
confidence 85% · agent's optimal strategy can be found efficiently using dynamic programming
SequentialExperimentation → appliedto → AntibioticDevelopment
confidence 80% · simulation experiments using publicly available data on antibiotic development
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Regulatory approval of products in high-stakes domains such as drug development requires statistical evidence of safety and efficacy through large-scale randomized controlled trials. However, the high financial cost of these trials may deter developers who lack absolute certainty in their product's efficacy, ultimately stifling the development of `moonshot' products that could offer high social utility. To address this inefficiency, in this paper, we introduce a statistical protocol for experimentation where the product developer (the agent) conducts a randomized controlled trial sequentially and the regulator (the principal) partially subsidizes its cost. By modeling the protocol using a belief Markov decision process, we show that the agent's optimal strategy can be found efficiently using dynamic programming. Further, we show that the social utility is a piecewise linear and convex function over the subsidy level the principal selects, and thus the socially optimal subsidy can also be found efficiently using divide-and-conquer. Simulation experiments using publicly available data on antibiotic development and approval demonstrate that our statistical protocol can be used to increase social utility by more than $35$$\%$ relative to standard, non-sequential protocols.
Tags
Links
- Source: https://arxiv.org/abs/2605.06520v1
- Canonical: https://arxiv.org/abs/2605.06520v1
Trouble viewing inline? Open PDF directly →
Full Text
163,175 characters extracted from source content.
Expand or collapse full text
Optimizing Social Utility in Sequential Experiments Ander Artola Velasco § , Stratis Tsirtsis † , and Manuel Gomez-Rodriguez § § Max Planck Institute for Software Systems, Kaiserslautern, Germany avelasco, manuel@mpi-sws.org † Hasso Plattner Institute, Potsdam, Germany stratis.tsirtsis@hpi.de Abstract Regulatory approval of products in high-stakes domains such as drug development requires statistical evidence of safety and efficacy through large-scale randomized controlled trials. However, the high financial cost of these trials may deter developers who lack absolute certainty in their product’s efficacy, ultimately stifling the development of ‘moonshot’ products that could offer high social utility. To address this inefficiency, in this paper, we introduce a statistical protocol for experimentation where the product developer (the agent) conducts a randomized controlled trial sequentially and the regulator (the principal) partially subsidizes its cost. By modeling the protocol using a belief Markov decision process, we show that the agent’s optimal strategy can be found efficiently using dynamic programming. Further, we show that the social utility is a piecewise linear and convex function over the subsidy level the principal selects, and thus the socially optimal subsidy can also be found efficiently using divide-and-conquer. Simulation experiments using publicly available data on antibiotic development and approval demonstrate that our statistical protocol can be used to increase social utility by more than 35% relative to standard, non-sequential protocols. 1 Introduction Access to markets in high-stakes domains, such as drug development, is strictly governed by regulatory bodies to ensure that new products meet rigorous safety and efficacy standards [1,2]. Randomized controlled trials (RCTs) serve as one of the primary mechanisms for access control, requiring developers to gather sufficient statistical evidence to prove that a product is safe and effective [3–5]. However, the high financial costs of RCTs can stifle the development of ‘moonshot’ products that could offer high social utility, as developers may hesitate to proceed if they lack absolute certainty in their product’s efficacy [6–8]. To avoid missing such opportunities, calls have emerged for adaptive RCTs [9–11] and targeted subsidies [12–15], particularly in the context of orphan and rare diseases where small patient populations often discourage private investment. Yet, the effectiveness of such interventions depends on how developers strategically respond to them. In this context, a recent line of work [16–18] has argued that the regulatory approval of products is best modeled as a principal-agent game [19], where the regulator (the principal) designs an approval protocol and a hypothesis testing rule to incentivize the product developer (the agent) to act in a way that aligns with the principal’s interests. In this paper, we extend this line of work to consider a setting in which the RCTs are conducted sequentially and are (partially) subsidized by the principal. Our contributions. We introduce a statistical protocol for experimentation where the product developer (the agent) conducts an RCT sequentially and the regulator (the principal) partially subsidizes its cost. At each step of the protocol, the agent and principal update their beliefs about the product’s effectiveness based on the latest experimental outcome. If there is sufficient statistical evidence to reject the null hypothesis [20], 1 arXiv:2605.06520v1 [cs.GT] 7 May 2026 the principal approves the product; otherwise, they request that the agent gather more evidence, and the agent may either proceed with the experiment or terminate without approval. By modeling the protocol using a belief Markov decision process [21], we show that the agent’s optimal strategy can be found efficiently using dynamic programming. Further, we show that the social utility is a piecewise linear and convex function over the subsidy level the principal selects, and thus the socially optimal subsidy can also be found efficiently via divide-and-conquer. To validate our statistical protocol, we conduct simulation experiments using publicly available data on antibiotic development and approval. The results show that our protocol can be used to increase social utility by more than 35% relative to standard, non-sequential protocols. 1 Further related work. Our work builds upon further related work on the economic aspects of statistical testing, sequential hypothesis testing, and Bayesian experimental design. A recent and closely related line of work studies the economic and strategic incentives arising in regulatory approval processes [16–18,22,23]. Therein, Shi et al. [17] and Hossain et al. [18] focus on determining the opti- mal hypothesis test that the principal can use to control false positives and false negatives, while Tetenov [22] and Bates et al. [16] study, respectively, the design of approval and payment rules to disincentivize agents who know their product is ineffective from participating in the approval process. Relatedly, McClellan [23] considers a setting in which the principal designs approval rules to encourage agent participation without monetary transfers, e.g., by lowering approval standards if previous experiments were not successful. In contrast to these works, we study how the principal can optimally subsidize the agent to increase social utility in a scenario where both are uncertain about the effectiveness of the product. Within the hypothesis testing literature, our work draws on an active line of research on anytime-valid statistical inference using e-values [20,24,25], which has been successfully applied to a wide range of statistical problems [16,26–35]. Most closely related to ours is the work of Bates et al. [16], who also uses e-values in the context of RCTs. However, their focus is on designing contracts that disincentivize agents with ineffective products from participating in the approval process. We instead use e-values to design an approval protocol that provides sequential error guarantees to the principal. Our work also connects to the broad literature on Bayesian experimental design [36–44]. Among these, the closest works to ours [42–44] model a Bayesian agent’s experiment selection problem using (partially observable) Markov decision processes. However, their objective is to design policies that maximize information gain while trading off experimental costs. In contrast, in the approval setting we study, the agent has direct economic incentives to conduct experiments, as product approval is financially beneficial. 2 A Protocol for Subsidized Sequential Experimentation We consider an agent (the product developer) who seeks regulatory approval for a product from a principal (the regulator). The product is characterized by an efficacy parameterθ ∗ ∈[0,1], unknown both to the agent and the principal, with higher values indicating a more effective product. 2 Motivated by multi-stage clinical trials [2,4], in this section, we introduce a sequential approval process in which the agent conducts a sequence of RCTs to provide sufficient evidence that the product meets the principal’s standards. In turn, the principal commits to subsidize a fraction of the agent’s total experimentation cost and, after each trial, they decide whether to approve the product or require the agent to conduct further experimentation. The agent begins the (sequential) approval process with a prior beliefB 0 about the efficacyθ ∗ of its product, which we model using a Beta distributionB 0 =Beta(α 0 ,β 0 ), whereα 0 ,β 0 >0 are given parameters. 3 At each time stept∈[T] =0,1,...,Tof the approval process, the agent’s action is to either (i) conduct a randomized trial with a sample sizen t ∈ 1,...,n max and incur a costc(n t ), wherec:N → R + is a 1 The code for our experiments is publicly available athttps://github.com/Human-Centric-Machine-Learning/strategic- experiments. 2 In certain settings, the principal may be interested not only in the efficacy of a product but also in other properties of the product such as its safety. In Appendix C, we discuss how to extend the approval process to such settings. 3 The parametersα 0 andβ 0 characterize prior information the agent may have about the product based on, e.g., preliminary tests. In Appendix C, we show that the approval process can be extended to more general settings with arbitrary prior beliefs. 2 non-decreasing cost function, or (i) opt out and stop the approval process at no additional cost, which we represent asn t = 0 withc(0) = 0. If the agent decides to conduct a trial by selectingn t >0, the agent observes a random outcomeX t ∼ Bin(n t ,θ ∗ ), which represents the number of successes in the trial and depends on the unknown efficacyθ ∗ , and this outcome is then revealed to the principal. 4 Then, based on the outcome X t , the agent updates their (posterior) belief B t about the true efficacy θ ∗ of the product, i.e., B t+1 = Beta(α t + X t |z α t+1 ,β t + n t − X t |z β t+1 ).(1) Throughout the process, the agent employs a (possibly randomized) policyπ ∈Π to select the sample sizes n t ∼ π(α t ,β t ,C t ,t) based on their beliefB t =Beta(α t ,β t ) about the efficacy of the product, the total cost C t = P t−1 k=0 c(n k ) they have incurred so far, and the time steptof the approval process. Moreover, the principal uses the revealed outcomesX t as evidence to decide on the approval of the product. Formally, the principal conducts a (sequential) hypothesis test with null and alternative hypotheses given by H 0 = θ ∗ : θ ∗ < θ b and H 1 = θ ∗ : θ ∗ ≥ θ b ,(2) whereθ b ∈(0,1) is a baseline efficacy mandated by the principal and known to the agent (e.g., the efficacy of the current standard-of-care treatment in the context of clinical trials). The null hypothesisH 0 therefore corresponds to the product failing to meet the principal’s standard, and we will equivalently refer to the principal approving the product as rejecting H 0 . In general, the principal can implement any decision rule to rejectH 0 . However, in what follows, we draw on the literature on sequential hypothesis testing [20,25,45], and consider a principal who aims to control the false positive rate, i.e., the probability of approving a product whose efficacy does not exceed the baseline θ b . Concretely, based on the sample sizen t and revealed outcomeX t , the principal computes—and shares with the agent—a non-negative quantityE(X t ,n t )∈ R + , referred to as an e-value, quantifying the observed evidence againstH 0 at timet(i.e., larger e-values correspond to stronger evidence againstH 0 ). 5 Then, the principal aggregates all available evidence multiplicatively via a (stochastic) processM, which we will refer to as the test process, and whose value at time t is defined as: M t = ( 1t = 0 E(X t−1 ,n t−1 )· M t−1 t≥ 1. (3) Finally, based on the test processM, the principal rejects the null hypothesisH 0 (approves the product) as soon as the accumulated evidence exceeds a fixed thresholdκ ∈(0,1) set in advance, i.e., as soon as M t+1 ≥1/κfor some time stept. Here,κacts as a tolerance parameter specifying how much evidence must be accumulated before rejectingH 0 , where smaller values ofκcorrespond to a more conservative approval standard set by the principal, and note that, if the process fails to yield sufficient evidence for approval within T time steps, it concludes without approval. In the above sequential test, the false positive rate is bounded byκas long as the e-values are such that E H 0 [E(X t ,n t )]≤ 1, as shown elsewhere [20], i.e., P H 0 (∃t∈ [T ]:M t+1 ≥ 1/κ)≤ κ.(4) Importantly, the above guarantee on the false positive rate holds throughout the entire approval process—a particularly desirable property known as any-time validity [24]. To ensure thatE H 0 [E(X t ,n t )]≤1 holds, we construct an e-value that exponentiates a sufficient statistic for the unknown efficacyθ ∗ [46,47] by comparing the empirical success rate X t /n t against a monotone transformation of the baseline θ b : 6 4 The specific meaning of the number of successes is application dependent. In the context of clinical trials, it may correspond to the number of patients who recover after receiving a treatment. 5 ComputingE(X t ,n t ) plays a role similar to that of a p-value for rejectingH 0 , while offering stronger guarantees in sequential settings. See Appendix B for a brief overview of sequential hypothesis testing with e-values. 6 See Appendix C for an extension to arbitrary e-values; proofs are deferred to Appendix F. 3 Proposition 1. Given the (unknown) efficacy parameterθ ∗ ∈[0,1],n t >0, and a binomial variable X t ∼ Bin(n t ,θ ∗ ), the positive random variable E(X t ,n t ) = exp X t − n t · log(1 + θ b (e− 1)) (5) is a valid e-value under H 0 =θ ∗ : θ ∗ < θ b , that is, E X t ∼Bin(n t ,θ ∗ ) [E(X t ,n t )]≤ 1 for any θ ∗ ∈ H 0 . Furthermore, given the specific form of e-value in Eq. 5, it is easy to verify that the value of the test process M t is uniquely determined by the parameters α t and β t characterizing the agent’s belief: Proposition 2. At any time step t∈ [T ], the value of the test process M t satisfies: M t = f (α t ,β t ) where f (α,β) = exp α− α 0 − (α + β− α 0 − β 0 )· log(1 + θ b (e− 1)) .(6) If the agent’s product is approved, the agent and the principal obtain benefitsρ A ,ρ S >0, respectively, and we refer to the latter as the social benefit upon approval. In the context of clinical trials,ρ A can be interpreted as the (estimated) economic benefit obtained by the agent—the pharmaceutical company—from drug sales if the drug is approved, whileρ S represents the corresponding benefit to society resulting from bringing an effective treatment to market. Since the principal also benefits from a product’s approval, it can be in their interest to incentivize the agent to continue experimenting, particularly in situations where the agent’s benefitρ A is not high enough to compensate for the total experimentation cost required to rejectH 0 . In the following, we focus our attention on a natural and widely-used mechanism through which the principal can provide such an incentive: subsidizing, conditional on approval, a fractionε ∈[0,ε max ] of the total cost incurred by the agent, where ε max ≤ 1 denotes the maximum fraction the principal is willing to subsidize [48, 49]. Given the above benefits, experimentation cost, and subsidies, the agent’s and the principal’s (expected) utilities U A and U S are given by U A (π;ε) = E π " ρ A + ε· τ X t=0 c(n t ) ! · 1M τ +1 ≥ 1/κ− τ X t=0 c(n t ) θ ∗ # , U S (ε;π) = E π " ρ S − ε· τ X t=0 c(n t ) ! · 1M τ +1 ≥ 1/κ θ ∗ # , (7) whereτ=T ∧ mint ∈[T]:n t = 0or M t+1 ≥1/κis the last step of the approval process,1•is the indicator function, and the expectation E π [•|θ ∗ ] is taken over the random outcomes X t ∼ Bin(n t ,θ ∗ ). 7 However, since the agent and the principal do not know the true efficacyθ ∗ a priori, they cannot find the policyπand subsidyεthat maximize their respective utilities, as defined in Eq. 7. In the next sections, we investigate how the agent and principal may leverage their evolving beliefs about efficacy to find near-optimal policies and subsidies as the approval process progresses. 3 Experimental Design Using Belief Markov Decision Processes If the agent does not know the true efficacyθ ∗ , we argue that, as the approval process progresses, they act based on their beliefs regardingθ ∗ given the trial outcomes observed [50]. Consequently, we consider an agent who determines their trial sample sizes by planning ahead—not according to the (unknown) actual evolution of the process, but according to how they anticipate their beliefs will evolve depending on their actions and anticipated outcomes. To formalize the agent’s planning strategy, we employ the framework of (belief) Markov decision processes [21,51]. Specifically, for a fixed subsidyεspecified by the principal, we define the process M ε = (S,A,P,r ε ,T ), whose components we describe next. 7 Throughout, we adopt the convention min∅ = +∞ and write x∧ y = min(x,y) for x,y ∈ R∪+∞. 4 The state spaceS=S in ∪S out includes statesS= (α,β,C)∈S in , whereαandβare the parameters characterizing the agent’s belief andCis the agent’s total running cost, as well as a special absorbing state S out , which indicates that the agent has opted out of the approval process. The action spaceA=0,...,n max consists of all possible sample sizes the agent may select. Further, the transition distributionPcharacterizes how the agent anticipates their belief and cost will evolve after they conduct a trial of sizen. Formally, a transition from a stateS= (α,β,C) to a stateS ′ follows from the (randomized) assignment S ′ = (α + X, β + n− X, C + c(n)) if n > 0 and f (S) < 1/κ Sif n > 0 and f (S)≥ 1/κ S out if n = 0 (8) whereX ∼ Bin(n,θ) withθ ∼ Beta(α,β), and the functionf(S) =f(α,β) as in Eq. 6 ifS ∈ S in and f(S out ) = 0. In words, the upper case captures the Bayesian belief update and the additional cost incurred by the agent for conducting an experiment of sizenand observing an outcomeX(see Eq. 1), the middle case captures the successful conclusion of the approval process, and the lower case captures the scenario in which the agent decides to opt out. Importantly, note that evaluating the likelihood of such transitions does not depend on the efficacy θ ∗ and thus is possible solely based on the agent’s belief. Finally, the rewardr ε characterizes the agent’s anticipated profit or loss due to conducting a trial under subsidy ε. Formally, the reward for transitioning from state S to state S ′ via action n is given by r ε (S,n,S ′ ) = ( −c(n) + (ρ A + ε· (C + c(n)))· 1f (S ′ )≥ 1/κ if S ̸= S out and f (S) < 1/κ 0if S = S out or f (S)≥ 1/κ. (9) In words, as long as the approval process is in progress, the reward is simply equal to the (negative) costc(n) the agent anticipates to incur by conducting a trial with their chosen sample sizen. If the updated stateS ′ leads to product approval (i.e.,f(S)<1/κandf(S ′ )≥1/κ), the agent anticipates receiving their one-off benefitρ A >0 and a subsidized fraction of their total incurred cost throughout the process, as determined by the subsidy ε selected by the principal. Now that we have defined all the components of the Markov decision processM ε , we can formalize the agent’s strategy within the process. Starting from the initial stateS 0 = (α 0 ,β 0 ,0), the agent aims to select a policy π :S× [T ]→ ∆(A) maximizing their anticipated utility ̄ U A (π;ε) = E π " T X t=0 r ε (S t ,n t ,S t+1 ) S 0 = (α 0 ,β 0 , 0) # ,(10) where the expectation is taken over the state transitions that the agent anticipates to occur throughout the approval process, given their initial belief. Here, it is important to note that both the rewardr ε and the transition distributionPare known to the agent, hence they can evaluate the anticipated utility of any given policyπbefore the approval process starts. Moreover, due to Bellman’s optimality principle in MDPs [51], for any S ∈S and time step l∈ [T ], the agent’s optimal policy π ε ∈ argmax π ̄ U A (π;ε) satisfies that π ε (S,l)∈ argmax π V ε π (S,l) where V ε π (S,l) = E π " T X t=l r ε (S t ,n t ,S t+1 ) S l = S # .(11) In the above equation, the functionV ε π is often referred as the value function. Moreover, note that ̄ U A (π;ε) = V ε π (α 0 ,β 0 , 0, 0). In this context, a natural question is how the anticipated utility ̄ U A (π;ε) relates to the agent’s true utility U A (π;ε) defined in Eq. 7. The following proposition shows that the anticipated utility is equal to true agent utility averaged over the agent’s initial belief about the true efficacyθ ∗ of their product at the start of the approval process. 5 Proposition 3. Let ε∈ [0,ε max ] be any subsidy set by the principal, and π be any agent policy. Then, ̄ U A (π;ε) = E θ ∗ ∼B 0 U A (π;ε) .(12) The above proposition reveals that an agent implementing the decision policyπ ε is, in fact, a Bayesian decision maker maximizing their expected utility [39, 41, 52]. Next, we formalize the principal’s strategy within the process. LetQbe a prior distribution characterizing the principal’s prior knowledge about the agent’s initial belief (α 0 ,β 0 ). Then, the principal aims to select a subsidy ε maximizing the anticipated social utility ̄ U S (ε;π) = E (α 0 ,β 0 )∼Q " E π " T X t=0 ρ S − ε· C t+1 · 10 < f (S t ) < 1/κ≤ f (S t+1 ) S 0 = (α 0 ,β 0 , 0) ## , (13) where the inner expectation is taken over the state transitions that the principal anticipates to occur throughout the approval process, given the prior knowledge about the agent’s initial belief. In the next sections, we analyze how the agent and the principal maximize their anticipated utilities ̄ U A (π;ε) and ̄ U S (ε;π) within a natural Stackelberg setting [53]. In this setting, the principal (the leader) first commits to a subsidy levelε ∗ maximizing the social utility ̄ U S , accounting for the fact that, given anyε, the agent (the follower) will implement the optimal policyπ ε that maximizes their utility ̄ U A . Then, after observing this commitment, the agent implements the optimal policy. Formally, the subsidy levelε ∗ is the solution to the following optimization problem: ε ∗ = argmax ε∈[0,ε max ] ̄ U S (ε;π ε ) subject to π ε ∈ argmax π∈Π ̄ U A (π;ε).(14) The above Stackelberg setting fits a variety of real-world applications. For example, in clinical trials, funding agencies act as leaders who commit to and announce a funding program; drug developers are the followers who apply to the program and, if successful, run a trial [54, 55]. 4 Optimal Experimental Design under Subsidies In this section, we characterize the agent’s optimal policy π ε along with its corresponding value function V ε (S,l) = max π∈Π V ε π (S,l) ∀S ∈S, l∈ [T ].(15) To this end, we first note that, once the processM ε reaches a state wheref(S)≥1/κ, or the stateS out , the actionn= 0 is always optimal. This is implied by the transition dynamics and the reward definition in Eqs. 8 and 9. Therefore, to find the policyπ ε , it suffices to consider statesSwhere 0< f(S)<1/κ. Further, we show that, in the process M ε , the number of (unique) reachable states S where f (S)≥ 1/κ is finite: Proposition 4. For any realization of the processM ε , any reached stateSwhere 0< f(S)<1/κbelongs to a finite set S r ⊂S. Moreover, under linear cost c(n), the set S r has size O((n max ) 2 · T 3 ). As an immediate consequence, we can find an optimal (deterministic) policyπ ε using standard planning methods. In particular, in Algorithm 2, we provide an adaptation of the classical value iteration algorithm [56], which is guaranteed to find an optimal policyπ ε inO((n max ) 4 · T 3 ). In the remainder of this section, we derive several key structural insights about the optimal policyπ ε and the optimal value functionV ε , which will be helpful to efficiently find the principal’s optimal subsidy ε ∗ . Our starting point is the observation that, in light of Proposition 2, the value of the test processMat a stateS= (α,β,C) increases withαand decreases withβ. This suggests that states with largerαand smallerβare more favorable. The following proposition formalizes this intuition by showing that the optimal value functionV ε satisfies a monotonicity property in both the belief parameters and the accumulated cost. 6 Proposition 5. For any time stepl ∈[T] and pair of statesS= (α,β,C) andS ′ = (α ′ ,β ′ ,C ′ ) such that f (S) < 1/κ and f (S ′ ) < 1/κ, the following holds: 1. V ε (α,β,C,l)≤ V ε (α ′ ,β,C,l) if α≤ α ′ ; 2. V ε (α,β,C,l)≥ V ε (α,β ′ ,C,l) if β ≤ β ′ ; 3. V ε (α,β,C,l)≤ V ε (α,β,C ′ ,l) if C ≤ C ′ . Leveraging the above proposition, we can characterize the conditions under which the optimal policyπ ε opts out of the approval process by selecting the actionn= 0. In particular, the following proposition shows thatπ ε exhibits a threshold structure: at a state with belief (α,β), it selectsn= 0 if and only ifβexceeds a threshold that depends on α (see Figure 3 in Appendix D for an illustration). Proposition 6. For any time stept∈[T] and total costC t , there exists a non-decreasing function ̃ β:R + → R + such that, for any stateS t = (α t ,β t ,C t )∈ S in , the optimal policyπ ε opts out of the approval process if β t > ̃ β(α t ), and it does not opt out if β t < ̃ β(α t ). Furthermore, we can also characterize how the agent’s anticipated utility ̄ U A (π ε ;ε) under the optimal policyπ ε depends on the subsidy. To this end, we first show that, for any policyπ, the value functionV ε π is linear in the subsidy ε: Proposition 7. For any policyπ, stateS ∈S, and time stepl∈[T], the value functionV ε π admits a linear decomposition V ε π (S,l) = V 0 π (S,l) + ε· A π (S,l),(16) whereV 0 π (S,l) is the value function in the unsubsidized processM 0 , andA π (S,l)≥0 denotes the expected total cost incurred conditional on approval, starting from state S at time l (see Eq. 57 in Appendix F.9). Building upon this result, the agent’s anticipated utility ̄ U A (π ε ;ε) under the optimal policyπ ε admits a concise structural characterization, as formalized by the following proposition: Proposition 8. The agent’s optimal anticipated utility ̄ U A (π ε ;ε), and its expectationE (α 0 ,β 0 )∼Q [ ̄ U A (π ε ;ε)] according to the principal’s beliefQ, are piecewise linear, continuous, and convex functions of the subsidyε over a partitionP=ε 0 ,ε 1 ,...,ε L of the interval [0,ε max ], with 0 =ε 0 < ε 1 <· < ε L =ε max . Moreover, for each interval of P, the agent’s optimal policy is constant, i.e., π ε = π i for all ε∈ [ε i ,ε i+1 ). In the next section, we leverage the above characterization of the agent’s anticipated optimal utility to develop an algorithm that computes the principal’s optimal subsidy ε ∗ , as defined in Eq. 14. 8 5 Finding Optimal Subsidies To find the optimal subsidyε ∗ , our starting point is the observation that, for any policyπ, the anticipated social utility ̄ U S (ε;π) is a decreasing function ofε. This is because, as the subsidyεincreases, the principal covers a higher fraction of the agent’s cost, but the probability that the product receives approval remains unchanged. More formally, we have the following proposition: Proposition 9. For any policyπand subsidyε∈[0,ε max ], the anticipated social utility ̄ U S (ε;π) admits a linear decomposition ̄ U S (ε;π) = ρ S · E (α 0 ,β 0 )∼Q [P π (∃t∈ [T ]:f (S t+1 )≥ 1/κ |S 0 )]− ε· E (α 0 ,β 0 )∼Q [A π (S 0 , 0)],(17) 8 In principle, Eq. 14 may allow for multiple solutions. However, our objective is not to characterize the full set of solutions, but rather to provide an algorithmic procedure to compute one such solution. 7 Algorithm 1 Finds the Principal’s Optimal Subsidy 1: Input: MDP solver SolveMDP, maximum subsidy ε max , principal’s belief Q 2: Initialize: I ←∅, U ←∅ 3: (π L ,V 0 L ,A L )← SolveMDP(M 0 ),(π R ,V 0 R ,A R )← SolveMDP(M ε max ) ▷ Compute optimal policies for ε = 0 and ε = ε max using the decomposition in Proposition 7 4: U ←U ∪(0, ̄ U S (0;π L )) 5: ̄ V 0 L ← E (α 0 ,β 0 )∼Q [V 0 L (α 0 ,β 0 , 0, 0)], ̄ A L ← E (α 0 ,β 0 )∼Q [A L (α 0 ,β 0 , 0, 0)] 6: ̄ V 0 R ← E (α 0 ,β 0 )∼Q [V 0 R (α 0 ,β 0 , 0, 0)], ̄ A R ← E (α 0 ,β 0 )∼Q [A R (α 0 ,β 0 , 0, 0)] ▷ Compute the average agent value using the principal’s belief. 7: Push (ε L ,π L , ̄ V 0 L , ̄ A L ,ε R ,π R , ̄ V 0 R , ̄ A R ) into I 8: while I is not empty do 9: Pop (ε L ,π L , ̄ V 0 L , ̄ A L ,ε R ,π R , ̄ V 0 R , ̄ A R ) from I 10:if A L ̸= A R then 11:ε int ← ( ̄ V 0 L − ̄ V 0 R )/( ̄ A R − ̄ A L )▷ Compute the candidate subsidy to evaluate 12:(π int ,V 0 int ,A int )← SolveMDP(M ε int )▷ Compute the optimal policy for the candidate subsidy 13: ̄ V 0 int ← E (α 0 ,β 0 )∼Q [V 0 int (α 0 ,β 0 , 0, 0)], ̄ A int ← E (α 0 ,β 0 )∼Q [A int (α 0 ,β 0 , 0, 0)] ▷ Compute the average agent value using the principal’s belief. 14:if ̄ V 0 int + ε int · ̄ A int ≤ ̄ V 0 L +·ε int ̄ A L then▷ Verify if the optimal policy improves over π R and π L 15:U ←U ∪(ε int , ̄ U S (ε int ;π R )) ▷ Found a vertex; compute social utility using Eq. 13 and principal’s belief Q 16:else▷ Split the subsidy interval 17:Push (ε L ,π L , ̄ V 0 L , ̄ A L ,ε int ,π int , ̄ V 0 int , ̄ A int ), (ε int ,π int , ̄ V 0 int , ̄ A int ,ε R ,π R , ̄ V 0 R , ̄ A R ) into I 18:end if 19:end if 20: end while 21: return ε ∗ where (ε ∗ ,u ∗ ) = argmax (ε,u)∈U u▷ Return the optimal subsidy In the above expression, the first term corresponds to the principal’s anticipated probability of rejecting H 0 under policyπ, and the second term corresponds to the anticipated total cost borne by the principal through the subsidy. As a consequence, and in light of Proposition 8, in each interval [ε i ,ε i+1 ) of the partitionPwhere a fixed policyπ i is optimal, the anticipated social utility ̄ U S (ε;π i ) is a decreasing (linear) function of the subsidy over [ε i ,ε i+1 ), and therefore it is maximized at the left point ε i : max ε∈[ε i ,ε i+1 ) ̄ U S (ε;π i ) = ̄ U S (ε i ;π i ).(18) Crucially, the principal can compute each policyπ i without knowing the agent’s initial belief (α 0 ,β 0 ), since the optimal policy for the processM ε i does not depend on the initial state. This stands in contrast to many settings in the literature on (Bayesian) Stackelberg games, where the principal typically must anticipate the agent’s best response by averaging over the agent’s private information [57, 58]. Leveraging the above results, we derive an efficient divide-and-conquer procedure to find the optimal subsidyε ∗ . The procedure (i) explicitly constructs the partitionPand (i) applies Proposition 9 to determine the optimal anticipated social utility on each interval [ε i ,ε i+1 ) by evaluating it at the left endpointε i . Algorithm 1 summarizes the overall procedure, and the following proposition establishes its correctness. Proposition 10. Algorithm 1 is guaranteed to find an optimal subsidy ε ∗ in a finite number of iterations. Algorithm 1 maintains a stackIof intervals, alongside their corresponding optimal policies at the endpoints and the linear value function decompositions provided by Proposition 7. We abstract the computation of these policies and value functions into a procedure,SolveMDP, which may be implemented using a value-iteration algorithm (e.g., Algorithm 3). The stackIis initialized to the full interval [0,ε max ] (lines 3 to 7). At each iteration, the algorithm calculates the intersection pointε int between the endpoint value functions, ̄ V 0 L +ε· ̄ A L and ̄ V 0 R +ε· ̄ A R (line 11), where the bar denotes evaluation atS 0 and averaging over the principal’s beliefQ. 8 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1540 1560 1580 1600 1620 Social utility, U S ( ε ; π ε ) ( $ M) ε ∗ = 0.108 PartitionP 0.16 0.17 0.18 0.19 0.20 0.21 0.22 Probability of opting out (a) Social utility vs. subsidy (ρ S = $2000 M) 0.0 0.1 0.2 0.3 Optimal subsidy, ε ∗ 010203040 Social-to-agent approval benefit ratio,ρ S /ρ A 30 40 50 60 Social utility gain vs. non-sequential (%) Non-sequential (no subsidy) Non-sequential (optimal subsidy) (b) Optimal subsidy and social utility gain vs. ρ S /ρ A Figure 1: Subsidizing antibiotic development. The figure shows the results of the approval process for an antibiotic with true (unknown) efficacyθ ∗ = 0.65. Panel (a) shows the result of running Algorithm 1 to compute the optimal subsidy for the principalε ∗ = 0.108 when the social benefit of approval isρ S = $2000M. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8), and the colors indicate the probability that the agent opts out by selecting n= 0 during the approval process (before the drug is approved). Panel (b) shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy, together with the percentage increase in social utility of the sequential approval protocol relative to a non-sequential approval protocol in which the agent is restricted to a single trial withn max = 800. The error bars represent 95% bootstrapped confidence intervals. Then, it computes the optimal value function atε int (line 12). Due to the convexity of the optimal value function, if the optimal value function atε int coincides with the value ̄ V 0 L +ε int · ̄ A L = ̄ V 0 R +ε int · ̄ A R , the policyπ L is optimal in [ε L ,ε int ), and the policyπ R is optimal in [ε int ,π R ]. In this case,ε int is a vertex ofP, and the algorithm stores the social utility atε int (line 15). Conversely, if the optimal policy atε int strictly improves over the policyπ L , then the algorithm has found a new interval of the partition, and the stackIis updated (line 17). The algorithm then iterates the same steps over all intervals in the stack I. 9 6 Experiments: Subsidizing Antibiotic Development Antimicrobial resistance is a major global threat, projected to cause 10 million deaths annually by 2050 [59,60]. Yet, FDA antibiotic approvals have dropped from 13% of all drugs in 1980 to 4% in the 2000s [61], largely for economic reasons: treatments are short, prices must stay low to ensure availability, use is restricted to limit resistance, and competition from existing or generic drugs is intense [62,63]. Consequently, many major pharmaceutical companies have exited or reduced antibiotic pipelines [64], while small biotech firms often struggle financially [65–67]. To counter this, public and private efforts have focused on incentivizing and subsidizing development [68–70]. In this section, we conduct a series of experiments to demonstrate the effectiveness of our approval protocol in optimally subsidizing antibiotic development. Experimental setup. The principal subsidizes the development of an antibiotic and conducts a hypothesis test withκ= 0.05 (i.e., a false positive rate of at most 0.05) to determine whether the antibiotic’s (unknown) efficacyθ ∗ = 0.65 exceeds the benchmarkθ b = 0.5. The agent can conduct up to four trials (T= 3), each with a maximum sample size ofn max = 200 patients. Although data on the economic cost and sales of antibiotic development are mostly private, recent reports estimate that the average present value of sales is approximately $240M[71], the per-patient Phase I cost is $66k[72], and the average fixed cost per trial is $48.9M[73]. Therefore, we setρ A = $240M, andc(n) = $48.9M+ $0.066M· nfor anyn ∈ 1,...,n max . Further, we assume the agent has a non-informative (uniform) prior withα 0 = 1,β 0 = 1, known to the 9 We report runtime measurements of Algorithm 1 in Appendix G. 9 principal, and varyρ S . Refer to Appendix G for additional details regarding our experimental setup, and to Appendix H for results under alternative parameter choices and extensive sensitivity analyses. Results. For an antibiotic with a social benefit upon approval ofρ S = $2000M(a ratioρ S /ρ A ≈8.3), Panel (a) of Figure 1 shows (i) the social utilityU S (ε;π ε ), (i) the partitionP, which consists of 49 intervals, and (i) the agent’s opt-out probability, over the entire range of subsidy levels. We find that, under the optimal subsidyε ∗ = 0.108, the social utility increases by∼5.5% and the agent’s opt-out probability decreases by∼22% compared to a scenario with no subsidies. Interestingly, we also find that the optimal subsidy ε ∗ —which maximizes the anticipated social utility ̄ U S (ε;π ε )—also maximizes the true social utilityU S (ε;π ε ). Further, Panel (b) of Figure 1 shows the optimal subsidyε ∗ and the social utility gain compared to two non-sequential baselines for different values of the social-to-agent approval benefit ratioρ S /ρ A . We find that the optimal subsidy increases with the social-to-agent approval benefit ratio, reflecting that higher societal utility strengthens the principal’s incentive to subsidize experimentation. We also find that, compared to two non-sequential baselines in which the agent is restricted to conducting at most a single clinical trial (with a larger maximum sample size ofn max = 800), our approval protocol yields substantial gains in social utility. Specifically, relative to a non-sequential protocol without subsidies, our protocol increases social utility by approximately 50%–60%, depending on the ratioρ S /ρ A and, relative to a non-sequential protocol with optimal subsidies, our protocol still achieves gains exceeding 35%. 7 Discussion and Limitations In this section, we highlight several limitations of our work and discuss avenues for future research. Methodology. In our work, we have considered a Beta–Binomial model, which is particularly natural in the context of RCTs. However, in other application domains, it may be desirable to consider more general models where the agent’s beliefs, the experimental outcomes, and the e-values exhibit greater complexity—for instance, through dependence across trials or through the inclusion of safety characteristics. In Appendix C, we outline how to extend our framework to these more general settings; however, this extension introduces significant computational challenges. Furthermore, although our results do not require the principal to know the agent’s initial belief (i.e., its private informationB 0 ), observing the agent’s experimental actions over time may provide information about its prior, suggesting that the principal could, in principle, dynamically update its belief about the agent’s prior. It would be interesting to incorporate such learning and elicitation into a sequential approval protocol in future work [74]. In addition, it would also be valuable to extend our protocol to allow for uncertainty in the agent’s benefitρ A , which in practice may vary across product developers and contexts. Finally, real-world regulatory settings may involve additional non-economic factors that are not explicitly captured in our model. For instance, clinical trial duration, patient follow-up requirements, and other operational constraints can affect both the feasibility and optimality of sequential experimentation protocols, and accounting for these factors remains an important direction for future work. Implementation and Evaluation. We have conducted a case study applying our approval protocol to antibiotic development, a well-known setting characterized by underinvestment and market failure, which is particularly suited to studying the effects of subsidies (see Appendix H for further experimental results using different parameter values). However, it would be interesting to extend this analysis to other settings—such as orphan drugs or rare disease treatments—where trial costs and sample sizes may differ substantially from those in standard antibiotic development and may therefore lead to different optimal subsidies. We also empirically evaluated the computational cost of computing the agent’s optimal decision policy and the optimal subsidy. In our experiments, both procedures were efficient and typically completed within a few minutes (see Appendix G for more details). However, as shown theoretically in Section 4, the complexity of computing the optimal agent policy scales polynomially with the number of actions and trials. Consequently, extending our methodology to more general settings—such as those described in Appendix C—might require developing approximate algorithms, as computing (Stackelberg) equilibria in general Bayesian games is known to be computationally intractable in many cases [57]. Broader Impact. Regulatory agencies and product developers are progressively exploring more flexible 10 and data-driven approval methodologies, including Bayesian approaches, to better balance safety and innovation [11]. Our work contributes to this direction by providing a principled framework for designing subsidy mechanisms that improve social utility in approval processes, which may inform future policy discussions and be of interest to both public and private regulators and decision makers. 8 Conclusions How can approval protocols be designed to incentivize experimentation? In this work, we have addressed this question by introducing a sequential approval protocol that allows the agent to continuously refine its knowledge about the product and the principal to subsidize a fraction of the agent’s experimental costs—all while maintaining anytime-valid guarantees on the false positive rate. Along the way, we have shown that the agent can efficiently compute the optimal experimental policy, and the principal can find the subsidy that maximizes social utility, even when anticipating that the agent selects its policy strategically. Finally, using real-world data on antibiotic development, we have demonstrated that our sequential, subsidized protocol can substantially improve social utility, yielding gains of up to 60% relative to non-sequential designs without subsidies. More broadly, we hope our work provides insights for designing approval protocols that better align agents’ incentives with social objectives. Acknowledgements. Gomez-Rodriguez acknowledges support from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 945719). Tsirtsis acknowledges supports from the Alexander von Humboldt Foundation in the framework of the Alexander von Humboldt Professorship (Humboldt Professor of Technology and Regulation awarded to Sandra Wachter) endowed by the Federal Ministry of Education and Research via the Hasso Plattner Institute. References [1] Dale H Gieringer. The safety and efficacy of new drug approval. Cato J., 5:177, 1985. [2] U.S. Food and Drug Administration. Demonstrating substantial evidence of effectiveness for human drug and biological products: Guidance for industry. Draft guidance, U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), December 2019. Draft—Not for Implementation. Clinical/Medical. [3]Alec B O’Connor. Building comparative efficacy and tolerability into the fda approval process. Jama, 303(10):979– 980, 2010. [4] Perrine Janiaud, Telba Irony, Estelle Russek-Cohen, and Steven N Goodman. U.S. food and drug administration reasoning in approval decisions when efficacy evidence is borderline, 2013-2018. Ann. Intern. Med., 174(11):1603– 1611, November 2021. [5]Alberto Farina, Federico Moro, Frederick Fasslrinner, Annahita Sedghi, Miluska Bromley, and Timo Siepmann. Strength of clinical evidence leading to approval of novel cancer medicines in europe: A systematic review and data synthesis. Pharmacology Research & Perspectives, 9(4):e00816, 2021. [6] Allan S Detsky. Are clinical trials a cost-effective investment? Jama, 262(13):1795–1800, 1989. [7] Simon Frantz. Why are clinical costs so high? Nature Reviews Drug Discovery, 2(11), 2003. [8]Linda Martin, Melissa Hutchens, Conrad Hawkins, and Alaina Radnov. How much do clinical trials cost? Nature Reviews Drug Discovery, 16(6):381–382, 2017. [9]C Hendricks Brown, Thomas R Ten Have, Booil Jo, Getachew Dagne, Peter A Wyman, Bengt Muthén, and Robert D Gibbons. Adaptive designs for randomized trials in public health. Annual review of public health, 30(1):1–25, 2009. 11 [10]Rajiv Mahajan and Kapil Gupta. Adaptive design clinical trials: Methodology, challenges and prospect. Indian journal of pharmacology, 42(4):201–207, 2010. [11]U.S. Food and Drug Administration. Use of bayesian methodology in clinical trials of drug and biological products. Draft guidance, Center for Biologics Evaluation and Research and Center for Drug Evaluation and Research, Food and Drug Administration, March 2026. [12]Clinical trial-specific funding opportunities.https://grants.nih.gov/policy-and-compliance/policy- topics/clinical-trials/specific-funding-opportunities. Accessed: 2026-03-31. [13] Clinical trials grants program.https://w.fda.gov/industry/orphan-products-grants-program/clinical- trials-grants-program. Accessed: 2026-03-31. [14] Clinical trials.https://w.dfg.de/en/research-funding/funding-opportunities/programmes/individual/ clinical-trials. Accessed: 2026-03-31. [15] The european and developing countries clinical trials partnership.https://w.edctp.org/. Accessed: 2026-03- 31. [16]Stephen Bates, Michael I Jordan, Michael Sklar, and Jake A Soloff. Principal-agent hypothesis testing. arXiv preprint arXiv:2205.06812, 2022. [17]Flora C Shi, Stephen Bates, and Martin J Wainwright. Sharp results for hypothesis testing with risk-sensitive agents. arXiv preprint arXiv:2412.16452, 2024. [18]Safwan Hossain, Yatong Chen, and Yiling Chen. Strategic hypothesis testing. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. [19]Sanford J Grossman and Oliver D Hart. An analysis of the principal-agent problem. In Foundations of insurance economics: Readings in economics and finance, pages 302–340. Springer, 1992. [20]Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference, 2023. [21]Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998. [22]Aleksey Tetenov. An economic theory of statistical testing. CeMMAP working papers 50/16, Institute for Fiscal Studies, Sep 2016. [23] Andrew McClellan. Experimentation and approval mechanisms. Econometrica, 90(5):2215–2247, 2022. [24]Peter Grünwald, Rianne de Heide, and Wouter Koolen. Safe testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(5):1091–1128, 03 2024. [25]Aaditya Ramdas and Ruodu Wang. Hypothesis testing with e-values. Foundations and Trends in Statistics, 1(1-2):1–390, 07 2025. [26] Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting, 2022. [27]Ziyu Xu and Aaditya Ramdas. Online multiple testing with e-values. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 3997–4005. PMLR, 02–04 May 2024. [28]Jaehyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo. E-detectors: A nonparametric framework for sequential change detection. The New England Journal of Statistics in Data Science, 2(2):229–260, 2024. [29]Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Trans. Inf. Theor., 70(2):1178–1203, February 2024. [30] Ian Waudby-Smith, Ricardo Sandoval, and Michael I. Jordan. Universal log-optimality for general classes of e-processes and sequential hypothesis tests, 2025. 12 [31]Ben Chugg, Etienne Gauthier, Michael I Jordan, Aaditya Ramdas, and Ian Waudby-Smith. Post-hoc large-sample statistical inference. arXiv preprint arXiv:2603.08002, 2026. [32]Etienne Gauthier, Francis Bach, and Michael I. Jordan. Backward conformal prediction. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [33] Baihe Huang, Eric Xu, Kannan Ramchandran, Jiantao Jiao, and Michael I Jordan. Towards anytime-valid statistical watermarking. arXiv preprint arXiv:2602.17608, 2026. [34]Ander Artola Velasco, Stratis Tsirtsis, and Manuel Gomez Rodriguez. Auditing pay-per-token in large language models. In The 29th International Conference on Artificial Intelligence and Statistics, 2026. [35]Guneet S. Dhillon, Javier Gonzalez, Teodora Pandeva, and Alicia Curth. E-scores for (in)correctness assessment of generative model outputs. In The 29th International Conference on Artificial Intelligence and Statistics, 2026. [36] Herbert Robbins. Optimal stopping. The American Mathematical Monthly, 77(4):333–343, 1970. [37]Goran Peskir and Albert N Shiryaev. Optimal stopping and free-boundary problems. Lectures in Mathematics. ETH Zürich. Birkhauser Verlag AG, Basel, Switzerland, 2006 edition, August 2006. [38] Warren B Powell and Ilya O Ryzhov. Optimal Learning. Wiley Series in Probability and Statistics. Wiley-Blackwell, Hoboken, NJ, March 2012. [39] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 8(5–6):359–483, November 2015. [40] Namhoon Cho, Seokwon Lee, Hyo-Sang Shin, and Antonios Tsourdos. Bayesian learning approach to model predictive control. arXiv preprint arXiv:2203.02720, 2022. [41]Thomas Kleine Buening, Christos Dimitrakakis, Hannes Eriksson, Divya Grover, and Emilio Jorge. Minimax-bayes reinforcement learning, 2023. [42] Wanggang Shen and Xun Huan. Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning. Computer Methods in Applied Mechanics and Engineering, 416:116304, 2023. [43]Chen Cheng and Xun Huan. Optimal stopping for sequential bayesian experimental design. arXiv preprint arXiv:2509.21734, 2025. [44]Wanggang Shen, Jiayuan Dong, and Xun Huan. Variational sequential optimal experimental design using reinforcement learning. Comput. Methods Appl. Mech. Eng., 444(118068):118068, September 2025. [45] A. Wald. Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics, 16(2):117 – 186, 1945. [46] Victor H. de la Peña. A General Class of Exponential Inequalities for Martingales and Ratios. The Annals of Probability, 27(1):537 – 564, 1999. [47]Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingales. Probability Surveys, 17(none):257 – 317, 2020. [48]Matthew J Renwick, David M Brogan, and Elias Mossialos. A systematic review and critical assessment of incentive strategies for discovery and development of novel antibiotics. J. Antibiot. (Tokyo), 69(2):73–88, February 2016. [49]Wan-Shu Wu and Kai Zhao. Government r&d subsidies and enterprise r&d activities: theory and evidence. Economic Research-Ekonomska Istraživanja, 35(1):391–408, 2022. [50]John C Harsanyi. Bayesian decision theory, rule utilitarianism, and arrow’s impossibility theorem. Theory and Decision, 11(3):289–317, 1979. 13 [51]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. [52] Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling, 2020. [53] Heinrich von Stackelberg. Market Structure and Equilibrium. Springer, Berlin, Germany, 2011 edition, August 2010. [54] U.S. Congress. 26 U.S.C. 45C — clinical testing expenses for certain drugs for rare diseases or conditions. Internal Revenue Code, 2024. Accessed: 2026-05-03. [55] National Institutes of Health. Small business funding.https://seed.nih.gov/small-business-funding, 2026. Accessed: 2026-05-03. [56] Onésimo Hernández-Lerma and Jean Bernard Lasserre. Discrete-Time Markov Control Processes. Springer New York, 1996. [57] Vincent Conitzer and Tuomas Sandholm. Computing the optimal strategy to commit to. In Proceedings of the 7th ACM Conference on Electronic Commerce, EC ’06, page 82–90, New York, NY, USA, 2006. Association for Computing Machinery. [58] Praveen Paruchuri, Jonathan P Pearce, Janusz Marecki, Milind Tambe, Fernando Ordonez, and Sarit Kraus. Playing games for security: An efficient exact algorithm for solving bayesian stackelberg games. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 2, pages 895–902, 2008. [59]GBD 2021 Antimicrobial Resistance Collaborators. Global burden of bacterial antimicrobial resistance 1990-2021: a systematic analysis with forecasts to 2050. Lancet, 404(10459):1199–1226, September 2024. [60]Laura J Shallcross, Simon J Howard, Tom Fowler, and Sally C Davies. Tackling the threat of antimicrobial resistance: from policy to sustainable action. Philos. Trans. R. Soc. Lond. B Biol. Sci., 370(1670):20140082, June 2015. [61]Kevin Outterson, John H Powers, Enrique Seoane-Vazquez, Rosa Rodriguez-Monguio, and Aaron S Kesselheim. Approval and withdrawal of new antibiotics and other antiinfectives in the U.S., 1980-2009. J. Law Med. Ethics, 41(3):688–696, 2013. [62]David M Shlaes. The economic conundrum for antibacterial drugs. Antimicrob. Agents Chemother., 64(1), December 2019. [63]Nupur Gargate, Mark Laws, and Khondaker Miraz Rahman. Current economic and regulatory challenges in developing antibiotics for gram-negative bacteria. NPJ Antimicrob. Resist., 3(1):50, June 2025. [64] Benjamin Plackett. Why big pharma has abandoned antibiotics. Nature, 586(7830):S50–S52, October 2020. [65]Gilles Courtemanche, Rohini Wadanamby, Amritanjali Kiran, Luisa Fernanda Toro-Alzate, Mathew Diggle, Dipanjan Chakraborty, Ariel Blocker, and Maarten van Dongen. Looking for solutions to the pitfalls of developing novel antibacterials in an economically challenging system. Microbiol. Res. (Pavia), 12(1):173–185, March 2021. [66]Laura J V Piddock, Yewande Alimi, James Anderson, Damiano de Felice, Catrin E Moore, John-Arne Røttingen, Henry Skinner, and Peter Beyer. Advancing global antibiotic research, development and access. Nat. Med., 30(9):2432–2443, September 2024. [67] Nadya Wells, Vinh-Kim Nguyen, and Stephan Harbarth. Novel insights from financial analysis of the failure to commercialise plazomicin: Implications for the antibiotic investment ecosystem. Humanit. Soc. Sci. Commun., 11(1), July 2024. [68] Kevin Outterson, John H Rex, Tim Jinks, Peter Jackson, John Hallinan, Steve Karp, Deborah T Hung, Francois Franceschi, Tyler Merkeley, Christopher Houchens, Dennis M Dixon, Michael G Kurilla, Rosemarie Aurigemma, and Joseph Larsen. Accelerating global innovation to address antibacterial resistance: introducing CARB-X. Nat. Rev. Drug Discov., 15(9):589–590, September 2016. 14 [69]Michael Anderson, Dimitra Panteli, Robin van Kessel, Gunnar Ljungqvist, Francesca Colombo, and Elias Mossialos. Challenges and opportunities for incentivising antibiotic research and development in europe. Lancet Reg. Health Eur., 33(100705):100705, October 2023. [70]United States Congress. H.R. 7352: PASTEUR Act of 2026, 2026. To amend the Public Health Service Act to establish a program to develop innovative antimicrobial drugs. [71]Sakib Rahman, Olof Lindahl, Chantal M Morel, and Aidan Hollis. Market concentration of new antibiotic sales. J. Antibiot. (Tokyo), 74(6):421–423, June 2021. [72] Stella Stergiopoulos, Sara B Calvert, Carrie A Brown, Josephine Awatin, Pamela Tenaerts, Thomas L Holland, Joseph A DiMasi, and Kenneth A Getz. Cost drivers of a hospital-acquired bacterial pneumonia and ventilator- associated bacterial pneumonia phase 3 clinical trial. Clin. Infect. Dis., 66(1):72–80, January 2018. [73]Thomas J Moore, Hanzhe Zhang, Gerard Anderson, and G Caleb Alexander. Estimated costs of pivotal trials for novel therapeutic agents approved by the US food and drug administration, 2015-2016. JAMA Intern. Med., 178(11):1451–1457, November 2018. [74]Flora C Shi, Martin J Wainwright, and Stephen Bates. Instance-adaptive hypothesis tests with heterogeneous agents. arXiv preprint arXiv:2510.21178, 2025. [75] B. Campbell, N Balakrishnan, and Brani Vidakovic. Encyclopedia of statistical sciences. Methods and Applications of Statistics. John Wiley & Sons, Nashville, TN, 2 edition, December 2005. [76] Joseph P Simmons, Leif D Nelson, and Uri Simonsohn. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci., 22(11):1359–1366, November 2011. [77]Etienne Gauthier, Francis Bach, and Michael I. Jordan. Betting on equilibrium: Monitoring strategic behavior in multi-agent systems, 2026. [78] J. Ville. Étude Critique de la Notion de Collectif. Collection des monographies des probabilités. Gauthier-Villars, 1939. [79]R.T. Rockafellar. Convex Analysis. Princeton landmarks in mathematics and physics. Princeton University Press, 1970. 15 A Summary of Notation In Table 1 we summarize the key symbols used in the main body of the paper. Table 1: Summary of notation. Symbol Description κPrincipal’s false positive rate bound θ ∗ True (unknown) product efficacy θ b Baseline efficacy H 0 ,H 1 Null and alternative hypotheses tTime index lInitial time index (if not 0) TMaximum number of trials (time horizon of the belief MDP) εPrincipal’s subsidy level ε max Maximum allowable subsidy ε ∗ Principal’s optimal subsidy B t Agent’s belief at time t α t ,β t Parameters of the agent’s belief at time t QPrincipal’s belief over the agent’s prior n t Agent’s action (sample size) at time t n max Maximum sample size πAgent’s policy π ε Optimal policy under subsidy ε V ε Optimal value function under subsidy ε V ε π Value function of policy π under subsidy ε A π Expected total cost conditional on approval under policy π cCost function X t Experimental outcome at time t E t e-value at time t M t Principal’s test process at time t fMapping from beliefs to the test process C t Cumulative experimental cost at time t M ε Belief MDP under subsidy ε r ε Reward function in the belief MDP under subsidy ε S t State of M ε at time t τStopping time (last non-absorbing state of M ε ) PPartition associated with fixed policies ρ A Agent’s approval benefit ρ S Social approval benefit U A Agent’s utility under θ ∗ U S Social utility under θ ∗ ̄ U A Agent’s anticipated utility computed via the belief MDP ̄ U S Principal’s anticipated social utility computed via the belief MDP 16 B Background on Sequential Hypothesis Testing Using e-values In this section, we provide a brief overview of sequential hypothesis testing using e-values. For a detailed exposition, we refer the reader to Ramdas et al. [25]. A central object in classical hypothesis testing is the p-value, used to assess a null hypothesisH 0 against an alternativeH 1 . Despite their widespread use across scientific disciplines, p-values suffer from important limitations when used without appropriate precautions. A canonical example is p-hacking (also called sampling to a foregone conclusion): if one repeatedly collects data, computes a p-value, and checks whether it falls below a fixed threshold (e.g., 0.05), rejection of H 0 is eventually guaranteed even when H 0 is true [75, 76]. Alongside classical methods based on p-values, a rich line of work has developed methods based on a game-theoretic formulation of statistics that is better suited to sequential hypothesis testing [20] and mitigates issues such as p-hacking. The key objects in this framework are e-values, which are nonnegative random variables E that satisfy: E H 0 [E]≤ 1,(19) where the expectation is taken under any distribution inH 0 . Typically, as with p-values, e-values are computed from some observed dataXthat is intended to provide evidence to rejectH 0 ; in the sequential approval protocol considered in Section 2, we make this dependence explicit by writingE(n,X), wheren is the sample size selected by the agent andXthe number of successes in the control trial. The intuition behind e-values is straightforward: sinceEhas expectation at most 1 underH 0 , it can take large values only with small probability. Hence, observing a large e-value can be interpreted as evidence againstH 0 —and the larger the e-value, the stronger the evidence. Consequently, one can construct a statistical test that rejects H 0 whenever the e-value takes high values, i.e., wheneverE ≥1/κfor a certain thresholdκ∈(0,1). More precisely, by Markov’s inequality, this threshold automatically controls the false positive rate: P H 0 (E ≥ 1/κ)≤ κ· E H 0 [E]≤ κ.(20) Proposition 11. If E is an e-value for H 0 , i.e., E H 0 [E]≤ 1, then the non-sequential hypothesis test φ = 1E ≥ 1/κ is a level-κ test for H 0 for any κ∈ (0, 1): its false positive rate is at most κ. Informally, e-values are as general as p-values in the sense that they exist under essentially the same technical conditions and can be transformed into one another. Their main advantage, however, arises in sequential settings—i.e., when data arrive over time [27–29,34,77]—as in clinical trials, which is the primary reason our formalism in Section 2 and 3 is built on e-values. One way to understand this advantage is to note that the defining condition of e-values (Eq. 19) is preserved under a wide range of operations. For example, convex combinations of e-values and multiplications of (independent) e-values remain valid e-values. In contrast, analogous operations do not generally preserve the validity of p-values. To formalize the use of e-values in a sequential setting, letF= (F t ) ∞ t=0 be a filtration on a given sample space, where eachF t is aσ-algebra representing the information available at timet. For instance,Fmay be the filtration generated by the observationsX 0 ,...,X t . Then, given a sequential data streamX 0 ,X 1 ,..., the goal of sequential hypothesis testing is to maintain and update a running measure of evidence against H 0 as new observations arrive. In the e-value framework, this can be achieved by constructing a sequence E 0 ,E 1 ,... adapted to F—so that each E t depends only on data observed up to time t—and satisfying E H 0 [E t |F t−1 ]≤ 1.(21) The above condition is the sequential analogue of the defining property in Eq. 19. For instance, if the observationsX t are independent, one may construct eachE t fromX t alone, in which case the e-values E 0 ,...,E t are mutually independent. In Section 4, we adopt this construction, but we emphasize that more complex e-values can be defined if the experimental protocol requires it. For example, in Eq. 28 we consider a data-dependent construction in which new evidence is collected only if a prior experiment satisfies certain conditions. 17 By interpreting the quantityE t as the new evidence againstH 0 obtained at timet, a canonical way to construct a sequential test is to define a stochastic test process M = (M t ) t≥0 as M t = ( 1t = 0 E t · M t−1 t≥ 1, (22) which simply corresponds to aggregating the previous e-values multiplicatively. 10 By contrast, combining p-values in a sequential or dependent setting typically requires specialized corrections to maintain validity [24]. Whenever the e-values satisfy Eq. 21, the above processMforms a (non-negative) supermartingale underH 0 , i.e., E H 0 [M t |F t−1 ]≤ M t−1 .(23) Intuitively, this property means that ifH 0 is true, the value ofM t does not, in expectation, increase over time. Conversely, sustained growth of the processM t provides evidence againstH 0 . Thus, a natural sequential test to rejectH 0 is to monitor whetherM t ≥1/κfor a chosenκ∈(0,1), and to rejectH 0 as soon as this condition is met. Ville’s inequality [78], a sequential extension of Markov’s inequality, guarantees that this procedure controls the false positive rate uniformly over time: Theorem 12 (Ville’s inequality). If the process M is a non-negative supermartingale, then, P H 0 (∃t∈ N : M t ≥ 1/κ) = P H 0 sup t≥0 M t ≥ 1/κ ≤ κ.(24) Moreover, validity is preserved if the test is stopped at any stopping timeτadapted toF(but possibly depending on the observed data): P H 0 (M τ ≥ 1/κ)≤ κ,(25) which is a property known as any-time validity and provides a principled solution to the problem of p-hacking. Although the above results hold for any choice of e-values satisfying Eq. 21, a natural practical question is which specific form to adopt. We discuss this, and how different choices affect the approval process, in Appendix C. 10 In Eq. 30 of Appendix C we discuss a different method to construct a test process M without multiplying e-values. 18 C Extension to Arbitrary Belief Functions and e-values The approval protocol introduced in Section 2 can be formulated in a more general setting (at the expense of losing tractability), as we now outline. Generalization of the approval process. In a general setting, the agent may begin the approval process with an arbitrary initial beliefB 0 ∈∆(Θ), where Θ denotes a general parameter space for the unknown parameterθ ∗ characterizing the product. At each step of the approval process, the agent selects an action n t ∈ A gen ∪0, whereA gen is a (potentially infinite) set describing the design of the next randomized controlled trial (e.g., sample size, participant characteristics, etc.), andn t = 0 again denotes the option to opt out. Ifn t ̸=0, an outcomeX t ∼ P(X t | n t ,θ ∗ ) is observed and the agent incurs a costc(n t ), where P(·| n t ,θ ∗ ) is a likelihood function characterizing the data-collection process specific to each experimental setting and c : A gen → R + is a cost function. The agent then updates its belief via the Bayesian posterior: B t+1 (θ) = B t (θ)· P (X t | n t ,θ) R Θ B t (θ)· P (X t | n t ,θ)dθ . To decide on approval, the principal partitions the parameter space as Θ = Θ 0 ⊔Θ 1 and defines the following null and alternative hypotheses: ( H 0 =θ ∗ ∈ Θ 0 (null) H 1 =θ ∗ ∈ Θ 1 (alternative). (26) This formulation allows for richer principal objectives. For instance, the principal may be concerned not only with a drug’s efficacy but also with its safety, in which caseH 0 may correspond to treatments that are either insufficiently effective or unsafe. To conduct the above hypothesis test sequentially, at each time step, the principal may compute any test process valueM t , subject only to two conditions: (i) the test process Mmust be predictable with respect to the filtrationF= (F t ) ∞ t=0 generated by the data and agent actions X 0 ,n 0 ,X 1 ,n 1 ,..., and (i) the test processMmust be a supermartingale underH 0 . The principal can then rejectH 0 wheneverM t ≥1/κat any time step, as described in Appendix B. A particular construction of the test process that generalizes Eq. 3 proceeds multiplicatively as M t = ( 1t = 0, E t−1 (X t−1 ,n t−1 )· M t−1 t≥ 1, (27) where eachE t−1 is an e-value adapted toFthat may depend on all previously observed experimental outcomes and actions, i.e., X 0 ,n 0 ,X 1 ,n 1 ,...,X t−1 ,n t−1 , and satisfies: E H 0 [E t (n t ,X t )|F t−1 ]≤ 1. This allows modeling adaptive experimental designs in which future trials depend on past outcomes. Such settings arise naturally in multi-stage clinical trials [2], where progression to a subsequent phase may be contingent on earlier success. This can be represented by defining: E t (n t ,X t ) = ( E(n t ,X t ) if g(n 0 ,X 0 ,...,n t−1 ,X t−1 ) = 1, 0if g(n 0 ,X 0 ,...,n t−1 ,X t−1 ) = 0, (28) whereE(·,·) is a fixed e-value andgis a decision rule determining whether the agent is permitted to conduct the next experiment. Ifg(n 0 ,X 0 ,...,n t−1 ,X t−1 ) = 0 at any timet−1, the test process stops accumulating evidence and approval becomes impossible after time t. Given a test processM, the utilities of both the agent and the principal can be defined analogously to Eq. 7, and a belief Markov decision processM ε can be formulated as in Section 3, with the difference that 19 the state space now includes any possible belief and any possible value of the test process, i.e., 11 S gen = ∆(Θ)×R + |z total cost C ×R + |z test process M ∪ S out , with a state S t transitioning to S t+1 after taking action n t and observing outcome X t according to: S t+1 = (B t+1 , C t + c(n t ), M t+1 (n t ,X t )) if n t ̸= 0 and 0 < f (S t ) < 1/κ, Sif f (S t )≥ 1/κ, S out if n t = 0 or S t = S out , (29) where forS t = (B t ,C t ,M t )̸= S out ,B t+1 (θ)∝ B t (θ)·P(X t | n t ,θ). In the above,fis the function that maps a state to the test process value, namelyf(S t ) =M t forS t = (B t ,C t ,M t )̸= S out andf(S out ) = 0. While this formulation is fully general, it also significantly complicates the subsequent analysis of the optimal agent policy. Alternative e-values and statistical power. For a given choice of test processM, it is important to note that the statistical power of the test—verifying whetherM t ≥1/κat any time step—need not be 1. That is, with nonzero probability it may occur that, even for θ ∗ ≥ θ b , the approval process is unsuccessful. To quantify the statistical power of sequential tests based on e-values (or supermartingales), there exists a canonical notion called e-power. For a given e-valueEfor the nullH 0 , its e-power against an alternative L∈ H 1 is defined asE L [logE]. Under this definition, likelihood ratios are the optimal e-values for simple nulls and alternatives, i.e., whenH 0 andH 1 each correspond to a single probability distribution. In contrast, whenH 0 andH 1 are composite, as in the approval process of Section 4, the choice of e-value becomes more intricate [30]. For concreteness, in Section 4 we therefore adopt a general strategy that constructs e-values by exponentiating random variables [26,46]. The rationale is as follows: if we wish to test whether an arbitrary random variableXhas meanE[X]< θ b , we consider the quantityX − θ b , which we expect to be large when E[X]> θ b and small otherwise. To form an e-value, we consider the positive quantityexp(X− θ b ); since this can have expectation exceeding 1 even whenE[X]< θ b , we obtain a valid e-value satisfying Eq. 19 by shifting the argument by an appropriately chosen constantλ, i.e.,exp(X − θ b +λ), as given by Proposition 1. This e-value is particularly simple and admits a closed-form expression that simplifies the exposition in Section 3; however, in the context of the approval protocol in Section 2, other choices of e-values may achieve better statistical power for certain values of the efficacy parameterθ ∗ . Indeed, forEas defined in Proposition 1 and using the notation therein, the e-power is: E θ ∗ [logE t ] = n t · θ ∗ − n t · log(1 + θ b (e− 1)). Thus,E t has positive powerE θ ∗ [logE t ]≥0 if and only ifθ ∗ ≥ log(1 +θ b (e−1))> θ b . This means that if θ b < θ ∗ < log(1 +θ b (e−1)), it may be thatE t ̸→∞even asn→∞. In other words, the processM t might never exceed the threshold 1/κifθ ∗ is sufficiently close to—yet strictly above—θ b , even for RCTs of arbitrary size. The standard remedy (see Chapter 3 of [25]) is to form a mixture of e-values or supermartingales that generalizes Proposition 1. We now outline how our approval protocol can be extended to use such mixtures, and refer the reader to Appendix H.2.7 for additional experimental results. Our starting point is the simple observation that the e-valueE(X t ,n t ) in Proposition 1 can be equivalently written as a likelihood ratio between a Binomial distribution evaluated atθ b and one evaluated at a particular alternative: E(X t ,n t ) = ̃ θ θ b ! X t · 1− ̃ θ 1− θ b ! n t −X t , where ̃ θ ≥ θ b is the unique efficacy value satisfying: e = ̃ θ· (1− θ b ) (1− ̃ θ)· θ b . 11 If the e-values depend on all past observations, as in Eq. 28, the state space may need to be augmented to explicitly track the history of experimental outcomes. 20 To generalize this construction, one can mix over other values of the alternative parameter. This is achieved by choosing any (smooth) distribution P mix over [θ b , 1] and defining the mixture test process as: 12 M mix t+1 = Z 1 θ b P mix (θ)· θ θ b P t i=0 X i · 1− θ 1− θ b P t i=0 n i − P t i=0 X i dθ.(30) It can be shown that, if the true efficacyθ ∗ > θ b lies in the support ofP mix , thenM mix t+1 →∞as P t i=0 n i →∞, i.e., the mixture achieves asymptotic power 1 [20]. In this context, Algorithm 3 can be used to compute the optimal agent policy for any subsidy with minimal modifications: it suffices to replace the test process function f with f mix (α,β) = Z 1 θ b P mix (θ)· θ θ b α−α 0 · 1− θ 1− θ b β−β 0 dθ, and Algorithm 1 can then be used to compute the principal’s optimal subsidy with no further modification. 13 0255075100 α−α 0 0 25 50 75 100 β − β 0 Uniform mixture Exponential mixture Non-mixture Figure 2: Rejection regions in belief space. The figure shows, for each agent belief with parameters (α,β), whether the conditionf(α,β)≥1/κis satisfied (i.e., whetherH 0 is rejected; shaded region), under different test processes: the test process defined using the non-mixed e-values in Proposition 1 (orange), a test process defined in Eq. 30 with a uniform mixtureP mix = U(θ b ,1) (blue), and a test process defined in Eq. 30 with an exponential mixtureP mix =Exp(10) restricted to (θ b ,1) (green). Here, we setκ= 0.05,θ b = 0.5, and α 0 = β 0 = 1. We have presented the non-mixture processMdefined in Section 2 in the main text for two reasons. First, it yields closed-form expressions which simplify the exposition and provide a more transparent (geometric) intuition for the belief Markov decision process (see Figure 3). Second, the multiplicative property in Eq. 3 is exploited extensively in the theoretical proofs in Appendix F, whereas the mixture processM mix does not satisfy this property in non-trivial cases whereP mix is not a point mass. However, we conjecture that all our theoretical results in Section 3 and 4 carry over to the mixture setting—in particular, because the belief (α t ,β t ) remains a sufficient statistic forM mix t via the monotone functionf mix , and the Bayesian update of the 12 An argument analogous to that of Proposition 1 shows that M mix t is a supermartingale under H 0 . 13 Note that the fact that Algorithm 1 returns an optimal subsidy only relies on the agent’s optimal utility being a piecewise linear and convex function of the subsidy (Proposition 8), which holds independently of the specific form off, as can be seen from its proof in Appendix F.10 21 agent is unchanged—but adapting the proofs would require routing arguments through the Markov property of the belief MDP rather than through the algebraic identityM t+1 =E t · M t . We leave a formal treatment of this extension for future work. In Appendix H.2.7, we present experimental results using the above mixture test process with a uniform mixtureP mix = U(θ b ,1). Further, in Figure 2, we compare the regions of the belief space wheref(α,β)≥1/κ, i.e., whereH 0 is rejected, for different test processes. The test process without mixture (based on Proposition 1) yields a linear rejection region, while the mixture processes yield nonlinear regions whose boundary slope increases for larger values of (α,β). 22 D Geometry of the Belief Markov Decision Process In Figure 3 we illustrate the geometry of the belief MDPM ε in the (α,β)-plane corresponding to all the possible beliefs for the agent. In blue, we represent the beliefs to which the agent may transition after selecting an actionn t >0 (the figure showsn t = 3 for concreteness). The exact next state depends on the realized outcomeX t of the experiment, as described in Eq. 8. Further, in light of Proposition 2, note that each pair (α,β) is associated with a valuef(α,β) for the test processM, with pairs such thatf(α,β)≥1/κ corresponding to a state where H 0 as been rejected by the principal. The condition f (α,β)≥ 1/κ ⇐⇒ α− α 0 − (α + β− α 0 − β 0 ) log(1 + θ b (e− 1))≥ log(1/κ)(31) corresponds to a linear region in the (α,β)-plane, represented in green in Figure 3 and labelled “RejectH 0 ”. Note that the reward function defined in Eq. 9 only includes the positive termρ A for a transition that crosses the boundary of this region. Similarly, the red region corresponds to beliefs for which—given a fixed total costC ≥0 and initial time stepl∈[T]— it is optimal for the agent to opt out of the approval process by choosingn= 0. This region is bounded by the curveα7→ ̃ β(α) in Proposition 6. Refer to Appendix H for concrete numerical solutions to the MDP M ε . α− α 0 β− β 0 n Opt out region ̃ β(α) log 1/κ 1−log(1+θ b (e−1)) Reject H 0 Figure 3: Illustration of the geometry of the state space in the MDP M ε . 23 E Value Iteration in the Belief Markov Decision Process In Algorithm 2, we present, for completeness, an adaptation of the value-iteration algorithm [51] to exactly compute an optimal policy π ε for the MDP M ε . 14 In the proof of Proposition 4 in Appendix F.5, we showed that the set of accessible states inttime steps, which we denoted by S r (t), satisfies: |S r (t)|=O (n max ) 2 · t 2 . Building on this bound, it is straightforward to verify that the value-iteration method in Algorithm 2 has a time complexityO (n max ) 4 · T 3 . Indeed, for a givenl, the set|S r (l)\S out |=O (n max ) 2 · l 2 , and for each such state, the inner loops (line 4 and 5) iterate over all actionsnand sums over possible outcomes x. That is, for eachl, the algorithm performsO (n max ) 4 · l 2 iterations. Summing overl, we conclude that the total complexity isO (n max ) 4 · T 3 . We refer the reader to Appendix G for further details regarding the implementation of the algorithm. Algorithm 2 Finds the Optimal Agent Policy π ε for the belief MDP M ε 1:Input Subsidyε, horizonT, max trialsn max , approval benefitρ A , cost functionc(n) =c 0 +c 1 n, threshold κ, prior parameters (α 0 ,β 0 ), test process function f 2: Initialize V ε (S,T + 1)← 0 for all states S ∈S r 3: for l = T down to 0 do 4: for each S = (α,β,C)∈S r (l)\S out such that f (α,β) < 1/κ do 5:for each action n∈1,...,n max do 6:Q ε (S,n,l)←−(c 0 + c 1 n) + P n x=0 B(n,α,β)(x)· NextValue(x,n) 7:where NextValue(x,n) = 8: ( ρ A + ε(C + c 0 + c 1 n)if f (α + x,β + n− x)≥ 1/κ V ε (α + x,β + n− x,C + c 0 + c 1 n,l + 1) otherwise 9:end for 10:V ε (S,l)← max 0, max n∈1,...,n max Q ε (S,n,l) 11:if V ε (S,l) > 0 then 12:π ε (S,l)← arg max n∈1,...,n max Q ε (S,n,l) 13:▷ Ties broken arbitrarily 14:else 15:π ε (S,l)← 0 16:end if 17: end for 18: end for 19: return V ε ,π ε Note that the above algorithm can readily be modified to return the linear decomposition of the optimal value function at a subsidy ε in Proposition 7, namely V ε π ε = V 0 π ε + ε· A π ε , as shown in Algorithm 3. 14 B(n,α,β) denotes the Beta–Binomial distribution. 24 Algorithm 3 Finds the Optimal Agent Policy π ε and Value Function Decomposition 1:Input Subsidyε, horizonT, max trialsn max , approval benefitρ A , costc(n) =c 0 +c 1 n, thresholdκ, prior (α 0 ,β 0 ), test process function f 2: Initialize V ε (S,T + 1)← 0, V 0 (S,T + 1)← 0, A(S,T + 1)← 0 for all S ∈S r 3: for l = T down to 0 do 4: for each S = (α,β,C)∈S r (l)\S out s.t. f (α,β) < 1/κ do 5:for each action n∈1,...,n max do 6:c n ← c 0 + c 1 n, (α ′ x ,β ′ x )← (α+x, β+n−x) 7:Q ε (S,n,l)←−c n + n X x=0 B(n,α,β)(x)· v ε (x,n) 8:Q 0 (S,n,l)←−c n + n X x=0 B(n,α,β)(x)· v 0 (x,n) 9:Q A (S,n,l)← n X x=0 B(n,α,β)(x)· a(x,n) 10:where (writing approved for f (α ′ x ,β ′ x )≥ 1/κ): v ε (x,n) = ( ρ A + ε(C + c n )if approved V ε (α ′ x ,β ′ x ,C+c n ,l+1) otherwise v 0 (x,n) = ( ρ A if approved V 0 (α ′ x ,β ′ x ,C+c n ,l+1) otherwise a(x,n) = ( C + c n if approved A(α ′ x ,β ′ x ,C+c n ,l+1) otherwise ▷ Note: Q ε = Q 0 + ε· Q A by construction 11:end for 12:n ∗ ← arg max n∈1,...,n max Q ε (S,n,l)▷ Ties broken arbitrarily 13:if Q ε (S,n ∗ ,l) > 0 then 14:V ε (S,l)← Q ε (S,n ∗ ,l), V 0 (S,l)← Q 0 (S,n ∗ ,l), A(S,l)← Q A (S,n ∗ ,l) 15:π ε (S,l)← n ∗ 16:else 17:V ε (S,l)← 0, V 0 (S,l)← 0, A(S,l)← 0 18:π ε (S,l)← 0 19:end if 20: end for 21: end for 22: return V ε , V 0 , A, π ε 25 F Proofs F.1 Proof of Proposition 1 Fix θ ∗ ∈ [0, 1] and n t > 0. Let X t ∼ Bin(n t ,θ ∗ ) and define E(X t ,n t ) = exp X t − n t · log(1 + θ b (e− 1)) . We explicitly show that the expectation of the above random variable is upper-bounded by 1 ifθ ∗ ∈ H 0 = θ ∗ : θ ∗ < θ b : E X t ∼Bin(n t ,θ ∗ ) [E(X t ,n t )] = E X t ∼Bin(n t ,θ ∗ ) [exp X t − n t · log(1 + θ b (e− 1)) ] = 1 (1 + θ b (e− 1)) n t · E X t ∼Bin(n t ,θ ∗ ) [exp (X t )] (∗) = 1 (1 + θ b (e− 1)) n t · (1− θ ∗ + θ ∗ · e) n t (∗) ≤ 1 (1 + θ b (e− 1)) n t · (1− θ b + θ b · e) n t = 1 where in (∗) we have used the formula for the moment-generating function of the Binomial distribution, and in (∗) we have used that, by definition, if θ ∗ ∈ H 0 , then θ ∗ < θ b . 26 F.2 Proof of Proposition 2 We begin by noting that the definition of the test process in Eq. 3 together with the e-value in Eq. 5 implies that, if the agent has continued the approval process up to timet≤ Tby selecting non-nulln 0 ,...,n t , then: M t+1 = t Y s=0 exp X s − n s · log(1 + θ b (e− 1)) (32) = exp t X s=0 X s − t X s=0 n s · log(1 + θ b (e− 1)) ! (33) Now, unfolding Eq. 1 forttime steps, we can write the parametersα t andβ t that characterize the belief of the agent at time t as: α t+1 = α 0 + t X s=0 X s β t+1 = β 0 + t X s=0 n s − t X s=0 X s , (34) ⇐⇒ t X s=0 X s = α t+1 − α 0 t X s=0 n s = β t+1 − β 0 + t X s=0 X s = α t+1 − α 0 + β t+1 − β 0 , (35) Thus, substituting the above in Eq. 32 we readily obtain: M t+1 = exp t X s=0 X s − t X s=0 n s · log(1 + θ b (e− 1)) ! = exp α t+1 − α 0 − (α t+1 − α 0 + β t+1 − β 0 )· log(1 + θ b (e− 1)) 27 F.3 Auxiliary Lemma to Proposition 3 Here, we prove a lemma that will be used later in the proof of Proposition 3. Lemma 1. Letπ ∈Π be a policy,S ∈ Sbe a state such thatS ̸= S out , denoteS= (α,β,C) and fix an integrable functionG:R 3 → R. Further, denote byπ(S,t)(•) the density overAgiven byπat timet, by Beta(α,β)(•) the density of the Beta distribution, byBin(n,θ)(•) the density of a Binomial distribution with parametersnandθ, and byBB(n,α,β)(•) the density of a Beta-Binomial distribution. Consider any random variables θ ∗ , n t and X t with a joint density P such that n t and θ ∗ are independent and: 15 P (n t ) = π(S,t)(n t ) P (θ ∗ ) = Beta(α,β)(θ ∗ ) P (X t |n t ,θ ∗ ) = Bin(n t ,θ ∗ )(X t ). (36) Then, it holds that: E θ ∗ ∼Beta(α,β) E n t ∼π(S,t) E X t ∼Bin(n t ,θ ∗ ) [G(n t ,θ ∗ ,X t )] = E n t ∼π(S,t) E X t ∼B(n t ,α,β) E θ ∗ ∼Beta(α+X t ,β+n t −X t ) [G(n t ,θ ∗ ,X t )]. (37) Proof. We first note thatX t |n t follows a Beta-Binomial distribution with parameters (n t ,α,β). Indeed, denoting by B the beta function and by Γ the gamma function: P (X t = k|n t ) = Z 1 0 P (X t = k|n t ,θ ∗ )P (θ ∗ )dθ ∗ = Z 1 0 n t k (θ ∗ ) k (1− θ ∗ ) n t −k 1 B(α,β) (θ ∗ ) α−1 (1− θ ∗ ) β−1 dθ ∗ = n t k B(α,β) Z 1 0 (θ ∗ ) k+α−1 (1− θ ∗ ) n t −k+β−1 dθ ∗ = n t k B(k + α,n t − k + β) B(α,β) = n t k Γ(k + α)Γ(n t − k + β) Γ(n t + α + β) Γ(α + β) Γ(α)Γ(β) , which is precisely the density of a Beta-Binomial distribution with parameters (n t ,α,β). Similarly,θ ∗ |n t ,X t follows a Beta distribution with parameters (α + X t ,β + n t − X t ): P (θ ∗ |X t ,n t ) = P (X t |θ ∗ ,n t )P (θ ∗ ) P (X t |n t ) = h n t X t (θ ∗ ) X t (1− θ ∗ ) n t −X t ih 1 B(α,β) (θ ∗ ) α−1 (1− θ ∗ ) β−1 i n t X t B(α+X t ,β+n t −Xt0) B(α,β) = (θ ∗ ) X t (1− θ ∗ ) n t −X t (θ ∗ ) α−1 (1− θ ∗ ) β−1 B(α + X t ,β + n t − Xt0) = (θ ∗ ) α+X t −1 (1− θ ∗ ) β+n t −X t −1 B(α + X t ,β + n t − X t ) . Since n t and θ ∗ are independent, the joint distribution can be written as: P (θ ∗ ,n t ,X t ) = P (n t )· P (θ ∗ )· P (X t |θ ∗ ,n t )(38) 15 We adopt the notationθ ∗ ,n t , andX t to align with the proof of Proposition 3, where this lemma is applied to the efficacy, sample size, and outcomes of the approval process. 28 Alternatively, we can factor the distribution P as: P (θ ∗ ,n t ,X t ) = P (n t )· P (X t |n t )· P (θ ∗ |n t ,X t ).(39) Then, we can conclude using the law of total expectations for the expressionE (n t ,θ ∗ ,X t )∼P [G(n t ,θ ∗ ,X t )] to obtain: E θ ∗ ∼Beta(α,β) E n t ∼π(S,t) E X t ∼Bin(n t ,θ ∗ ) [G(n t ,θ ∗ ,X t )] = E n t ∼π(S,t) E X t ∼B(n t ,α,β) E θ ∗ ∼Beta(α+X t ,β+n t −X t ) [G(n t ,θ ∗ ,X t )] (40) 29 F.4 Proof of Proposition 3 We fix a subsidyε∈[0,ε max ] and a policyπ ∈Π. Sinceεis fixed, for clarity, we omit it from the notation for the remainder of the proof, and we will simply use the notationr(•) to denote the (agent’s) reward inM ε . Similarly,V π will denote the value function inM ε for policyπ. To show that Proposition 3 holds, we will proceed by induction over the time steps of the MDP M ε . We first introduce some additional notation. Preliminaries. For anyt ≤ T(representing a duration) and anyl ≤ T − t(representing the initial time step), we define the following value function of π in the MDP M ε : V π (S,l,t) = E π " l+t X k=l r(S k ,n k ,S k+1 ) S l = S # ,(41) which is the total expected reward obtained by policy π starting from state S at time l and taking t steps. Moreover, given a stateS= (α,β,C)∈S\S out of the MDP such thatf(S)<1/κ,t≤ Tandl≤ T−t, consider the approval process described in Section 2 when: (i) the initial time index isl, (i) the agent has initial beliefB 0 =Beta(α,β), (i) the test processMstarts atM l =f(α,β)<1/κand (iv) the subsidy includes an additional total costC. Then, we define the agent’s utility ̄ U A l,t (π|S) (averaged over its initial belief) for the first t steps as: ̄ U A l,t (π|S) = E θ ∗ ∼Beta(α,β) " E π l " ρ A + ε· C + τ (S,l,t) X j=0 c(n j ) · 1∃j ∈ [τ (S,l,t)]:f (α,β)· M j+1 ≥ 1/κ(42) − τ (S,l,t) X j=0 c(n j ) θ ∗ ,B 0 = Beta(α,β),C 0 = C ## whereτ(S,l,t) =t∧ minj ∈ 0,...,t:n j = 0or f(α,β)· M j+1 ≥1/κis the last step of the approval process using the shifted policyπ l defined asπ l (•,j) =π(•,j+l). Here,E π l [•|θ ∗ ,B 0 =Beta(α,β),C 0 =C] indicates that: (i) all outcomesX t are drawn with a fixed efficacyθ ∗ , and (i) the agent has initial belief Beta(α,β) and cumulated costC. For instance, in the above expectation, the first action at indexj= 0 taken by the agent is sampled from the distributionπ(α,β,C,l). Alternatively, for a stateSsuch thatS=S out or f (S)≥ 1/κ, we simply define: ̄ U A l,t (π|S) = 0.(43) Importantly, note that, by definition: ̄ U A 0,T (π | (α 0 ,β 0 , 0)) =E θ ∗ ∼Beta(α 0 ,β 0 ) " E π 0 " ρ A + ε τ (α 0 ,β 0 ,0,0,0) X j=0 c(n j ) · 1∃j ∈ [τ (α 0 ,β 0 , 0, 0, 0)] : M j+1 ≥ 1/κ − τ (α 0 ,β 0 ,0,0,0) X j=0 c(n j ) θ ∗ ,B 0 = Beta(α 0 ,β 0 ),C 0 = 0 ## , = E θ ∗ ∼B 0 [U A (π,ε)]. In light of the above, our goal will be to show by induction over t that: ̄ U A l,t (π | S) = V π (S,l,t) for any S ∈S, t≤ T, l≤ T − t. 30 Then, by particularizing to t = T, l = 0 and S = (α 0 ,β 0 , 0) and using Eq. 10, we will obtain that ̄ U A (π;ε) = V π (α 0 ,β 0 , 0, 0,T ) = ̄ U A 0,T (π|α 0 ,β 0 , 0 ) = E θ ∗ ∼B 0 [U A (π,ε)], which is the statement in Proposition 3. Base caset= 0. We will show that ̄ U A l,0 (π|S) =V π (S,l,0) holds for anyS ∈Sandl≤ T. We first consider the case whereS=S out orf(S)≥1/κ. In this case, note that the transition dynamics of the MDP (Eq. 8) imply that at any time stepk ≥ l,S k =S out orS k =S. Thus, the expression in Eq. 41 only contains rewards that are null (see Eq. 9), and hence: V π (S,l, 0) = 0. Similarly, by definition (Eq. 43), ̄ U A l,0 (π|S) = 0. Thus, ̄ U A l,0 (π|S) = V π (S,l, 0). Consider now the non-trivial case whereS ̸= S out andf(S)<1/κ, and writeS= (α,β,C). In this case, the value function starting at timelfort= 0 can be expanded as (note thatt= 0 corresponds to a single step in the MDP): V π (S,l, 0) = E π [r(S l ,n l ,S l+1 )|S l = S] = E π −c(n l ) + (ρ A + ε· (C + c(n l )))· 1f (S l+1 )≥ 1/κ) S l = S = E n l ∼π(S,l) E X l ∼B(n l ,α,β) [− c(n l ) + (ρ A + ε· (C + c(n l )))· 1f (S)· E(X l ,n l )≥ 1/κ)] In the above, we denote byX l ∼ B(n l ,α,β) a sample from the Beta-Binomial distribution, corresponding to samplingθ l ∼ Beta(α,β), and thenX l ∼ Bin(n l ,θ l ), and we have used the definition of the test process in Eq. 3. On the other hand, let t = 0 in Eq. 42. Then, τ (S,l, 0) = 0, and we obtain: 16 16 Whenever we write nested expectations, such asE Y E Z [•] for arbitrary random variablesYandZ, the inner expectation is understood to be conditional on the outer variable; that is,E Y [E Z [•| Y]]. To simplify notation, we may omit the explicit conditioning when no confusion is likely to arise. 31 ̄ U A l,0 (π|S) = E θ ∗ ∼Beta(α,β) " E π l " − c(n 0 ) + ρ A + ε· (C + c(n 0 )) · 1f (α,β)· M 1 ≥ 1/κ θ ∗ ,B 0 = Beta(α,β),C 0 = C ## = E θ ∗ ∼Beta(α,β) " E π l " − c(n 0 ) + ρ A + ε· (C + c(n 0 )) · 1f (S)· M 1 ≥ 1/κ θ ∗ ,B 0 = Beta(α,β),C 0 = C ## = E θ ∗ ∼Beta(α,β) E n 0 ∼π(S,l) E X 0 ∼Bin(n 0 ,θ ∗ ) " − c(n 0 ) + ρ A + ε· (C + c(n 0 )) · 1f (S)· M 1 ≥ 1/κ # (∗) = E n 0 ∼π(S,l) E θ ∗ ∼Beta(α,β) E X 0 ∼Bin(n 0 ,θ ∗ ) " − c(n 0 ) + ρ A + ε· (C + c(n 0 )) · 1f (S)· E(n 0 ,X 0 )≥ 1/κ # = E n l ∼π(S,l) E X l ∼B(n l ,α,β) " − c(n l ) + ρ A + ε· (C + c(n l )) · 1f (S)· E(n l ,X l )≥ 1/κ # = V π (S,l, 0), where in (∗) we have used thatθ ∗ andn 0 are independent (sincen 0 is sampled independently fromπ(S,l)). This concludes the base case. Inductive stept→ t+ 1. For the inductive step, assume that given at < T, for anyl≤ T − tand for any state S ∈S it holds that: ̄ U A l,t (π | S) = V π (S,l,t). We will show that ̄ U A l,t+1 (π|S) =V π (S,l,t+ 1) for anyl ≤ T −(t+ 1) andS ∈ S. To this end, fix any such S and l. Firstly, we consider the case whereS=S out orf(S)≥1/κ, where the transition dynamics of the MDP (Eq. 8) imply that at any time stepk ≥ l,S k =S out orS k =S. Then, similarly to the base case,V π (S,l,t) = 0 and ̄ U A l,t (π|S) = 0 for any t≥ 0, so the equality holds. We focus now on the non-trivial case whereS ̸= S out andf(S)<1/κ, and writeS= (α,β,C). We begin 32 by expanding the value for the first t + 1 time steps in Eq. 41: V π (S,l,t + 1) = E π " l+t+1 X k=l r(S k ,n k ,S k+1 ) S l = S # = E π " r(S l ,n l ,S l+1 ) + l+t+1 X k=l+1 r(S k ,n k ,S k+1 ) S l = S # = E π [r(S l ,n l ,S l+1 ) + V π (S l+1 ,l + 1,t)|S l = S] (∗) = E π r(S l ,n l ,S l+1 ) + ̄ U A l+1,t (π|S l+1 ) S l = S (∗) = E π [r(S l ,n l ,S l+1 )|S l = S] | z † + E π ̄ U A l+1,t (π|S l+1 )· 1f (S l+1 ) < 1/κ S l = S | z ‡ (44) where (∗) follows from the induction hypothesis and (∗) because ̄ U A t (π|S l+1 ) = 0 iff(S l+1 )≥1/κ. Analo- gously to the base case, the term † above can be written as: † = E π [r(S l ,n l ,S l+1 )|S l = S] = E π −c(n l ) + (ρ A + ε· (C + c(n l )))· 1f (S l+1 )≥ 1/κ S l = S = E n l ∼π(S,l) E θ ∗ ∼Beta(α,β) E X l ∼Bin(n l ,θ ∗ ) h − c(n l ) + (ρ A + ε· (C + c(n l )))· 1f (S l+1 )≥ 1/κ i = E θ ∗ ∼Beta(α,β) E n l ∼π(S,l) E X l ∼Bin(n l ,θ ∗ ) h − c(n l ) + (ρ A + ε· (C + c(n l )))· 1f (α + X l ,β + n l − X l )≥ 1/κ i = E θ ∗ ∼Beta(α,β) E n 0 ∼π l (S,0) E X 0 ∼Bin(n 0 ,θ ∗ ) h − c(n 0 ) + (ρ A + ε· (C + c(n 0 )))· 1f (α + X 0 ,β + n 0 − X 0 )≥ 1/κ i = E θ ∗ ∼Beta(α,β) E n 0 ∼π l (S,0) E X 0 ∼Bin(n 0 ,θ ∗ ) h − c(n 0 ) + (ρ A + ε· (C + c(n 0 )))· 1f (α,β)· E(n 0 ,X 0 )≥ 1/κ i = E θ ∗ ∼Beta(α,β) h E π l h − c(n 0 ) + ρ A + ε· (C + c(n 0 )) · 1f (α,β)· M 1 ≥ 1/κ θ ∗ ,B 0 = Beta(α,β),C 0 = C i (45) We now focus on the term‡in Eq. 44. GivenS l =S, the stateS l+1 is fully determined by the actionn l and the value X l , namely (see Eq. 8), S l+1 = ( (α + X l ,β + n l − X l ,C + c(n l )) if n l > 0 S out if n l = 0. Thus, conditional on the policyπ(S,l) selecting actionn l = 0, we haveS l+1 =S out and ̄ U A l+1,t (π|S l+1 ) = 0. That is, conditioning on the the event n l = 0 inside the expectations † and ‡ results in: ( E π [r(S l ,n l ,S l+1 )|S l = S,n l = 0] = 0 E π ̄ U A l+1,t (π|S l+1 )· 1f (S l+1 ) < 1/κ S l = S,n l = 0 = 0. 33 Similarly, note that conditional on the policy π(α,β,C,l) selecting action n l = 0, we also have: E θ ∗ ∼Beta(α,β) " E π l " ρ A + ε· C + τ (S,l,t+1)) X j=0 c(n j ) · 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − τ (S,l,t+1) X j=0 c(n j ) θ ∗ ,B 0 = Beta(α,β),C 0 = C,n 0 = 0 ## = 0 As a consequence, in the following, we assume without loss of generality that the action 0 is not in the support of the distribution π(S,l), and therefore S l+1 ̸= S out . Then, under the above simplification, we expand the expectation in the term‡to average over the possible values ofn l andX l , and then substitute the expression for the utility ̄ U A l+1,t (π|S l+1 ) defined in Eq. 42 (emphasized below in blue for clarity): 17 ‡ = E π ̄ U A l+1,t (π|S l+1 )· 1f (S l+1 ) < 1/κ S l = S = E n ′ l ∼π(S,l) E X ′ l ∼B(n ′ l ,α,β) E θ ∗ ∼Beta(α+X ′ l ,β+n ′ l −X ′ l ) " E π l+1 " 1f (α + X ′ l ,β + n ′ l − X ′ l ) < 1/κ· ρ A + ε· C + c(n ′ l ) + ̃τ X j=0 c(n j ) · 1∃j ∈ [ ̃τ ]:f (α + X ′ l ,β + n ′ l − X ′ l )· M j+1 ≥ 1/κ − 1f (α + X ′ l ,β + n ′ l − X ′ l ) < 1/κ· ̃τ X j=0 c(n j ) θ ∗ ,B 0 = Beta(α + X ′ l ,β + n ′ l − X ′ l ),C 0 = C + c(n ′ l ) ## , where we have defined ̃τ=τ(S(n ′ l ,X ′ l ),l+ 1,t), and we useS(n ′ l ,X ′ l ) to denote the state to whichStransitions after selecting action n ′ l and observing the outcome X ′ l , as given by the transition dynamics (Eq. 8). 17 We also change notationn l → n ′ l andX l → X ′ l to avoid confusion with the actions and outcomes that appear when expanding ̄ U A l+1,t (π|S l+1 ). 34 We now leverage Lemma 1 to reorder the first three expectations as: ‡ = E θ ∗ ∼Beta(α,β) E n ′ l ∼π(S,l) E X ′ l ∼Bin(n ′ l ,θ ∗ ) " E π l+1 " 1f (α + X ′ l ,β + n ′ l − X ′ l ) < 1/κ· ρ A + ε· C + c(n ′ l ) + ̃τ X j=0 c(n j ) · 1∃j ∈ [ ̃τ ]:f (α + X ′ l ,β + n ′ l − X ′ l )· M j+1 ≥ 1/κ − 1f (α + X ′ l ,β + n ′ l − X ′ l ) < 1/κ· ̃τ X j=0 c(n j ) θ ∗ ,B 0 = Beta(α + X ′ l ,β + n ′ l − X ′ l ),C 0 = C + c(n ′ l ) ## = E θ ∗ ∼Beta(α,β) E n 0 ∼π(S,l) E X 0 ∼Bin(n 0 ,θ ∗ ) " 1f (α + X 0 ,β + n 0 − X 0 ) < 1/κ· E π l+1 " ρ A + ε· C + c(n 0 ) + ̃τ X j=0 c( ̃n j ) · 1∃j ∈ [ ̃τ ]:f (α + X 0 ,β + n 0 − X 0 )· ̃ M j+1 ≥ 1/κ − ̃τ X j=0 c( ̃n j ) # θ ∗ ,B 0 = Beta(α + X 0 ,β + n 0 − X 0 ),C 0 = C + c(n 0 ) # , (46) where, in the last equality, we have renamed the dummy variables appearing in the inner expectation using tildes, i.e., ̃n j , ̃ X j , and ̃ M j . 18 This is purely notational at this stage, but will be useful later in the proof when it becomes important to distinguish between different sets of variables. To keep our objective in view, recall that the goal of the inductive step is to show that ̄ U A l,t+1 (π | S) =† +‡. Our goal will now be to expand the term ̄ U A l,t+1 (π | S) and verify that this equality indeed holds. To this end, we will use the identity 1 = 1f (α,β)· M 1 ≥ 1/κ + 1f (α,β)· M 1 < 1/κ, which holds for anyα,βandM 1 since the two events in the indicator functions are complementary, and substitute it in the definition of ̄ U A l,t+1 (π | S) in Eq. 42: 18 Here, note that ̃τ(S(n 0 ,X 0 ),l+ 1,t) d = ̃τ(S(n ′ l ,X ′ l ),l+ 1,t), becausen 0 d = n ′ l andX 0 d = X ′ l by definition, and so we kept the notation ̃τ to denote τ (S(n 0 ,X 0 ),l + 1,t) 35 ̄ U A l,t+1 (π|S) = E θ ∗ ∼Beta(α,β) " E π l " ρ A + ε· C + τ (S,l,t+1) X j=0 c(n j ) · 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − τ (S,l,t+1) X j=0 c(n j ) θ ∗ ,B 0 = Beta(α,β),C 0 = C ## = E θ ∗ ∼Beta(α,β) " E π l " ρ A + ε· C + τ (S,l,t+1) X j=0 c(n j ) · 1f (α,β)· M 1 ≥ 1/κ· 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − 1f (α,β)· M 1 ≥ 1/κ· τ (S,l,t+1) X j=0 c(n j ) θ ∗ ,B 0 = Beta(α,β),C 0 = C ## + E θ ∗ ∼Beta(α,β) " E π l " ρ A + ε· C + τ (S,l,t+1) X j=0 c(n j ) · 1f (α,β)· M 1 < 1/κ· 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − 1f (α,β)· M 1 < 1/κ· τ (S,l,t+1) X j=0 c(n j ) θ ∗ ,B 0 = Beta(α,β),C 0 = C ## We now observe that iff(α,β)· M 1 ≥1/κ, thenτ(S,l,t+ 1) = 0, that is, the approval process stops after the first step, which simplifies the first summand and yields: ̄ U A l,t+1 (π|S) = E θ ∗ ∼Beta(α,β) " E π l " − c(n 0 ) + ρ A + ε· (C + c(n 0 )) · 1f (α,β)· M 1 ≥ 1/κ θ ∗ ,B 0 = Beta(α,β),C 0 = C ## + E θ ∗ ∼Beta(α,β) " E π l " ρ A + ε· C + τ (S,l,t+1) X j=0 c(n j ) · 1f (α,β)· M 1 < 1/κ· 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − 1f (α,β)· M 1 < 1/κ· τ (S,l,t+1) X j=1 c(n j ) θ ∗ ,B 0 = Beta(α,β),C 0 = C ## Further, we can identify that the first summand in the above expression corresponds to the form of the term † derived in Eq. 45, and replacing it, we obtain: 36 ̄ U A l,t+1 (π|S) =† +E θ ∗ ∼Beta(α,β) " E π l " ρ A + ε· C + τ (S,l,t+1) X j=0 c(n j ) ·1f (α,β)· M 1 < 1/κ· 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ −1f (α,β)· M 1 < 1/κ· τ (S,l,t+1) X j=1 c(n j ) θ ∗ ,B 0 = Beta(α,β),C 0 = C ## We now focus on the orange term in the expression above, which we denote by□. Our goal will be to show that□=‡. We start by factoring out the term1f(α,β)· M 1 <1/κ, and using that by definition M 1 = E(n 0 ,M 0 ), we obtain: □ = E θ ∗ ∼Beta(α,β) " E π l " 1f (α,β)· E(n 0 ,X 0 ) < 1/κ· ρ A + ε· C + c(n 0 ) + τ (S,l,t+1) X j=1 c(n j ) 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − τ (S,l,t+1) X j=1 c(n j ) ! θ ∗ ,B 0 = Beta(α,β),C 0 = C ## In the above expectationE π l [•|θ ∗ ,B 0 = Beta(α,β),C 0 = C] , we can use the law of iterated expectations by conditioning on the first samplesn 0 andX 0 . In particular, note thatn 0 ∼ π(α,β,C,l) and that 37 X 0 ∼ Bin(n 0 ,θ ∗ ), and thus: □ = E θ ∗ ∼Beta(α,β) E n 0 ∼π(S,l) E X 0 ∼Bin(n 0 ,θ ∗ ) " E π l " 1f (α,β)· E(n 0 ,X 0 ) < 1/κ· ρ A + ε· C + c(n 0 ) + τ (S,l,t+1) X j=1 c(n j ) 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − τ (S,l,t+1) X j=1 c(n j ) ! θ ∗ ,B 0 = Beta(α,β),C 0 = C,n 0 ,X 0 ## = E θ ∗ ∼Beta(α,β) E n 0 ∼π(S,l) E X 0 ∼Bin(n 0 ,θ ∗ ) " 1f (α + X 0 ,β + n 0 + X 0 ) < 1/κ· E π l " ρ A + ε· C + c(n 0 ) + τ (S,l,t+1) X j=1 c(n j ) 1∃j ∈ [τ (S,l,t + 1)]:f (α,β)· M j+1 ≥ 1/κ − τ (S,l,t+1) X j=1 c(n j ) # θ ∗ ,B 0 = Beta(α,β),C 0 = C,n 0 ,X 0 # (47) Here, note that the inner expectationE π l [•|θ ∗ ,B 0 = Beta(α,β),C 0 = C,n 0 ,X 0 ] is conditioned onn 0 and X 0 . Therefore, the first actionn 1 is distributed according toπ(α+X 0 ,β+n 0 − X 0 ,C+c(n 0 ),l+ 1), and any subsequent action n j is distributed according to: n 1 |n 0 ,X 0 ∼ π (α + X 0 ,β + n 0 − X 0 ,C + c(n 0 ),l + 1) X 1 |n 0 ,X 0 ∼ Bin(n 1 ,θ ∗ ) n 2 |n 0 ,X 0 ∼ π (α + X 0 + X 1 ,β + n 0 + n 1 − X 0 − X 1 ,C + c(n 0 ) + c(n 1 ),l + 2) X 2 |n 0 ,X 0 ∼ Bin(n 2 ,θ ∗ ) . . . n j |n 0 ,X 0 ∼ π(α + X 0 +· + X j−1 ,β + n 0 +· + n j−1 − X 0 −·− X j−1 , C + c(n 0 ) +· + c(n j−1 ),l + j) X j |n 0 ,X 0 ∼ Bin(n j ,θ ∗ ) . . . These are equal in distribution to the sequence of actions and experimental outcomes ̃n 1 , ̃ X 1 ,...in the inner 38 expectation of Eq. 46, i.e., we have the following equalities in distribution n 1 |n 0 ,X 0 d = ̃n 0 X 1 |n 0 ,X 0 d = ̃ X 0 E(n 1 ,X 1 )|n 0 ,X 0 d = E( ̃n 0 , ̃ X 0 ) . . . n j |n 0 ,X 0 d = ̃n j−1 X j |n 0 ,X 0 d = ̃ X j−1 E(n j ,X j )|n 0 ,X 0 d = E( ̃n j−1 , ̃ X j−1 ) . . . =⇒ M 1 |n 0 ,X 0 d = E(n 0 ,X 0 )· ̃ M 0 M 2 |n 0 ,X 0 d = E(n 0 ,X 0 )· ̃ M 1 . . . M j |n 0 ,X 0 d = E(n 0 ,X 0 )· ̃ M j−1 . . . .(48) Based on the above, we can also conclude the following about the stopping times under the eventf(α,β)· E(n 0 ,X 0 ) < 1/κ: τ (S,l,t + 1)|n 0 ,X 0 d = (t + 1)∧ minj ∈1,...,t + 1:n j = 0 orf (α,β)· M j+1 ≥ 1/κ d = (t + 1)∧ 1 + minj ∈0,...,t: ̃n j = 0 or f (α,β)· E(n 0 ,X 0 )· ̃ M j+1 ≥ 1/κ d = 1 + τ (S(n 0 ,X 0 ),l + 1,t) |z ̃τ (49) As a consequence, in Eq. 47, we can change the summation indexj → j−1 and use the equalities in distribution in Eq. 48 and Eq. 49 to finally conclude: □ = E θ ∗ ∼Beta(α,β) E n 0 ∼π(S,l) E X 0 ∼Bin(n 0 ,θ ∗ ) " 1f (α + X 0 ,β + n 0 + X 0 ) < 1/κ· E π l+1 " ρ A + ε· C + c(n 0 ) + ̃τ X j=0 c( ̃n j ) · 1∃j ∈ [ ̃τ ]:f (α + X 0 ,β + n 0 − X 0 )· ̃ M j+1 ≥ 1/κ − ̃τ X j=0 c( ̃n j ) θ ∗ ,B 0 = Beta(α + X 0 ,β + n 0 − X 0 ),C 0 = C + c(n 0 ) ## =‡ That is, we have shown that ̄ U A l,t+1 (π|S) =†+‡=V π (S,l,t+ 1). This concludes the induction step and the proof. 39 F.5 Proof of Proposition 4 Consider the state spaceSof the MDPM ε . We define the setS r of reachable states from the initial state (α 0 ,β 0 ,0) to be any state that can be reached with non-negative probability by a policyπ. More formally, S ∈ S r if an only ifS=S out orS= (α,β,C) and there exist a 0≤ t ≤ Tand a sequence of actions and outcomes n 0 ,X 0 ,...n t ,X t such that 0≤ X k ≤ n k , for k = 0,...,t α = α 0 + t X k=0 X k β = β 0 + t X k=0 (n k − X k ) C = t X k=0 c(n t ). (50) In words, the sequence (n 0 ,X 0 ,...n t ,X t ) allows the initial state to eventually transition to the stateS according to the transition dynamics (Eq. 8). Observe that then, any state visited under any realization of the MDP for any policyπis contained in the setS r . Then,S r is finite because, for a fixedt, any sequence (n 0 ,X 0 ,...n t ,X t ) can only take finitely many values (since|A|=n max + 1), and the MDP has only finitely many steps, namely, T + 1. We focus now on the case where the cost functioncis linear. Letc(n) =c 0 +c 1 · n, and consider any state that can be reached at time step t with a total number N = P t k=0 n t , X = P t k=0 X t . Then, t X k=0 c(n t ) = (t + 1)· c 0 + N · c 1 . In particular, in light of Eq. 50, the states inS r (exceptS out ) are in a bijection with the triplets (t,N,X). This is becauseαandβare uniquely determined byXandN, and ifc 0 ,c 1 ̸=0, thentandNuniquely determine C. Next, observe that if the policy never opts out, given a time step 0≤ t ≤ T, the minimum value that Ncan take isN=t+ 1, which corresponds to a sequence of actionsn 0 = 1,...,n t = 1. On the other hand, the maximum value thatNcan take isN= (t+ 1)· n max , which corresponds to a sequence of actions n 0 =n max ,...,n t =n max . Further, for a fixed value ofN, the total positive outcomesXcan take exactly N + 1 values, i.e., X ∈0,...,N. Thus, |S r | = 1 |z S out + T X t=0 (t+1)·n max X N =t+1 (N + 1) = 1 + T X t=0 (t + 1)· n max − t 2 ((t + 2) + ((t + 1)· n max + 1)) = 1 + T X t=0 (n max ) 2 − 1 2 · (t + 1) 2 + 3n max − 1 2 · (t + 1) + 1 = 1 + (n max ) 2 − 1 12 · (T + 1)(T + 2)(2T + 3) + 3n max − 1 4 · (T + 1)(T + 2) + (T + 1) =O (n max ) 2 · T 3 where we have used standard summation formulas for the arithmetic progression P T t=0 (t+1) and the quadratic progression P T t=0 (t+ 1) 2 . In particular, the above also shows that the set of reachable states inttime steps, which we denote by S r (t), satisfies |S r (t)|=O (n max ) 2 · t 2 . 40 F.6 Auxiliary Lemma for Proposition 5 Lemma 2. Consider a stateS ∈Ssuch thatS ̸= S out andf(S)<1/κ, any policyπand 0≤ l≤ T. Write S = (α,β,C). Then, E π " T X t=l r ε (S t ,n t ,S t+1 ) S l = S # ≤ ρ A + ε· C.(51) Note that as an immediate consequence of the above, it also holds for the optimal value function (Eq. 52): V ε (S,l)≤ ρ A + ε· C Proof. Fix any such S,l and policy π. Denote by τ the corresponding stopping time when S l = S, i.e., τ = T ∧ mint∈l,...,T:S t = S out orf (S t )≥ 1/κ| S l = S. Then, r ε (S t ,n t ,S t+1 ) = 0 for t≥ τ by definition of the rewards (Eq. 9). Thus, T X t=l r ε (S t ,n t ,S t+1 ) = τ−1 X t=l r ε (S t ,n t ,S t+1 ) = τ−1 X t=l −c(n t ) + (ρ A + ε· (C t + c(n t )))· 1f (S t+1 )≥ 1/κ =− τ−1 X t=l c(n t ) + τ−1 X t=l (ρ A + ε· (C t + c(n t )))· 1f (S t+1 )≥ 1/κ. Note that by definition ofτ, the second term is at most non-null for the summand witht=τ −1. Thus: T X t=l r ε (S t ,n t ,S t+1 )≤− τ−1 X t=l c(n t ) + (ρ A + ε· (C τ−1 + c(n τ−1 ))) =− τ−1 X t=l c(n t ) + ρ A + ε· (C τ−1 + c(n τ−1 )) =− τ−1 X t=l c(n t ) + ρ A + ε· C τ = ρ A − τ−1 X t=l c(n t ) + ε· C + τ−1 X t=l c(n t ) ! ≤ ρ A + ε· C, where in the last step we have used ε≤ ε max ≤ 1. This concludes the proof. 41 F.7 Proof of Proposition 5 We fix any subsidyε ∈[0,ε max ]. We will use the Bellman optimality equation [51], which states that the optimal value function V ε for M ε , defined by: V ε (S,l) = sup π∈Π E π " T X t=l r ε (S t ,n t ,S t+1 ) S l = S # if 0≤ l≤ T 0if l = T + 1, (52) satisfies the following recursive condition: V ε (S,l) = max n∈A ( X S ′ ∈S P (S ′ |S,n)· (r ε (S,n,S ′ ) + V ε (S ′ ,l + 1)) ) , wherePare the transition dynamics of the MDP, defined implicitly in Eq. 8. We can now particularize to the action spaceAinM ε . Since the actionn= 0 results in a null reward (see Eq. 9), we obtain forl≤ T and any state S = (α,β,C) such that f (S) < 1/κ (and S ̸= S out ): V ε (S,l) = max 0,max n∈1,...,n max E X∼B(n,α,β) [r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)] (53) whileV ε (S,l) = 0 iff(S)≥1/κorS=S out . In the above, to simplify notation, we denote byS(n,X) the state to which the MDP transitions from S after selecting n > 0 and observing outcome X, i.e., S(n,X) = (α + X,β + n− X,C + c(n)). F.7.1 Monotonicity on the belief We focus on proving the monotonicity property in Proposition 5 ofV ε in the parameterα. 19 More precisely, fix a stateS= (α,β,C) such thatf(α,β)<1/κandα ′ ≥ αsuch thatf(α ′ ,β)<1/κ. DenoteS ′ = (α ′ ,β,C). We want to show that V ε (α,β,C,l)≤ V ε (α ′ ,β,C,l) for any 0≤ l≤ T. We proceed by induction over l. Base case l = T. Using Eq. 53 we obtain: 19 The monotonicity in the parameter β follows using a completely symmetric argument. 42 V ε (α ′ ,β,C,T ) = max 0,max n∈1,...,n max E X∼B(n,α ′ ,β) [r ε (S ′ ,n,S ′ (n,X))] = max 0,max n∈1,...,n max E X∼B(n,α ′ ,β) h − c(n) + (ρ A + ε· (C + c(n)))· 1f (α ′ ,β)· E(n,X)≥ 1/κ i (∗) ≥ max 0,max n∈1,...,n max E X∼B(n,α ′ ,β) h − c(n) + (ρ A + ε· (C + c(n)))· 1f (α,β)· E(n,X)≥ 1/κ i (∗) ≥ max 0,max n∈1,...,n max E X∼B(n,α,β) h − c(n) + (ρ A + ε· (C + c(n)))· 1f (α,β)· E(n,X)≥ 1/κ i = V ε (α,β,C,T ) where in (∗) we have used that the functionf(•,•) defined in Eq. 3 is non-decreasing in its first component 20 . In step (∗) we have used that the distributionBB(α ′ ,β,n) stochastically dominates (in the first-order sense) B(α,β,n) if α ′ ≥ α. Induction stepl+ 1→ l. Suppose now thatV ε (α,β,C,l+ 1)≤ V ε (α ′ ,β,C,l+ 1) holds for a certain l+ 1≤ Tand for allα,β,α ′ such thatα ′ ≥ α,f(α,β)<1/κandf(α ′ ,β)<1/κ. Using Eq. 53 again, note that: V ε (α ′ ,β,C,l) = max 0,max n∈1,...,n max E X∼B(n,α ′ ,β) [r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)] | z † Consider a given n > 0 and expand the term † above: 20 This is immediate to verify since θ b ∈ (0, 1) =⇒ log(1 + θ b (e− 1))∈ (0, 1). 43 † = E X∼B(n,α ′ ,β) [r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)] = E X∼B(n,α ′ ,β) h 1f (S ′ (n,X))≥ 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)) i (∗) ≥ E X∼B(n,α ′ ,β) h 1f (S ′ (n,X))≥ 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S(n,X),l + 1)) i (∗) = E X∼B(n,α ′ ,β) h 1f (S ′ (n,X))≥ 1/κ,f (S(n,X))≥ 1/κ · (r ε (S ′ ,n,S ′ (n,X)) + V ε (S(n,X),l + 1)) + 1f (S ′ (n,X))≥ 1/κ,f (S(n,X)) < 1/κ · (r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S(n,X),l + 1)) i where in (∗) we have used that iff(S ′ (n,X))<1/κ, thenf(S(n,X))<1/κand hence the induction hypothesis appliesV ε (S ′ (n,X),l+ 1)≥ V ε (S(n,X),l+ 1). In (∗) we have used that iff(S ′ (n,X))≥1/κ and f (S(n,X))≥ 1/κ, then V ε (S(n,X),l + 1) = V ε (S ′ (n,X),l + 1) = 0. Lastly, in †, consider the case f (S ′ (n,X))≥ 1/κ and f (S(n,X)) < 1/κ. Then, r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1) =−c(n) + (ρ A + ε· (C + c(n))) and, r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1) =−c(n) + V ε (S(n,X),l + 1) ⋄ ≤−c(n) + ρ A + ε· (C + c(n))), where ⋄ follows from Lemma 2. Finally, noting thatr ε (S ′ ,n,S ′ (n,X))≥ r ε (S,n,S(n,X)) and using the first-order stochastic dominance for the Beta-Binomial again, we conclude that: †≥ E X∼B(n,α ′ ,β) h 1f (S ′ (n,X))≥ 1/κ, f (S(n,X))≥ 1/κ · (r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)) + 1f (S ′ (n,X))≥ 1/κ, f (S(n,X)) < 1/κ · (r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ · (r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)) i ≥ E X∼B(n,α,β) [r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)]. That is, E X∼B(n,α ′ ,β) [r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)]≥ E X∼B(n,α,β) [r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)], (54) and taking maximum over the action n and using Eq. 53: V ε (α ′ ,β,C,l)≥ V ε (α,β,C,l). This concludes the induction step and thus the proof of the monotonicity in the belief. 44 F.7.2 Monotonicity on the cost We now focus on proving the monotonicity property in Proposition 5 ofV ε in the cumulated costC. More precisely, fix a stateS= (α,β,C) such thatf(α,β)<1/κand considerC ′ ≥ C. DenoteS ′ = (α,β,C ′ ). We want to show that V ε (α,β,C,l)≤ V ε (α,β,C ′ ,l) for any 0≤ l≤ T. Again, we proceed by induction over l. Base l = T. Using Eq. 53 we obtain: V ε (α,β,C ′ ,T ) = max 0,max n∈1,...,n max E X∼B(n,α,β) [r ε (S ′ ,n,S ′ (n,X))] = max 0,max n∈1,...,n max E X∼B(n,α,β) h − c(n) + (ρ A + ε(C ′ + c(n)))· 1f (α,β)· E(n,X)≥ 1/κ i ≥ max 0,max n∈1,...,n max E X∼B(n,α,β) h − c(n) + (ρ A + ε(C + c(n)))· 1f (α,β)· E(n,X)≥ 1/κ i = V ε (α,β,C,T ). Induction stepl+ 1→ l. Suppose now thatV ε (α,β,C,l+ 1)≤ V ε (α,β,C ′ ,l+ 1) holds for a certain l + 1≤ T and for all α,β such that f (α,β) < 1/κ and 0≤ C ≤ C ′ . Using Eq. 53 again, note that: V ε (α,β,C ′ ,l) = max 0,max n∈1,...,n max E X∼B(n,α,β) [r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)] | z † Consider a given n > 0 and expand the term † above: † = E X∼B(n,α,β) [r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)] = E X∼B(n,α,β) h 1f (S ′ (n,X))≥ 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)) i (∗) ≥ E X∼B(n,α,β) h 1f (S ′ (n,X))≥ 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S(n,X),l + 1)) i (∗) = E X∼B(n,α,β) h 1f (S ′ (n,X))≥ 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S(n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ (r ε (S ′ ,n,S ′ (n,X)) + V ε (S(n,X),l + 1)) i where in (∗) we have used that iff(S ′ (n,X))<1/κ, then the induction hypothesis appliesV ε (S ′ (n,X),l+1)≥ V ε (S(n,X),l+ 1) because the total cost in stateS ′ (n,X) isC ′ +c(n)≥ C+c(n), which equals the total 45 cost inS(n,X). In (∗) we have used that iff(S ′ (n,X))≥1/κ, thenf(S(n,X))≥1/κand hence V ε (S ′ (n,X),l + 1) = V ε (S(n,X),l + 1) = 0. Finally, noting that r ε (S ′ ,n,S ′ (n,X))≥ r ε (S,n,S(n,X)), we conclude that: †≥ E X∼B(n,α,β) h 1f (S ′ (n,X))≥ 1/κ (r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)) + 1f (S ′ (n,X)) < 1/κ (r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)) i = E X∼B(n,α,β) [r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)] That is, E X∼B(n,α,β) [r ε (S ′ ,n,S ′ (n,X)) + V ε (S ′ (n,X),l + 1)]≥ E X∼B(n,α,β) [r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)], and taking maximum over the action n and using Eq. 53: V ε (α,β,C,l)≥ V ε (α,β,C ′ ,l). This concludes the induction step and thus the proof of the monotonicity in the cost. 46 F.8 Proof of Proposition 6 Fix any subsidy levelε∈[0,ε max ], total costC ≥0, and an initial time step 0≤ l≤ T. Consider any state S ̸= S out and writeS= (α,β,C). Then, the Bellman optimality condition [51] establishes that the action n= 0 is optimal if and only if the value function coincides with the value of actionn= 0 (which is 0 by definition of the reward function in Eq. 9), that is, V ε (S,l) = 0, whereV ε (S,l) is the optimal value function defined in Eq. 52. Using Eq. 53, this is equivalent to the condition: max 0,max n∈1,...,n max E X∼B(n,α,β) [r ε (S,n,S(n,X)) + V ε (S(n,X),l + 1)] | z =H(α,β,C) = 0 ⇐⇒ H(α,β,C)≤ 0. Now, we observe that in proving Proposition 5 in Appendix F.7 (see Eq. 54), we precisely showed by induction that the functionH(α,β,C) is non-decreasing inα, and an analogous argument shows that it is non-increasing in β. We leverage this property in what follows. For any α > 0, define the quantity ̃ β(α) = supβ : H(α,β,C) > 0 and β > 0. As a consequence of the monotonicity of H(α,β,C), if α ′ ≥ α, then H(α,β,C) > 0 =⇒ H(α ′ ,β,C) > 0, and thus the following set inclusion holds: β : H(α,β,C) > 0 and β > 0⊆β : H(α ′ ,β,C) > 0 and β > 0. Taking the supremum, the above implies: supβ : H(α,β,C) > 0 and β > 0 | z ̃ β(α) ≤ supβ : H(α ′ ,β,C) > 0 and β > 0 |z ̃ β(α ′ ) , which shows that ̃ β(α) is non-decreasing in α. Next, observe that: •Ifβ > ̃ β(α), then by definition of the supremum we have thatH(α,β,C)≤0, and thus the action n = 0 is optimal. •Ifβ < ̃ β(α), then again by the definition of the supremum, there existsβ ′ such thatβ < β ′ ≤ ̃ β(α) withH(α,β ′ ,C)>0. Then, using the monotonicity,H(α,β,C)≥ H(α,β ′ ,C)>0, which means that opting out is strictly sub-optimal: there exists an actionn >0 that leads to a strictly higher expected value if taken at state S and time l. This proves the claim in Proposition 6. 47 F.9 Proof of Proposition 7 Recall that the value function in the MDP M ε is defined by: V ε π (S,l) = E π " T X t=l r ε (S t ,n t ,S t+1 ) S l = S # ,(55) with r ε defined by: r ε (S,n,S ′ ) = ( −c(n) + (ρ A + ε· (C + c(n)))· 1f (S ′ )≥ 1/κ if 0 < f (S) < 1/κ 0if S = S out or f (S)≥ 1/κ. (56) Firstly, note that all rewards in Eq. 55 become 0 as soon asS t =S out orf(S t )≥1/κ. This is because, from the transition dynamics in Eq. 8, any such stateS t is absorbing, i.e.,S t+1 =S t ifS t =S out orf(S t )≥1/κ, and satisfiesr ε (S t ,n t ,S t+1 ) = 0. Then, the terms in Eq. 55 that are non-zero correspond to time stepst≤ τ, where τ is the stopping time defined by: τ = T ∧ mint∈l,...,T:n t = 0 orf (S t+1 )≥ 1/κ. For l≤ t≤ τ, the first case of the reward function in Eq. 56 applies: V ε π (S,l) = E π " τ X t=l −c(n t ) + (ρ A + εC t+1 )1f (S t+1 )≥ 1/κ S l = S # = E π " τ X t=l (−c(n t ) + ρ A 1f (S t+1 )≥ 1/κ) + ε τ X t=l C t+1 1f (S t+1 )≥ 1/κ S l = S # Note that the first part of the expectation corresponds exactly to the value function under no subsidy,V 0 π (S,l), and thus, V ε π (S,l) = V 0 π (S,l) + ε· E π " τ X t=l C t+1 · 1f (S t+1 )≥ 1/κ S l = S # Now, define: A π (S,l) = E π " τ X t=l C t+1 · 1f (S t+1 )≥ 1/κ S l = S # = E π [C τ +1 · 1f (S τ +1 )≥ 1/κ| S l = S](57) This term corresponds to the (expected) total cost incurred by the agent conditional on the product being approved, when the MDP starts from stateSat timel. 21 Indeed, by the definition of the stopping timeτ, approval (i.e.,f(S t+1 )≥1/κ) can only happen at exactlyt=τ. If the agent opts out or the horizonTis reached without approval, the indicator 1f (S t+1 )≥ 1/κ is 0 for all t. We conclude that V ε π (S,l) = V 0 π (S,l) + ε· A π (S,l). 21 We use the term “conditional on approval” informally and for didactic purposes, sinceA π (S,l) is the quantity that naturally appears in the value decompositionV ε π (S,l) =V 0 π (S,l) +ε· A π (S,l), representing the expected subsidy paid by the principal. Strictly speaking, however,A π (S,l) is the expected cost weighted by the indicator of approval, rather than a conditional expectation in the measure-theoretic sense. 48 F.10 Proof of Proposition 8 Denote by Π r ⊂Π the set of all deterministic policies that select actionn= 0 atS out , any state such that f(S)≥1/κ, or any state such thatS /∈ S r , and observe that Π r is finite. Recall from Proposition 7 that, for any policy π ∈ Π r , V ε π (S,l) = V 0 π (S,l) + ε· A π (S,l).(58) In particular, at the initial state S 0 = (α 0 ,β 0 , 0) and time l = 0, we have ̄ U A (π;ε) = V ε π (S 0 , 0) = V 0 π (S 0 , 0) + ε· A π (S 0 , 0).(59) Since Π r is finite, for every (S,l)∈S r × [T ] the optimal value function V ε (S,l) = max π∈Π r V 0 π (S,l) + ε· A π (S,l)(60) is the point-wise maximum of finitely many affine functions inε. It is well-known that such a point-wise maximum is convex, continuous, and piecewise linear [79]. Here, we particularize to our problem, with the goal of constructing a single partition of [0,ε max ] on which a single optimal policy (forM ε at every state S ∈S r and time l∈ [T ] simultaneously) is optimal in each interval of the partition. For any two distinct policies π,π ′ ∈ Π r and any (S,l)∈S r × [T ], consider the difference (V 0 π (S,l)− V 0 π ′ (S,l)) + ε(A π (S,l)− A π ′ (S,l)). If (V 0 π (S,l),A π (S,l)) = (V 0 π ′ (S,l),A π ′ (S,l)) for all (S,l), then the two policies yield identical values for allε and all (S,l), and we may break ties arbitrarily and retain only one of them. Similarly, ifV 0 π (S,l)+ε·A π (S,l)≥ V 0 π ′ (S,l) +ε· A π ′ (S,l) for allε∈[0,ε max ] and all (S,l), then policyπ ′ can be removed without changing the optimal value function at any (S,l). Thus, without loss of generality, we restrict our attention to a subset ̃ Π r ⊂Π r such that for any distinctπ,π ′ ∈ ̃ Π r and any (S,l), the corresponding affine functions intersect exactly once in [0,ε max ], and each policy is optimal for some value of ε. Then, let K =ε∈ [0,ε max ] :∃π ̸= π ′ ∈ ̃ Π r , ∃ (S,l)∈S r × [T ] such that V 0 π (S,l) + ε· A π (S,l) = V 0 π ′ (S,l) + ε· A π ′ (S,l). Since ̃ Π r ,S r , and [T] are all finite, and each quadruple (π,π ′ ,S,l) contributes at most one point toK, the set K is finite. Ordering its elements and adding the endpoints if necessary, we obtain a partition 0 = ε 0 < ε 1 <· < ε L = ε max . By construction, no two affine functionsε7→ V 0 π (S,l)+ε·A π (S,l) intersect in any open interval (ε i ,ε i+1 ) at any (S,l). Hence, for every (S,l), the ordering ofV 0 π (S,l) +ε·A π (S,l) π∈ ̃ Π r is constant on each such interval. It follows that there exists a single policyπ i ∈ ̃ Π r that is optimal forM ε at every (S,l) simultaneously, and such that V ε (S,l) = V 0 π i (S,l) + ε· A π i (S,l) for all ε∈ [ε i ,ε i+1 ) and all (S,l). In particular, specializing to (S,l) = (S 0 , 0), ̄ U A (π ε ;ε) = V 0 π i (S 0 , 0) + ε· A π i (S 0 , 0) for all ε∈ [ε i ,ε i+1 ). This establishes the result for ̄ U A (π ε ;ε). Note thatE (α 0 ,β 0 )∼Q [ ̄ U A (π ε ;ε)] is then also convex and continuous because it is an average of convex continuous functions that are uniformly bounded (observe that by Lemma 2, we have 0≤ ̄ U A (π ε ;ε)≤ ρ A uniformly for any initial belief parameters (α 0 ,β 0 ), where the lower bound follows from the fact that the agent can always opt out at no cost); continuity of the expectation then follows from the dominated convergence theorem. Lastly, in each interval [ε i ,ε i+1 ), ̄ U A (π ε ;ε) is linear, which implies that E (α 0 ,β 0 )∼Q [ ̄ U A (π ε ;ε)] is also linear: E (α 0 ,β 0 )∼Q [ ̄ U A (π ε ;ε)] = E (α 0 ,β 0 )∼Q [V 0 π i (S 0 , 0)] + ε· E (α 0 ,β 0 )∼Q [A π i (S 0 , 0)] for all ε∈ [ε i ,ε i+1 ). This concludes the proof. 49 F.11 Proof of Proposition 9 Recall that the social utility ̄ U S (ε;π) is defined as (Eq. 13): ̄ U S (ε;π) = E π,(α 0 ,β 0 )∼Q " T X t=0 ρ S − ε· C t+1 · 10 < f (S t ) < 1/κ≤ f (S t+1 ) S 0 = (α 0 ,β 0 , 0) # . Let τ be the stopping time defined by: τ = T ∧ mint∈0,...,T:n t = 0 or f (S t+1 )≥ 1/κ, that is, the last time step before reaching an absorbing state—eitherS out , for whichf(S out ) = 0, or any state S such that f (S)≥ 1/κ. Then, using the linearity of the expectation: ̄ U S (ε;π) = E π,(α 0 ,β 0 )∼Q " T X t=0 ρ S − ε· C t+1 · 10 < f (S t ) < 1/κ≤ f (S t+1 ) S 0 = (α 0 ,β 0 , 0) # = ρ S · E π,(α 0 ,β 0 )∼Q " T X t=0 10 < f (S t ) < 1/κ≤ f (S t+1 ) S 0 = (α 0 ,β 0 , 0) # − ε· E π,(α 0 ,β 0 )∼Q " τ X t=0 C t+1 · 10 < f (S t ) < 1/κ≤ f (S t+1 ) S 0 = (α 0 ,β 0 , 0) # = ρ S · E (α 0 ,β 0 )∼Q E π [1∃t∈ [T ]: 0 < f (S t ) < 1/κ≤ f (S t+1 )| S 0 = (α 0 ,β 0 , 0)] − ε· E (α 0 ,β 0 )∼Q E π [C τ +1 · 11/κ≤ f (S τ +1 )| S 0 = (α 0 ,β 0 , 0)] = ρ S · E (α 0 ,β 0 )∼Q [P π (∃t∈ [T ]: 0 < f (S t ) < 1/κ≤ f (S t+1 )| S 0 = (α 0 ,β 0 , 0))] − ε· E (α 0 ,β 0 )∼Q E π [C τ +1 · 11/κ≤ f (S τ +1 )| S 0 = (α 0 ,β 0 , 0)] = ρ S · E (α 0 ,β 0 )∼Q [P π (∃t∈ [T ]: 1/κ≤ f (S t+1 )| S 0 = (α 0 ,β 0 , 0))] − ε· E (α 0 ,β 0 )∼Q [A π (α 0 ,β 0 , 0, 0)] where we have used the definition ofA π in Eq. 57, and the fact that the condition 0< f(S t )<1/κ≤ f(S t+1 ) can occur at most once at time step t = τ. 50 F.12 Proof of Proposition 10 In this section, we will show that Algorithm 1 recovers (in a finite number of steps) the partition of the interval [0,ε max ] given by Proposition 8, which we denote by P: 22 P =0 = ε 0 < ε 1 <· < ε L = ε max , where for each interval [ε i ,ε i+1 ), there exists a (deterministic) policyπ i that is optimal for anyε∈[ε i ,ε i+1 ), that is (writing S 0 = (α 0 ,β 0 , 0)), ( ̄ U A (π ε ;ε) = V 0 π i (S 0 , 0) + ε· A π i (S 0 , 0)∀ε∈ [ε i ,ε i+1 ) E (α 0 ,β 0 )∼Q [ ̄ U A (π ε ;ε)] = E (α 0 ,β 0 )∼Q [V 0 π i (S 0 , 0)] + ε· E (α 0 ,β 0 )∼Q [A π i (S 0 , 0)] ∀ε∈ [ε i ,ε i+1 ). Since the above partition does not depend on the beliefQof the principal (Proposition 8), we assume without loss of generality that the principal knows the agent’s initial beliefBeta(α 0 ,β 0 ). Consequently, the expectationsE (α 0 ,β 0 )∼Q [•] simply correspond to evaluating the integrand at the true belief parameters (α 0 ,β 0 ). We begin by proving the following lemma. 23 Lemma 3. Letε L < ε R be two subsidy levels with, respectively, optimal policiesπ L andπ R , and value functions (evaluated at the initial state (α 0 ,β 0 ,0) and initial time stepl= 0, which we omit for notational convenience) V 0 L + ε· A L and V 0 R + ε· A R (see Proposition 7). The following holds: 1. If A L = A R , then ̄ U A (π ε ;ε) = V 0 L + ε· A L for all ε∈ [ε L ,ε R ]. 2. IfA L < A R , letε int = V 0 L −V 0 R A R −A L . Then, ̄ U A (π ε ;ε) =max(V 0 L +ε· A L ,V 0 R +ε· A R ) for allε∈[ε L ,ε R ] if and only if ̄ U A (π ε int ;ε int ) = V 0 L + ε int · A L . Proof. For part 1., ifA L =A R , then since ̄ U A (π ε ;ε) is convex, its subgradient must be non-decreasing inε. Thus, for anyε∈(ε L ,ε R ), we must haveA L ≤ A π ε ≤ A R , which impliesA π ε =A L . By continuity and the fact that ̄ U A (π ε ;ε) is the point-wise maximum of affine functions, it follows thatV 0 π ε =V 0 L =V 0 R , and the value function is a single affine segment on this interval. For part 2., the direct implication follows because if ̄ U A (π ε ;ε) =max(V 0 L +ε· A L ,V 0 R +ε· A R ) for all ε ∈[ε L ,ε R ], then evaluating atε int , and sinceV 0 L +ε int · A L =V 0 R +ε int · A R by definition, we obtain ̄ U A (π ε int ;ε int ) =V 0 L +ε int · A L . For the backward direction, we argue by contradiction and suppose ̄ U A (π ε int ;ε int ) =V 0 L +ε int · A L but there exists someε ′ ∈(ε L ,ε R ) and a policyπ ′ such thatV 0 π ′ +ε ′ · A π ′ > max(V 0 L +ε ′ ·A L ,V 0 R +ε ′ ·A R ). Assume without loss of generalityε ′ ≤ ε int (the argument for the caseε ′ ≥ ε int is symmetric, withπ R in place ofπ L ). Sinceπ L is optimal atε L , we haveV 0 L +ε L · A L ≥ V 0 π ′ +ε L · A π ′ . Combined with the assumptionV 0 π ′ +ε ′ ·A π ′ > V 0 L +ε ′ ·A L and subtracting, we obtain (ε ′ −ε L )(A π ′ −A L )>0, which implies A π ′ > A L since ε ′ > ε L . Then, at ε int ≥ ε ′ , V 0 π ′ + ε int · A π ′ = (V 0 π ′ + ε ′ · A π ′ ) + (ε int − ε ′ )· A π ′ > (V 0 L + ε ′ · A L ) + (ε int − ε ′ )· A L = V 0 L + ε int · A L , where the strict inequality usesV 0 π ′ +ε ′ · A π ′ > V 0 L +ε ′ · A L together withA π ′ > A L andε int ≥ ε ′ . This contradicts the assumption that ̄ U A (π ε int ;ε int ) = V 0 L + ε int · A L . To prove Proposition 10, we begin by showing that any point added to the setUin Algorithm 1 corresponds to a point ofP. Consider any iteration of the algorithm whereV 0 int +ε int ·A int ≤ V 0 L +ε int ·A L andA L ̸= A R . Then, from Lemma 3, it follows that in this case, for any possible subsidyε ∈[ε L ,ε R ], the optimal value function is given by ̄ U A (π ε ;ε) =max(V 0 L +ε· A L ,V 0 R +ε· A R ), with a change in slope at their intersection, 22 We denote by P(ε) the interval of P containing ε. 23 For convenience, we use L, R and int as subscripts instead of using π L , π R and π int . 51 i.e., atε int . That is,ε int is a point of the partitionP, and the value ̄ U S (ε int ;π R ) =max ε∈P(ε int ) ̄ U S (ε;π ε ) is stored in the set U. Reciprocally, consider any iteration of the algorithm whereV 0 int +ε int · A int > V 0 L +ε int · A L . Then, Lemma 3 implies thatP(ε L )̸= P(ε int ) andP(ε R )̸= P(ε int ). That is,P(ε int ) is a new interval in the partitionPwhere the optimal value function is given by the linear componentV 0 π int +ε·A π int . The algorithm pushes the two sub-intervals [ε L ,ε int ] and [ε int ,ε R ] onto the stack (line 17), along with their respective linear componentsV 0 L +ε· A L ,V 0 π int +ε· A π int andV 0 R +ε· A R . Since by Proposition 8, there are a finite number of such linear components (or equivalently, P is finite), Algorithm 1 terminates in a finite number of steps. To conclude, we argue that, upon termination, the setUobtained from Algorithm 1 contains every point ofP. LetNdenote the total number of pop operations performed by the algorithm beforeIbecomes empty (finite by the argument above). Fork= 0,1,...,N, denote byI k the state of the stack afterkpops, and byD k ⊆Pthe set of points ofPfound by the algorithm during the firstkiterations, withD 0 =0,ε max . Observe first that whenever Algorithm 1 pushes two new intervals ontoI, the intersection pointε int lies strictly interior to the intervalP(ε int ) (since by Lemma 3,P(ε L )̸=P(ε int )̸=P(ε R ), andε L < ε int < ε R ), soε int /∈P. Consequently, every endpoint of an interval ever pushed ontoI k is either inD k or not inP. In light of this, we prove by induction on k the following condition: 24 P ⊆ D k ∪ [ [ε L ,ε R ]∈I k (ε L ,ε R ).(61) Base casek= 0.I 0 =[0,ε max ]andD 0 =0,ε max . Everyε∈Pis either in0,ε max =D 0 or in (0,ε max ), so the condition in Eq. 61 holds for k = 0. Inductive stepk → k+ 1. Suppose the condition in Eq. 61 holds and consider the (k+ 1)-th pop of some [ε L ,ε R ]∈I k . •IfV 0 int +ε int ·A int ≤ V 0 L +ε int ·A L , thenε int is added toD k+1 , and the popped interval is not replaced. By Lemma 3, (ε L ,ε R )∩P ⊆ε int ⊆D k+1 . Hence anyε∈Ppreviously covered by (ε L ,ε R ) is now in D k+1 . The endpointsε L andε R , if they belong toP, were endpoints of the popped interval and hence are already contained in D k ⊆D k+1 . Thus, the condition in Eq. 61 holds for k + 1. •IfV 0 int +ε int ·A int > V 0 L +ε int ·A L , then [ε L ,ε R ] is replaced by [ε L ,ε int ] and [ε int ,ε R ], whileD k+1 =D k . Ifε int ∈D k , Eq. 61 holds fork+1, and ifε int /∈P, we have that (ε L ,ε R )∩P=[(ε L ,ε int )∪(ε int ,ε R )]∩P, and the condition in Eq. 61 also holds for k + 1. Lastly, at the final iteration,I N =∅impliesP ⊆D N ; combined withD N ⊆Pby construction,D N =P. Since everyε∈D N has its social utility saved inU(at line 7 for the initial endpoints and line 15 for each new point ofPfound),Ucontains(ε, ̄ U S (ε;π ε )) :ε∈P. The maximizer of ̄ U S over [0,ε max ] is attained at some left endpoint ε i ∈P, and arg max (ε,u)∈U u returns it. 24 With a slight abuse of notation, we identify the elements ofIwith intervals, meaning that if (ε L ,π L , ̄ V 0 L , ̄ A L ,ε R ,π R , ̄ V 0 R , ̄ A R )∈ I, we write that [ε L ,ε R ]∈I. 52 G Additional Experimental Details The complete code used for our experiments, including the implementation of Algorithm 3 and Algorithm 1, is available as supplementary material. We will publicly release it with the final version of the paper. Hardware setup. Our experiments are executed on a compute server equipped with 2×Intel Xeon Gold 5317 CPU, 1,024 GB main memory, and 2×H100 NVIDIA GPU (80 GB, Hopper Architecture). In each experiment, a single Nvidia H100 GPU is used. Software setup. All experiments are implemented inPython3.13.5 usingPyTorch2.1.1 andNumPy2.4.4. Computations are performed on an NVIDIA GPU with CUDA 13.0 support. Runtime. For the setting used in Section 6, Algorithm 3 computes the optimal policy of the agent (for any given subsidy) in∼2.4s, and Algorithm 1 computes the optimal subsidy in∼271s, which involves solving 114 times a different belief MDPM ε . In Figure 4, we evaluate the runtime of Algorithm 1 across multiple configurations for the parametersTandn max , which determine the size of the state and action space (Proposition 4). 50100150200250300 n max 0 1 2 3 4 T 5.2s5.2s5.2s5.3s5.3s5.3s 5.4s5.4s6.2s9.0s12.3s19.6s 2.8s4.0s12.6s43.4s123.0s369.6s 1.6s10.6s62.8s271.8s729.5s1771.0s 6.5s39.3s239.4s1043.9s2402.1s5875.7s 50100150200250300 n max 0 1 2 3 4 T 235555 3211191721 91531476294 133871114134163 3977132207208254 10 1 10 2 10 3 Runtime (s) 50 100 150 200 250 MDPs solved Figure 4: Runtime of Algorithm 1. The figure shows, across multiple values of the maximum number of trialsTand the maximum sample size per trialn max , the runtime of Algorithm 1 (left panel) and the number of belief MDPs solved by the algorithm (right panel). All other parameters are fixed as specified in Tables 2 and 3. The experiments are run on an NVIDIA H100 GPU. Parameter details. Tables 2 and 3 report the values of all parameters required to specify the sequential approval protocol used in our fiducial setting (Section 6). Unless otherwise stated, all results are obtained using these values. When any parameter is varied (e.g., in Panel (b) of Figure 1 or in Appendix H), we explicitly indicate it. Table 2: Non-economic parameters T n max θ b κ θ ∗ ε max (α 0 ,β 0 ) Q 3 200 0.5 0.05 0.65 0.9(1,1) δ (1,1) Table 3: Economic parameters ρ S ρ A c(n) = c 0 + c 1 n, n̸= 0 c 0 c 1 $2000 M $240 M $48.9 M $0.066 M Implementation details. Our Python implementation of Algorithm 3 leverages the bijection between pairs 53 (α,β) and pairs (X,N), whereXis the total number of successes andNthe total number of patients (the total sample size), as we detail in Eq. 34. Furthermore, since the cost function is linear, we also use the fact that the total cumulated costCat a given state (X,N) at timelcan be written asC=l· c 0 +N · c 1 . As a result, our implementation does not explicitly keep track of the cumulated costC, which significantly reduces the computational overhead. A similar idea is used in the proof of Proposition 4. Our implementation of Algorithm 1 computes the optimal subsidyε ∗ (and solves each belief MDPM ε ) exactly. Consequently, there is no associated uncertainty inε ∗ or in any of the quantities that we report computed using the belief MDP, including ̄ U A and ̄ U S . In contrast, the true quantities under the approval process usingθ ∗ described in Section 2—such as the true utilitiesU A andU S (Eq. 7), as well as the probability that the agent opts out during the approval process—are estimated using 100,000−200,000 Monte Carlo rollouts, and we report 95% confidence error bars computed via bootstrapping with 1000 resamples. 54 H Additional Experimental Results H.1 Additional results complementing Section 6 In this section, we provide complementary results to the approval process considered in Section 6, whose parameters are given in Table 2 and Table 3. In Figure 5, we show the optimal value function and policy for the MDPM ε ∗ , withε ∗ = 1.08, and in Figure 6 we show how the belief of the agent evolves in 300 realizations of the approval process, for different values of the true efficacy θ ∗ . 16111621 α 1 6 11 16 21 β 16111621 α 1 6 11 16 21 β 0 25 50 75 100 125 150 175 200 Optimal value function ( $ M) 0 25 50 75 100 125 150 175 200 Optimal action n Figure 5: Optimal value function and policy in the belief MDPM ε ∗ for the optimal subsidy ε ∗ = 0.108. The left panel shows the optimal value function in the belief MDP,V ε ∗ (α,β,C(α,β),1), at time stepl= 1, where the cost of each state is given byC(α,β) = 1· c 0 + (α+β− α 0 − β 0 )· c 1 (see Eq. 34). The right panel shows the optimal actionntaken by the optimal policy at time stepl= 1 for each belief, i.e., π ε ∗ (α,β,C(α,β), 1). The optimal action at time step l = 0 (not shown here) is n = 79. 050100150200 α 0 50 100 150 β (a) θ ∗ = 0.3 050100150200 α 0 50 100 150 β (b) θ ∗ = 0.65 050100150200 α 0 50 100 150 β (c) θ ∗ = 0.8 Figure 6: Trajectories of the approval process. Each panel shows 300 realizations of the approval process for different true efficaciesθ ∗ of the antibiotic. Each blue segment corresponds to the agent conducting a new trial and updating its belief (see Figure 3 for an illustration of the geometry). In the left panel, the agent conducts a first trial and then always opts out; in the middle panel, the agent can conduct multiple trials, and in the right panel, the antibiotic is always approved after the first trial. In Figure 7, we show, for the initial action taken by the agent at time stepl= 0 (and belief (α 0 ,β 0 ) = (1,1)), 55 the expected cumulative future reward inM ε ∗ for the optimal subsidyε ∗ = 0.108, defined for eachn >0 as E π ε ∗ [ P T t=0 r ε ∗ (S t ,n t ,S t+1 )|S 0 ,n 0 = n] . The sample size maximizing this curve isn= 79, which is the size of the first trial conducted by the agent. Notably, although the expected reward is unimodal, it exhibits small-scale oscillations. These oscillations are not numerical artifacts. In fact, their approximate period is given by 1/log(1 +θ b (e−1)). The reason is that the agent’s actions are discrete, whereas the functionf defined in Proposition 2, which determines the approval condition, decreases by exactlylog(1 +θ b (e−1)) whenever the agent selects a new actionn(recall thatα t+1 +β t+1 − α t − β t =n t ). Since this change is not an integer quantity, the expected reward only exhibits a small decrease afternincreases by approximately 1/log(1 + θ b (e− 1)), which gives rise to the observed oscillations. 121416181101121141161181 Action,n 0 5 10 15 20 25 30 35 Expected reward of the action ( $ M) Figure 7: Expected reward for each sample size. The figure show, for the initial action taken by the agent at time stepl= 0 and state (α 0 = 1,β 0 = 1,0), the total expected reward in the MDPM ε ∗ under the optimal subsidy ε ∗ = 0.108 when the agent takes action n and then follows the optimal policy π ε ∗ . 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0.16 0.18 0.20 0.22 Probability of opting out (a) Opting out before approval vs. subsidy 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0.77 0.78 0.79 0.80 0.81 0.82 Probability of approval (b) Approval probability vs subsidy Figure 8: Opt out and approval probabilities. The figure shows, for an antibiotic withθ ∗ = 0.65, the probability that the agent opts out of the approval process by selectingn= 0 before approval, as well as the probability that the antibiotic is ultimately approved. For each subsidy level, the agent follows the optimal policy. Note that, in principle, the agent may never opt out during the approval process; however, the antibiotic may still fail to be approved if the maximum number of trials is reached. In Figure 9 and Figure 10, we show, respectively, the utility of the agent and the social utility when the agent selects its optimal policy for each possible subsidy, both computed using the belief MDP, and the true 56 realized utilities when the antibiotic has efficacy θ ∗ = 0.65 (Eq. 7). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 35 40 45 50 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 120 140 160 Agent utility, U A ( π ε ; ε ) ( $ M) 0.16 0.18 0.20 0.22 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 9: Agent utilities. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 745 750 755 760 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1540 1560 1580 1600 1620 Social utility, U S ( ε ; π ε ) ( $ M) 0.16 0.18 0.20 0.22 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 10: Social utilities. The left panel shows the social utility (Eq. 13) computed using the belief MDP M ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). Lastly, in Figure 11 we show that the optimal social utility (that is, the social utility under the optimal subsidy) increases monotonically as the true efficacy θ ∗ increases. 57 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 0 2000 4000 6000 8000 10000 Optimal social utility, U S ( ε ∗ ; π ε ∗ ) ( $ M) 0.3 0.4 0.5 0.6 0.7 0.8 0.9 θ ∗ Figure 11: Optimal social utility for different antibiotic efficacies. The figure shows how social utility U S (ε ∗ ;π ε ∗ )—when the principal chooses the optimal subsidyε ∗ and the agent adopts the corresponding optimal policyπ ε ∗ —varies as a function of the ratioρ S /ρ A across different levels of efficacyθ ∗ . The dashed line corresponds to the social utility ̄ U S (ε ∗ ;π ε ∗ ) computed using the belief MDP, which does not depend on the true efficacy θ ∗ . 58 H.2 Additional results using different parameters In this section, we present further experimental results for the antibiotic approval process described in Section 6, where we vary selected parameters (see Tables 2 and 3). H.2.1 Approval under increased experimental costs Here, we show the result of an antibiotic approval process with an increased experimental cost. In particular, we take the parameters in Table 2 and Table 3 but increase the fixed costc 0 of a trial to $100Mand the per-patient costc 1 to $0.1M. We find that the agent opts out at the beginning of the process unless the principal subsidizes a fraction higher than the optimal subsidy, ε ∗ = 0.551. 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0 5 10 15 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0 50 100 150 Agent utility, U A ( π ε ; ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 12: Agent utilities under increased experimental costs. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex, and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0 200 400 600 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0 500 1000 1500 Social utility, U S ( ε ; π ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 13: Social utilities under increased experimental costs. The left panel shows the social utility (Eq. 13) computed using the belief MDP M ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionP where the agent’s optimal policy is constant (Proposition 8). 59 In Figure 14 we show that this optimal subsidy is constant asρ S increases, and in Figure 15 that our sequential protocol yields social utility gains > 20% relative to a non-sequential protocol. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 0.0 0.2 0.4 0.6 Optimal subsidy, ε ∗ Figure 14: Optimal subsidy vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 25 30 35 Social utility gain vs. non-sequential (%) Non-sequential (optimal subsidy) Figure 15: Social utility gain vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non- sequential approval protocol in which the agent is restricted to a single trial withn max = 800, under the optimal subsidy computed using Algorithm 1 (in the non-sequential protocol without subsidy the agent always opts out, yielding zero social utility). 60 H.2.2 Approval under increased agent approval benefit Here, we show the result of an antibiotic approval process with an increased approval utility for the agent, ρ A = $5000M(and the rest of the parameters in Table 2 and Table 3 fixed). In this case, the approval utility for the agent covers the expected cost of approval by a large margin, and we find that no subsidy is needed, namely, ε ∗ = 0. 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1900 1905 1910 1915 1920 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 4500 4520 4540 4560 4580 4600 Agent utility, U A ( π ε ; ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 16: Agent utilities under increased approval utility. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 765 770 775 780 785 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1760 1780 1800 1820 1840 1860 Social utility, U S ( ε ; π ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 17: Social utilities under increased approval agent utility. The left panel shows the social utility (Eq. 13) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8). 61 0.000.250.500.751.001.251.501.752.00 Social-to-agent approval benefit ratio,ρ S /ρ A −0.02 −0.01 0.00 0.01 0.02 Optimal subsidy, ε ∗ Figure 18: Optimal subsidy vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1. 0.000.250.500.751.001.251.501.752.00 Social-to-agent approval benefit ratio,ρ S /ρ A 8.25 8.50 8.75 Social utility gain vs. non-sequential (%) Non-sequential (no subsidy) Non-sequential (optimal subsidy) Figure 19: Social utility gain vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non- sequential approval protocol in which the agent is restricted to a single trial withn max = 800, under (i) the optimal subsidy computed using Algorithm 1 and (i) no subsidy (ε= 0). In this case, the optimal non-sequential policy is the same for all subsidies, and we apply a small vertical jitter to improve visibility. 62 H.2.3 Approval under pessimistic prior Here, we show the results of an antibiotic approval process where the agent’s prior is (α 0 ,β 0 ) = (1,1.5), that is, the agent is slightly pessimistic about its product. 25 Note that the mean efficacy of the drug according to its prior is then 0.4< θ ∗ = 0.65. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. In this case, we find that the agent opts out at the beginning of the approval process as long as the subsidy is ε≲ 0.1. The optimal subsidy is ε ∗ ≈ 0.4. 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0.0 2.5 5.0 7.5 10.0 12.5 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0 50 100 150 Agent utility, U A ( π ε ; ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 20: Agent utilities under a pessimistic prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0 100 200 300 400 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 0 500 1000 1500 Social utility, U S ( ε ; π ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 21: Social utilities under a pessimistic prior. The left panel shows the social utility (Eq. 13) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). In Figure 22 we show that the optimal subsidy increases withρ S /ρ A , and in Figure 23 that our sequential protocol yields gains > 40% in social utility relative to a non-sequential protocol. 25 We select β 0 = 1.5 as higher values lead the agent to opt out at the beginning of the process, regardless of the subsidy. 63 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 0.00 0.25 0.50 0.75 Optimal subsidy, ε ∗ Figure 22: Optimal subsidy vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 40 50 60 Social utility gain vs. non-sequential (%) Non-sequential (optimal subsidy) Figure 23: Social utility gain vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non- sequential approval protocol in which the agent is restricted to a single trial withn max = 800, under (i) the optimal subsidy computed using Algorithm 1 (in the non-sequential protocol without subsidy the agent always opts out, yielding zero social utility). 64 H.2.4 Approval under optimistic prior Here, we show the results of an antibiotic approval process where the agent’s prior is (α 0 ,β 0 ) = (4,1), that is, the agent is slightly optimistic about the antibiotic. Note that the mean efficacy of the drug according to its prior is then 0.8> θ ∗ = 0.65. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. In this case, the optimal subsidy isε ∗ = 0, and as can be seen in Figure 25, the optimal subsidy maximizing ̄ U S (ε;π ε ) does not necessarily maximize the true (unknown) utilityU S (ε;π ε ). Nevertheless, Figure 27 shows that the proposed sequential subsidized protocol can still substantially improve social utility relative to a non-sequential protocol, with gains exceeding approximately 18%. 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 150 160 170 180 190 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 120 140 160 180 200 Agent utility, U A ( π ε ; ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 24: Agent utilities under an optimist prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1670 1680 1690 1700 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1640 1660 1680 1700 1720 Social utility, U S ( ε ; π ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 25: Social utilities under an optimist prior. The left panel shows the social utility (Eq. 13) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 65 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 0.00 0.25 0.50 0.75 Optimal subsidy, ε ∗ Figure 26: Optimal subsidy vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 20 40 60 Social utility gain vs. non-sequential (%) Non-sequential (no subsidy) Non-sequential (optimal subsidy) Figure 27: Social utility gain vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non- sequential approval protocol in which the agent is restricted to a single trial withn max = 800, under (i) the optimal subsidy computed using Algorithm 1 and (i) no subsidy (ε = 0). 66 H.2.5 Approval under calibrated prior Here, we show the results of an antibiotic approval process where the agent’s prior is (α 0 ,β 0 ) = (130,70). This corresponds to a very informative prior that is calibrated to the true efficacyθ ∗ = 0.65, since the mean of the prior is precisely 0.65. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. In this case, we find that the calibrated prior allows the agent to increase its utilityU A (π ε ;ε) for any possible subsidy, as can be seen by comparing Figure 28 to the fiducial setting in Figure 9. However, and perhaps surprisingly, we also find that it is still optimal for the principal to subsidize a non-negligible fraction ε ∗ ≈ 0.234 of the agent’s cost, despite the prior belief supporting that the drug should be approved. 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 120 140 160 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 140 160 180 200 Agent utility, U A ( π ε ; ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 28: Agent utilities under a calibrated informative prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1565 1570 1575 1580 1585 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1820 1830 1840 Social utility, U S ( ε ; π ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 29: Social utilities under a calibrated informative prior. The left panel shows the social utility (Eq. 13) computed using the belief MDP M ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policy π ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionP where the agent’s optimal policy is constant (Proposition 8). 67 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 0.0 0.5 1.0 Optimal subsidy, ε ∗ Figure 30: Optimal subsidy vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1. Lastly, Figure 31 shows that the social utility gain relative to a non-sequential protocol with optimal subsidies is small for high values ofρ S /ρ A . This is expected because under a calibrated prior, the agent already has an accurate estimate of the product’s efficacy and therefore gathering new information brings little benefit. As a result, the agent can effectively select an optimal sample size and complete the process in a single trial. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 5 10 15 Social utility gain vs. non-sequential (%) Non-sequential (no subsidy) Non-sequential (optimal subsidy) Figure 31: Social utility gain vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non- sequential approval protocol in which the agent is restricted to a single trial withn max = 800, under (i) the optimal subsidy computed using Algorithm 1 and (i) no subsidy (ε = 0). 68 H.2.6 Approval under uncalibrated prior Here, we show the results of an antibiotic approval process where the agent’s prior is (α 0 ,β 0 ) = (130,30). This corresponds to a very informative and optimistic prior that is uncalibrated to the true efficacyθ ∗ = 0.65, since the mean of the prior is precisely 0.8125. The principal knows such prior, and the rest of the parameters in Table 2 and Table 3 are fixed. We find that, across subsidies, the utilityU A (π ε ;ε) achieved by the agent slightly decreased compared to an agent with a calibrated prior (Figure 32 vs. Figure 28). However, surprisingly, we also find that the social utility is greater in this case compared to the case where the agent has a calibrated prior (Figure 33 vs. Figure 29). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 190 200 210 220 230 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 120 140 160 180 200 Agent utility, U A ( π ε ; ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 32: Agent utilities under an uncalibrated informative prior. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1950 1960 1970 1980 1990 2000 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1800 1820 1840 1860 Social utility, U S ( ε ; π ε ) ( $ M) 0.0 0.2 0.4 0.6 0.8 1.0 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 33: Social utilities under an uncalibrated informative prior. The left panel shows the social utility (Eq. 13) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partition P where the agent’s optimal policy is constant (Proposition 8). Interestingly, Figure 34 shows that the optimal subsidy remains zero asρ S /ρ A increases. At the same time, Figure 35 indicates that our sequential protocol achieves gains exceeding 90% in social utility relative to a 69 non-sequential protocol. In other words, the protocol can substantially improve social utility even in the absence of subsidies. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A −0.02 −0.01 0.00 0.01 0.02 Optimal subsidy, ε ∗ Figure 34: Optimal subsidy vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 90 95 100 Social utility gain vs. non-sequential (%) Non-sequential (no subsidy) Non-sequential (optimal subsidy) Figure 35: Social utility gain vs.ρ S /ρ A . The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non- sequential approval protocol in which the agent is restricted to a single trial withn max = 800, under (i) the optimal subsidy computed using Algorithm 1 and (i) no subsidy (ε = 0). 70 H.2.7 Approval under a different test process In this section, we present additional experimental results for the antibiotic approval process described in Section 6, using an alternative test process to the processMdefined in Proposition 1 and Eq. 3, while keeping all other parameters fixed as in Tables 2 and 3. Here, our goal is to illustrate that Algorithm 1 can provide insight into how to optimally subsidize agents under different statistical tests. More concretely, following Appendix C, we consider the mixed processM mix defined in Eq. 30 with a uniform mixtureP mix = U(θ b ,1). We find that the sequential subsidized protocol yields social utility gains of up to 15% compared to a non-sequential baseline without subsidies. Relative to a non-sequential but optimally subsidized baseline, the gains can also reach up to 15% when the ratioρ S /ρ A is low, but diminish rapidly as this ratio increases. The initial optimal action taken by the agent isn≤114 for all subsidies. Lastly, in Figure 40, we confirm that the mixed process M mix has higher power than M for values of θ ∗ close to θ b , as discussed in Appendix C. 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 40 45 50 55 60 65 Agent’s utility, ̄ U A ( π ε ; ε ) ( $ M) (a) Agent utility ̄ U A (π ε ;ε) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 160 180 200 220 Agent utility, U A ( π ε ; ε ) ( $ M) 0.000 0.005 0.010 0.015 0.020 Probability of opting out (b) Agent utility U A (π ε ;ε) in the approval process Figure 36: Agent utilities under a mixed test process. The left panel shows the agent’s utility (Eq. 10) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy, which is a piece-wise linear, convex, and continuous function in accordance with Proposition 8. The right panel shows the true utility of the agent (Eq. 7) in the approval process when using the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 845 850 855 Social utility, ̄ U S ( ε ; π ε ) ( $ M) (a) Social utility ̄ U S (ε;π ε ) computed using M ε 0.00.10.20.30.40.50.60.70.80.9 Subsidy,ε 1920 1940 1960 Social utility, U S ( ε ; π ε ) ( $ M) 0.000 0.005 0.010 0.015 0.020 Probability of opting out (b) Social utility U S (ε;π ε ) in the approval process Figure 37: Social utilities under a mixed test process. The left panel shows the social utility (Eq. 13) computed using the belief MDPM ε when the agent uses the optimal policy for each subsidy. The right panel shows the true social utility (Eq. 7) in the approval process when the agent uses the optimal policyπ ε for each subsidy andθ ∗ = 0.65. The dashed vertical lines correspond to the intervals of the partitionPwhere the agent’s optimal policy is constant (Proposition 8). 71 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 0.0 0.3 0.6 0.9 Optimal subsidy, ε ∗ Figure 38: Optimal subsidy vs. ρ S /ρ A using a mixed test process. The figure shows, as a function of the social-to-agent approval benefit ratio, the optimal subsidy obtained using Algorithm 1. 0510152025303540 Social-to-agent approval benefit ratio,ρ S /ρ A 0 5 10 15 Social utility gain vs. non-sequential (%) Non-sequential (no subsidy) Non-sequential (optimal subsidy) Figure 39: Social utility gain vs.ρ S /ρ A using a mixed test process. The figure shows, as a function of the social-to-agent approval benefit ratio, the percentage increase in social utility of the sequential approval protocol relative to a non-sequential approval protocol in which the agent is restricted to a single trial with n max = 800, under (i) the optimal subsidy computed using Algorithm 1 and (i) no subsidy (ε = 0). 00.25 θ b 0.751 True efficacy,θ ∗ 0.0 0.2 0.4 0.6 0.8 1.0 Probability of approval Non-mixture Uniform mixture Figure 40: Probability of approval under the optimal policy and subsidy. The figure shows, across various efficaciesθ ∗ of the antibiotic, the probability of approval (that is, of rejectingH 0 ) when the principal selects the optimal subsidy and the agent its optimal policy. The dashed (orange) curve corresponds to the processM mix defined in Eq. 30 for a uniform mixture (optimal subsidyε ∗ = 0.027), while the solid (green) curve corresponds to the process M defined in Proposition 1 and Eq. 3 (optimal subsidy ε ∗ = 0.108). 72