Paper deep dive
To Defer or Not to Defer? Online Learning with LLMs and Congested Experts
Vasileios Saketos, William Reveillard, Alexandre Proutiere, Sebastian Dalleiger, Ming Xiao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern AI systems increasingly combine large language models (LLMs) with human experts, using the LLM to handle routine queries while deferring uncertain cases for expert review. When expert capacity is limited, however, deferral decisions must account not only for the reliability of the LLM but also for congestion in the expert queue. We formulate this as an online sequential decision-making problem in which, for each query, the system observes the confidence signal of the LLM's answer and must decide whether to output the answer or defer the query to a capacity-constrained expert. The reliability of the confidence signal is initially unknown and must be learned from expert feedback, which is both selectively observed (only deferred queries are labeled) and delayed by queuing. We propose \texttt{KL-PBD}, a learning algorithm that uses Kullback--Leibler lower confidence bounds on the unknown reliability parameters to guide deferral decisions. For $K$ confidence levels and horizon $T$, \texttt{KL-PBD} achieves $O(K\log T)$ gap-dependent regret when the accuracy at every confidence level is bounded away from the deferral boundary, and $\widetilde{O}(\sqrt{KT})$ worst-case regret. We establish matching lower bounds, showing that these rates are optimal up to logarithmic factors. Finally, we validate our approach on synthetic instances and real-world question-answering datasets, namely GSM8K, NuminaMath, and MedMCQA, using Qwen LLMs.
Tags
Links
- Source: https://arxiv.org/abs/2610.15128v1
- Canonical: https://arxiv.org/abs/2610.15128v1
Trouble viewing inline? Open PDF directly →
Full Text
254,969 characters extracted from source content.
Expand or collapse full text
To Defer or Not to Defer? Online Learning with LLMs and Congested Experts Vasileios Saketos William Réveillard11footnotemark: 1 Alexandre Proutière Sebastian Dalleiger Ming Xiao †thanks: Equal contribution. Affiliation: KTH Royal Institute of Technology Affiliation: Stockholm, Sweden Email: saketos,wilrev,alepro,sdall,mingx@kth.se Abstract Modern AI systems increasingly combine large language models (LLMs) with human experts, using the LLM to handle routine queries while deferring uncertain cases for expert review. When expert capacity is limited, however, deferral decisions must account not only for the reliability of the LLM but also for congestion in the expert queue. We formulate this as an online sequential decision-making problem in which, for each query, the system observes the confidence signal of the LLM’s answer and must decide whether to output the answer or defer the query to a capacity-constrained expert. The reliability of the confidence signal is initially unknown and must be learned from expert feedback, which is both selectively observed (only deferred queries are labeled) and delayed by queuing. We propose KL-PBD, a learning algorithm that uses Kullback–Leibler lower confidence bounds on the unknown reliability parameters to guide deferral decisions. For K confidence levels and horizon T, KL-PBD achieves O(KlogT)O(K T) gap-dependent regret when the accuracy at every confidence level is bounded away from the deferral boundary, and O~(KT) O( KT) worst-case regret. We establish matching lower bounds, showing that these rates are optimal up to logarithmic factors. Finally, we validate our approach on synthetic instances and real-world question-answering datasets, namely GSM8K, NuminaMath, and MedMCQA, using Qwen LLMs. 1 Introduction Large language models (LLMs) are increasingly deployed in settings where automated predictions can be complemented by human expertise, including customer support and healthcare (Huang et al., 2025; Feng et al., 2024; Yang et al., 2025b; Bhattacharya et al., 2025). A natural design is to let the LLM handle routine queries while referring uncertain or high-risk cases to a human expert. Expert attention, however, is limited: reviewing a query takes time, and queries sent for review may have to wait behind previously deferred requests. Consequently, a deferral decision must account not only for whether the LLM is likely to be correct, but also for the congestion created by sending another query to the expert queue. Determining when the LLM is reliable is itself challenging: the correctness of a particular answer is not known unless a ground-truth label is obtained, which in our setting requires expert review. LLMs nevertheless provide signals that may be informative about their reliability: a prominent example is self-consistency (Wang et al., 2023), which measures agreement across multiple generated answers. Such signals can be correlated with correctness, but high confidence or agreement does not guarantee that the answer is correct (Aggarwal et al., 2023; Wang et al., 2024). Thus, a deferral system must learn how the observed confidence signal relates to the probability that the LLM’s answer is correct. We are interested in doing so online, without requiring a separate task-specific calibration dataset or retraining the LLM before deployment. Instead, the system learns from expert feedback generated during operation. This gives rise to a coupled learning-and-control problem with an unusual feedback structure. If the system accepts an LLM response, it observes no ground-truth label. If it defers, the label is revealed only after the query has been served by the expert. Feedback is therefore both selective and delayed. Moreover, acquiring information is itself costly: every deferred query consumes expert capacity, enters the queue, and may increase congestion for future queries. Deferral is therefore simultaneously a prediction decision, an information-acquisition decision, and a queue-control decision. We formalize this interaction as a finite-horizon online sequential decision problem. At each round, a query arrives together with the LLM’s answer and a confidence level, and the system decides whether to accept the answer or defer the query to a capacity-constrained expert queue. The conditional accuracy associated with each confidence level is initially unknown and must be learned from completed expert reviews. Our goal is to design an online policy that minimizes the expected cumulative cost over the horizon, balancing the risk of incorrect LLM answers, the congestion created by expert review, and the value of deferrals as a source of information for future decisions. Our contributions are as follows. (a) We propose KL-PBD, a queue-aware algorithm that learns the unknown conditional accuracies from delayed expert feedback. The algorithm acts optimally for a pessimistic estimate of the LLM’s reliability, favoring deferral, and hence information acquisition, when the available evidence is still limited, while explicitly accounting for the congestion cost of doing so. (b) We establish near-optimal regret guarantees for KL-PBD. For K confidence levels, the algorithm achieves O(KlogT/γ)O(K T/γ) regret when the accuracy at every confidence level is at least γ away from the deferral boundary, and O~(KT) O( KT) regret in the worst case. (c) We derive complementary lower bounds showing that logarithmic regret and inverse-gap dependence are unavoidable in separated instances, while the worst-case KT KT dependence is optimal up to logarithmic factors. (d) We evaluate KL-PBD on synthetic and real-world question-answering tasks based on GSM8K, NuminaMath, and MedMCQA. We demonstrate the importance of pessimism under selective feedback and show that explicitly accounting for the expert queue is paramount when expert service is congested (see Fig. 1 (right) for an illustrative example). 2 Related Work Learning to defer. Learning to defer studies systems that either make an automated prediction or delegate the decision to a human expert (Madras et al., 2018; Mozannar and Sontag, 2020). These foundational formulations jointly learn a predictor and a deferral rule, with subsequent work developing calibrated deferral objectives (Verma and Nalisnick, 2022). Post-hoc approaches instead learn when to defer the predictions of a fixed model (Narasimhan et al., 2022). These approaches are formulated as offline supervised learning problems, where the deferral policy is learned from a fixed labeled dataset and evaluated through test-time prediction risk. In our problem, the LLM response rule is fixed, but the system learns a deferral policy online from delayed, selectively observed human feedback generated by its own deferral decisions. Moreover, expert intervention is capacity constrained, making the cost of deferral state dependent rather than fixed. Further supervised and multi-expert formulations are discussed in Appendix D.3. Online learning and selective feedback. Reid et al. (2024) study online deferral under a total consultation budget. Their bandit feedback model reveals the reward of the selected action, including automated predictions. Online selective classification is closer to our setting: labels are observed only when the learner abstains from answering, and regret guarantees trade off prediction errors against selective label acquisition (Gangrade et al., 2021; Baharav et al., 2026). In our setting, labels are similarly observed only on deferrals, but a deferred query first enters the expert queue, and its label becomes available only once it has been served. Broader active-learning and oracle-query settings are discussed in Appendix D.6. Deferral under limited review capacity. Alves et al. (2024) consider cost-sensitive deferral to multiple experts under individual workload constraints, optimizing assignments using offline estimates of expert error. More directly related, Lykouris and Weng (2025) learn a binary classifier online while controlling admission to human review and scheduling; as in our model, feedback is selective and becomes available only after a review is completed. In this work, by contrast, the LLM response rule is fixed and the order of service is not controlled by the learner; what is learned is the LLM’s accuracy at each self-confidence level, which determines the deferral policy. Their benchmark allocates review capacity using expected arrival and service rates and assumes admitted jobs are reviewed immediately. We instead compare with the optimal finite-horizon policy subject to the same queue dynamics as the learner, with expert-review and holding costs included in the objective. Additional related work on uncertainty signals and probabilistic calibration, LLM routing, structured admission-control policies, and human–AI decision support is discussed in Appendix D. 3 Problem formulation and optimal deferral 3.1 LLM–expert system model Figure 1: Left: expert-aided LLM system with deferral policy π and delayed feedback from completed expert reviews. Right: regret after T=5×104T=5× 10^4 queries on MedMCQA with Qwen3-4B against the expert service rate μ, comparing the queue-aware KL-PBD with the queue-unaware KL-UCB-4P. For μ=1μ=1, deferred queries are served before the next round and the queue does not build up, so queue awareness provides no benefit. For μ<1μ<1, deferred queries accumulate and create congestion, making queue-aware deferral essential for low regret. Details in Appendix H.3. We consider a sequential deferral problem over a known horizon T. At each round t=1,…,Tt=1,…,T, a user query XtX_t may arrive. Upon receiving a query, the LLM produces a candidate answer RtR_t together with an index Jt∈[K]:=1,…,KJ_t∈[K]:=\1,…,K\, which we call its self-confidence level. Our model is agnostic to how (Rt,Jt)(R_t,J_t) is generated. A canonical example is self-consistency (Wang et al., 2023): the LLM generates K−1K-1 answers, RtR_t is chosen from the largest answer-equivalence class, and JtJ_t is one plus the size of that class, so that Jt=1J_t=1 when no answer can be extracted. Alternative constructions of (Rt,Jt)(R_t,J_t) are discussed in Appendix D.1. Let Rt⋆R_t denote the ground-truth answer and define Yt:=Rt≡Rt⋆Y_t:=1\R_t≡ R_t \, where R1≡R2R_1≡ R_2 means that the two answers are equivalent. We set Jt=0J_t=0, Xt=Rt=∅X_t=R_t= , Yt=1Y_t=1 when no query arrives and assume that (Jt,Yt)t≥1(J_t,Y_t)_t≥ 1 is an i.i.d. sequence. For j∈[K]∪0j∈[K]∪\0\, let ωj:=ℙ(Jt=j) _j:=P(J_t=j), and for j∈[K]j∈[K], define11 1 If ωj=0 _j=0, hj⋆h_j may be assigned arbitrarily. hj⋆:=ℙ(Yt=1∣Jt=j)h_j :=P(Y_t=1 J_t=j). Thus, hj⋆h_j is the LLM’s conditional accuracy at self-confidence level j, and we call h⋆=(h1⋆,…,hK⋆)h =(h_1 ,…,h_K ) the conditional accuracy vector. When a query arrives, the decision maker, who observes the self-confidence level JtJ_t but neither XtX_t nor RtR_t, either accepts the LLM answer or defers the query to a queue served by a pool of experts, who always provide the correct answer. We encode this decision by At∈0,1A_t∈\0,1\, where At=1A_t=1 denotes deferral. We set At=0A_t=0 if Jt=0J_t=0. Let BtB_t be the number of queries in the queue at the beginning of round t, and let CtC_t be the number of queries that can be served by the experts between rounds t and t+1t+1. We assume that (Ct)t≥1(C_t)_t≥ 1 is i.i.d. with distribution ρC _C and independent of (Jt,Yt)t≥1(J_t,Y_t)_t≥ 1 and the algorithm’s internal randomization, and that μ:=ℙ(Ct≥1)>0μ:=P(C_t≥ 1)>0. The order in which pending queries are reviewed may depend on the observations and the service capacity, but not on unrevealed ground-truth labels. The queue is initially empty, B1=0B_1=0, and has known capacity B¯≥1 B≥ 1, so deferral is infeasible when Bt=B¯B_t= B. Its dynamics are Bt+1=max0,Bt+At−CtB_t+1= \0,B_t+A_t-C_t\. The cost incurred at round t is ct:=choldBt+cexpAt+ce(1−At)(1−Yt)c_t:=c_holdB_t+c_expA_t+c_e(1-A_t)(1-Y_t), where chold≥0c_hold≥ 0 is the per-round holding cost of each query in the queue, cexp≥0c_exp≥ 0 is the cost of deferring a query to the experts, and ce>0c_e>0 is the cost of returning an incorrect LLM answer. At the end of the horizon, we account for the cost of serving the queries remaining in the queue. These cost parameters are known to the decision maker. Let D(b)D(b) denote the expected holding cost of clearing a queue of size b. It satisfies D(0)=0D(0)=0 and, for b≥1b≥ 1, D(b)=choldb+C∼ρC[D((b−C)+)]D(b)=c_ holdb+E_C _C\! [D ((b-C)^+ ) ]. The terminal cost is therefore D(BT+1)D(B_T+1). For example, if C∼Bernoulli(μ)C (μ), then D(b)=choldb(b+1)/(2μ)D(b)=c_ holdb(b+1)/(2μ). The resulting problem is a finite-horizon Markov decision process (MDP) with state (Bt,Jt)(B_t,J_t), action AtA_t, per-round cost ctc_t, and terminal cost D(BT+1)D(B_T+1). A policy π=(πt)t=1Tπ=( _t)_t=1^T is a sequence of possibly randomized decision rules, where πt _t maps the current state (Bt,Jt)(B_t,J_t) to a distribution over feasible actions. Our goal is to learn, online, a policy that performs nearly as well as the optimal policy that knows the conditional accuracy vector h⋆h . We formalize this objective through regret after first characterizing the optimal policy under known h⋆h . 3.2 Optimal deferral policy Fix h∈[0,1]Kh∈[0,1]^K and set h0:=1h_0:=1. For j∈[K]∪0j∈[K]∪\0\, define Lh(j):=ce(1−hj)L^h(j):=c_e(1-h_j), which, for j∈[K]j∈[K], is the expected error cost of accepting an LLM response with self-confidence level j. Let VthV_t^h and QthQ_t^h denote the finite-horizon value and state-action value functions of the MDP induced by h, with terminal value VT+1h((b,j))=D(b)V_T+1^h((b,j))=D(b). In other words, Qth((b,j),a)Q_t^h((b,j),a) is the expected cost-to-go from state (b,j)(b,j) when action a is taken at round t, and Vth((b,j))V_t^h((b,j)) is its minimum over feasible actions. To simplify the notation, let Pa(b,b′)P_a(b,b ) denote the transition probability from queue state b to b′b under action a∈0,1a∈\0,1\. Since the queue dynamics depend on the current queue state and action, but not on the current self-confidence level, define, for b<B¯b< B, Mth(b):=cexp+∑b′=0B¯(P1(b,b′)−P0(b,b′))∑j=0KωjVt+1h((b′,j)).M_t^h(b):=c_ exp+ _b =0 B (P_1(b,b )-P_0(b,b ) ) _j=0^K _jV_t+1^h((b ,j)). (1) We refer to Mth(b)M_t^h(b) as the marginal cost of deferral: it captures both the immediate expert-review cost and the expected increase in future cost caused by adding a query to the expert queue. We adopt the convention Mth(B¯):=+∞M_t^h( B):=+∞. For any b<B¯b< B and j∈[K]j∈[K], the difference between the costs of deferring and accepting is Qth((b,j),1)−Qth((b,j),0)=Mth(b)−Lh(j).Q_t^h((b,j),1)-Q_t^h((b,j),0)=M_t^h(b)-L^h(j). Hence, the optimal decision compares the expected error cost of the LLM response with the marginal cost of deferral. The optimal policy has a threshold structure: a query is deferred whenever the LLM’s conditional accuracy is below a threshold determined by the current time and expert backlog. As stated in the following lemma, the threshold is endogenous: through Mth(b)M_t^h(b), it accounts for the future congestion created by the current deferral decision. Lemma 1 (Optimal deferral policy). Fix h∈[0,1]Kh∈[0,1]^K. Resolving ties in favor of acceptance, an optimal action at round t is Ath=Jt∈[K],Lh(Jt)>Mth(Bt)A_t^h=1\! \J_t∈[K],\;L^h(J_t)>M_t^h(B_t) \. Equivalently, defining the accuracy thresholds τth(b):=1−Mth(b)/ce _t^h(b):=1-M_t^h(b)/c_e, for all b≤B¯b≤ B, the optimal policy defers if and only if hJt<τth(Bt)h_J_t< _t^h(B_t). In particular, τth(B¯)=−∞ _t^h( B)=-∞, so deferral is infeasible when the queue is full. 3.3 Learning problem and regret decomposition We now turn to the setting in which the true conditional accuracy vector h⋆h is unknown. The statistical challenge is to learn h⋆h while making deferral decisions. Feedback is both selective and delayed. If the learner accepts the LLM response (At=0A_t=0), its ground-truth label YtY_t is not revealed, and hence the corresponding error cost is not observed. If the learner defers (At=1A_t=1), the ground-truth label becomes available only once the query is processed by an expert. Thus, information about h⋆h can be acquired only through the capacity-constrained experts, and the resulting feedback is delayed by the same queue that governs the deferral decisions. In particular, exploration affects not only the information available to the learner but also the future state of the system through congestion. To isolate this learning challenge, our main analysis assumes that the distributions of the confidence levels ω and of the service capacity ρC _C are known. This assumption does not require analogous active exploration: the self-confidence level JtJ_t is observed whenever a query arrives, so ω can be estimated passively from arrival frequencies, while observed service capacities provide samples from ρC _C independently of the deferral decisions. In Appendix B.6, we extend our analysis to the setting where ω and ρC _C are unknown and show that our worst-case regret guarantees are preserved. A learning algorithm π is analogous to a policy, except that since it learns on the way, the decision taken at time t may depend on all observations made until time t. The expected regret of an algorithm π is defined by: RT(π)=π[∑t=1Tct+D(BT+1)]−π⋆[∑t=1Tct+D(BT+1)].R_T(π)=E^π\! [ _t=1^Tc_t+D(B_T+1) ]-E^π \! [ _t=1^Tc_t+D(B_T+1) ]. where π⋆π is the optimal deferral policy for the conditional accuracy vector h⋆h , identified in Lemma 1. Note that the oracle, applying π⋆π , and the learner use the same K self-confidence levels; hence, regret measures only the additional cost incurred from learning the unknown conditional accuracy vector h⋆h , rather than any approximation error induced by the choice of confidence discretization. For brevity, a superscript ⋆ on a quantity denotes evaluation at h=h⋆h=h (e.g. L⋆=Lh⋆L =L^h ). Define the instantaneous decision gap Δt:=|L⋆(Jt)−Mt⋆(Bt)|At≠At⋆ _t:= |L (J_t)-M_t (B_t) |1\A_t≠ A_t \ when Jt∈[K]J_t∈[K] and Bt<B¯B_t< B, and set Δt:=0 _t:=0 otherwise. Lemma 2 (Regret decomposition). For any learning algorithm π, RT(π)=π[∑t=1TΔt].R_T(π)=E^π [ _t=1^T _t ]. Lemma 2 shows that regret is the expected cumulative decision gap over the rounds in which the learner takes a strictly suboptimal action. Consequently, the learning problem reduces to estimating h⋆h accurately enough to identify on which side of the time- and backlog-dependent oracle threshold τt⋆(Bt) _t (B_t) each self-confidence level lies. 4 The KL-PBD algorithm and its regret 4.1 Pessimistic Bellman deferral We now introduce our algorithm KL-PBD, which learns the unknown conditional accuracy vector h⋆h from delayed expert feedback while accounting for the congestion induced by deferral. KL-PBD is summarized in Algorithm 1. For each self-confidence level j∈[K]j∈[K], let Nj,tN_j,t denote the number of level-j queries whose expert reviews have been completed before the decision at round t. Importantly, Nj,tN_j,t need not equal the number of level-j queries deferred before t: a deferred query contributes to Nj,tN_j,t only once it is served and its ground-truth label becomes available. The selective and delayed nature of the feedback thus enters the algorithm through Nj,tN_j,t. Let Yj,nY_j,n denote the label revealed by the nnth completed level-j review, and, for n≥1n≥ 1, define h^j,n:=1n∑ℓ=1nYj,ℓ. h_j,n:= 1n _ =1^nY_j, . For T≥3T≥ 3, let ϕ(T):=logT+loglogTφ(T):= T+ T denote the confidence budget. Given Nj,t≥1N_j,t≥ 1 observed labels, we form the KL lower confidence bound h¯j,t:=infq∈[0,h^j,Nj,t]:Nj,tkl(h^j,Nj,t,q)≤ϕ(T) h_j,t:= \q∈[0, h_j,N_j,t]:N_j,t\,kl\! ( h_j,N_j,t,q )≤φ(T) \ where klkl denotes the Bernoulli relative entropy, and set h¯j,t=0 h_j,t=0 when Nj,t=0N_j,t=0. At round t, KL-PBD treats h¯t h_t as the current conditional accuracy vector and applies the corresponding optimal policy (Algorithm 1, lines 5–6). Equivalently, the learner defers whenever the pessimistic expected error cost of the LLM exceeds the Bellman marginal cost of expert review. The use of a lower confidence bound serves both learning and control. Because correctness is observed only for deferred queries, accepting a query provides no information about h⋆h . A plug-in estimate can therefore become prematurely optimistic about a self-confidence level, causing the learner to accept subsequent queries from that level and thereby suppress the feedback needed to correct the estimate (see Appendix H.5). Algorithm 1 KL-PBD 1: Set h¯j,1=0 h_j,1=0 for every j∈[K]j∈[K]. 2: for t=1,…,Tt=1,…,T do 3: Incorporate all expert labels received before round t and update Nj,tN_j,t, h^j,Nj,t h_j,N_j,t, and h¯j,t h_j,t. 4: Observe (Bt,Jt)(B_t,J_t). 5: Compute τ^t(Bt)=1−Mth¯t(Bt)/ce τ_t(B_t)=1-M_t h_t(B_t)/c_e. 6: Choose At=Jt∈[K],h¯Jt,t<τ^t(Bt).A_t=1\! \J_t∈[K],\; h_J_t,t< τ_t(B_t) \. 7: end for In contrast, KL-PBD treats uncertain levels pessimistically: before sufficient expert feedback has accumulated, a smaller h¯j,t h_j,t makes deferral more attractive and hence promotes the collection of additional labels. This exploration is nevertheless congestion-aware, since the decision is made through the threshold τ^t(Bt) τ_t(B_t) and therefore accounts for the current queue and the future cost of expert review. 4.2 Regret guarantees We first give a gap-dependent regret guarantee. Recall that the oracle defers a level-j query at state (t,b)(t,b) when hj⋆<τt⋆(b)=1−Mt⋆(b)/ceh_j < _t (b)=1-M_t (b)/c_e. The distance |hj⋆−τt⋆(b)| |h_j - _t (b) | therefore measures how far the conditional accuracy of level j is from the oracle threshold at time t and queue length b. Equivalently, for b<B¯b< B, |L⋆(j)−Mt⋆(b)|=ce|hj⋆−τt⋆(b)| |L (j)-M_t (b) |=c_e |h_j - _t (b) | is the difference between the oracle Q-values of accepting and deferring. Thus, the farther hj⋆h_j is from the threshold, the less accurately it needs to be estimated to identify the oracle action. Importantly, states at which hj⋆=τt⋆(b)h_j = _t (b) should not be treated as having a problematic “zero gap.” At such a state, L⋆(j)=Mt⋆(b)L (j)=M_t (b), so accepting and deferring have the same oracle Q-value. Choosing either action therefore incurs no regret. Accordingly, the relevant notion of separation is the smallest nonzero distance from an oracle threshold. To formalize this, define the set of nonzero oracle gaps over the horizon by T:=|hj⋆−τt⋆(b)|:j∈[K],t≤T,b<B¯,hj⋆≠τt⋆(b).G_T:= \ |h_j - _t (b) |:j∈[K],\;t≤ T,\;b< B,\;h_j ≠ _t (b) \. If T=∅G_T= , every feasible oracle comparison is a tie. Hence the decision gap in Lemma 2 is zero at every round, and every algorithm has zero regret. We therefore restrict attention to the nontrivial case T≠∅G_T≠ and define γT:=min1,minT _T:= \1, _T\. This quantity captures the smallest separation that the learner must resolve in order to distinguish a strict oracle preference between accepting and deferring. When this separation is bounded away from zero, sufficiently accurate estimates of h⋆h identify the oracle action. Theorem 1 (Gap-dependent regret of KL-PBD). Suppose that T≠∅G_T≠ . For every T≥3T≥ 3, RT(πKL-PBD)≤Kκϕ(T)+2ce∑j=1KminωjT,B¯+4ϕ(T)γT+2B¯ce[1+4∑j=1Klog(1+ωjTϕ(T))]R_T(π KL-PBD)≤ Kκφ(T)+2c_e _j=1^K \ _jT, B+ 4φ(T) _T \+2 Bc_e [1+4 _j=1^K (1+ _jTφ(T) ) ] where κ:=4e(maxce,cexp+choldB¯/μ)κ:=4e( \c_e,c_exp\+c_hold B/μ). In particular, if γT≥γ>0 _T≥γ>0 uniformly in T, then RT(πKL-PBD)=O(KlogTγ)R_T(π KL-PBD)=O( K Tγ). Thus, when the conditional accuracies remain separated from the oracle thresholds, KL-PBD incurs only logarithmic regret. A stronger version of Theorem 1 is proven in Appendix B.4. We next give a gap-free guarantee, which holds without any separation from the oracle thresholds. A stronger version of Theorem 2 is proven in Appendix B.3. Theorem 2 (Worst-case regret of KL-PBD). For every T≥3T≥ 3, RT(πKL-PBD)≤Kκϕ(T)+ce∑j=1KminωjT,B¯+22ωjTϕ(T)+B¯ce[1+2∑j=1Klog(1+ωjTϕ(T))].R_T(π KL-PBD)≤ Kκφ(T)+c_e _j=1^K \ _jT, B+2 2 _jTφ(T) \+ Bc_e [1+2 _j=1^K (1+ _jTφ(T) ) ]. Up to logarithmic factors, Theorem 2 gives RT(πKL-PBD)=O~(K+∑j=1KminωjT,ωjT)R_T(π KL-PBD)= O(K+ _j=1^K \ _jT, _jT\). In particular, the worst case over arrival distributions yields O~(KT) O( KT) for K≤TK≤ T (Remark 1). The main difficulty in the regret analysis is that h¯t h_t also enters the marginal cost of deferral, so pessimism may cause KL-PBD not only to defer queries that the oracle accepts, but also to accept queries that the oracle defers. For a high-level summary of our proof technique, see Appendix B.1. 5 Lower bounds We now show that the dependence on K, T, and the gaps in Theorems 1 and 2 is optimal in general (up to logarithmic factors for worst-case regret) by deriving matching lower bounds, which are proven in Appendix C. Write RT(π,h)R_T(π;h) for the regret when the conditional accuracy vector is h, and define τ:=1−[cexp+(1−μ)chold/μ]/ceτ:=1-[c_ exp+(1-μ)c_ hold/μ]/c_e. This is the largest accuracy threshold the oracle ever uses: τth(b)≤τ _t^h(b)≤τ for all h, t and b, with equality at the empty queue in the last round (Lemma 18). We first give an instance-dependent lower bound. We call an algorithm π uniformly efficient if, run with horizon T, it satisfies RT(π,h)=o(Tα)R_T(π;h)=o(T^α) for every h∈[0,1]Kh∈[0,1]^K and every α>0α>0. Let τ¯⋆:=limT−t→∞τt⋆(0) τ := _T-t→∞ _t (0) be the long-horizon oracle threshold at the empty queue, which exists by Lemma 18. For every t and b, τt⋆(b)≤τt⋆(0) _t (b)≤ _t (0) (Proposition 1) and τ¯⋆≤τt⋆(0)≤τ τ ≤ _t (0)≤τ (Lemma 18). Thus, τ¯⋆ τ is the long-horizon limit of the largest threshold the oracle uses. Theorem 3 (Instance-dependent lower bound). If τ¯⋆∈(0,1) τ ∈(0,1), every uniformly efficient algorithm π satisfies lim infT→∞RT(π,h⋆)logT≥ce∑j∈[K]:hj⋆>τ¯⋆,ωj>0hj⋆−τ¯⋆kl(hj⋆,τ¯⋆). _T→∞ R_T(π;h ) T≥ c_e _ subarraycj∈[K]:\\ h_j > τ ,\ _j>0 subarray h_j - τ kl(h_j , τ ). (2) As the positive gaps hj⋆−τ¯⋆h_j - τ tend to zero, with τ¯⋆ τ bounded away from 00 and 11, each summand in the lower bound scales as 2τ¯⋆(1−τ¯⋆)/(hj⋆−τ¯⋆)2 τ (1- τ )/(h_j - τ ). Hence, the logarithmic growth in T and inverse-gap dependence of Theorem 1 are necessary in general. When μ=1μ=1, τ¯⋆=τ τ =τ and KL-PBD attains equation 2 with equality (Proposition 3). Additionally, the stronger version of Theorem 1 proved in Appendix B.4 matches the dependence of Theorem 3 on the conditional accuracies for small gaps. Our proof relies on a change-of-measure argument, as in Lai and Robbins (1985). The main difficulty in our setting is that the marginal cost of deferral depends on the round and the queue state, and that the oracle threshold changes with the instance. For a high-level summary of our proof technique, see Appendix C.1. We next give a minimax lower bound. Theorem 4 (Minimax lower bound). Assume τ∈(0,1)τ∈(0,1). For every T≥1T≥ 1, infπsuph∈[0,1]KRT(π,h)≥1256∑j=1Kminμ3ceminτ,1−τminωjT,ωjT,μ2choldωjT. _π _h∈[0,1]^KR_T(π;h)≥ 1256 _j=1^K \! \μ^3c_e \τ,1-τ\ \ _jT, _jT\,\ μ^2c_ hold\, _jT \. (3) When μ=1μ=1, the last term can be dropped and the constant 1/2561/256 improved to 1/121/12. If chold>0c_ hold>0 or μ=1μ=1, the lower bound scales as ∑j=1KminωjT,ωjT _j=1^K \ _jT, _jT\. Thus, its dependence on T and ω agrees with Theorem 2 up to logarithmic factors and the upper bound’s additive K term. The lower bound is largest for balanced levels, ω0=0 _0=0 and ωj=1/K _j=1/K (Remark 3), giving minT,KT \T, KT\ and matching the upper bound up to logarithmic factors for K≤TK≤ T. The main difficulty in the minimax lower-bound construction is that hard instances cannot be obtained by perturbing the conditional accuracies around a fixed threshold, since the threshold itself depends on them, and that nearby instances are only guaranteed to have different optimal actions when the queue is empty. For a high-level summary of our proof technique, see Appendix C.1. 6 Numerical experiments We now compare KL-PBD with baselines and examine the structure of the oracle policy on instances built from LLM outputs on GSM8K, NuminaMath-TIR and MedMCQA. Additional results and details are deferred to Appendices G and H. Instances. We use two LLMs, Qwen3-4B-Instruct-2507 and Qwen3-0.6B, and build one instance per dataset–LLM pair; results for Qwen2.5-0.5B-Instruct are reported in Appendix H. Each instance uses the self-consistency construction of Section 3: for every question, the LLM generates 2020 answers, so there are K=21K=21 levels. For a query at level j, let pjp_j denote its self-consistency score, i.e., the fraction of generated answers that agree with RtR_t, so that p=(p1,…,pK)=(0,.05,…,1)p=(p_1,…,p_K)=(0,.05,…,1), where p1=0p_1=0 corresponds to queries for which no answer could be extracted. We take ωj _j and hj⋆h_j to be the empirical frequency and conditional accuracy of self-consistency level j over the whole dataset, and sample queries independently with replacement from that dataset, so that a query arrives at every round (ω0=0 _0=0). Regret is evaluated against the optimal policy for this empirical instance. Figure 2 shows three calibration regimes. On GSM8K and NuminaMath the score is informative: Qwen3-0.6B is especially close to calibrated on GSM8K, while Qwen3-4B is underconfident at low scores and overconfident at high scores. On MedMCQA, the score is nearly uninformative: accuracy barely varies with the score below 100%100\%, and the 100%100\% level has accuracy only .79.79 (Qwen3-4B) and .68.68 (Qwen3-0.6B). Figure 2: Conditional accuracy of the self-consistency score (dashed line: perfect calibration). Setup and baselines. We set (ce,cexp,chold)=(1,0.25,0.03)(c_e,c_ exp,c_ hold)=(1,0.25,0.03), B¯=8 B=8, Ct∼Bernoulli(0.35)C_t (0.35), and T=105T=10^5. The arrival distribution ω is unknown to the algorithms and is estimated as ω^j,t=t−1∑s=1tJs=j ω_j,t=t^-1 _s=1^t1\J_s=j\ by all methods that require it, except FQT, which is tuned under a uniform ω. We compare KL-PBD with: i) SALMUT-SF, a threshold-based actor–critic policy adapted from Roy et al. (2021) to handle unknown rewards and selective feedback; i) Fixed Queue Threshold (FQT), a non-learning baseline that uses a queue threshold optimized under the surrogate h=ph=p; i) variants of KL-PBD, namely KL-PBD+, which uses a more aggressive confidence budget inspired by KL-UCB+ type rules (Garivier and Cappé, 2011; Achab et al., 2018), Greedy Bellman Deferral (GBD), which replaces the lower confidence bound h¯t h_t by the empirical mean h^j,Nj,t h_j,N_j,t, and Naive Bellman Deferral (NBD), which freezes the estimate at h=ph=p, i.e., treats the score as calibrated. These baselines are described in more detail in Appendix F. Regret of KL-PBD against baselines. In Figure 3, we report regret averaged over 100100 independent runs with pointwise 95%95\% confidence bands. Solid lines are algorithms that learn h⋆h , and dashed lines are the baselines using the h=ph=p surrogate. The results illustrate the impact of score calibration: on GSM8K and NuminaMath, NBD is generally the best non-learning baseline and competes with KL-PBD over this horizon, whereas on MedMCQA it incurs 2020 to 6565 times the regret of KL-PBD+. Among learning algorithms, GBD suffers from the selective-feedback failure described in Appendix H.5, and SALMUT-SF adapts too slowly to be competitive. KL-PBD+ attains the lowest regret in all six instances, up to a factor 44 below KL-PBD: its more aggressive level-dependent confidence budget reduces the conservatism of the deferral rule. Appendix H reports regret values at horizons T∈104,5⋅104,105,2⋅105T∈\10^4,5· 10^4,10^5,2· 10^5\; the gap between KL-PBD+ and the h=ph=p baselines keeps widening with T. Comparisons to queue-unaware baselines, including adaptations of bandit algorithms from Achab et al. (2018), are deferred to Appendix H.3: KL-PBD and KL-PBD+ generally beat them by a wide margin when μ<1μ<1, and remain competitive under immediate service (μ=1μ=1). Figure 3: Regret averaged over 100 runs. Curves leaving the frame are listed with their final regret. Structure of the oracle policy. Figure 4 shows the oracle threshold and the resulting decisions for Qwen3-0.6B. For every μ, the threshold decreases with the queue state, as established in Proposition 1. Higher values of μ lead to higher deferral thresholds, as faster expert service reduces the congestion cost of deferral: the oracle then defers even queries on which the LLM is fairly accurate. The bottom row shows the oracle deferral decisions for the service rate μ=0.35μ=0.35 used in the regret experiments. Accordingly, the oracle defers fewer levels as the queue becomes more congested. In contrast, it does not always defer fewer levels as self-consistency increases: since the oracle defers exactly the levels whose conditional accuracy lies below the threshold, this occurs because the conditional accuracy is non-monotone and the threshold falls between the accuracies of two levels whose order is reversed relative to their self-consistency. Similar plots for Qwen3-4B and comparisons with the thresholds and decisions learned by KL-PBD are deferred to Appendix H.4. Figure 4: Oracle thresholds for varying service rate μ (top) and oracle decisions for μ=0.35μ=0.35 (bottom), for Qwen3-0.6B at round t=T/2t=T/2. 7 Conclusion We studied online deferral to a congested expert, where deferral costs depend on the queue state and expert feedback is both selective and delayed. This creates a feedback loop: learning when to trust the LLM requires expert labels, but acquiring those labels consumes the same limited resource that the learner is trying to manage. KL-PBD addresses this through pessimistic estimates of the unknown conditional accuracies, preventing premature acceptance from cutting off the feedback needed for learning. Its regret guarantees are near-optimal in both the worst-case and gap-dependent regimes. Several extensions are natural. First, expert effort could depend on the query: low-confidence queries may also be harder to review, making service times confidence-level dependent. Second, our benchmark fixes a discretization of the confidence score; for continuous scores especially, choosing the number of levels would trade approximation error against statistical complexity. Third, the number of LLM generations could itself be treated as a design variable, trading the quality of the confidence signal against inference cost. AI use statement In this work, we used generative AI tools to assist with: i) writing and editing, and in particular, to improve the presentation of the proofs; i) coding, to help implement our algorithms and write data-processing and parsing scripts; i) literature search, to help identify potentially relevant work. The coding assistance was used strictly to implement our ideas and methods. We conducted our own literature review, and AI was used as a complementary tool for discovering potentially relevant references that we might have missed, not for synthesizing the literature on our behalf. We checked and revised all AI-assisted text and code, and independently verified all AI-suggested references and literature-related claims against the original sources. We take full responsibility for the final content of this work. Reproducibility statement We provide a complete description of our problem setting and assumptions in Section 3, and of our algorithm in Section 4. Full proofs of our results are given in Appendices A–C, with accompanying proof sketches to help the reader navigate them. Our experimental setup is described in Section 6 and Appendices E–H, including the datasets used and how they were parsed, the baselines we compare our algorithm to, and our evaluation protocol. The code used to reproduce our experiments is included in the supplementary material. References Achab et al. (2018) M. Achab, S. Clémençon, and A. Garivier Profitable bandits. In Proceedings of The 10th Asian Conference on Machine Learning, J. Zhu and I. Takeuchi (Eds.), Proceedings of Machine Learning Research, Vol. 95, p. 694–709. External Links: Link Cited by: 2nd item, Appendix F, §6, §6. Aggarwal et al. (2023) P. Aggarwal, A. Madaan, Y. Yang, and Mausam Let’s sample step by step: adaptive-consistency for efficient reasoning and coding with LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 12375–12396. External Links: Link, Document Cited by: §1. Alves et al. (2024) J. V. Alves, D. Leitão, S. Jesus, M. O. P. Sampaio, J. Liébana, P. Saleiro, M. A. T. Figueiredo, and P. Bizarro Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints. Transactions on Machine Learning Research. External Links: Link Cited by: §2. Baharav et al. (2026) T. Z. Baharav, S. Dragazis, and A. Pacchiano The good, the bad, and the sampled: a no-regret approach to safe online classification. International Conference on Artificial Intelligence and Statistics. Cited by: §2. Benz et al. (2026) N. C. Benz, E. Straitouri, and M. Gomez-Rodriguez Learning to decide with AI assistance under human-alignment. arXiv preprint arXiv:2605.12646. Cited by: §D.7. Bhattacharya et al. (2025) A. Bhattacharya, S. Stumpf, R. De Croon, and K. Verbert Explanatory debiasing: involving domain experts in the data generation process to mitigate representation bias in AI systems. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA. External Links: ISBN 9798400713941, Link, Document Cited by: §1. Charusaie et al. (2022) M. Charusaie, H. Mozannar, D. Sontag, and S. Samadi Sample efficient learning of predictors that complement humans. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 2972–3005. External Links: Link Cited by: §D.3. Chen et al. (2024) L. Chen, M. Zaharia, and J. Zou FrugalGPT: how to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research. Note: Featured Certification External Links: ISSN 2835-8856, Link Cited by: §D.4. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. External Links: 2110.14168, Link Cited by: 1st item. Cortes et al. (2026) C. Cortes, A. Mao, M. Mohri, and Y. Zhong Optimized deferral for imbalanced settings. In Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), External Links: Link Cited by: §D.3. Corvelo Benz and Gomez Rodriguez (2025) N. L. Corvelo Benz and M. Gomez Rodriguez Human-alignment influences the utility of AI-assisted decision making. Scientific Reports 15 (1), p. 29154. External Links: ISSN 2045-2322, Document, Link Cited by: §D.7. Corvelo Benz and Rodriguez (2023) N. Corvelo Benz and M. Rodriguez Human-aligned calibration for AI-assisted decision making. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 14609–14636. External Links: Link Cited by: §D.7. Dasgupta (2011) S. Dasgupta Two faces of active learning. Theor. Comput. Sci. 412 (19), p. 1767–1781. External Links: ISSN 0304-3975, Link, Document Cited by: §D.6. Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. External Links: ISSN 1476-4687, Document, Link Cited by: §D.1. Feng et al. (2024) X. Feng, Z. Chen, Y. Qin, Y. Lin, X. Chen, Z. Liu, and J. Wen Large language model-based human-agent collaboration for complex task solving. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 1336–1357. Cited by: §1. Gangrade et al. (2021) A. Gangrade, A. Kag, A. Cutkosky, and V. Saligrama Online selective classification with limited feedback. Advances in Neural Information Processing Systems 34, p. 14529–14541. Cited by: §2. Garivier and Cappé (2011) A. Garivier and O. Cappé The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th Annual Conference on Learning Theory, S. M. Kakade and U. von Luxburg (Eds.), Proceedings of Machine Learning Research, Vol. 19, Budapest, Hungary, p. 359–376. External Links: Link Cited by: 2nd item, §6. Garivier (2013) A. Garivier Informational confidence bounds for self-normalized averages and applications. In 2013 IEEE Information Theory Workshop (ITW), Vol. , p. 1–5. External Links: Document Cited by: §B.5.1. Gupta et al. (2024) N. Gupta, H. Narasimhan, W. Jitkrittum, A. S. Rawat, A. K. Menon, and S. Kumar Language model cascades: token-level uncertainty and beyond. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 4147–4180. External Links: Link Cited by: §D.1. Hanneke (2014) S. Hanneke Theory of disagreement-based active learning. Found. Trends Mach. Learn. 7 (2–3), p. 131–309. External Links: ISSN 1935-8237, Link, Document Cited by: §D.6. Hao et al. (2018) S. Hao, P. Hu, P. Zhao, S. C. H. Hoi, and C. Miao Online active learning with expert advice. ACM Trans. Knowl. Discov. Data 12 (5). External Links: ISSN 1556-4681, Link, Document Cited by: §D.6. Hemmer et al. (2022) P. Hemmer, S. Schellhammer, M. Vössing, J. Jakubik, and G. Satzger Forming effective human-AI teams: building machine learning models that complement the capabilities of multiple experts. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, L. D. Raedt (Ed.), p. 2478–2484. Note: Main Track External Links: Document, Link Cited by: §D.3. Hemmer et al. (2023) P. Hemmer, L. Thede, M. Vössing, J. Jakubik, and N. Kühl Learning to defer with limited expert predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 6002–6011. Cited by: §D.3. Huang et al. (2025) C. Huang, Y. Deng, W. Lei, J. Lv, T. Chua, and J. X. Huang How to enable effective cooperation between humans and NLP models: a survey of principles, formalizations, and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 466–488. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. Hugging Face (2025) Hugging Face Math-Verify. Note: https://github.com/huggingface/Math-Verify Cited by: §E.2. Jiang et al. (2026) C. Jiang, Y. Zhang, Y. Cai, C. Chan, Y. Liu, M. Chen, W. Xue, and Y. Guo Semantic voting: a self-evaluation-free approach for efficient LLM self-improvement on unverifiable open-ended tasks. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, p. 36536–36556. External Links: Link Cited by: §D.1. Jitkrittum et al. (2026) W. Jitkrittum, H. Narasimhan, A. S. Rawat, J. Juneja, C. Wang, Z. Wang, A. Go, C. Lee, P. Shenoy, R. Panigrahy, A. K. Menon, and S. Kumar Universal model routing for efficient LLM inference. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, p. 10169–10218. External Links: Link Cited by: §D.4. Keswani et al. (2021) V. Keswani, M. Lease, and K. Kenthapadi Towards unbiased and accurate deferral to multiple experts. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, p. 154–165. Cited by: §D.3. Kuhn et al. (2023) L. Kuhn, Y. Gal, and S. Farquhar Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §D.1. Lai and Robbins (1985) T. Lai and H. Robbins Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics 6 (1), p. 4–22. External Links: ISSN 0196-8858, Document, Link Cited by: §C.1, §5. Lattimore and Szepesvári (2020) T. Lattimore and Cs. Szepesvári Bandit algorithms. Cambridge University Press. External Links: Link Cited by: §B.5.1, §C.3, §C.3, §C.5, §C.5, §C.6.1. Li et al. (2024) J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. C. Huang, K. Rasul, L. Yu, A. Jiang, Z. Shen, Z. Qin, B. Dong, L. Zhou, Y. Fleureau, G. Lample, and S. Polu NuminaMath TIR. Numina. Note: https://huggingface.co/datasets/AI-MO/NuminaMath-TIR Cited by: 2nd item. Lykouris and Weng (2025) T. Lykouris and W. Weng Learning to defer in congested systems: the AI-human interplay. External Links: 2402.12237, Link Cited by: §2. Madras et al. (2018) D. Madras, T. Pitassi, and R. Zemel Predict responsibly: improving fairness and accuracy by learning to defer. Advances in neural information processing systems 31. Cited by: §2. Mao et al. (2023) A. Mao, C. Mohri, M. Mohri, and Y. Zhong Two-stage learning to defer with multiple experts. Advances in neural information processing systems 36, p. 3578–3606. Cited by: §D.3. Mao et al. (2024a) A. Mao, M. Mohri, and Y. Zhong Realizable H-consistent and Bayes-consistent loss functions for learning to defer. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 73638–73671. External Links: Document, Link Cited by: §D.3. Mao et al. (2024b) A. Mao, M. Mohri, and Y. Zhong Regression with multi-expert deferral. arXiv preprint arXiv:2403.19494. Cited by: §D.3. Mao et al. (2025) A. Mao, M. Mohri, and Y. Zhong Mastering multiple-expert routing: realizable H-consistency and strong guarantees for learning to defer. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 43035–43066. External Links: Link Cited by: §D.3. Montreuil et al. (2026) Y. Montreuil, A. Carlier, L. X. Ng, and W. T. Ooi Why ask one when you can ask k? learning-to-defer to the top-k experts. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, p. 66968–67011. External Links: Link Cited by: §D.3. Mozannar and Sontag (2020) H. Mozannar and D. Sontag Consistent estimators for learning to defer to an expert. In International conference on machine learning, p. 7076–7087. Cited by: §2. Narasimhan et al. (2022) H. Narasimhan, W. Jitkrittum, A. K. Menon, A. Rawat, and S. Kumar Post-hoc estimators for learning to defer to an expert. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 29292–29304. External Links: Document, Link Cited by: §2. Nikitin et al. (2024) A. Nikitin, J. Kossen, Y. Gal, and P. Marttinen Kernel language entropy: fine-grained uncertainty quantification for LLMs from semantic similarities. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 8901–8929. External Links: Document, Link Cited by: §D.1. Ok et al. (2018) J. Ok, A. Proutiere, and D. Tranos Exploration in structured reinforcement learning. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, p. . External Links: Link Cited by: Remark 2. Ong et al. (2025) I. Ong, A. Almahairi, V. Wu, W. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica RouteLLM: learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §D.4. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, G. Flores, G. H. Chen, T. Pollard, J. C. Ho, and T. Naumann (Eds.), Proceedings of Machine Learning Research, Vol. 174, p. 248–260. External Links: Link Cited by: 3rd item. Platt (1999) J. C. Platt Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, A. J. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans (Eds.), p. 61–74. Cited by: §D.2. Qwen et al. (2025) Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §E.2. Ramírez et al. (2024) G. Ramírez, A. Birch, and I. Titov Optimising calls to large language models with uncertainty-based two-tier selection. In First Conference on Language Modeling, External Links: Link Cited by: §D.1. Reid et al. (2024) M. Reid, T. Sühr, C. Vernade, and S. Samadi Online decision deferral under budget constraints. External Links: 2409.20489, Link Cited by: §2. Réveillard et al. (2026) W. Réveillard, V. Saketos, A. Proutiere, and R. Combes Minimizing human intervention in online classification. AISTATS 2026. Cited by: §D.6. Robertson et al. (1988) T. Robertson, F. T. Wright, and R. L. Dykstra Order restricted statistical inference. John Wiley & Sons, Chichester. External Links: ISBN 9780471917878 Cited by: §D.2. Roy et al. (2021) A. Roy, V. Borkar, A. Karandikar, and P. Chaporkar Online reinforcement learning of optimal threshold policies for Markov decision processes. IEEE Transactions on Automatic Control 67 (7), p. 3722–3729. Cited by: §D.5, Appendix F, Appendix F, §6. Russo et al. (2024) M. Russo, A. Celli, R. Colini-Baldeschi, F. Fusco, D. Haimovich, D. Karamshuk, S. Leonardi, and N. Tax Online learning with sublinear best-action queries. Advances in Neural Information Processing Systems 37, p. 40407–40433. Cited by: §D.6. Tailor et al. (2024) D. Tailor, A. Patra, R. Verma, P. Manggala, and E. Nalisnick Learning to defer to a population: a meta-learning approach. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, S. Dasgupta, S. Mandt, and Y. Li (Eds.), Proceedings of Machine Learning Research, Vol. 238, p. 3475–3483. External Links: Link Cited by: §D.3. Verma et al. (2023) R. Verma, D. Barrejón, and E. Nalisnick Learning to defer to multiple experts: consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics, p. 11415–11434. Cited by: §D.3. Verma and Nalisnick (2022) R. Verma and E. Nalisnick Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning, p. 22184–22202. Cited by: §2. Wang et al. (2024) A. Wang, L. Song, Y. Tian, B. Peng, L. Jin, H. Mi, J. Su, and D. Yu Self-consistency boosts calibration for math reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 6023–6029. External Links: Link, Document Cited by: §1. Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. ICLR. Cited by: §1, §3.1. Weber et al. (2024) L. Weber, A. Busic, and J. Zhu Reinforcement learning and regret bounds for admission control. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 52403–52427. External Links: Link Cited by: §D.5. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §E.2. Yang et al. (2025b) X. Yang, Z. Liu, C. Huang, J. Zhang, T. Zhang, Y. Zhang, and W. Lei ELABORATION: a comprehensive benchmark on human-LLM competitive programming. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 59–104. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. Yue et al. (2024) M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao Large language model cascades with mixture of thought representations for cost-efficient reasoning. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 21691–21728. External Links: Link Cited by: §D.1. Zadrozny and Elkan (2001) B. Zadrozny and C. Elkan Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, San Francisco, CA, USA, p. 609–616. External Links: ISBN 1558607781 Cited by: §D.2. Zadrozny and Elkan (2002) B. Zadrozny and C. Elkan Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 694–699. External Links: Document Cited by: §D.2. Zhang et al. (2025) T. Zhang, A. Mehradfar, D. Dimitriadis, and S. Avestimehr Leveraging uncertainty estimation for efficient LLM routing. In ICML 2025 Workshop on Collaborative and Federated Agentic Workflows, External Links: Link Cited by: §D.1. Appendix Content Appendix A Proofs for Section 3 Throughout the appendices, write ℬ:=0,…,B¯B:=\0,…, B\ and St:=(Bt,Jt)S_t:=(B_t,J_t). Let (b,j):=0,1A(b,j):=\0,1\ when j∈[K]j∈[K] and b<B¯b< B, and (b,j):=0A(b,j):=\0\ otherwise. We recall that action 11 corresponds to deferral, and action 00 to acceptance. Let ℱtF_t be the sigma-field generated by the self-confidence levels, actions, service capacities, completed-query identities and their revealed ground-truth labels through the end of round t, together with past internal randomization. The algorithm chooses AtA_t using only ℱt−1F_t-1, JtJ_t, and independent randomization. The queries completed in each round are selected using only the observed history and service capacity, before any of their ground-truth labels are revealed. Thus BtB_t is ℱt−1F_t-1-measurable, whereas JtJ_t is independent of ℱt−1F_t-1 with distribution ω. For a real-valued function f on a finite domain, we write sp(f):=maxxf(x)−minxf(x)sp(f):= _xf(x)- _xf(x). A.1 Optimal deferral and regret decomposition Proof of Lemma 1. For b<B¯b< B, j∈[K]j∈[K], and s=(b,j)s=(b,j), Qth(s,1)−Qth(s,0)=Mth(b)−Lh(j).Q_t^h(s,1)-Q_t^h(s,0)=M_t^h(b)-L^h(j). (4) Hence deferral is optimal when Lh(j)>Mth(b)L^h(j)>M_t^h(b), acceptance is optimal when Lh(j)<Mth(b)L^h(j)<M_t^h(b), and both actions are optimal at equality. On no-query and full-queue states, acceptance is the only feasible action. Resolving ties in favor of acceptance gives the stated action rule and its equivalent threshold formulation. ∎ Proof of Lemma 2. All expectations are under the true conditional accuracy vector h⋆h . By the Bellman definition and the observation model, Qt⋆(St,At)=π[ct+Vt+1⋆(St+1)∣ℱt−1,Jt,At].Q_t (S_t,A_t)=E^π\! [c_t+V_t+1 (S_t+1) _t-1,J_t,A_t ]. Taking expectations and summing over t, the value terms telescope: π[∑t=1T(Qt⋆(St,At)−Vt⋆(St))]= ^π\! [ _t=1^T (Q_t (S_t,A_t)-V_t (S_t) ) ]= π[∑t=1Tct+D(BT+1)]−π[V1⋆(S1)]. ^π\! [ _t=1^Tc_t+D(B_T+1) ]-E^π[V_1 (S_1)]. Since B1=0B_1=0 and J1∼ωJ_1 ω under both the algorithm and the oracle policy, π[V1⋆(S1)]=∑j=0KωjV1⋆((0,j))=π⋆[∑t=1Tct+D(BT+1)].E^π[V_1 (S_1)]= _j=0^K _jV_1 ((0,j))=E^π \! [ _t=1^Tc_t+D(B_T+1) ]. Consequently, RT(π)=π[∑t=1T(Qt⋆(St,At)−Vt⋆(St))].R_T(π)=E^π\! [ _t=1^T (Q_t (S_t,A_t)-V_t (S_t) ) ]. (5) When Jt∈[K]J_t∈[K] and Bt<B¯B_t< B, equation 4 gives Qt⋆(St,At)−Vt⋆(St)=|L⋆(Jt)−Mt⋆(Bt)|At≠At⋆=ΔtQ_t (S_t,A_t)-V_t (S_t)= |L (J_t)-M_t (B_t) |1\A_t≠ A_t \= _t since the action space is binary. Otherwise, acceptance is the only feasible action, and both sides are zero. Substitution into equation 5 proves the claim. ∎ A.2 Service-capacity structure In this section, we prove several structural properties induced by our service-capacity model. The proposition below states that the marginal cost of deferral increases with the expert queue state. Proposition 1 (Monotone marginal cost). For every conditional accuracy vector h and every t≤Tt≤ T, the Bellman marginal cost is nondecreasing in the queue state: cexp≤Mth(0)≤Mth(1)≤⋯≤Mth(B¯−1).c_exp≤ M_t^h(0)≤ M_t^h(1)≤·s≤ M_t^h( B-1). Consequently, for every round t and self-confidence level j∈[K]j∈[K], there exists a threshold τt,jqueue(h)∈0,…,B¯ _t,j queue(h)∈\0,…, B\ such that, resolving ties in favor of acceptance, the optimal policy defers a query with self-confidence level j if and only if Bt<τt,jqueue(h).B_t< _t,j queue(h). For a conditional accuracy vector h, consider the queue value Uth(b):=∑j=0KωjVth((b,j)),U_t^h(b):= _j=0^K _jV_t^h((b,j)), i.e., the value at queue state b before the level of the incoming query is revealed. For t≤Tt≤ T and b∈ℬb , let Fth(b):=[Ut+1h((b−Ct)+)]F_t^h(b):=E[U_t+1^h((b-C_t)^+)], and, for b<B¯b< B, define dth(b):=Fth(b+1)−Fth(b)d_t^h(b):=F_t^h(b+1)-F_t^h(b). Then the Bellman marginal cost can be written as Mth(b)=cexp+dth(b),b<B¯.M_t^h(b)=c_exp+d_t^h(b), b< B. (6) We first establish the structural properties of the queue value function that will also be useful later. Define its adjacent increments by xth(q):=Uth(q)−Uth(q−1),q=1,…,B¯.x_t^h(q):=U_t^h(q)-U_t^h(q-1), q=1,…, B. Lemma 3 (Queue-value increments under service capacity). For every conditional accuracy vector h and every t≤T+1t≤ T+1, 0≤xth(1)≤xth(2)≤⋯≤xth(B¯).0≤ x_t^h(1)≤ x_t^h(2)≤·s≤ x_t^h( B). Consequently, UthU_t^h is nondecreasing and discrete convex in the queue state. Moreover, for every t≤Tt≤ T and b<B¯b< B, dth(b)=∑c=0bρC(c)xt+1h(b+1−c).d_t^h(b)= _c=0^b _C(c)\,x_t+1^h(b+1-c). (7) Proof. Equation 7 follows directly from the service-capacity dynamics. Indeed, dth(b)=[Ut+1h((b+1−Ct)+)−Ut+1h((b−Ct)+)].d_t^h(b)=E\! [U_t+1^h((b+1-C_t)^+)-U_t+1^h((b-C_t)^+) ]. If Ct=c≤bC_t=c≤ b, the two queue states differ by one and the difference equals xt+1h(b+1−c)x_t+1^h(b+1-c). If c>bc>b, both queue states are zero. Hence equation 7 follows. We prove the monotonicity of the increments by backward induction. At t=T+1t=T+1, we have UT+1h=DU_T+1^h=D, independently of h. Let S0:=0S_0:=0 and Sk:=C1+⋯+CkS_k:=C_1+·s+C_k for k≥1k≥ 1, where C1,C2,…C_1,C_2,… are i.i.d. with law ρC _C. Iterating the definition of D gives D(b)=chold∑k≥0[(b−Sk)+].D(b)=c_ hold _k≥ 0E[(b-S_k)^+]. The series is finite since μ>0μ>0. Let g(q):=D(q)−D(q−1)g(q):=D(q)-D(q-1). Since (q−Sk)+−(q−1−Sk)+=Sk<q,(q-S_k)^+-(q-1-S_k)^+=1\S_k<q\, we have g(q)=chold∑k≥0ℙ(Sk<q).g(q)=c_ hold _k≥ 0P(S_k<q). Thus g(q)≥0g(q)≥ 0 and is nondecreasing in q, so 0≤g(1)≤g(2)≤⋯≤g(B¯).0≤ g(1)≤ g(2)≤·s≤ g( B). Therefore UT+1h=DU_T+1^h=D is nondecreasing and discrete convex, establishing the base case. Suppose now that 0≤xt+1h(1)≤⋯≤xt+1h(B¯).0≤ x_t+1^h(1)≤·s≤ x_t+1^h( B). Equation 7 immediately gives dth(b)≥0d_t^h(b)≥ 0. Moreover, for b<B¯−1b< B-1, dth(b+1)−dth(b)= d_t^h(b+1)-d_t^h(b)= ∑c=0bρC(c)[xt+1h(b+2−c)−xt+1h(b+1−c)] _c=0^b _C(c) [x_t+1^h(b+2-c)-x_t+1^h(b+1-c) ] +ρC(b+1)xt+1h(1)≥0. + _C(b+1)x_t+1^h(1)≥ 0. Thus 0≤dth(0)≤dth(1)≤⋯≤dth(B¯−1).0≤ d_t^h(0)≤ d_t^h(1)≤·s≤ d_t^h( B-1). (8) For j=0,…,Kj=0,…,K, write rjh:=Lh(j)−cexp.r_j^h:=L^h(j)-c_exp. In particular, r0h=−cexpr_0^h=-c_exp because Lh(0)=0L^h(0)=0. For b<B¯b< B, the Bellman equation for the queue value function gives Uth(b)=choldb+Fth(b)+cexp+∑j=0Kωjminrjh,dth(b).U_t^h(b)=c_holdb+F_t^h(b)+c_exp+ _j=0^K _j \r_j^h,d_t^h(b)\. Indeed, for j∈[K]j∈[K] this represents the minimum between acceptance and deferral costs. For j=0j=0, action 00 is forced; since r0h=−cexpr_0^h=-c_exp and dth(b)≥0d_t^h(b)≥ 0, the corresponding term satisfies cexp+minr0h,dth(b)=0c_exp+ \r_0^h,d_t^h(b)\=0, and therefore represents the no-query state correctly. Consequently, for 1≤q<B¯1≤ q< B, subtracting the Bellman equations for the queue value function at q and q−1q-1 gives xth(q)= x_t^h(q)= chold+dth(q−1)+∑j=0Kωj[minrjh,dth(q)−minrjh,dth(q−1)] c_hold+d_t^h(q-1)+ _j=0^K _j [ \r_j^h,d_t^h(q)\- \r_j^h,d_t^h(q-1)\ ] (9) = = chold+∑j=0Kωjmeddth(q−1),dth(q),rjh, c_hold+ _j=0^K _jmed\d_t^h(q-1),d_t^h(q),r_j^h\, where the second equality uses ∑j=0Kωj=1 _j=0^K _j=1 and dth(q−1)≤dth(q)d_t^h(q-1)≤ d_t^h(q), and medx,y,zmed\x,y,z\ denotes the median of its three arguments. At q=B¯q= B, action 11 is infeasible. Hence Uth(B¯)=choldB¯+Fth(B¯)+∑j=0KωjLh(j),U_t^h( B)=c_hold B+F_t^h( B)+ _j=0^K _jL^h(j), which together with the Bellman equation at B¯−1 B-1 yields xth(B¯)=chold+∑j=0Kωjmaxdth(B¯−1),rjh.x_t^h( B)=c_hold+ _j=0^K _j \d_t^h( B-1),r_j^h \. (10) By equation 8, for every fixed r, meddth(q−1),dth(q),rmed\d_t^h(q-1),d_t^h(q),r\ is nonnegative and nondecreasing in q. Therefore 0≤xth(1)≤⋯≤xth(B¯−1).0≤ x_t^h(1)≤·s≤ x_t^h( B-1). Finally, maxdth(B¯−1),r≥meddth(B¯−2),dth(B¯−1),r, \d_t^h( B-1),r \ \d_t^h( B-2),d_t^h( B-1),r \, whenever B¯≥2 B≥ 2. Combining equation 9 and equation 10 therefore gives xth(B¯)≥xth(B¯−1)x_t^h( B)≥ x_t^h( B-1). For B¯=1 B=1, the conclusion follows directly from equation 10. This completes the backward induction. ∎ Proof of Proposition 1. By Lemma 3, xt+1h(q)x_t+1^h(q) is nonnegative and nondecreasing in q. Hence equation 7 and equation 6 give 0≤dth(0)≤dth(1)≤⋯≤dth(B¯−1),cexp≤Mth(0)≤Mth(1)≤⋯≤Mth(B¯−1). gathered0≤ d_t^h(0)≤ d_t^h(1)≤·s≤ d_t^h( B-1),\\ c_exp≤ M_t^h(0)≤ M_t^h(1)≤·s≤ M_t^h( B-1). gathered Fix t and self-confidence level j, and define τt,jqueue(h):=min(b∈0,…,B¯−1:Lh(j)≤Mth(b)∪B¯). _t,j queue(h):= ( \b∈\0,…, B-1\:L^h(j)≤ M_t^h(b) \∪\ B\ ). Since Mth(b)M_t^h(b) is nondecreasing in b, Lh(j)>Mth(b)L^h(j)>M_t^h(b) if and only if b<τt,jqueue(h)b< _t,j queue(h). By equation 4, the optimal policy with ties resolved in favor of acceptance therefore defers if and only if Bt<τt,jqueue(h)B_t< _t,j queue(h). ∎ Appendix B Proofs for Section 4.2 B.1 Proof sketch To ease navigation within this appendix, we first provide a high-level proof sketch of our gap-dependent and worst-case upper bounds. They are proven in Appendices B.4 and B.3 respectively. Proof sketch. The analysis exploits that pessimism makes KL-PBD almost one-sided. On the high-probability event h¯t≤h⋆ h_t≤ h , if the marginal cost of deferral were computed under the true model h⋆h , the learner would accept only if the oracle does, and regret would come from deferrals only. This ordering is not exact, because h¯t h_t is also plugged into the marginal cost of deferral. To account for these additional errors, we track the difference between the value functions of the pessimistic and true models: Proposition 2 relates this misspecification to the estimation error. Updating h¯t h_t leaves an additional term controlled by its total variation, which is logarithmic. The remaining estimation error is charged only on deferred rounds, which are exactly the rounds that produce labels, so the confidence widths shrink as they are charged. Summing these widths incurs only an additive B¯ B per level from delayed feedback, since at most B¯ B previously deferred queries can still be unlabeled, and yields Theorem 2. For Theorem 1, we exploit that, by Proposition 2(i), estimation errors smaller than γT/2 _T/2 cannot change strict oracle decisions. We therefore compare h¯t h_t with the intermediate model obtained by replacing h¯j,t h_j,t with maxh¯j,t,hj⋆−γT/2 \ h_j,t,h_j - _T/2\, which agrees with the oracle on every strict comparison. Only the confidence width beyond γT/2 _T/2 needs to be charged, yielding the 1/γT1/ _T dependence. ∎ B.2 Notation All expectations in this appendix are under h⋆h and πKL-PBDπ KL-PBD. For j∈[K]j∈[K] and t≥0t≥ 0, define Dj,t:=∑s=1tJs=j,As=1D_j,t:= _s=1^t1\J_s=j,A_s=1\. Recall that Nj,tN_j,t counts completed self-confidence level j labels available before round t. In this appendix, we prove more general regret upper bounds from which the bounds of Theorems 2 and 1 directly follow. The technical lemmas used in the regret proofs are stated and proved in Section B.5. We first establish bounds for a general confidence budget ϕ(T)>1φ(T)>1, then specialize to ϕ(T)=logT+loglogTφ(T)= T+ T to obtain the main-text bounds. For T≥2T≥ 2, define ηT:=Tmin1, 2KTe−ϕ(T), 2eK⌈ϕ(T)logT⌉e−ϕ(T), _T:=T \1,\,2KTe^-φ(T),\,2eK φ(T) T e^-φ(T) \, (11) set η1:=min1,2Ke−ϕ(1) _1:= \1,2Ke^-φ(1)\, and write ΛT:=∑j=1Klog(1+ωjTϕ(T)). _T:= _j=1^K (1+ _jTφ(T) ). These two quantities are used only to state the appendix bounds. For a fixed budget ϕ(T)>1φ(T)>1, let h^j,n h_j,n denote the empirical mean of the first n ground-truth labels of self-confidence level j, extended to an infinite i.i.d. Bernoulli(hj⋆)Bernoulli(h_j ) sequence as in Lemma 8, and define the concentration event ℰ:=⋂j=1K⋂n=1Tnkl(h^j,n,hj⋆)≤ϕ(T).E:= _j=1^K _n=1^T \nkl( h_j,n,h_j )≤φ(T) \. (12) Let also gj,0:=0,gj,n:=infq∈[0,h^j,n]:nkl(h^j,n,q)≤ϕ(T),n≥1.g_j,0:=0, g_j,n:= \q∈[0, h_j,n]:nkl( h_j,n,q)≤φ(T) \, n≥ 1. (13) Thus h¯j,t=gj,Nj,t h_j,t=g_j,N_j,t, where Nj,tN_j,t counts completed labels available before round t. B.3 Worst-case regret of KL-PBD We first prove a refined bound that retains the Bernoulli KL geometry. The main-text theorem follows by bounding the KL confidence width using Pinsker’s inequality. For a Bernoulli parameter p∈[0,1]p∈[0,1] and x≥0x≥ 0, define wp(x):=supp−q:0≤q≤p,∃z∈[0,1],kl(z,p)≤x,kl(z,q)≤x.w_p(x):= \p-q:0≤ q≤ p,\ ∃ z∈[0,1],\ kl(z,p)≤ x,\ kl(z,q)≤ x \. (14) This function is nondecreasing in x and takes values in [0,p][0,p]. For Γ>0 >0 and d≥0d≥ 0, let Fp,Γ(d):=∫0dwp(Γ/u)u,F_p, (d):= _0^dw_p( /u)\,du, (15) with the integrand at zero defined by its right limit. Theorem 5 (Refined regret bound). For any fixed budget ϕ(T)>1φ(T)>1, RT(πKL-PBD)≤ R_T(π KL-PBD)≤ ηTΔ+ce∑j=1Kminhj⋆ωjT,B¯hj⋆+Fhj⋆,ϕ(T)(ωjT) _T +c_e _j=1^K \h_j _jT,\, Bh_j +F_h_j ,φ(T)( _jT) \ (16) +B¯ce(1+2ΛT). + Bc_e(1+2 _T). where Δ:=maxce,cexp+choldB¯μ. := \c_e,c_exp\+ c_hold Bμ. Recall the history ℱtF_t from Appendix A. Immediately before JtJ_t is observed, BtB_t, the completed-label counts, and h¯t h_t are ℱt−1F_t-1-measurable, while JtJ_t is independent of ℱt−1F_t-1 with distribution ω. Define ℰt:=Nj,tkl(h^j,Nj,t,hj⋆)≤ϕ(T) for every j with Nj,t≥1,σ:=inft∈[T]:ℰtc∧(T+1).E_t:= \N_j,tkl( h_j,N_j,t,h_j )≤φ(T) for every $j$ with $N_j,t≥ 1$ \, σ:= \t∈[T]:E_t^c\ (T+1). Then t<σ∈ℱt−1\t<σ\ _t-1. Since Nj,t≤TN_j,t≤ T, we have ℰtc⊆ℰcE_t^c ^c for every t, so Lemma 8 gives ℙ(σ≤T)≤ℙ(ℰc)≤ηT.P(σ≤ T) (E^c)≤ _TT. (17) For t<σt<σ, let b0,t:=0b_0,t:=0 and for j∈[K]j∈[K], define bj,t:=ce(hj⋆−h¯j,t).b_j,t:=c_e(h_j - h_j,t). (18) For j∈[K],j∈[K], Lemma 9 gives 0≤bj,t≤cehj⋆,Nj,t=0,cewhj⋆(ϕ(T)/Nj,t),Nj,t≥1.0≤ b_j,t≤ casesc_eh_j ,&N_j,t=0,\\[2.84526pt] c_ew_h_j (φ(T)/N_j,t),&N_j,t≥ 1. cases (19) Regret decomposition. Define ℛfail _ fail :=[∑t=σTΔtσ≤T], :=E\! [ _t=σ^T _t 1\σ≤ T\ ], ℛdefer _ defer :=[∑t<σbJt,tAt=1], :=E\! [ _t<σb_J_t,t 1\A_t=1\ ], ℛupdate _ update :=B¯ce[1+[∑t=2Tt<σ‖h¯t−h¯t−1‖∞]]. := Bc_e [1+E\! [ _t=2^T 1\t<σ\ \| h_t- h_t-1 \|_∞ ] ]. Lemma 4 (Regret decomposition). The regret satisfies RT(πKL-PBD)≤ℛfail⏟(I)+ℛdefer⏟(I)+ℛupdate⏟(I).R_T(π KL-PBD)≤ R_ fail_(I)+ R_ defer_(I)+ R_ update_(I). (20) Proof of Lemma 4. By Lemma 2, RT(πKL-PBD)=[∑t<σΔt]+ℛfail.R_T(π KL-PBD)=E\! [ _t<σ _t ]+R_ fail. Thus it remains to control the regret accumulated before σ. On these rounds, h¯t≤h⋆ h_t≤ h coordinatewise. Apply Lemma 13 with h~t=h⋆ h_t=h . Then Δ~t=Δt _t= _t, ej,t=bj,te_j,t=b_j,t, and the reference vector does not change with t. Consequently, [∑t<σΔt] \! [ _t<σ _t ] ≤[∑t<σbJt,tAt=1]+B¯ce(1+[∑t=2σ−1∥h¯t−h¯t−1∥∞]) \! [ _t<σb_J_t,t1\A_t=1\ ]+ Bc_e (1+E\! [ _t=2^σ-1\| h_t- h_t-1\|_∞ ] ) =ℛdefer+ℛupdate. =R_ defer+R_ update. Adding ℛfailR_ fail proves equation 20. ∎ Lemma 5 (Term (I): concentration failure). We have ℛfail≤ηTΔR_ fail≤ _T . Proof. Lemma 12 and equation 17 give ℛfail≤TΔℙ(σ≤T)≤ηTΔ.R_ fail≤ T (σ≤ T)≤ _T . ∎ Lemma 6 (Term (I): deferred-round contribution). We have ℛdefer≤ce∑j=1Kminhj⋆ωjT,B¯hj⋆+Fhj⋆,ϕ(T)(ωjT).R_ defer≤ c_e _j=1^K \h_j _jT,\, Bh_j +F_h_j ,φ(T)( _jT) \. (21) Proof. Fix a self-confidence level j. Define fj(0):=hj⋆f_j(0):=h_j and fj(u):=whj⋆(ϕ(T)/u)f_j(u):=w_h_j (φ(T)/u) for u>0u>0. Since whj⋆w_h_j is nondecreasing and takes values in [0,hj⋆][0,h_j ], fjf_j is nonnegative and nonincreasing. By equation 19, bj,t≤cefj(Nj,t),t<σ.b_j,t≤ c_ef_j(N_j,t), t<σ. Moreover, the definition of Fp,ΓF_p, gives ∫0ωjTfj(u)u=Fhj⋆,ϕ(T)(ωjT) _0 _jTf_j(u)\,du=F_h_j ,φ(T)( _jT). Applying Lemma 14, we obtain [∑t<σbj,tJt=j,At=1] \! [ _t<σb_j,t1\J_t=j,A_t=1\ ] ≤ce[∑t<σJt=j,At=1fj(Nj,t)] ≤ c_eE\! [ _t<σ1\J_t=j,A_t=1\f_j(N_j,t) ] ≤ceminhj⋆ωjT,B¯hj⋆+Fhj⋆,ϕ(T)(ωjT). ≤ c_e \h_j _jT,\ Bh_j +F_h_j ,φ(T)( _jT) \. Summing over j∈[K]j∈[K] proves equation 21. ∎ Lemma 7 (Term (I): Bellman-model updates). We have ℛupdate≤B¯ce(1+2ΛT).R_ update≤ Bc_e(1+2 _T). Proof. Since t<σ≤11\t<σ\≤ 1, Lemma 10 gives ℛupdate _ update ≤B¯ce[1+[∑t=2T‖h¯t−h¯t−1‖∞]] ≤ Bc_e [1+E\! [ _t=2^T \| h_t- h_t-1 \|_∞ ] ] ≤B¯ce(1+2ΛT). ≤ Bc_e(1+2 _T). ∎ Proof of Theorem 5. Combine Lemmas 4, 5, 6, and 7. ∎ Proof of Theorem 2. For the main-text budget ϕ(T)=logT+loglogTφ(T)= T+ T, Lemma 8 gives ηT≤4eKϕ(T) _T≤ 4eKφ(T). Using hj⋆≤1h_j ≤ 1 and Fhj⋆,ϕ(T)(ωjT)≤22ωjTϕ(T)F_h_j ,φ(T)( _jT)≤ 2 2 _jTφ(T) in Theorem 5 gives RT(πKL-PBD)≤ R_T(π KL-PBD)≤ 4eKϕ(T)Δ+ce∑j=1KminωjT,B¯+22ωjTϕ(T) 4eKφ(T) +c_e _j=1^K \ _jT,\, B+2 2 _jTφ(T) \ +B¯ce[1+2∑j=1Klog(1+ωjTϕ(T))]. + Bc_e [1+2 _j=1^K (1+ _jTφ(T) ) ]. Since κ=4eΔκ=4e , this is the main-text bound. ∎ Remark 1 (Worst case over ω). Since ∑j=1Kωj≤1 _j=1^K _j≤ 1, Jensen’s inequality gives ΛT _T =∑j=1Klog(1+ωjTϕ(T)) = _j=1^K (1+ _jTφ(T) ) ≤Klog(1+TKϕ(T)∑j=1Kωj) ≤ K (1+ TKφ(T) _j=1^K _j ) ≤Klog(1+TKϕ(T)). ≤ K (1+ TKφ(T) ). Moreover, ∑j=1KωjT≤KT∑j=1Kωj≤KT,∑j=1KωjT≤T. _j=1^K _jT≤ KT _j=1^K _j≤ KT, _j=1^K _jT≤ T. Hence ∑j=1KminωjT,B¯+22ωjTϕ(T)≤minT,KB¯+22KTϕ(T), _j=1^K \ _jT,\, B+2 2 _jTφ(T) \≤ \T,\,K B+2 2KTφ(T) \, and therefore RT(πKL-PBD)≤ R_T(π KL-PBD)≤ 4eKϕ(T)Δ+ceminT,KB¯+22KTϕ(T) 4eKφ(T) +c_e \T,\,K B+2 2KTφ(T)\ +B¯ce[1+2Klog(1+TKϕ(T))]. + Bc_e [1+2K (1+ TKφ(T) ) ]. These inequalities hold with equality when ω0=0 _0=0 and ωj=1/K _j=1/K for every j∈[K]j∈[K], so this bound is the largest value of the bound of Theorem 2 over ω. Alternatively, retaining the variance-sensitive term in equation 30 gives, for any fixed budget ϕ(T)>1φ(T)>1, RT(πKL-PBD)≤ R_T(π KL-PBD)≤ ηTΔ+B¯ce(K+1) _T + Bc_e(K+1) +4ce2Tϕ(T)∑j=1Khj⋆(1−hj⋆)ωj+ce(7ϕ(T)+2B¯)ΛT. +4c_e 2Tφ(T) _j=1^K h_j (1-h_j ) _j+c_e(7φ(T)+2 B) _T. The term proportional to ϕ(T)ΛTφ(T) _T is a statistical remainder; it is separate from the Bellman-model update contribution. B.4 Gap-dependent regret We use the gap definitions from the main text. For p∈[0,1]p∈[0,1] and a>0a>0, define Ip(a):=∫0∞(wp(1/u)−a)+u,I_p(a):= _0^∞ (w_p(1/u)-a )_+\,du, (22) with the integrand at zero defined by its right limit. Theorem 6 (Refined gap-dependent regret bound). If T=∅G_T= , the regret is zero. Otherwise, let aT=γT/2a_T= _T/2. For any budget ϕ(T)>1φ(T)>1 held fixed throughout the run, RT(πKL-PBD)≤ R_T(π KL-PBD)≤ ηTΔ+2ce∑j=1Kmin(hj⋆−aT)+ωjT,B¯(hj⋆−aT)++ϕ(T)Ihj⋆(aT) _T +2c_e _j=1^K \(h_j -a_T)_+ _jT,\, B(h_j -a_T)_++φ(T)I_h_j (a_T) \ (23) +2B¯ce(1+4ΛT). +2 Bc_e(1+4 _T). Proof of Theorem 6. If T=∅G_T= , then hj⋆=τt⋆(b)h_j =τ _t(b) for every j∈[K]j∈[K], t≤Tt≤ T, and b<B¯b< B. Hence every feasible action contributes 00 to regret and RT(πKL-PBD)=0R_T(π KL-PBD)=0. We therefore assume below that T≠∅G_T≠ . Set a:=aT=γT2a:=a_T= _T2, θ:=cea.θ:=c_ea. Recall from equation 18 that, for t<σt<σ, bj,t=ce(hj⋆−h¯j,t)≥0.b_j,t=c_e(h_j - h_j,t)≥ 0. Clipped intermediate model. For t<σt<σ and j∈[K]j∈[K], define bj,t∘:=minbj,t,θ,hj,t∘:=hj⋆−bj,t∘ce,b_j,t := \b_j,t,θ\, h_j,t :=h_j - b_j,t c_e, and let xj,t:=bj,t−bj,t∘=(bj,t−θ)+.x_j,t:=b_j,t-b_j,t =(b_j,t-θ)_+. For notational convenience, set x0,t:=0.x_0,t:=0. Thus h¯t≤ht∘≤h⋆,‖ht∘−h⋆‖∞≤a. h_t≤ h_t ≤ h , \|h_t -h \|_∞≤ a. For j∈[K]j∈[K], t≤Tt≤ T, and b<B¯b< B, define the action margin Dth(j,b):=Mth(b)−ce(1−hj).D_t^h(j,b):=M_t^h(b)-c_e(1-h_j). Under h⋆h , the margin satisfies Dt⋆(j,b)=ce(hj⋆−τt⋆(b)).D_t (j,b)=c_e (h_j -τ _t(b) ). Since ht∘≤h⋆h_t ≤ h , Proposition 2(i) gives, for every b<B¯,b< B, 0≤Mtht∘(b)−Mt⋆(b)≤ce‖ht∘−h⋆‖∞≤θ.0≤ M_t^h_t (b)-M_t (b)≤ c_e\|h_t -h \|_∞≤θ. Moreover, for every j∈[K],j∈[K], ce(1−hj,t∘)−ce(1−hj⋆)=bj,t∘∈[0,θ].c_e(1-h_j,t )-c_e(1-h_j )=b_j,t ∈[0,θ]. Consequently, for every j∈[K]j∈[K] and b<B¯,b< B, |Dtht∘(j,b)−Dt⋆(j,b)|≤θ. |D_t^h_t (j,b)-D_t (j,b) |≤θ. Whenever Dt⋆(j,b)≠0D_t (j,b)≠ 0, the definition of γT _T gives |Dt⋆(j,b)|=ce|hj⋆−τt⋆(b)|≥ceγT=2θ|D_t (j,b)|=c_e|h_j - _t (b)|≥ c_e _T=2θ. Hence Dtht∘(j,b)D_t^h_t (j,b) has the same sign as Dt⋆(j,b)D_t (j,b) and |Dtht∘(j,b)|≥12|Dt⋆(j,b)|. |D_t^h_t (j,b) |≥ 12 |D_t (j,b) |. For t<σt<σ, define Δt∘:=Qtht∘(St,At)−Vtht∘(St). _t :=Q_t^h_t (S_t,A_t)-V_t^h_t (S_t). If AtA_t is optimal under h⋆h , then Δt=0≤2Δt∘ _t=0≤ 2 _t . Otherwise, necessarily Jt∈[K]J_t∈[K] and Bt<B¯B_t< B, because action 00 is forced on no-query and full-queue states. The sign comparison above then shows that AtA_t is also suboptimal under ht∘h_t , and Δt∘=|Dtht∘(Jt,Bt)|≥12|Dt⋆(Jt,Bt)|=12Δt. _t = |D_t^h_t (J_t,B_t) |≥ 12 |D_t (J_t,B_t) |= 12 _t. Therefore Δt≤2Δt∘,t<σ. _t≤ 2 _t , t<σ. (24) Regret relative to the clipped model. For every j∈[K]j∈[K] and t<σt<σ, the conditional accuracy vectors h¯t h_t and ht∘h_t satisfy ce(hj,t∘−h¯j,t)=ce(hj⋆−h¯j,t)−bj,t∘=bj,t−bj,t∘=xj,t.c_e(h_j,t - h_j,t)=c_e(h_j - h_j,t)-b_j,t =b_j,t-b_j,t =x_j,t. Apply Lemma 13 with h~j,t:=maxh¯j,t,hj⋆−a h_j,t:= \ h_j,t,h_j -a\. This vector is ℱt−1F_t-1-measurable, satisfies h¯t≤h~t h_t≤ h_t, and equals ht∘h_t for t<σt<σ. Thus ej,t=xj,te_j,t=x_j,t and Δ~t=Δt∘ _t= _t on these rounds, giving [∑t<σΔt∘]≤ \! [ _t<σ _t ]≤ [∑t<σxJt,tAt=1] \! [ _t<σx_J_t,t1\A_t=1\ ] +B¯ce(1+[∑t=2σ−1(‖h¯t−h¯t−1‖∞+‖ht∘−ht−1∘‖∞)]). + Bc_e (1+E\! [ _t=2^σ-1 (\| h_t- h_t-1\|_∞+\|h_t -h_t-1 \|_∞ ) ] ). It remains to control the variation of the clipped model. The clipping map is 11-Lipschitz: for every j∈[K]j∈[K] and 2≤t<σ2≤ t<σ, |hj,t∘−hj,t−1∘| |h_j,t -h_j,t-1 | =|maxh¯j,t,hj⋆−a−maxh¯j,t−1,hj⋆−a| = | \ h_j,t,h_j -a\- \ h_j,t-1,h_j -a\ | ≤|h¯j,t−h¯j,t−1|. ≤| h_j,t- h_j,t-1|. Hence ‖ht∘−ht−1∘‖∞≤‖h¯t−h¯t−1‖∞\|h_t -h_t-1 \|_∞≤\| h_t- h_t-1\|_∞. By Lemma 10, [∑t=2σ−1(‖h¯t−h¯t−1‖∞+‖ht∘−ht−1∘‖∞)] \! [ _t=2^σ-1 (\| h_t- h_t-1\|_∞+\|h_t -h_t-1 \|_∞ ) ] ≤2[∑t=2T‖h¯t−h¯t−1‖∞]≤4ΛT. ≤ 2E\! [ _t=2^T\| h_t- h_t-1\|_∞ ]≤ 4 _T. It follows that [∑t<σΔt∘]≤[∑t<σxJt,tAt=1]+B¯ce(1+4ΛT).E\! [ _t<σ _t ] \! [ _t<σx_J_t,t1\A_t=1\ ]+ Bc_e(1+4 _T). Combining this with equation 24, [∑t<σΔt]≤2[∑t<σxJt,tAt=1]+2B¯ce(1+4ΛT).E\! [ _t<σ _t ]≤ 2E\! [ _t<σx_J_t,t1\A_t=1\ ]+2 Bc_e(1+4 _T). (25) Residual error on deferred rounds. Fix a self-confidence level j. Define fj(0):=(hj⋆−a)+f_j(0):=(h_j -a)_+ and fj(u):=(whj⋆(ϕ(T)/u)−a)+f_j(u):=(w_h_j (φ(T)/u)-a)_+ for u>0u>0. This function is nonnegative and nonincreasing. Since θ=ceaθ=c_ea, equation 19 gives xj,t=(bj,t−θ)+≤ce(hj⋆−a)+,Nj,t=0,ce(whj⋆(ϕ(T)/Nj,t)−a)+,Nj,t≥1,t<σ.x_j,t=(b_j,t-θ)_+≤ casesc_e(h_j -a)_+,&N_j,t=0,\\ c_e (w_h_j (φ(T)/N_j,t)-a )_+,&N_j,t≥ 1, cases t<σ. Thus xj,t≤cefj(Nj,t)x_j,t≤ c_ef_j(N_j,t). Applying Lemma 14, [∑t<σxj,tJt=j,At=1] \! [ _t<σx_j,t1\J_t=j,A_t=1\ ] ≤ce[∑t<σJt=j,At=1fj(Nj,t)] ≤ c_eE\! [ _t<σ1\J_t=j,A_t=1\f_j(N_j,t) ] ≤cemin(hj⋆−a)+ωjT,B¯(hj⋆−a)++∫0ωjTfj(u)du. ≤ c_e \(h_j -a)_+ _jT,\ B(h_j -a)_++ _0 _jTf_j(u)\,du \. The integral is bounded using nonnegativity and the change of variables u=ϕ(T)vu=φ(T)v: ∫0ωjTfj(u)u _0 _jTf_j(u)\,du ≤∫0∞(whj⋆(ϕ(T)/u)−a)+u ≤ _0^∞ (w_h_j (φ(T)/u)-a )_+\,du =ϕ(T)∫0∞(whj⋆(1/v)−a)+v =φ(T) _0^∞ (w_h_j (1/v)-a )_+\,dv =ϕ(T)Ihj⋆(a). =φ(T)I_h_j (a). Substituting this bound and summing over j∈[K]j∈[K] gives [∑t<σxJt,tAt=1]≤ce∑j=1Kmin(hj⋆−a)+ωjT,B¯(hj⋆−a)++ϕ(T)Ihj⋆(a).E\! [ _t<σx_J_t,t1\A_t=1\ ]≤ c_e _j=1^K \(h_j -a)_+ _jT,\ B(h_j -a)_++φ(T)I_h_j (a) \. Substituting into equation 25 and adding the failure contribution ℛfail≤ηTΔR_ fail≤ _T from Lemma 5 proves equation 23. ∎ Proof of Theorem 1. If T=∅G_T= , the refined theorem gives zero regret. Otherwise, we apply Theorem 6 with aT=γT/2a_T= _T/2. We have (hj⋆−aT)+≤1(h_j -a_T)_+≤ 1 and, by Lemma 11, Ihj⋆(aT)≤2/aT=4/γTI_h_j (a_T)≤ 2/a_T=4/ _T. For ϕ(T)=logT+loglogTφ(T)= T+ T, Lemma 8 gives ηT≤4eKϕ(T) _T≤ 4eKφ(T). Consequently, RT(πKL-PBD)≤ R_T(π KL-PBD)≤ 4eKϕ(T)Δ+2ce∑j=1KminωjT,B¯+4ϕ(T)γT 4eKφ(T) +2c_e _j=1^K \ _jT,\, B+ 4φ(T) _T \ +2B¯ce[1+4∑j=1Klog(1+ωjTϕ(T))], +2 Bc_e [1+4 _j=1^K (1+ _jTφ(T) ) ], which proves the main-text theorem since κ=4eΔκ=4e . ∎ In particular, the main-text bound implies RT(πKL-PBD)≤2B¯ce(K+1)+Kϕ(T)(8ceγT+4eΔ)+8B¯ceΛT.R_T(π KL-PBD)≤ 2 Bc_e(K+1)+Kφ(T) ( 8c_e _T+4e )+8 Bc_e _T. When the gap is bounded away from zero, the regret remains logarithmic in T. The failure contribution now has the same logarithmic order, but does not worsen the inverse-gap dependence. The refined theorem and equation 35 retain the variance-sensitive improvement. B.5 Technical lemmas B.5.1 Concentration and KL confidence bounds Lemma 8 (Concentration within each self-confidence level under delayed feedback). For every j∈[K]j∈[K], the observed-label sequence can be extended to an infinite i.i.d. Bernoulli(hj⋆)Bernoulli(h_j ) sequence. For any fixed budget ϕ(T)>1φ(T)>1, the event ℰE of equation 12 satisfies ℙ(ℰc)≤ηT/TP(E^c)≤ _T/T. For T≥3T≥ 3 and ϕ(T)=logT+loglogTφ(T)= T+ T, we also have ηT≤4eKϕ(T). _T≤ 4eKφ(T). (26) Proof. Conditional on the self-confidence level sequence, the ground-truth labels are independent, with Yt∼Bernoulli(hj⋆)Y_t (h_j ) when Jt=jJ_t=j. Since deferral and completion decisions use only self-confidence levels, service information and previously revealed labels, conditional on the history immediately before a level-j label is revealed, that label has distribution Bernoulli(hj⋆)Bernoulli(h_j ). Revealing completed labels in submission order within each round, and appending independent Bernoulli(hj⋆)Bernoulli(h_j ) labels after the observed sequence for level j, gives the claimed infinite i.i.d. sequence (Lattimore and Szepesvári, 2020, Section 4.6). For a Bernoulli sample mean, Chernoff’s inequality and a union bound give ℙ(ℰc)≤2KTe−ϕ(T).P(E^c)≤ 2KTe^-φ(T). For T≥2T≥ 2, applying the Bernoulli maximal inequality (Garivier, 2013, equation (5)) to the ground-truth label sequence for each self-confidence level and taking a union bound over levels gives ℙ(ℰc)≤2eK⌈ϕ(T)logT⌉e−ϕ(T),ϕ(T)>1.P(E^c)≤ 2eK φ(T) T e^-φ(T), φ(T)>1. (27) If hj⋆∈0,1h_j ∈\0,1\, then h^j,n=hj⋆ h_j,n=h_j almost surely for every n≥1n≥ 1, so the KL concentration event holds automatically for that j. Combining these bounds with ℙ(ℰc)≤1P(E^c)≤ 1 gives ℙ(ℰc)≤ηT/TP(E^c)≤ _T/T for T≥2T≥ 2. For T=1T=1, the same conclusion follows from the Chernoff union bound and the definition of η1 _1. Picking ϕ(T)=logT+loglogTφ(T)= T+ T gives, using Te−ϕ(T)=(logT)−1Te^-φ(T)=( T)^-1, ηT≤2eK⌈ϕ(T)logT⌉logT≤2eK(ϕ(T)+1logT)≤4eKϕ(T). _T≤ 2eK φ(T) T T≤ 2eK (φ(T)+ 1 T )≤ 4eKφ(T). The last inequality uses logT>1 T>1 and ϕ(T)>1φ(T)>1. ∎ Lemma 9 (KL diameter and its integral bounds). For every p∈[0,1]p∈[0,1] and x≥0x≥ 0, wp(x)≤minp,2x.w_p(x)≤ \p, 2x\. (28) Whenever nkl(h^j,n,hj⋆)≤ϕ(T)nkl( h_j,n,h_j )≤φ(T) with n≥1n≥ 1, 0≤hj⋆−gj,n≤whj⋆(ϕ(T)/n).0≤ h_j -g_j,n≤ w_h_j (φ(T)/n). (29) Moreover, Fp,ΓF_p, is nondecreasing and concave, and Fp,Γ(d)≤minpd, 22Γd.F_p, (d)≤ \pd,\,2 2 d \. (30) Proof. The full confidence set q∈[0,1]:nkl(h^j,n,q)≤ϕ(T)\q∈[0,1]:nkl( h_j,n,q)≤φ(T)\ is nonempty and compact. Its minimum is gj,ng_j,n, since it contains h^j,n h_j,n. Under the stated KL constraint, it also contains hj⋆h_j . Thus gj,n≤hj⋆g_j,n≤ h_j , and the definition of whj⋆w_h_j gives equation 29. The bound wp(x)≤pw_p(x)≤ p is immediate. Pinsker’s inequality applied to each of the two KL constraints gives |z−p|,|z−q|≤x/2|z-p|,|z-q|≤ x/2, hence p−q≤2xp-q≤ 2x, proving equation 28. As wpw_p is nondecreasing, the integrand defining Fp,ΓF_p, is nonincreasing in u, so Fp,ΓF_p, is nondecreasing and concave. Integrating the two bounds in equation 28 gives equation 30. ∎ Lemma 10 (Pathwise total variation). For any binary sequence and any fixed Γ>0 >0, let g0=0g_0=0 and let gng_n be its Bernoulli KL lower confidence bound with budget Γ . For every integer N≥1N≥ 1, ∑n=1N|gn−gn−1|≤−2log(1−e−Γ/N)≤2log(1+N/Γ). _n=1^N|g_n-g_n-1|≤-2 (1-e^- /N)≤ 2 (1+N/ ). (31) For KL-PBD, on every sample path, ∑t=2T‖h¯t−h¯t−1‖∞≤2∑j=1Klog(1+Nj,Tϕ(T)). _t=2^T\| h_t- h_t-1\|_∞≤ 2 _j=1^K (1+ N_j,Tφ(T) ). (32) Consequently, [∑t=2T‖h¯t−h¯t−1‖∞]≤2ΛT.E [ _t=2^T\| h_t- h_t-1\|_∞ ]≤ 2 _T. (33) Proof. We interpolate the number of successes s and failures f continuously. For s>0s>0, put m=s+fm=s+f, z=s/mz=s/m, and let q=q(s,f)∈(0,z)q=q(s,f)∈(0,z) solve mkl(z,q)=Γ.mkl(z,q)= . For s=0s=0, set q(0,f)=0q(0,f)=0, and set q(0,0)=0q(0,0)=0. These definitions agree continuously at the boundaries relevant below: q≤zq≤ z gives continuity as s↓0s 0 with f>0f>0, while kl(z,q)≤kl(1,q)=−logqkl(z,q) (1,q)=- q gives q≤e−Γ/mq≤ e^- /m as m↓0m 0. In particular, q(s,0)=e−Γ/sq(s,0)=e^- /s for s>0s>0. Consider first adding a success, keeping f fixed. In the interior, implicit differentiation gives ∂s[mkl(z,q)]q fixed=log(z/q),∂q[mkl(z,q)]=−m(z−q)q(1−q). ∂ s [mkl(z,q) ]_q fixed= (z/q), ∂ q [mkl(z,q) ]=- m(z-q)q(1-q). Writing a=log(z/q)>0a= (z/q)>0 and x=Γ/mx= /m, we obtain ∂q∂s=1−qmaea−1≤Γm2(eΓ/m−1). ∂ q∂ s= 1-qm ae^a-1≤ m^2(e /m-1). (34) Indeed, x=za+(1−z)log((1−z)/(1−q))≤ax=za+(1-z) ((1-z)/(1-q))≤ a, and a↦a/(ea−1)a a/(e^a-1) is decreasing on (0,∞)(0,∞). The same inequality holds when f=0f=0, either directly from q(s,0)=e−Γ/sq(s,0)=e^- /s or by a limit. Adding a failure while keeping s fixed gives instead the partial derivative log((1−z)/(1−q))≤0 ((1-z)/(1-q))≤ 0 at fixed q. Implicit differentiation therefore gives ∂q/∂f≤0∂ q/∂ f≤ 0. Continuity extends this monotonicity to f=0f=0 and s=0s=0. Thus successes can only increase the lower confidence bound, and failures can only decrease it. We interpolate each observation over its own interval m∈[n−1,n]m∈[n-1,n]. The total upward variation is at most the integral of the right-hand side of equation 34 over all intervals: ∑n=1N(gn−gn−1)+≤∫0NΓm2(eΓ/m−1)dm=−log(1−e−Γ/N). _n=1^N(g_n-g_n-1)_+≤ _0^N m^2(e /m-1)\,dm=- (1-e^- /N). By continuity of the lower confidence bound at m=0m=0 and s=0s=0, the integrals may start at 00. Since g0=0g_0=0 and gN≥0g_N≥ 0, downward variation is no larger than upward variation. Finally, ex≥1+xe^x≥ 1+x implies −log(1−e−x)≤log(1+1/x)- (1-e^-x)≤ (1+1/x), proving equation 31. Fix a self-confidence level j. Since Nj,tN_j,t is nondecreasing and h¯j,t=gj,Nj,t h_j,t=g_j,N_j,t, the triangle inequality gives ∑t=2T|h¯j,t−h¯j,t−1|≤∑n=1Nj,T|gj,n−gj,n−1|. _t=2^T | h_j,t- h_j,t-1 |≤ _n=1^N_j,T|g_j,n-g_j,n-1|. Applying equation 31, with an empty sum when Nj,T=0N_j,T=0, and using ‖h¯t−h¯t−1‖∞≤∑j=1K|h¯j,t−h¯j,t−1| \| h_t- h_t-1 \|_∞≤ _j=1^K | h_j,t- h_j,t-1 | proves equation 32. It remains to take expectations. Since Nj,TN_j,T counts completed self-confidence level j labels available before round T, it is bounded by the number of self-confidence level j arrivals before that round. Thus [Nj,T]≤ωjTE[N_j,T]≤ _jT. By concavity of x↦log(1+x)x (1+x), [∑t=2T‖h¯t−h¯t−1‖∞] \! [ _t=2^T \| h_t- h_t-1 \|_∞ ] ≤2∑j=1K[log(1+Nj,Tϕ(T))] ≤ 2 _j=1^KE\! [ (1+ N_j,Tφ(T) ) ] ≤2∑j=1Klog(1+ωjTϕ(T))=2ΛT. ≤ 2 _j=1^K (1+ _jTφ(T) )=2 _T. ∎ Lemma 11 (Integrated residual width). If p≤ap≤ a, then Ip(a)=0I_p(a)=0. For p>a>0p>a>0, Ip(a)≤min2a, 8p(1−p)(1a−1p)+12logpa.I_p(a)≤ \ 2a,\,8p(1-p) ( 1a- 1p )+12 pa \. (35) Proof. For p≤ap≤ a, Ip(a)=0I_p(a)=0. Pinsker’s bound gives, for a>0a>0, Ip(a)≤∫0∞(2/u−a)+u=2a.I_p(a)≤ _0^∞( 2/u-a)_+\,du= 2a. For the other bound, assume p>ap>a and put A=22p(1−p)A=2 2p(1-p). By Tonelli’s theorem, Ip(a)=∫apLebu>0:wp(1/u)>rr.I_p(a)= _a^pLeb\u>0:w_p(1/u)>r\\,dr. (36) If wp(1/u)>rw_p(1/u)>r, then equation 28 gives r<A/u+6/ur<A/ u+6/u. By 2xy≤x2+y22xy≤ x^2+y^2, applied with x=rx= r and y=A/ruy=A/ ru, Au≤r2+A22ru. A u≤ r2+ A^22ru. Consequently, r<r2+A22ru+6u,sou<A2r2+12r.r< r2+ A^22ru+ 6u, u< A^2r^2+ 12r. Thus Lebu>0:wp(1/u)>r≤A2r2+12r.Leb\u>0:w_p(1/u)>r\≤ A^2r^2+ 12r. This argument also applies when A=0A=0. Substituting this length bound into equation 36 yields Ip(a)≤8p(1−p)(1a−1p)+12logpa,I_p(a)≤ 8p(1-p) ( 1a- 1p )+12 pa, as claimed. In particular, all integrals used in the theorems are finite. ∎ B.5.2 Service-capacity stability bounds Recall from Appendix A the queue value Uth(b):=∑j=0KωjVth((b,j)).U_t^h(b):= _j=0^K _jV_t^h((b,j)). Since VT+1h((b,j))=D(b)V_T+1^h((b,j))=D(b) for every j=0,…,Kj=0,…,K and ∑j=0Kωj=1 _j=0^K _j=1, we have UT+1h=DU_T+1^h=D, independently of h, and the Bellman recursion can be written as Uth(b)=choldb+∑j=0Kωjmina∈(b,j)cexpa+Lh(j)(1−a)+[Ut+1h((b+a−Ct)+)].U_t^h(b)=c_holdb+ _j=0^K _j _a (b,j) \c_expa+L^h(j)(1-a)+E\! [U_t+1^h((b+a-C_t)^+) ] \. We first show that the service-capacity structure yields a stability property useful to control the effect of a pessimistic conditional accuracy vector on the Bellman recursion. Proposition 2 (Bellman stability). Let h,h~∈[0,1]Kh, h∈[0,1]^K. For every t≤Tt≤ T: (i) For arbitrary h,h~h, h, sp(Uth−Uth~)≤B¯ce‖h−h~‖∞sp (U_t^h-U_t h )≤ Bc_e\|h- h\|_∞, and |Mth(b)−Mth~(b)|≤ce‖h−h~‖∞ |M_t^h(b)-M_t h(b) |≤ c_e\|h- h\|_∞ for every b<B¯b< B. (i) If h≤h~h≤ h coordinatewise, then Uth−Uth~U_t^h-U_t h is nondecreasing in the queue state and, for every b<B¯b< B, 0≤Mth(b)−Mth~(b)≤ce‖h−h~‖∞.0≤ M_t^h(b)-M_t h(b)≤ c_e\|h- h\|_∞. Part (i) controls changes in the Bellman quantities under arbitrary conditional accuracy perturbations, while part (i) shows that a pessimistic conditional accuracy vector increases the Bellman marginal deferral costs and induces an ordered change in the queue values. Proof of Proposition 2. Let h,h~∈[0,1]Kh, h∈[0,1]^K and set ε:=ce‖h−h~‖∞ :=c_e\|h- h\|_∞. Recall from Appendix A the notations xth(q)=Uth(q)−Uth(q−1)x_t^h(q)=U_t^h(q)-U_t^h(q-1) and dth(b)=Fth(b+1)−Fth(b)d_t^h(b)=F_t^h(b+1)-F_t^h(b). We prove both statements by a common backward induction. For arbitrary h,h~h, h, we show that |xth(q)−xth~(q)|≤ε, |x_t^h(q)-x_t h(q) |≤ , (37) and, if h≤h~h≤ h coordinatewise, that 0≤xth(q)−xth~(q)≤ε,0≤ x_t^h(q)-x_t h(q)≤ , (38) for every t≤T+1t≤ T+1 and q=1,…,B¯q=1,…, B. At t=T+1t=T+1, both queue-value functions are equal. Hence xT+1h(q)=xT+1h~(q)x_T+1^h(q)=x_T+1 h(q) for every q=1,…,B¯q=1,…, B, so both claims hold. Suppose they hold at time t+1t+1. By equation 7, for every b<B¯b< B, dth(b)−dth~(b)=∑c=0bρC(c)[xt+1h(b+1−c)−xt+1h~(b+1−c)].d_t^h(b)-d_t h(b)= _c=0^b _C(c) [x_t+1^h(b+1-c)-x_t+1 h(b+1-c) ]. The weights are nonnegative and sum to at most one. Therefore, |dth(b)−dth~(b)|≤ε, |d_t^h(b)-d_t h(b) |≤ , and, when h≤h~h≤ h, 0≤dth(b)−dth~(b)≤ε.0≤ d_t^h(b)-d_t h(b)≤ . Recall the notation rjh=Lh(j)−cexpr_j^h=L^h(j)-c_exp. For j∈[K]j∈[K], we have rjh−rjh~=ce(h~j−hj)r_j^h-r_j h=c_e( h_j-h_j), while r0h=r0h~=−cexpr_0^h=r_0 h=-c_exp. Thus |rjh−rjh~|≤ε|r_j^h-r_j h|≤ for j∈0∪[K]j∈\0\∪[K] and these differences are nonnegative when h≤h~h≤ h. For 1≤q<B¯1≤ q< B, the preceding bounds and equation 9 give |xth(q)−xth~(q)| |x_t^h(q)-x_t h(q)| ≤∑j=0Kωj|meddth(q−1),dth(q),rjh−meddth~(q−1),dth~(q),rjh~| ≤ _j=0^K _j |med\d_t^h(q-1),d_t^h(q),r_j^h\-med\d_t h(q-1),d_t h(q),r_j h\ | ≤∑j=0Kωjε=ε, ≤ _j=0^K _j = , where we used that for any a,a~∈ℝ3a, a ^3, |meda1,a2,a3−meda~1,a~2,a~3|≤maxi=1,2,3|ai−a~i|. |med\a_1,a_2,a_3\-med\ a_1, a_2, a_3\ |≤ _i=1,2,3|a_i- a_i|. At q=B¯q= B, the same conclusion follows from equation 10 and |maxa1,a2−maxa~1,a~2|≤maxi=1,2|ai−a~i|.| \a_1,a_2\- \ a_1, a_2\|≤ _i=1,2|a_i- a_i|. This proves equation 37 at time t. If h≤h~h≤ h, each argument of the median or maximum under h is at least the corresponding argument under h~ h. Since increasing any argument cannot decrease either its median or its maximum, equation 9 and equation 10 also give xth(q)≥xth~(q)x_t^h(q)≥ x_t h(q) for q=1,…,B¯q=1,…, B. Combining this with the absolute bound proves equation 38 at time t, completing the common induction. Now define Zt(b):=Uth(b)−Uth~(b)Z_t(b):=U_t^h(b)-U_t h(b). By equation 37, |Zt(b)−Zt(b−1)|≤ε|Z_t(b)-Z_t(b-1)|≤ for b=1,…,B¯.b=1,…, B. Hence, sp(Zt)≤B¯ε=B¯ce‖h−h~‖∞.sp(Z_t)≤ B = Bc_e\|h- h\|_∞. Moreover, by equation 6, Mth(b)−Mth~(b)=dth(b)−dth~(b),M_t^h(b)-M_t h(b)=d_t^h(b)-d_t h(b), so |Mth(b)−Mth~(b)|≤ce‖h−h~‖∞,b<B¯.|M_t^h(b)-M_t h(b)|≤ c_e\|h- h\|_∞, b< B. This proves part (i). If h≤h~h≤ h, equation 38 shows that ZtZ_t is nondecreasing in the queue state. The corresponding bound on the smoothed increments also gives 0≤Mth(b)−Mth~(b)≤ce‖h−h~‖∞,b<B¯.0≤ M_t^h(b)-M_t h(b)≤ c_e\|h- h\|_∞, b< B. This proves part (i). ∎ We next record a uniform bound on the gaps Δt _t. Lemma 12 (Uniform gap bound). For every conditional accuracy vector h, every t≤T+1t≤ T+1, and every q=1,…,B¯q=1,…, B, 0≤Uth(q)−Uth(q−1)≤qcholdμ+(ce−cexp)+.0≤ U_t^h(q)-U_t^h(q-1)≤ q\,c_holdμ+(c_e-c_exp)_+. (39) Consequently, for every t≤Tt≤ T and every b<B¯b< B, cexp≤Mth(b)≤maxce,cexp+choldB¯μ.c_exp≤ M_t^h(b)≤ \c_e,c_exp\+ c_hold Bμ. (40) In particular, 0≤Δt≤Δ0≤ _t≤ on every sample path. Proof. The lower bound in equation 39 follows from Lemma 3. For the upper bound, fix t and q≥1q≥ 1, and couple two systems starting from queue states q and q−1q-1, respectively, using the same self-confidence level and service realizations through round T, and the same service realizations thereafter. Let the lower system follow an optimal policy. To account for the terminal value, we regard both systems as continuing after round T with no further arrivals and incurring holding costs until their queues are empty; by definition, the expected cost of this continuation from queue state b is D(b)D(b). While the queues differ, they differ by exactly one job. If the lower system takes action 00, let the upper system also take action 00. If the lower system defers, let the upper system take action 00 instead. In the latter case, the two pre-service queue states after the deferral decision coincide, and hence the queues couple at the end of that round. Once they have coupled, let the upper system follow the same actions as the lower system. Before coupling, the difference in holding cost is choldc_hold per round. If the lower system never triggers coupling by deferring, the queues couple once q rounds with Ct≥1C_t≥ 1 have occurred. The expected time required for q such service opportunities is q/μq/μ. If coupling instead occurs because the lower system defers, the only additional non-holding cost incurred by the upper system on that round is ce(1−hJt)−cexp≤(ce−cexp)+.c_e(1-h_J_t)-c_exp≤(c_e-c_exp)_+. This can occur at most once. Thus the constructed algorithm for the upper system has expected cost at most Uth(q−1)+qcholdμ+(ce−cexp)+.U_t^h(q-1)+ q\,c_holdμ+(c_e-c_exp)_+. Optimality of Uth(q)U_t^h(q) proves equation 39. By equation 7, dth(b)=∑c=0bρC(c)xt+1h(b+1−c).d_t^h(b)= _c=0^b _C(c)\,x_t+1^h(b+1-c). Using equation 39 and b+1≤B¯b+1≤ B, 0≤dth(b)≤choldB¯μ+(ce−cexp)+.0≤ d_t^h(b)≤ c_hold Bμ+(c_e-c_exp)_+. Together with equation 6, this yields equation 40. Finally, for j∈[K]j∈[K], ce(1−hj⋆)∈[0,ce]c_e(1-h_j )∈[0,c_e], while equation 40 gives 0≤Mt⋆(b)≤Δ0≤ M_t (b)≤ . Hence, for b<B¯b< B, |ce(1−hj⋆)−Mt⋆(b)|≤Δ. |c_e(1-h_j )-M_t (b) |≤ . On no-query or full-queue rounds Δt=0 _t=0. Thus the definition of Δt _t gives 0≤Δt≤Δ0≤ _t≤ on every sample path. ∎ B.5.3 Model comparison and pending labels Lemma 13 (Bellman comparison). Let h~t∈[0,1]K h_t∈[0,1]^K be ℱt−1F_t-1-measurable and satisfy h¯t≤h~t h_t≤ h_t coordinatewise for t<σt<σ. For j∈[K]j∈[K], define ej,t:=ce(h~j,t−h¯j,t)e_j,t:=c_e( h_j,t- h_j,t), set e0,t:=0e_0,t:=0, and let Δ~t:=Qth~t(St,At)−Vth~t(St) _t:=Q_t h_t(S_t,A_t)-V_t h_t(S_t) be the decision gap of the learner’s action under h~t h_t. Then [∑t<σΔ~t]≤ \! [ _t<σ _t ]≤ [∑t<σeJt,tAt=1] \! [ _t<σe_J_t,t1\A_t=1\ ] +B¯ce(1+[∑t=2σ−1(‖h¯t−h¯t−1‖∞+‖h~t−h~t−1‖∞)]). + Bc_e (1+E\! [ _t=2^σ-1 (\| h_t- h_t-1\|_∞+\| h_t- h_t-1\|_∞ ) ] ). (41) Proof. Fix t<σt<σ. For t≤s≤T+1t≤ s≤ T+1, define Zst(b):=Ush¯t(b)−Ush~t(b)Z_s^t(b):=U_s h_t(b)-U_s h_t(b). By Proposition 2(i), ZstZ_s^t is nondecreasing in b. Define the centered potential Φst(b):=Zst(B¯)−Zst(b)≥0 _s^t(b):=Z_s^t( B)-Z_s^t(b)≥ 0, and write e¯t:=∑j=1Kωjej,t e_t:= _j=1^K _je_j,t. At queue state B¯ B, action 00 is the only feasible action. Therefore, for t≤s≤Tt≤ s≤ T, Zst(B¯)=e¯t+[Zs+1t((B¯−Cs)+)∣ℱt−1]≤e¯t+Zs+1t(B¯),Z_s^t( B)= e_t+E\! [Z_s+1^t(( B-C_s)^+) _t-1 ]≤ e_t+Z_s+1^t( B), and hence Zst(B¯)−e¯t≤Zs+1t(B¯).Z_s^t( B)- e_t≤ Z_s+1^t( B). (42) Since the KL-PBD action at round t is optimal under h¯t h_t, the definitions of QthQ_t^h and VthV_t^h give Δ~t= _t= Vth¯t(St)−Vth~t(St)−eJt,t(1−At) V_t h_t(S_t)-V_t h_t(S_t)-e_J_t,t(1-A_t) −[Zt+1t(Bt+1)∣ℱt−1,Jt,At]. -E\! [Z_t+1^t(B_t+1) _t-1,J_t,A_t ]. (43) Because JtJ_t is independent of ℱt−1F_t-1 with distribution ω, [eJt,t∣ℱt−1]=∑j=0Kωjej,t=e¯t,E[e_J_t,t _t-1]= _j=0^K _je_j,t= e_t, and [Vth¯t(St)−Vth~t(St)∣ℱt−1]=Ztt(Bt).E\! [V_t h_t(S_t)-V_t h_t(S_t) _t-1 ]=Z_t^t(B_t). Taking conditional expectation in equation 43 therefore yields [Δ~t∣ℱt−1]= [ _t _t-1]= Ztt(Bt)−e¯t+[eJt,tAt=1∣ℱt−1] Z_t^t(B_t)- e_t+E\! [e_J_t,t1\A_t=1\ _t-1 ] −[Zt+1t(Bt+1)∣ℱt−1]. -E\! [Z_t+1^t(B_t+1) _t-1 ]. Using Ztt(Bt)=Ztt(B¯)−Φt(Bt)Z_t^t(B_t)=Z_t^t( B)- _t^t(B_t) and equation 42, we obtain [Δ~t∣ℱt−1]≤ [ _t _t-1]≤ [eJt,tAt=1∣ℱt−1] \! [e_J_t,t1\A_t=1\ _t-1 ] +[Φt+1t(Bt+1)∣ℱt−1]−Φt(Bt). +E\! [ _t+1^t(B_t+1) _t-1 ]- _t^t(B_t). (44) Since t<σ∈ℱt−1\t<σ\ _t-1, we may multiply equation 44 by t<σ1\t<σ\, take expectations, and sum over t. Let m:=minT,σ−1m:= \T,σ-1\. On paths with m≥1m≥ 1, ∑t=1m[Φt+1t(Bt+1)−Φt(Bt)]= _t=1^m [ _t+1^t(B_t+1)- _t^t(B_t) ]= −Φ11(B1)+Φm+1m(Bm+1) - _1^1(B_1)+ _m+1^m(B_m+1) +∑t=2m[Φt−1(Bt)−Φt(Bt)]. + _t=2^m [ _t^t-1(B_t)- _t^t(B_t) ]. The first term is nonpositive. If m=Tm=T, then ZT+1T=D−D=0Z_T+1^T=D-D=0, and therefore ΦT+1T≡0 _T+1^T≡ 0. Otherwise, m+1=σm+1=σ, and Proposition 2(i) gives 0≤Φσ−1(Bσ)≤B¯ce‖h¯σ−1−h~σ−1‖∞≤B¯ce.0≤ _σ^σ-1(B_σ)≤ Bc_e\| h_σ-1- h_σ-1\|_∞≤ Bc_e. It remains to control the change of potential when the conditional accuracy vectors are updated. Successive vectors need not be coordinatewise ordered, so here we use the two-sided stability in Proposition 2(i). For 2≤t<σ2≤ t<σ, |Φt−1(Bt)−Φt(Bt)| | _t^t-1(B_t)- _t^t(B_t)| =|(Ztt−1−Ztt)(B¯)−(Ztt−1−Ztt)(Bt)| = |(Z_t^t-1-Z_t^t)( B)-(Z_t^t-1-Z_t^t)(B_t) | ≤sp(Ztt−1−Ztt) (Z_t^t-1-Z_t^t) ≤sp(Uth¯t−1−Uth¯t)+sp(Uth~t−1−Uth~t) (U_t h_t-1-U_t h_t )+sp (U_t h_t-1-U_t h_t ) ≤B¯ce(‖h¯t−h¯t−1‖∞+‖h~t−h~t−1‖∞). ≤ Bc_e (\| h_t- h_t-1\|_∞+\| h_t- h_t-1\|_∞ ). Here the third line uses subadditivity of the span. Combining these bounds controls the potential sum by B¯ce(1+∑t=2σ−1(‖h¯t−h¯t−1‖∞+‖h~t−h~t−1‖∞)). Bc_e (1+ _t=2^σ-1 (\| h_t- h_t-1\|_∞+\| h_t- h_t-1\|_∞ ) ). On paths with m=0m=0, the potential sum is empty and the same upper bound holds. Taking expectations and substituting into the summed one-step inequality proves equation 41. ∎ Lemma 14 (Pending-label bound for deferred queries). For j∈[K]j∈[K] and any nonnegative nonincreasing function f:[0,∞)→[0,∞)f:[0,∞)→[0,∞), [∑t<σJt=j,At=1f(Nj,t)]≤minf(0)ωjT,B¯f(0)+∫0ωjTf(u)u.E\! [ _t<σ1\J_t=j,A_t=1\f(N_j,t) ]≤ \f(0) _jT,\ Bf(0)+ _0 _jTf(u)\,du \. Proof. Fix a self-confidence level j, write Dj:=Dj,σ−1D_j:=D_j,σ-1, and enumerate its deferrals before σ as tj,1,…,tj,Djt_j,1,…,t_j,D_j. Immediately before the ℓ -th such deferral, ℓ−1 -1 previous self-confidence level j jobs have been submitted. Since deferral is feasible at tj,ℓt_j, , at most B¯−1 B-1 of them can still be pending. Hence Nj,tj,ℓ≥(ℓ−B¯)+.N_j,t_j, ≥( - B)_+. (45) For the first at most B¯ B deferrals, use f(Nj,tj,ℓ)≤f(0)f(N_j,t_j, )≤ f(0). For ℓ>B¯ > B, monotonicity of f and equation 45 give f(Nj,tj,ℓ)≤f(ℓ−B¯)f(N_j,t_j, )≤ f( - B). Therefore, ∑ℓ=1Djf(Nj,tj,ℓ)≤f(0)minDj,B¯+∑n=1(Dj−B¯)+f(n). _ =1^D_jf(N_j,t_j, )≤ f(0) \D_j, B\+ _n=1^(D_j- B)_+f(n). Since f is nonincreasing, f(n)≤∫n−1nf(u)uf(n)≤ _n-1^nf(u)\,du. Thus ∑ℓ=1Djf(Nj,tj,ℓ) _ =1^D_jf(N_j,t_j, ) ≤f(0)minDj,B¯+∫0(Dj−B¯)+f(u)u ≤ f(0) \D_j, B\+ _0^(D_j- B)_+f(u)\,du ≤B¯f(0)+∫0Djf(u)du. ≤ Bf(0)+ _0^D_jf(u)\,du. Every such deferral occurs on a self-confidence level j arrival, so [Dj]≤ωjTE[D_j]≤ _jT. The map d↦∫0df(u)ud _0^df(u)\,du is nondecreasing and concave because f is nonnegative and nonincreasing. Consequently, Jensen’s inequality gives [∑ℓ=1Djf(Nj,tj,ℓ)] \! [ _ =1^D_jf(N_j,t_j, ) ] ≤B¯f(0)+[∫0Djf(u)du] ≤ Bf(0)+E\! [ _0^D_jf(u)\,du ] ≤B¯f(0)+∫0[Dj]f(u)u ≤ Bf(0)+ _0^E[D_j]f(u)\,du ≤B¯f(0)+∫0ωjTf(u)du. ≤ Bf(0)+ _0 _jTf(u)\,du. On the other hand, f(Nj,t)≤f(0)f(N_j,t)≤ f(0) gives [∑ℓ=1Djf(Nj,tj,ℓ)]≤f(0)[Dj]≤f(0)ωjT.E\! [ _ =1^D_jf(N_j,t_j, ) ]≤ f(0)E[D_j]≤ f(0) _jT. Taking the better of these two bounds proves the claim. ∎ B.6 Extension to unknown arrival and service distributions The arrival distribution ω and the truncated service distribution defined below can be learned without additional deferrals when their realizations are observed independently of the deferral decisions. In this section, we show that using KL-PBD with plug-in empirical estimates of these distributions preserves the worst-case regret rate of Theorem 2, including its dependence on the arrival distribution ω. Throughout this section, assume chold>0c_hold>0. For unknown ρC _C, additionally assume that C¯t:=minCt,B¯ C_t:= \C_t, B\ is observed at the end of every round, including rounds with an empty queue. Write ρ¯c:=ℙ(C¯t=c) ρ_c:=P( C_t=c) for c=0,…,B¯c=0,…, B, so μ=1−ρ¯0>0μ=1- ρ_0>0, and set Ω:=ℙ(Jt≠0)=1−ω0 :=P(J_t≠ 0)=1- _0. Only this truncated service distribution affects the queue dynamics and terminal cost. We write Qth,ω,ρ¯Q_t^h,ω, ρ, Vth,ω,ρ¯V_t^h,ω, ρ and Mth,ω,ρ¯M_t^h,ω, ρ for the corresponding Bellman quantities. Estimators of ω and ρ¯ ρ. For t≥1t≥ 1 and t≥2t≥ 2 respectively, we form the estimates ω^j,t=1t∑s≤tJs=j,j=0,…,K,ρ^c,t=1t−1∑s<tC¯s=c,c=0,…,B¯. ω_j,t= 1t _s≤ t1\J_s=j\, j=0,…,K, ρ_c,t= 1t-1 _s<t1\ C_s=c\, c=0,…, B. (46) The arrival estimate uses the current level JtJ_t, which is observed before the decision; the capacity C¯t C_t is realized after it. When ρ¯ ρ is known, set ρ~t:=ρ¯ ρ_t:= ρ. When service is estimated and t≥2t≥ 2, set μ^t:=1−ρ^0,t μ_t:=1- ρ_0,t and, if μ^t>0 μ_t>0, set ρ~t:=ρ^t ρ_t:= ρ_t. Whenever ρ~t ρ_t is defined, we solve the remaining-horizon Bellman recursion using ω^t ω_t and ρ~t ρ_t, with terminal cost Dρ~tD_ ρ_t, and choose At=Jt∈[K],h¯Jt,t<τ^t(Bt),τ^t(b):=1−Mth¯t,ω^t,ρ~t(b)ce,A_t=1 \J_t∈[K],\ h_J_t,t< τ_t(B_t) \, τ_t(b):=1- M_t h_t, ω_t, ρ_t(b)c_e, (47) with the convention Mt⋅(B¯)=+∞M_t^·( B)=+∞ of Section 3. If service is estimated and either t=1t=1 or μ^t=0 μ_t=0, we set At=0A_t=0 instead. Define ΨT(ρ¯):=∑c=0B¯minρ¯cT,ρ¯c(1−ρ¯c)T,ST(ω):=∑j=1KminωjT,ωjT. _T( ρ):= _c=0 B \ ρ_cT, ρ_c(1- ρ_c)T\, S_T(ω):= _j=1^K \ _jT, _jT\. Denote the known-parameter worst-case bound by ℬT:=Kκϕ(T)+ce∑j=1KminωjT,B¯+22ωjTϕ(T)+B¯ce[1+2∑j=1Klog(1+ωjTϕ(T))]B_T:=Kκφ(T)+c_e _j=1^K \ _jT, B+2 2 _jTφ(T)\+ Bc_e [1+2 _j=1^K (1+ _jTφ(T) ) ] (48) where we recall κ=4eΔ=4e(maxce,cexp+choldB¯/μ)κ=4e =4e( \c_e,c_exp\+c_hold B/μ). For μ0>0 _0>0, set Lμ0:=B¯choldμ0+(ce−cexp)+,aω(μ0):=ceLμ0chold,aC(μ0):=Lμ02chold+Lμ0+choldB¯2μ02.L_ _0:= Bc_hold _0+(c_e-c_exp)_+, a_ω( _0):= c_eL_ _0c_hold, a_C( _0):= L_ _0^2c_hold+L_ _0+ c_hold B^2 _0^2. (49) In the rest of this section, we prove the following theorem. Theorem 7 (Regret of KL-PBD for unknown ω and ρC _C). For T≥3T≥ 3, the plug-in variants of KL-PBD defined above satisfy RT R_T ≤ℬT+4aω(μ)ST(ω), _T+4\,a_ω(μ)S_T(ω), if ρ¯ is known, $ ρ$ is known, (50) RT R_T ≤ℬT+ΩΔ(1+8/μ)+4aω(μ/2)ST(ω)+2ΩaC(μ/2)ΨT(ρ¯), _T+ (1+8/μ)+4\,a_ω(μ/2)S_T(ω)+2 \,a_C(μ/2) _T( ρ), if ρ¯ is estimated. $ ρ$ is estimated. (51) Both bounds are O~(K+ST(ω)) O(K+S_T(ω)), uniformly over arrival distributions. Theorem 7 shows that learning these distributions requires no additional exploration under the stated observation assumptions and does not worsen the worst-case rate of Theorem 2. Proof of Theorem 7. All Bellman quantities used for comparison below are evaluated at the fixed true distributions (ω,ρ¯)(ω, ρ). We use the history from Appendix A and set et:=Qth¯t,ω,ρ¯(St,At)−Vth¯t,ω,ρ¯(St)≥0.e_t:=Q_t h_t,ω, ρ(S_t,A_t)-V_t h_t,ω, ρ(S_t)≥ 0. By Lemma 12, et≤ΔJt≠0e_t≤ 1\J_t≠ 0\; also et=0e_t=0 when Bt=B¯B_t= B. Whenever AtA_t is computed from equation 47, equation 4 gives et≤Jt≠0supb<B¯|Mth¯t,ω^t,ρ~t(b)−Mth¯t,ω,ρ¯(b)|.e_t 1\J_t≠ 0\ _b< B |M_t h_t, ω_t, ρ_t(b)-M_t h_t,ω, ρ(b) |. (52) The proof of Theorem 2 then extends as RT≤ℬT+∑t=1T[et].R_T _T+ _t=1^TE[e_t]. (53) To see this, we use the proof of Lemma 13 with h~t=h⋆ h_t=h . The only use of exact optimality under h¯t h_t is in equation 43. Here, by definition of ete_t, Qth¯t,ω,ρ¯(St,At)=Vth¯t,ω,ρ¯(St)+et,Q_t h_t,ω, ρ(S_t,A_t)=V_t h_t,ω, ρ(S_t)+e_t, so equation 43 has the additional term ete_t, and equation 44 has the additional term [et∣ℱt−1]E[e_t _t-1]. The potential bounds from Proposition 2 and the subsequent telescoping argument are unchanged. Therefore, equation 41, and hence the decomposition in equation 20, hold with the additional term [∑t<σet]E[ _t<σe_t]. The bounds in Lemmas 5, 6 and 7 still apply: the concentration, pending-label counting and lower-confidence-bound variation arguments in Lemmas 8, 14 and 10 hold for adaptive feasible actions. The same simplification as in the proof of Theorem 2, together with et≥0e_t≥ 0, proves equation 53. No variation bound for ω^t ω_t or ρ^t ρ_t is needed, because the Bellman comparison in the proof is carried out under the fixed true distributions (ω,ρ¯)(ω, ρ); the effect of using the empirical distributions to choose AtA_t is isolated in ete_t. For an empirical distribution v^n v_n based on n i.i.d. samples from v, we have [v^i,n]=viE[ v_i,n]=v_i and Var(v^i,n)=vi(1−vi)/nVar( v_i,n)=v_i(1-v_i)/n. Cauchy–Schwarz and |v^i,n−vi|≤v^i,n+vi| v_i,n-v_i|≤ v_i,n+v_i therefore give [|v^i,n−vi|]≤min2vi,vi(1−vi)/n,∑n=1T[|v^i,n−vi|]≤2minviT,vi(1−vi)T,E[| v_i,n-v_i|]≤ \2v_i, v_i(1-v_i)/n\, _n=1^TE[| v_i,n-v_i|]≤ 2 \v_iT, v_i(1-v_i)T\, (54) where the second inequality uses ∑n=1Tn−1/2≤2T _n=1^Tn^-1/2≤ 2 T. Since ω^t ω_t and ω are probability vectors, |ω^0,t−ω0|=|∑j=1K(ω^j,t−ωj)|| ω_0,t- _0|= | _j=1^K( ω_j,t- _j) |, so ‖ω^t−ω‖1≤2∑j=1K|ω^j,t−ωj|,∑t=1T[‖ω^t−ω‖1]≤4ST(ω).\| ω_t-ω\|_1≤ 2 _j=1^K| ω_j,t- _j|, _t=1^TE[\| ω_t-ω\|_1]≤ 4S_T(ω). (55) Similarly, since ρ^t ρ_t is based on t−1t-1 samples, equation 54 gives ∑t=2T[‖ρ^t−ρ¯‖1]≤2ΨT(ρ¯). _t=2^TE[\| ρ_t- ρ\|_1]≤ 2 _T( ρ). (56) For known ρ¯ ρ, applying Lemma 15 with μ0=μ _0=μ in equation 52 and bounding the indicator by one gives [et]≤aω(μ)[‖ω^t−ω‖1]E[e_t]≤ a_ω(μ)E[\| ω_t-ω\|_1] for every t≥1t≥ 1. Equation 53 and equation 55 prove equation 50. For estimated ρ¯ ρ, let Gt:=μ^t≥μ/2G_t:=\ μ_t≥μ/2\ for t≥2t≥ 2. On GtG_t, Lemma 15 applies with μ0=μ/2 _0=μ/2; on GtcG_t^c, we have et≤ΔJt≠0e_t≤ 1\J_t≠ 0\. Both GtG_t and ρ^t ρ_t are ℱt−1F_t-1-measurable, whereas JtJ_t is independent of ℱt−1F_t-1. Bounding Jt≠01\J_t≠ 0\ by one in the ω^t ω_t term and using independence in the other two terms therefore gives, for t≥2t≥ 2, [et]≤aω(μ/2)[‖ω^t−ω‖1]+ΩaC(μ/2)[‖ρ^t−ρ¯‖1]+ΩΔℙ(Gtc).E[e_t]≤ a_ω(μ/2)E[\| ω_t-ω\|_1]+ \,a_C(μ/2)E[\| ρ_t- ρ\|_1]+ (G_t^c). The binomial lower-tail bound gives ℙ(Gtc)≤e−(t−1)μ/8P(G_t^c)≤ e^-(t-1)μ/8, so that ∑t=2Tℙ(Gtc)≤8/μ _t=2^TP(G_t^c)≤ 8/μ. Together with [e1]≤ΩΔE[e_1]≤ , equation 55, equation 56 and equation 53 prove equation 51. Finally, for T≥1T≥ 1, ST(ω)≥ΩT,ΩΨT(ρ¯)≤ΩB¯T≤B¯ST(ω).S_T(ω)≥ T, _T( ρ)≤ BT≤ B\,S_T(ω). Indeed, ST(ω)≥ΩTS_T(ω)≥ T follows from minωjT,ωjT≥ωjT \ _jT, _jT\≥ _j T for every j∈[K]j∈[K]. Moreover, Cauchy–Schwarz gives ∑c=0B¯ρ¯c(1−ρ¯c)≤B¯ _c=0 B ρ_c(1- ρ_c)≤ B and hence ΨT(ρ¯)≤B¯T _T( ρ)≤ BT. Thus the additional terms are O(ST(ω))O(S_T(ω)), with constants independent of ω. If Ω=0 =0, all actions are forced and regret is zero. ∎ Lemma 15 (Uniform stability of the marginal cost). Fix h∈[0,1]Kh∈[0,1]^K, with h0=1h_0=1. For two models22 2 Each model uses its own terminal draining cost. (ω,ρ)(ω,ρ) and (ω′,ρ′)(ω ,ρ ) satisfying 1−ρ0,1−ρ0′≥μ0>01- _0,1-ρ _0≥ _0>0, for every t≤Tt≤ T and b<B¯b< B, |Mth,ω,ρ(b)−Mth,ω′,ρ′(b)|≤aω(μ0)‖ω−ω′‖1+aC(μ0)‖ρ−ρ′‖1. |M_t^h,ω,ρ(b)-M_t^h,ω ,ρ (b) |≤ a_ω( _0)\|ω-ω \|_1+a_C( _0)\|ρ-ρ \|_1. (57) Proof. We use the notation xtx_t, dtd_t and rjr_j from the proof of Lemma 3, suppressing model superscripts. The recursions in equation 7, equation 9 and equation 10 apply with ρC(c)=ρc _C(c)= _c for c<B¯c< B and terminal cost DρD_ρ. We interpolate linearly between the two models: (ωλ,ρλ):=(1−λ)(ω,ρ)+λ(ω′,ρ′),λ∈[0,1].(ω^λ,ρ^λ):=(1-λ)(ω,ρ)+λ(ω ,ρ ), λ∈[0,1]. Since 1−ρ0λ≥μ01- _0^λ≥ _0, equation 39 gives 0≤xt(i)≤Lμ00≤ x_t(i)≤ L_ _0 for every λ∈[0,1]λ∈[0,1]. We bound dMt(b)/dλdM_t(b)/dλ uniformly in λ and then integrate over [0,1][0,1]. For the model with service law ρλρ^λ, define g(i):=Dρλ(i)−Dρλ(i−1)g(i):=D_ρ^λ(i)-D_ρ^λ(i-1). Subtracting the defining recursions for Dρλ(i)D_ρ^λ(i) and Dρλ(i−1)D_ρ^λ(i-1) gives, for 1≤i≤B¯1≤ i≤ B, (1−ρ0λ)g(i)=chold+∑c=1i−1ρcλg(i−c),(1- _0^λ)g(i)=c_hold+ _c=1^i-1 _c^λg(i-c), where the sum is empty for i=1i=1. Hence each g(i)g(i) is absolutely continuous in λ. The recursions in equation 7, equation 9 and equation 10 then imply by backward induction that all dt(b)d_t(b) and xt(i)x_t(i) are absolutely continuous in λ, since these recursions involve only finite sums, products, and the Lipschitz maps min , max and medmed. Below, derivatives are taken at values of λ where these functions are differentiable, which holds almost everywhere. Let F:=ce‖ω′−ω‖1+Lμ0‖ρ′−ρ‖1,G:=choldB¯2μ02‖ρ′−ρ‖1.F:=c_e\|ω -ω\|_1+L_ _0\|ρ -ρ\|_1, G:= c_hold B^2 _0^2\|ρ -ρ\|_1. We first bound dg(i)/dλdg(i)/dλ. Since ∑c=1i−1ρcλ≤1−ρ0λ _c=1^i-1 _c^λ≤ 1- _0^λ and 1−ρ0λ≥μ01- _0^λ≥ _0, the preceding recursion gives g(i)≤chold/μ0+maxk<ig(k)g(i)≤ c_hold/ _0+ _k<ig(k), where the maximum over an empty set is zero, so that 0≤g(i)≤ichold/μ00≤ g(i)≤ ic_hold/ _0. Differentiating the recursion gives (1−ρ0λ)dg(i)dλ=∑c=1i−1ρcλdg(i−c)dλ+∑c=0i−1(ρc′−ρc)g(i−c),(1- _0^λ) dg(i)dλ= _c=1^i-1 _c^λ dg(i-c)dλ+ _c=0^i-1(ρ _c- _c)g(i-c), so that |dg(i)dλ|≤maxk<i|dg(k)dλ|+choldB¯μ02‖ρ′−ρ‖1. | dg(i)dλ |≤ _k<i | dg(k)dλ |+ c_hold B _0^2\|ρ -ρ\|_1. A second induction on i≤B¯i≤ B gives |dg(i)/dλ|≤G |dg(i)/dλ |≤ G. We next bound dxt(i)/dλdx_t(i)/dλ recursively in terms of dxt+1(k)/dλdx_t+1(k)/dλ. Define ψj,t(i):=meddt(i−1),dt(i),rj,i<B¯,maxdt(B¯−1),rj,i=B¯. _j,t(i):= casesmed\d_t(i-1),d_t(i),r_j\,&i< B,\\[2.84526pt] \d_t( B-1),r_j\,&i= B. cases By equation 9 and equation 10, xt(i)=chold+∑j=0Kωjλψj,t(i),dxt(i)dλ=∑j=0K(ωj′−ωj)ψj,t(i)+∑j=0Kωjλdψj,t(i)dλ.x_t(i)=c_hold+ _j=0^K _j^λ _j,t(i), dx_t(i)dλ= _j=0^K(ω _j- _j) _j,t(i)+ _j=0^K _j^λ d _j,t(i)dλ. (58) At almost every λ, the derivative of each median or maximum equals the derivative of one of the arguments attaining its value (when several arguments attain it on a set of positive measure, their derivatives agree almost everywhere there). For each j, choose such an argument, and let t,iJ_t,i be the set of j for which it is some dt(bj)d_t(b_j). Since h is fixed, drj/dλ=0dr_j/dλ=0. Hence ψj,t(i)=dt(bj) _j,t(i)=d_t(b_j) and dψj,t(i)/dλ=ddt(bj)/dλd _j,t(i)/dλ=dd_t(b_j)/dλ for j∈t,ij _t,i, while ψj,t(i)=rj _j,t(i)=r_j and dψj,t(i)/dλ=0d _j,t(i)/dλ=0 for j∉t,ij _t,i. By equation 7, ddt(b)dλ=∑c=0bρcλdxt+1(b+1−c)dλ+∑c=0b(ρc′−ρc)xt+1(b+1−c). dd_t(b)dλ= _c=0^b _c^λ dx_t+1(b+1-c)dλ+ _c=0^b(ρ _c- _c)x_t+1(b+1-c). (59) Substituting equation 59 into equation 58 for j∈t,ij _t,i gives dxt(i)dλ=∑k=1B¯at,i,kdxt+1(k)dλ+ft(i), dx_t(i)dλ= _k=1 Ba_t,i,k dx_t+1(k)dλ+f_t(i), (60) where at,i,k:=∑j∈t,i∑c=0bjωjλρcλk=bj+1−c≥0,∑k=1B¯at,i,k=∑j∈t,iωjλ∑c=0bjρcλ≤1,a_t,i,k:= _j _t,i _c=0^b_j _j^λ _c^λ1\k=b_j+1-c\≥ 0, _k=1 Ba_t,i,k= _j _t,i _j^λ _c=0^b_j _c^λ≤ 1, and ft(i):=∑j=0K(ωj′−ωj)ψj,t(i)+∑j∈t,iωjλ∑c=0bj(ρc′−ρc)xt+1(bj+1−c).f_t(i):= _j=0^K(ω _j- _j) _j,t(i)+ _j _t,i _j^λ _c=0^b_j(ρ _c- _c)x_t+1(b_j+1-c). The second term of ft(i)f_t(i) is at most Lμ0‖ρ′−ρ‖1L_ _0\|ρ -ρ\|_1 in absolute value, since 0≤xt+1(k)≤Lμ00≤ x_t+1(k)≤ L_ _0 and ∑jωjλ≤1 _j _j^λ≤ 1. For the first, both meddt(i−1),dt(i),rmed\d_t(i-1),d_t(i),r\ and maxdt(B¯−1),r \d_t( B-1),r\ are 11-Lipschitz in r for fixed dtd_t, and rj=Lh(j)−cexpr_j=L^h(j)-c_exp with Lh(j)∈[0,ce]L^h(j)∈[0,c_e], so the values ψj,t(i) _j,t(i) lie in an interval of length at most cec_e. Since ∑j(ωj′−ωj)=0 _j(ω _j- _j)=0, the first term is at most ce‖ω′−ω‖1c_e\|ω -ω\|_1 in absolute value. Hence |ft(i)|≤F|f_t(i)|≤ F. The same coefficients satisfy xt(i)≥chold+∑k=1B¯at,i,kxt+1(k).x_t(i)≥ c_hold+ _k=1 Ba_t,i,kx_t+1(k). (61) Indeed, for j∈t,ij _t,i, ψj,t(i)=dt(bj)=∑c=0bjρcλxt+1(bj+1−c) _j,t(i)=d_t(b_j)= _c=0^b_j _c^λx_t+1(b_j+1-c), so these terms of equation 58 sum to ∑kat,i,kxt+1(k) _ka_t,i,kx_t+1(k). For j∉t,ij _t,i, ψj,t(i)=rj≥0 _j,t(i)=r_j≥ 0: rjr_j attaining the median (resp. maximum) implies rj≥dt(i−1)r_j≥ d_t(i-1) (resp. rj≥dt(B¯−1)r_j≥ d_t( B-1)), which is nonnegative by equation 8. We now prove by backward induction that |dxt(i)dλ|≤Fcholdxt(i)+G. | dx_t(i)dλ |≤ Fc_holdx_t(i)+G. (62) At t=T+1t=T+1, this follows from xT+1(i)=g(i)≥0x_T+1(i)=g(i)≥ 0 and |dg(i)/dλ|≤G |dg(i)/dλ |≤ G. If equation 62 holds at t+1t+1, then equation 60 and equation 61 give |dxt(i)dλ|≤∑k=1B¯at,i,k(Fcholdxt+1(k)+G)+F≤Fchold(xt(i)−chold)+G+F=Fcholdxt(i)+G. | dx_t(i)dλ |≤ _k=1 Ba_t,i,k ( Fc_holdx_t+1(k)+G )+F≤ Fc_hold (x_t(i)-c_hold )+G+F= Fc_holdx_t(i)+G. Thus |dxt(i)/dλ|≤FLμ0/chold+G |dx_t(i)/dλ |≤ FL_ _0/c_hold+G. Finally, Mt(b)=cexp+dt(b)M_t(b)=c_exp+d_t(b) by equation 6, so equation 59 gives |dMt(b)dλ|≤FLμ0chold+G+Lμ0‖ρ′−ρ‖1=aω(μ0)‖ω′−ω‖1+aC(μ0)‖ρ′−ρ‖1. | dM_t(b)dλ |≤ FL_ _0c_hold+G+L_ _0\|ρ -ρ\|_1=a_ω( _0)\|ω -ω\|_1+a_C( _0)\|ρ -ρ\|_1. Integrating over λ∈[0,1]λ∈[0,1] proves equation 57. ∎ Appendix C Proofs for Section 5 C.1 Proof sketches To ease navigation within this appendix, we first provide high-level proof sketches of our instance-dependent and minimax lower bounds. They are proven in Appendices C.3 and C.5 respectively. Proof sketch of Theorem 3. We first reduce to the long-horizon problem, in which the oracle threshold no longer depends on the round; passing from long-horizon to finite-horizon regret costs only an O(1)O(1) additive term (Lemma 16). For a fixed level j with hj⋆>τ¯⋆h_j > τ , the long-horizon optimal policy accepts level j at every queue state, whereas deferring incurs an extra cost of at least ce(hj⋆−τ¯⋆)c_e(h_j - τ ). We compare h⋆h with an alternative instance in which only hj⋆h_j is moved slightly below τ¯⋆ τ . Although the deferral threshold changes with the instance, we show that deferring level j is strictly beneficial in the long run under the alternative (equation 78), so a uniformly efficient learner must behave differently on the two instances. Since only completed level-j reviews distinguish them (Lemma 17), a change-of-measure argument (Lai and Robbins, 1985) forces order logT/kl(hj⋆,τ¯⋆) T/kl(h_j , τ ) level-j deferrals under h⋆h . Multiplying by their extra cost ce(hj⋆−τ¯⋆)c_e(h_j - τ ) yields the corresponding term in Theorem 3. Levels with hj⋆<τ¯⋆h_j < τ are instead always deferred by the oracle at the empty queue, so they can be reviewed without taking a suboptimal action. ∎ Proof sketch of Theorem 4. When μ=1μ=1, we move each conditional accuracy hjh_j a distance δj _j above or below the oracle’s accuracy threshold, so that two neighboring instances differing only at level j have different optimal actions for level-j queries. With δj≍min1,(ωjT)−1/2 _j \1,( _jT)^-1/2\, Lemma 17 implies that the corresponding histories remain statistically indistinguishable, so on one of the two instances the learner must take the suboptimal action on a constant fraction of these queries, each costing order ceδjc_e _j. When μ<1μ<1, perturbing the accuracies moves the threshold they are compared with. We therefore shift all coordinates of each perturbed instance by a common amount until the long-horizon empty-queue threshold coincides with the center of the perturbation (Lemma 21). An additional difficulty is that the two neighboring instances are only guaranteed to have different optimal actions when the queue is empty; by keeping the queue occupied, the learner could reduce the number of rounds on which indistinguishability translates into regret. Lemmas 20 and 22 show that this is not free: if the queue is rarely empty, the learner must defer sufficiently often while it is congested, at a cost of order choldc_ hold. Capping δj _j at the holding-cost scale keeps this cost uniformly positive and produces the second term in the minimum, yielding Theorem 4. ∎ C.2 Common ingredients We first collect the average-cost and information identities used by both lower-bound proofs. Their proofs and the supporting technical lemmas are given in Subsection C.6. Throughout this appendix, ω, ρC _C, B¯ B, (ce,cexp,chold)(c_e,c_ exp,c_ hold) and the review order are fixed; only the conditional accuracy vector h varies. Write ℙhπP^π_h and hπE^π_h for probability and expectation under h and algorithm π. For a function u:ℬ→ℝu:B and a∈0,1a∈\0,1\ with b+a≤B¯b+a≤ B, write (Pau)(b):=∑b′Pa(b,b′)u(b′)=[u((b+a−Ct)+)](P_au)(b):= _b P_a(b,b )u(b )=E[u((b+a-C_t)^+)], and define the expected one-step cost ℓh(b,j,a):=choldb+cexpa+Lh(j)(1−a) _h(b,j,a):=c_ holdb+c_ expa+L^h(j)(1-a). In this notation, the terminal cost satisfies D(0)=0D(0)=0 and D(b)=choldb+P0D(b),b≥1.D(b)=c_ holdb+P_0D(b), b≥ 1. (63) We recall that a stationary policy φ is a policy whose action distribution φ(⋅∣b,j) (· b,j) depends only on the current state (b,j)(b,j), not on time or past history. A pair (gh,uh)∈ℝ×ℝℬ(g_h,u_h) ×R^B with uh(0)=0u_h(0)=0 solves the average-cost optimality equation (ACOE) for h if gh+uh(b)=∑j=0Kωjmina∈(b,j)ℓh(b,j,a)+(Pauh)(b),b∈ℬ.g_h+u_h(b)= _j=0^K _j _a (b,j)\ _h(b,j,a)+(P_au_h)(b)\, b . (64) Given such a pair, write xh(q):=uh(q)−uh(q−1)x_h(q):=u_h(q)-u_h(q-1) for q∈[B¯]q∈[ B], dh(b):=(P1−P0)uh(b)d_h(b):=(P_1-P_0)u_h(b) and M¯h(b):=cexp+dh(b) M^h(b):=c_ exp+d_h(b) for b<B¯b< B; recall rjh=Lh(j)−cexpr_j^h=L^h(j)-c_ exp. For a feasible action, define its average-cost disadvantage by h(b,j,a):=ℓh(b,j,a)+(Pauh)(b)−mina′∈(b,j)ℓh(b,j,a′)+Pa′uh(b)≥0. d_h(b,j,a):= _h(b,j,a)+(P_au_h)(b)- _a (b,j)\ _h(b,j,a )+P_a u_h(b)\\ ≥ 0. (65) For s≥0s≥ 0, let Wsh(b)W_s^h(b) be the queue value function with s decision rounds remaining and terminal cost D. Thus W0h=DW_0^h=D and WT+1−th=UthW_T+1-t^h=U_t^h. Define ξsh(q):=Wsh(q)−Wsh(q−1) _s^h(q):=W_s^h(q)-W_s^h(q-1), so that xth(q)=ξT+1−th(q)x_t^h(q)= _T+1-t^h(q). Lemma 16 (Average-cost comparison). For every h∈[0,1]Kh∈[0,1]^K, the limits uh:=lims→∞[Wsh−Wsh(0)]u_h:= _s→∞[W_s^h-W_s^h(0)] and gh:=lims→∞[Ws+1h(0)−Wsh(0)]g_h:= _s→∞[W_s+1^h(0)-W_s^h(0)] exist. The pair (gh,uh)(g_h,u_h) solves equation 64; uhu_h is nondecreasing and discrete convex, and uh≥Du_h≥ D. Moreover, (i) the following increment identities hold: dh(b) d_h(b) =∑c=0bρC(c)xh(b+1−c), = _c=0^b _C(c)\,x_h(b+1-c), (66) xh(q) x_h(q) =chold+∑j=0Kωjmeddh(q−1),dh(q),rjh,1≤q<B¯, =c_ hold+ _j=0^K _jmed\d_h(q-1),d_h(q),r^h_j\, 1≤ q< B, (67) xh(B¯) x_h( B) =chold+∑j=0Kωjmaxdh(B¯−1),rjh; =c_ hold+ _j=0^K _j \d_h( B-1),r^h_j\; (68) (i) cexp≤M¯h(0)≤⋯≤M¯h(B¯−1)c_ exp≤ M^h(0)≤·s≤ M^h( B-1), and M¯h(0)=cexp+(1−μ)uh(1); M^h(0)=c_ exp+(1-μ)\,u_h(1); (69) (i) for every algorithm π, |RT(π,h)−hπ[∑t=1Th(Bt,Jt,At)]|≤sp(D−uh); |R_T(π;h)-E_h^π [ _t=1^T d_h(B_t,J_t,A_t) ] | (D-u_h); (70) (iv) for every stationary policy φ with a stationary distribution λ, gh≤∑s,aλ(s)φ(a∣s)ℓh(s,a).g_h≤ _s,aλ(s) (a s)\, _h(s,a). (71) Recall Dj,t:=∑s=1tJs=j,As=1D_j,t:= _s=1^t1\J_s=j,A_s=1\, the number of self-confidence level j deferrals submitted through round t, and that Nj,tN_j,t is the number of completed self-confidence level j labels available before round t. Let Phπ,tP_h^π,t denote the law of ℱtF_t under h and algorithm π. Lemma 17 (Information from completed reviews). For every t≥0t≥ 0, any algorithm π, and conditional accuracy vectors h,h~h, h such that h~j∈(0,1) h_j∈(0,1) whenever hj≠h~jh_j≠ h_j, KL(Phπ,t∥Ph~π,t)=∑j=1Khπ[Nj,t+1]kl(hj,h~j)≤∑j=1Khπ[Dj,t]kl(hj,h~j).KL(P_h^π,t\|P_ h^π,t)= _j=1^KE_h^π[N_j,t+1]kl(h_j, h_j)≤ _j=1^KE_h^π[D_j,t]kl(h_j, h_j). (72) Consequently, KL(Phπ,t∥Ph~π,t)≤t∑j=1Kωjkl(hj,h~j).KL(P_h^π,t\|P_ h^π,t)≤ t _j=1^K _jkl(h_j, h_j). (73) Define Mτ:=ce(1−τ)=cexp+1−μchold,h0:=τ.M_τ:=c_e(1-τ)=c_ exp+ 1-μc_ hold, h^0:= 1. (74) Lemma 18 (Empty-queue threshold bounds). For every h∈[0,1]Kh∈[0,1]^K and 1≤t≤T1≤ t≤ T, Mth(0)=cexp+(1−μ)xt+1h(1),M_t^h(0)=c_ exp+(1-μ)x_t+1^h(1), (75) and Mτ=MTh(0)≤Mth(0)≤M¯h(0),Mth(0)⟶M¯h(0)as T−t→∞.M_τ=M_T^h(0)≤ M_t^h(0)≤ M^h(0), M_t^h(0) M^h(0) T-t→∞. (76) Moreover, Mth(0)≤Mt−1h(0)M_t^h(0)≤ M_t-1^h(0) for 2≤t≤T2≤ t≤ T. If τ∈[0,1]τ∈[0,1], then at h0h^0 all marginal costs in equation 76 equal MτM_τ, always accepting is optimal, and uh0=Du_h^0=D. In particular, writing τ¯(h):=1−M¯h(0)/ce τ(h):=1- M^h(0)/c_e, we have τ¯(h)≤1−Mth(0)/ce≤τ τ(h)≤ 1-M^h_t(0)/c_e≤τ for every h and t, with equality at h0h^0 when τ∈[0,1]τ∈[0,1]. C.3 Instance-dependent regret lower bound We show that every uniformly efficient algorithm must collect order logT T expert labels at each self-confidence level where the oracle accepts, by comparing the true instance with an alternative on which deferral is strictly beneficial. Proof. For each horizon T, write ℙh:=ℙhπP_h:=P^π_h and h:=hπE_h:=E^π_h for probability and expectation under h and π run with horizon T, and use the subscript ⋆ when h=h⋆h=h . Let u⋆:=uh⋆u :=u_h be the bias from Lemma 16, set ⋆:=h⋆ d_ := d_h , so that M¯⋆(0)=cexp+(P1−P0)u⋆(0) M (0)=c_ exp+(P_1-P_0)u (0) and τ¯⋆=1−M¯⋆(0)/ce τ =1- M (0)/c_e by Lemma 18. Finally, let I:=j∈[K]:ωj>0,hj⋆>τ¯⋆I:=\j∈[K]: _j>0,\ h_j > τ \. For j∈Ij∈ I, Lemma 16(i) implies that acceptance is the unique average-cost minimizer at every queue state. For j∈Ij∈ I and b<B¯b< B, h⋆(b,j,1)=M¯⋆(b)−Lh⋆(j)≥ce(hj⋆−τ¯⋆). d_h (b,j,1)= M (b)-L^h (j)≥ c_e(h_j - τ ). Thus equation 70 gives RT(π,h⋆)≥ce∑j∈I(hj⋆−τ¯⋆)⋆[Dj,T]−C⋆,C⋆:=sp(D−u⋆).R_T(π;h )≥ c_e _j∈ I(h_j - τ )E_ [D_j,T]-C_ , C_ :=sp(D-u ). (77) By equation 77, uniform efficiency implies ⋆[Dj,T]≤[RT(π,h⋆)+C⋆]/[ce(hj⋆−τ¯⋆)]=o(Tα)E_ [D_j,T]≤[R_T(π;h )+C_ ]/[c_e(h_j - τ )]=o(T^α) for every α>0α>0 and j∈Ij∈ I. A confusing alternative for self-confidence level j. Fix j∈Ij∈ I and ε∈(0,τ¯⋆) ∈(0, τ ). We construct an alternative on which deferral at level j is strictly beneficial while all other conditional accuracies are unchanged: set h~j:=τ¯⋆−ε h_j:= τ - and h~k:=hk⋆ h_k:=h_k for k≠jk≠ j. We use the transformed costs khk_h, gains g¯h g_h, and state values vhv_h of Lemma 24, which isolate the effect of deferring at level j, and write g¯⋆:=g¯h⋆ g_ := g_h and v⋆:=vh⋆v_ :=v_h . Let φ⋆ be a deterministic stationary minimizing policy for h⋆h ; since j∈Ij∈ I, it accepts at level j at every queue state. Define a stationary policy φ′ that agrees with φ⋆ everywhere except that φ′(0,j)=1 (0,j)=1. From any positive queue state, B¯ B consecutive rounds with Jt=jJ_t=j and Ct≥1C_t≥ 1 empty the queue because φ′ accepts at level j whenever the queue state is positive. Thus every state reaches the recurrent class containing queue zero with positive probability at least (ωjμ)B¯( _jμ) B, so this class is unique. Let λ be its stationary distribution; since JtJ_t is drawn independently with law ω and ωj>0 _j>0, λ(0,j)>0λ(0,j)>0. Let g¯h~(φ′) g_ h( ) denote the stationary average transformed cost of φ′ under h~ h. Stationarity of λ and equation 111 give g¯h~(φ′)−g¯⋆=∑s,aλ(s)φ′(a∣s)[kh~(s,a)−g¯⋆+[v⋆(St+1)∣St=s,At=a]−v⋆(s)]. g_ h( )- g_ = _s,aλ(s) (a s) [k_ h(s,a)- g_ +E[v_ (S_t+1) S_t=s,A_t=a]-v_ (s) ]. For every action used by φ⋆ , the bracket is zero. The only additional action used by φ′ is deferral at (0,j)(0,j), where the bracket equals M¯⋆(0)−Lh~(j)=−ceε M (0)-L h(j)=-c_e . Therefore g¯h~(φ′)=g¯⋆−λ(0,j)ceε g_ h( )= g_ -λ(0,j)c_e . By equation 71 for the transformed problem, g¯h~≤g¯⋆−λ(0,j)ceε,aε:=g¯⋆−g¯h~≥λ(0,j)ceε>0. g_ h≤ g_ -λ(0,j)c_e , a_ := g_ - g_ h≥λ(0,j)c_e >0. (78) Regret of under-reviewing at level j under h~ h. Let rε:=Lh~(j)−Lh⋆(j)=ce(hj⋆−h~j)>0r_ :=L h(j)-L^h (j)=c_e(h_j - h_j)>0, w:=vh~−v⋆w:=v_ h-v_ , and sw:=sp(w)s_w:=sp(w). Subtracting equation 111 under the two conditional accuracy vectors gives h~(s,a)−⋆(s,a)=−rεak=j+aε+[w(St+1)∣St=s,At=a]−w(s). d_ h(s,a)- d_ (s,a)=-r_ a1\k=j\+a_ +E[w(S_t+1) S_t=s,A_t=a]-w(s). (79) Generate JT+1∼ωJ_T+1 ω independently after the last decision and set ST+1=(BT+1,JT+1)S_T+1=(B_T+1,J_T+1). By Lemma 24, ℳt:=∑s=1t(w(Ss+1)−[w(Ss+1)∣Ss,As])M_t:= _s=1^t (w(S_s+1)-E[w(S_s+1) S_s,A_s] ) is a martingale under h~ h, with |ℳt−ℳt−1|≤sw|M_t-M_t-1|≤ s_w. Summing equation 79, telescoping, and using ⋆≥0 d_ ≥ 0 gives rεDj,T+∑t=1Th~(St,At)≥Taε−sw−ℳT.r_ D_j,T+ _t=1^T d_ h(S_t,A_t)≥ Ta_ -s_w-M_T. (80) Define ET:=Dj,T<aεT/(4rε)E_T:=\D_j,T<a_ T/(4r_ )\. For T such that sw≤aεT/4s_w≤ a_ T/4, equation 80 implies ET⊆∑t=1Th~(St,At)≥aεT4∪ℳT≥aεT4.E_T \ _t=1^T d_ h(S_t,A_t)≥ a_ T4 \∪ \M_T≥ a_ T4 \. By equation 70 and uniform efficiency at h~ h, h~[∑t=1Th~(St,At)]≤RT(π,h~)+sp(D−uh~)=o(Tα)E_ h[ _t=1^T d_ h(S_t,A_t)]≤ R_T(π; h)+sp(D-u_ h)=o(T^α). Markov’s inequality and the Hoeffding–Azuma inequality (Lattimore and Szepesvári, 2020, Exercise 20.6) therefore give, for every α∈(0,1)α∈(0,1), ℙh~(ET)≤4h~[∑t=1Th~(St,At)]aεT+exp−aε2T32(1+sw)2=o(Tα−1).P_ h(E_T)≤ 4E_ h[ _t=1^T d_ h(S_t,A_t)]a_ T+ \! \- a_ ^2T32(1+s_w)^2 \=o(T^α-1). (81) Under the true instance, ℙ⋆(ETc)≤4rε⋆[Dj,T]aεT⟶0.P_ (E_T^c)≤ 4r_ E_ [D_j,T]a_ T 0. (82) Information from completed reviews. Lemma 17 gives KL(Ph⋆π,T∥Ph~π,T)≤⋆[Dj,T]kl(hj⋆,h~j).KL(P_h ^π,T\|P_ h^π,T) _ [D_j,T]kl(h_j , h_j). Since ET∈ℱTE_T _T, the data-processing inequality (Lattimore and Szepesvári, 2020, Exercise 14.10) applies. Moreover, kl(p,q)≥plog(1/q)−log2kl(p,q)≥ p (1/q)- 2 because the binary entropy is at most log2 2. Hence ⋆[Dj,T]kl(hj⋆,h~j)≥ℙ⋆(ET)log1ℙh~(ET)−log2.E_ [D_j,T]kl(h_j , h_j) _ (E_T) 1P_ h(E_T)- 2. By equation 82, ℙ⋆(ET)→1P_ (E_T)→ 1, while equation 81 gives log(1/ℙh~(ET))≥(1−α−o(1))logT (1/P_ h(E_T))≥(1-α-o(1)) T. Therefore, for every α∈(0,1)α∈(0,1), lim infT→∞⋆[Dj,T]logT≥1−αkl(hj⋆,τ¯⋆−ε). _T→∞ E_ [D_j,T] T≥ 1-αkl(h_j , τ - ). Letting α↓0α 0 and then ε↓0 0, using continuity of Bernoulli relative entropy in its second argument on (0,1)(0,1), yields lim infT→∞⋆[Dj,T]logT≥1kl(hj⋆,τ¯⋆). _T→∞ E_ [D_j,T] T≥ 1kl(h_j , τ ). Substituting this into equation 77 and summing over j∈Ij∈ I proves equation 2. ∎ Remark 2. When μ=1μ=1, τ¯⋆=1−cexp/ce τ =1-c_ exp/c_e, and Theorem 3 can alternatively be derived from the Graves–Lai lower bound for structured MDPs (Ok et al., 2018, Theorem 1). C.4 KL-PBD is asymptotically optimal under immediate-service In this section, we show that KL-PBD matches Theorem 3 when service is immediate (μ=1)(μ=1). Throughout this subsection, Ct≥1C_t≥ 1 almost surely and 0<cexp<ce0<c_ exp<c_e. Thus μ=1μ=1 and τ=1−cexp/ce∈(0,1)τ=1-c_ exp/c_e∈(0,1). We consider KL-PBD with a fixed budget ϕ(T)>0φ(T)>0. All probabilities and expectations in this subsection are under h⋆h and πKL-PBDπ KL-PBD. By equation 75 with μ=1μ=1, Mth(0)=cexpM_t^h(0)=c_exp for every h and t. Thus, after n completed level j reviews, the algorithm defers exactly when gj,n<τg_j,n<τ, with ties resolved in favor of acceptance. Every deferral yields a label before the next round. The sample counts before successive level j deferrals are therefore 0,1,…,Dj,T−10,1,…,D_j,T-1. We use the infinite i.i.d. ground-truth label sequences for each self-confidence level from Lemma 8. Lemma 19 (Erroneous acceptance under immediate service). For a level j with 0<hj⋆<τ0<h_j <τ, [∑t=1TJt=j,At=0]≤Te−ϕ(T)expkl(τ,hj⋆)−1.E [ _t=1^T1\J_t=j,A_t=0\ ]≤ Te^-φ(T) \kl(τ,h_j )\-1. (83) If hj⋆=0h_j =0, the left-hand side is zero. Proof. Fix j with 0<hj⋆<τ0<h_j <τ. The algorithm accepts only when gj,n≥τg_j,n≥τ and hence n≥1n≥ 1, since gj,0=0g_j,0=0. By the definition of the lower confidence bound and ϕ(T)>0φ(T)>0, gj,n≥τg_j,n≥τ implies h^j,n>τ h_j,n>τ and nkl(h^j,n,τ)≥ϕ(T)nkl( h_j,n,τ)≥φ(T). For every z∈[τ,1]z∈[τ,1], kl(z,hj⋆)−kl(z,τ) (z,h_j )-kl(z,τ) =kl(τ,hj⋆)+(z−τ)log(τ(1−hj⋆)hj⋆(1−τ)) =kl(τ,h_j )+(z-τ) ( τ(1-h_j )h_j (1-τ) ) ≥kl(τ,hj⋆). (τ,h_j ). If ϕ(T)/n≤kl(1,τ)φ(T)/n (1,τ), let zn∈(τ,1]z_n∈(τ,1] solve kl(zn,τ)=ϕ(T)/nkl(z_n,τ)=φ(T)/n. The Bernoulli Chernoff bound gives ℙ(gj,n≥τ)≤ℙ(h^j,n≥zn)≤e−nkl(zn,hj⋆)≤e−ϕ(T)−nkl(τ,hj⋆).P(g_j,n≥τ) ( h_j,n≥ z_n)≤ e^-nkl(z_n,h_j )≤ e^-φ(T)-nkl(τ,h_j ). If ϕ(T)/n>kl(1,τ)φ(T)/n>kl(1,τ), the event is empty and the same bound holds. Taking a union bound over sample counts, ℙ(∃n≥1:gj,n≥τ)≤∑n=1∞e−ϕ(T)−nkl(τ,hj⋆)=e−ϕ(T)expkl(τ,hj⋆)−1.P(∃ n≥ 1:g_j,n≥τ)≤ _n=1^∞e^-φ(T)-nkl(τ,h_j )= e^-φ(T) \kl(τ,h_j )\-1. Since ∑t=1TJt=j,At=0≤T∃n≥1:gj,n≥τ, _t=1^T1\J_t=j,A_t=0\≤ T1\∃ n≥ 1:g_j,n≥τ\, taking expectations proves the bound. If hj⋆=0h_j =0, all sample means and lower confidence bounds are zero, so every level j query is deferred. ∎ Proposition 3 (Asymptotic optimality under immediate service). For fixed K, any budget sequence satisfying ϕ(T)/logT→1φ(T)/ T→ 1 makes KL-PBD uniformly efficient on the immediate-service class. If, in addition, Te−ϕ(T)/logT→0Te^-φ(T)/ T→ 0, then, for every fixed immediate-service instance, limT→∞RT(πKL-PBD)logT=ce∑j∈[K]hj⋆>τ,ωj>0hj⋆−τkl(hj⋆,τ). _T→∞ R_T(π KL-PBD) T=c_e _ subarraycj∈[K]\\ h_j >τ,\ _j>0 subarray h_j -τkl(h_j ,τ). In particular, the equality holds for ϕ(T)=logT+loglogTφ(T)= T+ T and ϕ(T)=logTφ(T)= T for T≥3T≥ 3, with arbitrary positive budgets for smaller horizons. If hj⋆≥τh_j ≥τ for every j∈[K]j∈[K] with ωj>0 _j>0, the condition Te−ϕ(T)/logT→0Te^-φ(T)/ T→ 0 is unnecessary. Proof. Fix a level j with hj⋆>τh_j >τ and ε∈(0,hj⋆−τ) ∈(0,h_j -τ), and write d:=kl(hj⋆−ε,τ)>0d:=kl(h_j - ,τ)>0. If n≥ϕ(T)/dn≥φ(T)/d and h^j,n≥hj⋆−ε h_j,n≥ h_j - , then nkl(h^j,n,τ)≥nkl(hj⋆−ε,τ)≥ϕ(T).nkl( h_j,n,τ)≥ nkl(h_j - ,τ)≥φ(T). Since h^j,n>τ h_j,n>τ, this implies gj,n≥τg_j,n≥τ, so the algorithm accepts. Hence each deferral at a sample count n≥ϕ(T)/dn≥φ(T)/d requires h^j,n<hj⋆−ε h_j,n<h_j - . Since each sample count occurs at most once before a deferral, Dj,T≤⌈ϕ(T)d⌉+∑n=1∞h^j,n<hj⋆−ε.D_j,T≤ φ(T)d + _n=1^∞1\ h_j,n<h_j - \. Taking expectations and applying the Bernoulli Chernoff bound gives [Dj,T] [D_j,T] ≤⌈ϕ(T)d⌉+∑n=1∞e−nkl(hj⋆−ε,hj⋆) ≤ φ(T)d + _n=1^∞e^-nkl(h_j - ,h_j ) ≤1+ϕ(T)kl(hj⋆−ε,τ)+1expkl(hj⋆−ε,hj⋆)−1 ≤ 1+ φ(T)kl(h_j - ,τ)+ 1 \kl(h_j - ,h_j )\-1 (84) where the final term is interpreted as zero when the divergence is infinite. In particular, this covers hj⋆=1h_j =1. We next prove uniform efficiency. By Lemma 2, under immediate service, RT(πKL-PBD)= R_T(π KL-PBD)= ce∑j∈[K]hj⋆>τ,ωj>0(hj⋆−τ)[Dj,T] c_e _ subarraycj∈[K]\\ h_j >τ,\ _j>0 subarray(h_j -τ)E[D_j,T] +ce∑j∈[K]hj⋆<τ,ωj>0(τ−hj⋆)[∑t=1TJt=j,At=0]. +c_e _ subarraycj∈[K]\\ h_j <τ,\ _j>0 subarray(τ-h_j )E\! [ _t=1^T1\J_t=j,A_t=0\ ]. Levels j with hj⋆=τh_j =τ or ωj=0 _j=0 contribute zero regret. If hj⋆=0h_j =0, KL-PBD never accepts at level j. By Lemma 19, the second term is at most ceTe−ϕ(T)∑j∈[K]0<hj⋆<τ,ωj>0τ−hj⋆expkl(τ,hj⋆)−1.c_eTe^-φ(T) _ subarraycj∈[K]\\ 0<h_j <τ,\ _j>0 subarray τ-h_j \kl(τ,h_j )\-1. (85) If ϕ(T)/logT→1φ(T)/ T→ 1, then Te−ϕ(T)=expo(logT)=o(Tα)Te^-φ(T)= \o( T)\=o(T^α) for every α>0α>0, while equation 84 gives [Dj,T]=O(logT)E[D_j,T]=O( T) for every j with hj⋆>τh_j >τ. Thus KL-PBD is uniformly efficient on the immediate-service class. It remains to identify the asymptotic coefficient. Dividing equation 84 by logT T, using ϕ(T)/logT→1φ(T)/ T→ 1, and then letting ε↓0 0 gives, for every j with hj⋆>τh_j >τ, lim supT→∞[Dj,T]logT≤1kl(hj⋆,τ). _T→∞ E[D_j,T] T≤ 1kl(h_j ,τ). If, in addition, Te−ϕ(T)/logT→0Te^-φ(T)/ T→ 0, then equation 85 is o(logT)o( T). Since K is finite, lim supT→∞RT(πKL-PBD)logT≤ce∑j∈[K]hj⋆>τ,ωj>0hj⋆−τkl(hj⋆,τ). _T→∞ R_T(π KL-PBD) T≤ c_e _ subarraycj∈[K]\\ h_j >τ,\ _j>0 subarray h_j -τkl(h_j ,τ). Uniform efficiency and Theorem 3 give the matching lower bound, proving the claimed limit. If hj⋆≥τh_j ≥τ for every j∈[K]j∈[K] with ωj>0 _j>0, the acceptance contribution is identically zero. Hence the same upper coefficient follows from equation 84 under ϕ(T)/logT→1φ(T)/ T→ 1 alone, and the lower bound again gives equality. Finally, both ϕ(T)=logT+loglogTφ(T)= T+ T and ϕ(T)=logTφ(T)= T satisfy ϕ(T)/logT→1φ(T)/ T→ 1 and Te−ϕ(T)/logT→0Te^-φ(T)/ T→ 0. ∎ C.5 Worst-case regret lower bound The main idea of the proof is to construct statistically indistinguishable instances with empty-queue states at which the optimal actions differ, while controlling how often congestion prevents the process from visiting these informative states. Proof. Fix an algorithm π and write m:=minτ,1−τm:= \τ,1-τ\. If ∑j=1Kωj=0 _j=1^K _j=0, both sides of equation 3 are zero, so assume otherwise. We first prove a sharper lower bound for μ=1μ=1. In this setting, the queue stays empty and every deferred label is observed before the next decision. Define the hard family δj:=m4min1,1ωjT,hjσ:=τ+σjδj, _j:= m4 \! \1, 1 _jT \, h_j^σ:=τ+ _j _j, (86) with δj=0 _j=0 when ωj=0 _j=0. Since Bt=0B_t=0 for every t, the next-state law is the same under both actions, so Mth(0)=cexp=ce(1−τ)M_t^h(0)=c_ exp=c_e(1-τ) for every h and t≤Tt≤ T, and choldBt=D(BT+1)=0c_ holdB_t=D(B_T+1)=0 along every trajectory. Resolving ties in favor of acceptance, the oracle accepts at level j when σj=+1 _j=+1 and defers when σj=−1 _j=-1, and Lemma 2 gives RT(π,hσ) R_T(π;h^σ) =∑j=1KRT,j(π,hσ), = _j=1^KR_T,j(π;h^σ), RT,j(π,hσ) R_T,j(π;h^σ) :=ceδj∑t=1Tℙπhσ(Jt=j,At≠σj=−1). :=c_e _j _t=1^TP^π_h^σ (J_t=j,\ A_t 1\ _j=-1\ ). Fix j with ωj>0 _j>0 and neighboring sign vectors σ,σ(j)σ,σ^(j) with σj=+1 _j=+1 and σj(j)=−1σ^(j)_j=-1. Since the two conditional accuracy vectors differ only in coordinate j, Lemma 17 gives, for every t≤Tt≤ T, KL(Phσπ,t−1∥Phσ(j)π,t−1)≤ωj(t−1)kl(τ+δj,τ−δj)≤32δj29m2ωjT≤29.KL\! (P_h^σ^π,t-1 \|P_h^σ^(j)^π,t-1 )≤ _j(t-1)kl(τ+ _j,τ- _j)≤ 32 _j^29m^2\, _jT≤ 29. For the middle inequality, every u∈[τ−δj,τ+δj]u∈[τ- _j,τ+ _j] satisfies minu,1−u≥3m/4 \u,1-u\≥ 3m/4, so Taylor’s theorem in the first argument of the Bernoulli relative entropy gives kl(τ+δj,τ−δj)≤(2δj)22supu∈[τ−δj,τ+δj]1u(1−u)≤32δj29m2,kl(τ+ _j,τ- _j)≤ (2 _j)^22 _u∈[τ- _j,τ+ _j] 1u(1-u)≤ 32 _j^29m^2, and δj2ωjT≤m2/16 _j^2 _jT≤ m^2/16 by equation 86. Pinsker’s inequality (Lattimore and Szepesvári, 2020, Eq. (14.12)) therefore gives TV(Phσπ,t−1,Phσ(j)π,t−1)≤(2/9)/2=1/3TV(P_h^σ^π,t-1,P_h^σ^(j)^π,t-1)≤ (2/9)/2=1/3. Let αt:=πt(At=1∣ℱt−1,Jt=j) _t:= _t(A_t=1 _t-1,J_t=j) be the algorithm’s conditional probability of deferral after the arrival of a query at level j. Since JtJ_t is independent of ℱt−1F_t-1, for σj=+1 _j=+1, ℙhσπ(Jt=j,At=1)+ℙhσ(j)π(Jt=j,At=0) _h^σ^π(J_t=j,A_t=1)+P_h^σ^(j)^π(J_t=j,A_t=0) =ωj[1+hσπ[αt]−hσ(j)π[αt]] = _j [1+E_h^σ^π[ _t]-E_h^σ^(j)^π[ _t] ] ≥ωj(1−TV(Phσπ,t−1,Phσ(j)π,t−1)) ≥ _j (1-TV(P_h^σ^π,t-1,P_h^σ^(j)^π,t-1) ) ≥2ωj3, ≥ 2 _j3, using 0≤αt≤10≤ _t≤ 1 and that αt _t is ℱt−1F_t-1-measurable. Summing over t gives RT,j(π,hσ)+RT,j(π,hσ(j))≥2ceδjωjT/3R_T,j(π;h^σ)+R_T,j(π;h^σ^(j))≥ 2c_e _j _jT/3. Pairing the 2K−12^K-1 sign vectors with identical σ−j _-j and summing over j gives 2−K∑σRT(π,hσ)≥ce3∑j=1KδjωjT=cem12∑j=1KminωjT,ωjT.2^-K _σR_T(π;h^σ)≥ c_e3 _j=1^K _j _jT= c_em12 _j=1^K \ _jT, _jT\. The supremum over h is at least this average, proving the μ=1μ=1 sharpening stated after Theorem 4. Assume from now on that μ<1μ<1. If chold=0c_ hold=0, the right-hand side of equation 3 is zero, so there is nothing to prove. We therefore take chold>0c_ hold>0. Self-centered heterogeneous family. We next recenter each sign pattern so that its empty-queue action gaps remain exactly controlled despite the queue-dependent threshold. For ωj>0 _j>0, set δj:=124minμ2mmin1,1ωjT,μcholdce, _j:= 124 \! \μ^2m \! \1, 1 _jT \, μ c_ holdc_e \, (87) and set δj=0 _j=0 if ωj=0 _j=0. Write δmax:=maxjδj _ := _j _j. For each σ∈−1,+1Kσ∈\-1,+1\^K, Lemma 21 provides a unique center aσa_σ, with S−σ:=∑j:σj=−1ωjδjS^-_σ:= _j: _j=-1 _j _j. Set hjσ:=τ−aσ+σjδj.h_j^σ:=τ-a_σ+ _j _j. (88) Write uσ:=uhσu_σ:=u_h^σ, ℙσs:=ℙhσπ,sP^s_σ:=P^π,s_h^σ, ℙσ:=ℙhσπP_σ:=P^π_h^σ and σ:=hσπE_σ:=E^π_h^σ. By Lemma 21, τ¯(hσ)=τ−aσ, τ(h^σ)=τ-a_σ, (89) so M¯hσ(0)−Lhσ(j)=σjceδj M^h^σ(0)-L^h^σ(j)= _jc_e _j. Moreover, 0≤aσ≤1−μSσ−.0≤ a_σ≤ 1-μS^-_σ. (90) Testing neighboring instances. We now show that flipping one sign changes the observations too little to identify the optimal empty-queue action reliably. By equation 90 and equation 87, every hard parameter lies within m/12m/12 of τ. The inequality logx≤x−1 x≤ x-1 gives kl(p,q)≤(p−q)2/[q(1−q)]kl(p,q)≤(p-q)^2/[q(1-q)]; since q(1−q)≥(3m/4)2q(1-q)≥(3m/4)^2 on the hard family, kl(p,q)≤16(p−q)2/(9m2)kl(p,q)≤ 16(p-q)^2/(9m^2). Using the neighboring-center bound equation 110 from Lemma 21, for neighboring sign vectors with σj=+1 _j=+1 and σj(j)=−1σ^(j)_j=-1, write Δa:=aσ(j)−aσ a:=a_σ^(j)-a_σ and Ω:=∑k=1Kωk := _k=1^K _k. Then ∑k=1Kωk|hkσ−hkσ(j)|2 _k=1^K _k|h_k^σ-h_k^σ^(j)|^2 =4ωjδj2+4ωjδjΔa+Ω(Δa)2 =4 _j _j^2+4 _j _j a+ ( a)^2 ≤4μ2ωjδj2. ≤ 4μ^2 _j _j^2. Lemma 17 and equation 87 therefore give KL(Pσt−1∥Pσ(j)t−1)≤μ2/81KL(P_σ^t-1\|P_σ^(j)^t-1)≤μ^2/81 uniformly over t≤Tt≤ T. Pinsker’s inequality (Lattimore and Szepesvári, 2020, Eq. (14.12)) then yields TV(Pσt−1,Pσ(j)t−1)≤μ12,t≤T.TV(P_σ^t-1,P_σ^(j)^t-1)≤ μ12, t≤ T. (91) Empty-queue occupancy and average-cost disadvantage. We combine equation 92 and equation 91: either the queue is often empty, forcing decision errors, or keeping it occupied requires costly deferrals. Let N0(T):=∑t=1TBt=0N_0(T):= _t=1^T1\B_t=0\. For each hard instance and level j, define σ,j:=σ[∑t=1Thσ(Bt,Jt,At)Jt=j],σ:=∑j=1Kσ,j.D_σ,j:=E_σ\! [ _t=1^T d_h^σ(B_t,J_t,A_t)1\J_t=j\ ], _σ:= _j=1^KD_σ,j. Let r be the conditional service median in Lemma 20. By equation 102, equation 89, and equation 87, for j∈[K]j∈[K] and r≤b<B¯r≤ b< B, hσ(b,j,1)≥chold2−ceδmax≥1124chold=:gbar. d_h^σ(b,j,1)≥ c_ hold2-c_e _ ≥ 1124c_ hold=:g_ bar. (92) Let HT:=∑t=1TBt≥r,At=1H_T:= _t=1^T1\B_t≥ r,A_t=1\. Lemma 22 and equation 92 give σ[N0(T)]≥μT2−2σ[HT]≥μT2−4811choldσ.E_σ[N_0(T)]≥ μ T2-2E_σ[H_T]≥ μ T2- 4811c_ holdD_σ. (93) Pair σ and σ(j)σ^(j). For the positive sign, every level j deferral has disadvantage at least ceδjc_e _j; for the negative sign, accepting a level-j query at an empty queue has disadvantage exactly ceδjc_e _j. With αt:=πt(At=1∣ℱt−1,Jt=j) _t:= _t(A_t=1 _t-1,J_t=j) as in the case μ=1μ=1, and since Bt=0αt1\B_t=0\ _t and Bt=0(1−αt)1\B_t=0\(1- _t) are ℱt−1F_t-1-measurable and [0,1][0,1]-valued, the variational characterization of total variation gives σ[Bt=0αt]+σ(j)[Bt=0(1−αt)] _σ[1\B_t=0\ _t]+E_σ^(j)[1\B_t=0\(1- _t)] ≥ℙσ(Bt=0)+ℙσ(j)(Bt=0)2−TV(Pσt−1,Pσ(j)t−1). ≥ P_σ(B_t=0)+P_σ^(j)(B_t=0)2-TV(P_σ^t-1,P_σ^(j)^t-1). Summing this inequality and using equation 91 yields 12(σ,j+σ(j),j)≥ceδjωj2[σ[N0(T)]+σ(j)[N0(T)]2−μT12]. 12 (D_σ,j+D_σ^(j),j )≥ c_e _j _j2 [ E_σ[N_0(T)]+E_σ^(j)[N_0(T)]2- μ T12 ]. Averaging over all neighboring pairs and summing over j, with Sδ:=∑j=1KωjδjS_δ:= _j=1^K _j _j and ¯:=2−K∑σ D:=2^-K _σD_σ, gives ¯≥5μT24ceSδ−24ceSδ11chold¯. D≥ 5μ T24c_eS_δ- 24c_eS_δ11c_ hold D. Since ceSδ≤μchold/24c_eS_δ≤μ c_ hold/24, we conclude that ¯≥55288μTceSδ. D≥ 55288\,μ Tc_eS_δ. (94) From disadvantage to finite-horizon regret. It remains to control the terminal bias correction without introducing any dependence on the capacity B¯ B. For a general conditional accuracy vector h, write h(π):=hπ[∑t=1Th(Bt,Jt,At)]D_h(π):=E_h^π[ _t=1^T d_h(B_t,J_t,A_t)]. We only need a one-sided terminal comparison. By Lemma 16, uh≥Du_h≥ D. Because B1=0B_1=0, equation 100 shows that a stationary average-cost minimizer has finite-horizon cost at most TghTg_h, while an arbitrary algorithm has cost Tgh+h(π)+hπ[(D−uh)(BT+1)]Tg_h+D_h(π)+E_h^π[(D-u_h)(B_T+1)]. Consequently, RT(π,h)≥h(π)−hπ[(uh−D)(BT+1)].R_T(π;h) _h(π)-E_h^π[(u_h-D)(B_T+1)]. (95) For the hard instance hσh^σ, Lemma 23 gives 0≤uσ(b)−D(b)≤2ceSσ−μ⌈br⌉.0≤ u_σ(b)-D(b)≤ 2c_eS^-_σμ br . Pathwise, ⌈BT+1/r⌉≤1+HT B_T+1/r ≤ 1+H_T, because a deferral below r cannot raise the queue above r, while each deferral at or above r increases this coarse queue state by at most one. Together with equation 92, equation 95 yields RT(π,hσ)≥911σ−2ceSδμ.R_T(π;h^σ)≥ 911D_σ- 2c_eS_δμ. Averaging over the hard family and using equation 94, suphRT(π,h)≥ceSδ(5μT32−2μ). _hR_T(π;h)≥ c_eS_δ ( 5μ T32- 2μ ). (96) Independently, the first decision gives a complementary bound valid for every T. Consider the family hjσ=τ+σjm/4h_j^σ=τ+ _jm/4. By equation 75, Lemma 18 at h0h^0, and the adjacent-increment estimate equation 37 from the proof of Proposition 2, |M1hσ(0)−Mτ|≤(1−μ)cem/4|M_1^h^σ(0)-M_τ|≤(1-μ)c_em/4. Since |Lhσ(j)−Mτ|=cem/4|L^h^σ(j)-M_τ|=c_em/4, opposite signs in coordinate j induce opposite optimal actions at B1=0B_1=0, each with gap at least μcem/4μ c_em/4. Before the first action the algorithm has no accuracy-dependent observations, so averaging each sign pair forces an error probability at least 1/21/2 conditional on the arrival of a query at level j. Summing over j gives suphRT(π,h)≥μcem8∑j=1Kωj. _hR_T(π;h)≥ μ c_em8 _j=1^K _j. (97) Finally, define ℒT:=∑j=1Kminμ3cemminωjT,ωjT,μ2choldωjT.L_T:= _j=1^K \! \μ^3c_em \ _jT, _jT\,μ^2c_ hold _jT \. By equation 87, ℒT=24μTceSδL_T=24μ Tc_eS_δ. Therefore equation 96 gives suphRT(π,h)≥ℒT(5768−112μ2T), _hR_T(π;h) _T ( 5768- 112μ^2T ), (98) while equation 97 and ℒT≤μ3cemT∑jωjL_T≤μ^3c_emT _j _j give suphRT(π,h)≥ℒT8μ2T. _hR_T(π;h)≥ L_T8μ^2T. (99) If μ2T≥32μ^2T≥ 32, equation 98 is at least ℒT/256L_T/256; if μ2T≤32μ^2T≤ 32, equation 99 is at least ℒT/256L_T/256. Since π was arbitrary, taking the infimum over algorithms proves equation 3. ∎ Remark 3 (Worst case over ω). The function f(x)=minx,xf(x)= \x, x\ is nondecreasing and concave on [0,∞)[0,∞), so Jensen’s inequality gives ∑j=1Kf(ωjT)≤Kf(T/K)=minT,KT _j=1^Kf( _jT)≤ Kf(T/K)= \T, KT\, with equality when ω0=0 _0=0 and ωj=1/K _j=1/K for every j∈[K]j∈[K]. C.6 Technical lemmas C.6.1 Proofs of the common ingredients Proof of Lemma 16. By Lemma 3, each ξsh _s^h is nonnegative and nondecreasing in q. Its update is given by equation 7, equation 9, and equation 10. Write gD(q):=D(q)−D(q−1)g_D(q):=D(q)-D(q-1) and d0(b):=∑c=0bρC(c)gD(b+1−c)d_0(b):= _c=0^b _C(c)g_D(b+1-c). Differencing equation 63 gives gD(q)=chold+d0(q−1)g_D(q)=c_ hold+d_0(q-1). Since D is discrete convex, d0d_0 is nondecreasing. The median or maximum in the increment update is therefore at least d0(q−1)d_0(q-1), giving ξ1h(q)≥gD(q)=ξ0h(q) _1^h(q)≥ g_D(q)= _0^h(q). The update is coordinatewise nondecreasing, so ξs+1h≥ξsh _s+1^h≥ _s^h for every s≥0s≥ 0. By equation 39, ξsh(q)≤qchold/μ+(ce−cexp)+ _s^h(q)≤ qc_ hold/μ+(c_e-c_ exp)_+, uniformly in s. Hence ξsh(q)↑xh(q) _s^h(q) x_h(q), and Wsh(b)−Wsh(0)→uh(b):=∑q=1bxh(q)W_s^h(b)-W_s^h(0)→ u_h(b):= _q=1^bx_h(q). In particular, uh≥Du_h≥ D because ξ0h=gD _0^h=g_D. Let hT_h be the queue Bellman operator. With vsh:=Wsh−Wsh(0)v_s^h:=W_s^h-W_s^h(0) and gsh:=Ws+1h(0)−Wsh(0)g_s^h:=W_s+1^h(0)-W_s^h(0), translation invariance gives gsh+vs+1h=hvshg_s^h+v_s+1^h=T_hv_s^h and gsh=(hvsh)(0)g_s^h=(T_hv_s^h)(0). The operator is continuous because the state and action spaces are finite. Thus gsh→gh:=(huh)(0)g_s^h→ g_h:=(T_hu_h)(0), and passing to the limit yields equation 64. Passing the increment identities to the limit proves (i) and preserves monotonicity and discrete convexity. For (i), equation 66 shows that dh(b)d_h(b) is nonnegative and nondecreasing in b. At the empty queue, dh(0)=ρC(0)xh(1)=(1−μ)uh(1)d_h(0)= _C(0)x_h(1)=(1-μ)u_h(1), proving equation 69. For (i), using equation 64, independence of JtJ_t from ℱt−1F_t-1, and the tower property, hπ[∑t=1Tℓh(Bt,Jt,At)+D(BT+1)] _h^π\! [ _t=1^T _h(B_t,J_t,A_t)+D(B_T+1) ] =Tgh+[uh(B1)]+hπ[(D−uh)(BT+1)] =Tg_h+E[u_h(B_1)]+E_h^π[(D-u_h)(B_T+1)] +hπ[∑t=1Th(Bt,Jt,At)]. +E_h^π\! [ _t=1^T d_h(B_t,J_t,A_t) ]. (100) A stationary policy attaining the minima in equation 64 has zero disadvantage. Therefore every algorithm has cost at least Tgh+[uh(B1)]+minb(D−uh)(b)Tg_h+E[u_h(B_1)]+ _b(D-u_h)(b), while that stationary policy has cost at most the same expression with the minimum replaced by the maximum. The finite-horizon optimum lies between these two quantities, which proves equation 70. For (iv), evaluating the minimum in equation 64 at the action distribution of φ gives gh+uh(b)≤∑j=0Kωj∑aφ(a∣b,j)ℓh(b,j,a)+(Pauh)(b)g_h+u_h(b)≤ _j=0^K _j _a (a b,j)\ _h(b,j,a)+(P_au_h)(b)\ for every b. Averaging under λ, stationarity cancels the two uhu_h terms and leaves equation 71. ∎ Proof of Lemma 17. For s≥1s≥ 1, set ΔNj,s:=Nj,s+1−Nj,s N_j,s:=N_j,s+1-N_j,s, order the labels completed in round s by submission time, and expose them one at a time. Conditional on ℱs−1F_s-1, JsJ_s, AsA_s, CsC_s, and the labels already exposed in round s, each newly exposed level j label has law Bernoulli(hj)Bernoulli(h_j) under h and Bernoulli(h~j)Bernoulli( h_j) under h~ h; the remaining observations have the same conditional law under the two instances. The KL chain rule (Lattimore and Szepesvári, 2020, Exercise 14.12, Eq. (14.17)) therefore gives KL(Phπ,t∥Ph~π,t) (P_h^π,t\|P_ h^π,t) =∑s=1thπ[∑j=1KΔNj,skl(hj,h~j)] = _s=1^tE_h^π\! [ _j=1^K N_j,skl(h_j, h_j) ] =∑j=1Khπ[Nj,t+1]kl(hj,h~j). = _j=1^KE_h^π[N_j,t+1]kl(h_j, h_j). Since Nj,t+1≤Dj,t≤∑s=1tJs=jN_j,t+1≤ D_j,t≤ _s=1^t1\J_s=j\ pathwise, hπ[Nj,t+1]≤hπ[Dj,t]≤tωjE_h^π[N_j,t+1] _h^π[D_j,t]≤ t _j, proving equation 72 and equation 73. ∎ Proof of Lemma 18. Equation 75 follows from equation 7 at b=0b=0. Since xt+1h(1)=ξT−th(1)x_t+1^h(1)= _T-t^h(1), the monotone convergence proved in Lemma 16 gives both the time monotonicity and the limit, using equation 69. Also, equation 63 gives D(1)=chold+(1−μ)D(1)D(1)=c_ hold+(1-μ)D(1), so ξ0h(1)=chold/μ _0^h(1)=c_ hold/μ and MTh(0)=MτM_T^h(0)=M_τ. Suppose now that τ∈[0,1]τ∈[0,1]. At h0h^0, Lh0(j)=MτL^h^0(j)=M_τ for every j∈[K]j∈[K]. By Proposition 1 and equation 76, Mth0(b)≥MτM_t^h^0(b)≥ M_τ whenever deferral is feasible. Hence always accepting is optimal for every horizon. Under this policy, the difference in expected cost between starting from queue state b and from zero is exactly D(b)D(b). Thus Wsh0(b)−Wsh0(0)=D(b)W_s^h^0(b)-W_s^h^0(0)=D(b) for every s. Taking the limit gives uh0=Du_h^0=D, and the finite-horizon empty-queue marginals all equal MτM_τ. ∎ C.6.2 Lemmas for the worst-case lower bound For the following four lemmas, assume 0<μ<10<μ<1, chold>0c_ hold>0, and τ∈(0,1)τ∈(0,1), and write m:=minτ,1−τm:= \τ,1-τ\. Choose an integer r≥1r≥ 1 that is a median of CtC_t conditional on Ct>0C_t>0, which exists since μ>0μ>0, so that p−:=ℙ(1≤Ct≤r)≥μ2,p+:=ℙ(Ct≥r)≥μ2.p_-:=P(1≤ C_t≤ r)≥ μ2, p_+:=P(C_t≥ r)≥ μ2. (101) Let :=h:‖h−h0‖∞<minm2,chold4ce.N:= \h:\|h-h^0\|_∞< \! \ m2, c_ hold4c_e \ \. Starting from B1=bB_1=b under a stationary policy φ , write bφE_b for expectation and let τ0:=infs≥0:Bs+1=0 _0:= \s≥ 0:B_s+1=0\ be the number of rounds until the queue first becomes empty. Lemma 20 (Local queue estimates). Fix h∈h . For r≤b<B¯r≤ b< B, M¯h(b)−M¯h(0)≥chold2, M^h(b)- M^h(0)≥ c_ hold2, (102) every stationary minimizing policy accepts at b, and h(b,j,1)≥chold/4 d_h(b,j,1)≥ c_ hold/4 for j∈[K]j∈[K]. Under every stationary minimizing policy, and under the policy that always accepts, b[τ0]≤⌈b/r⌉p+≤2⌈b/r⌉μ,b≥1.E_b[ _0]≤ b/r p_+≤ 2 b/r μ, b≥ 1. (103) For every stationary policy φ with bφ[τ0]<∞E_b [ _0]<∞ for every b, uh(b)≤bφ[∑t=1τ0(ℓh(Bt,Jt,At)−gh)],u_h(b) _b \! [ _t=1 _0 ( _h(B_t,J_t,A_t)-g_h ) ], (104) with equality when φ selects an ACOE-minimizing action at every state. Proof. For the solution uhu_h from Lemma 16, equation 66, monotonicity of (xh(q))q(x_h(q))_q, and equation 76 give, for every b≥rb≥ r with b<B¯b< B, M¯h(b)−M¯h(0)≥∑c=1bρC(c)xh(1)≥choldμℙ(1≤Ct≤b)≥chold2, M^h(b)- M^h(0)≥ _c=1^b _C(c)x_h(1)≥ c_ holdμP(1≤ C_t≤ b)≥ c_ hold2, which is equation 102. Because M¯h(0)≥Mτ M^h(0)≥ M_τ and ‖h−h0‖∞<chold/(4ce)\|h-h^0\|_∞<c_ hold/(4c_e), we have Lh(j)≤Mτ+chold/4L^h(j)≤ M_τ+c_ hold/4. Thus every stationary minimizing policy for h∈h accepts every query at b≥rb≥ r. For j∈[K]j∈[K] and r≤b<B¯r≤ b< B, we also have M¯h(b)−Lh(j)≥chold/4 M^h(b)-L^h(j)≥ c_ hold/4. From queue state one the queue then stays at most r until it hits zero, so under any stationary minimizing policy 1[τ0]≤1p+≤2μ.E_1[ _0]≤ 1p_+≤ 2μ. (105) More generally, positive-service events of size at least r reduce any queue state by at least r until it enters 0,…,r\0,…,r\. This proves equation 103 under every stationary minimizing policy for h∈h . The same drainage bounds hold for the policy that always accepts. Let φ be a stationary policy with bφ[τ0]<∞E_b [ _0]<∞ for every b. Evaluating the minimum in equation 64 at φ , summing over rounds 1,…,τ0∧s1,…, _0 s, and letting s→∞s→∞ by dominated convergence gives equation 104, with equality when φ selects an ACOE-minimizing action at every state. ∎ Lemma 21 (Self-centered family). Fix T≥1T≥ 1, let δj _j be given by equation 87, with δj=0 _j=0 when ωj=0 _j=0, and write δmax:=maxjδj _ := _j _j. Let r and N be as in Lemma 20. For σ∈−1,+1Kσ∈\-1,+1\^K, let S−σ:=∑j:σj=−1ωjδjS^-_σ:= _j: _j=-1 _j _j and Aσ:=(1−μ)Sσ−/μA_σ:=(1-μ)S^-_σ/μ. There is a unique aσ∈[0,Aσ]a_σ∈[0,A_σ] for which the vector hjσ:=τ−aσ+σjδjh^σ_j:=τ-a_σ+ _j _j satisfies τ¯(hσ)=τ−aσ τ(h^σ)=τ-a_σ. It belongs to N. If σj=+1 _j=+1 and σ(j)σ^(j) is obtained by flipping the jjth sign, then 0≤aσ(j)−aσ≤2(1−μ)ωjδj/μ0≤ a_σ^(j)-a_σ≤ 2(1-μ) _j _j/μ. Proof. We need the ordered stability from Proposition 2, applied to the values with s rounds remaining: for h′≤h ≤ h coordinatewise and every s≥0s≥ 0, 0≤[Wsh′(1)−Wsh′(0)]−[Wsh(1)−Wsh(0)]≤ce‖h−h′‖∞.0≤ [W_s^h (1)-W_s^h (0) ]- [W_s^h(1)-W_s^h(0) ]≤ c_e\|h-h \|_∞. (106) For a≥0a≥ 0 such that h−a∈[0,1]Kh-a1∈[0,1]^K, let s→∞s→∞ with h′:=h−ah :=h-a1. Using equation 69 yields 0≤τ¯(h)−τ¯(h−a)≤(1−μ)a.0≤ τ(h)- τ(h-a1)≤(1-μ)a. (107) Define, for a∈[0,Aσ]a∈[0,A_σ], Fσ(a):=τ−τ¯(τ−a+σ⊙δ)F_σ(a):=τ- τ( 1-a1+σ δ) where ⊙ denotes componentwise multiplication. Since Aσ≤δmax/μA_σ≤ _ /μ, for a∈[0,Aσ]a∈[0,A_σ] we have a+δmax≤μm24+μ2m24≤m12,a+δmax≤chold24ce+μchold24ce≤chold12ce,a+ _ ≤ μ m24+ μ^2m24≤ m12, a+ _ ≤ c_ hold24c_e+ μ c_ hold24c_e≤ c_ hold12c_e, so every conditional accuracy vector in this interval belongs to N. Equation 107 gives |Fσ(a)−Fσ(a′)|≤(1−μ)|a−a′||F_σ(a)-F_σ(a )|≤(1-μ)|a-a |, so FσF_σ is a contraction. We next check that FσF_σ maps [0,Aσ][0,A_σ] into itself. Fix a in this interval, write h=τ−a+σ⊙δh= 1-a1+σ δ, and set z:=Fσ(a)z:=F_σ(a). Set wh:=(dh(1)−dh(0))/ce≥0w_h:=(d_h(1)-d_h(0))/c_e≥ 0 when B¯≥2 B≥ 2 and wh:=+∞w_h:=+∞ when B¯=1 B=1. By equation 69 and the definition of z, dh(0)=(1−μ)uh(1)=(1−μ)chold/μ+cezd_h(0)=(1-μ)u_h(1)=(1-μ)c_ hold/μ+c_ez, hence rjh−dh(0)=ce(a−σjδj−z)r_j^h-d_h(0)=c_e(a- _j _j-z). Since meddh(0),dh(1),r−dh(0)=min(r−dh(0))+,dh(1)−dh(0)med\d_h(0),d_h(1),r\-d_h(0)= \(r-d_h(0))_+,d_h(1)-d_h(0)\, equation 67 at q=1q=1 (equation 68 when B¯=1 B=1) gives μz=(1−μ)∑j=1Kωjmin(a−σjδj−z)+,wh.μ z=(1-μ) _j=1^K _j \! \(a- _j _j-z)_+,w_h \. (108) In particular z≥0z≥ 0. If z>Aσz>A_σ, then z>az>a and only negative coordinates can contribute to the right-hand side, each by at most δj _j. Thus μz≤(1−μ)Sσ−=μAσμ z≤(1-μ)S^-_σ=μ A_σ, a contradiction. Since FσF_σ maps the closed interval [0,Aσ][0,A_σ] into itself and is a contraction, it has a unique fixed point aσ∈[0,Aσ]a_σ∈[0,A_σ]. Set hjσ:=τ−aσ+σjδjh_j^σ:=τ-a_σ+ _j _j, as in equation 88, and write gσ:=ghσg_σ:=g_h^σ, uσ:=uhσu_σ:=u_h^σ. The fixed-point identity gives equation 89, so M¯hσ(0)−Lhσ(j)=σjceδj M^h^σ(0)-L^h^σ(j)= _jc_e _j. Moreover, setting z=aσz=a_σ in equation 108 yields equation 90. We need one sharper comparison between neighboring centers. Let σj=+1 _j=+1 and let σ(j)σ^(j) be obtained by flipping the jjth sign. At a common value of a, the conditional accuracy vector for σ(j)σ^(j) is coordinatewise no larger than that for σ. Applying equation 106 and letting s→∞s→∞ gives Fσ(j)(a)≥Fσ(a)F_σ^(j)(a)≥ F_σ(a). Both maps are nondecreasing contractions. Starting the iteration of Fσ(j)F_σ^(j) at aσa_σ therefore produces a nondecreasing sequence whose limit is its fixed point, so aσ(j)≥aσa_σ^(j)≥ a_σ. Write Δa:=aσ(j)−aσ a:=a_σ^(j)-a_σ. Evaluating equation 64 at b=0b=0, where uσ(0)=P0uσ(0)=0u_σ(0)=P_0u_σ(0)=0 and P1uσ(0)=M¯hσ(0)−cexpP_1u_σ(0)= M^h^σ(0)-c_ exp, and using equation 89 gives gσ=Ω(Mτ+ceaσ)−ce∑k:σk=+1ωkδk,Ω:=∑k=1Kωk.g_σ= (M_τ+c_ea_σ)-c_e _k: _k=+1 _k _k, := _k=1^K _k. (109) Let φσ _σ be a stationary minimizing policy under hσh^σ. Starting from queue state one, the queue stays at most r until its first hitting time τ0 _0 of zero, so equation 105 gives 1φσ[τ0]≤2/μE_1 _σ[ _0]≤ 2/μ. From equation 109, gσ(j)−gσ=ceΩΔa+ceωjδjg_σ^(j)-g_σ=c_e a+c_e _j _j. For the same queue state and action rule φσ _σ, the conditional one-step cost difference satisfies [ℓhσ(j)(Bt,Jt,At)−ℓhσ(Bt,Jt,At)|Bt]≤ceΩΔa+2ceωjδj,E\! [ _h^σ^(j)(B_t,J_t,A_t)- _h^σ(B_t,J_t,A_t)\, |\,B_t ]≤ c_e a+2c_e _j _j, because acceptance losses increase by ceΔac_e a at every self-confidence level and by an additional 2ceδj2c_e _j at level j, while deferral costs do not change. Hence [(ℓhσ(j)−gσ(j))−(ℓhσ−gσ)|Bt]≤ceωjδj.E\! [( _h^σ^(j)-g_σ^(j))-( _h^σ-g_σ)\, |\,B_t ]≤ c_e _j _j. By equation 104 at b=1b=1, with equality for hσh^σ under φσ _σ and as an inequality for hσ(j)h^σ^(j) under the same φσ _σ, subtracting the two relations yields uσ(j)(1)−uσ(1)≤ceωjδj1φσ[τ0]≤2ceμωjδj.u_σ^(j)(1)-u_σ(1)≤ c_e _j _j\,E_1 _σ[ _0]≤ 2c_eμ _j _j. Since equation 89 and equation 69 give ceΔa=(1−μ)uσ(j)(1)−uσ(1)c_e a=(1-μ)\u_σ^(j)(1)-u_σ(1)\, we obtain 0≤Δa≤2(1−μ)μωjδj.0≤ a≤ 2(1-μ)μ\, _j _j. (110) ∎ Lemma 22 (Empty-queue occupancy). Fix a conditional accuracy vector h and an algorithm π, and let r be as in Lemma 20. Write N0(T):=∑t=1TBt=0N_0(T):= _t=1^T1\B_t=0\ and HT:=∑t=1TBt≥r,At=1H_T:= _t=1^T1\B_t≥ r,A_t=1\. Then hπ[N0(T)]≥μT/2−2hπ[HT]E_h^π[N_0(T)]≥μ T/2-2E_h^π[H_T]. Proof. Write p+:=ℙ(Ct≥r)p_+:=P(C_t≥ r), as in equation 101, and drop the sub- and superscripts of hπE_h^π. With Wt:=⌈(Bt−r)+/r⌉W_t:= (B_t-r)_+/r , pathwise Wt+1−Wt≤Bt≥r,At=1−Bt>r,At=0,Ct≥rW_t+1-W_t 1\B_t≥ r,A_t=1\-1\B_t>r,A_t=0,C_t≥ r\. Thus p+[∑t=1TBt>r,At=0]≤[HT].p_+\,E\! [ _t=1^T1\B_t>r,A_t=0\ ] [H_T]. A round with Bt+At≤rB_t+A_t≤ r and Ct≥rC_t≥ r leaves the next queue empty; the number of rounds with Bt+At>rB_t+A_t>r is at most the count in the preceding display plus HTH_T. Since B1=0B_1=0, the number of empty next-round queues Bt+1=0B_t+1=0 is at most N0(T)N_0(T). Using equation 101 therefore gives [N0(T)]≥μT/2−2[HT]E[N_0(T)]≥μ T/2-2E[H_T], as claimed. ∎ Lemma 23 (Terminal comparison for the hard family). For every hσh^σ from Lemma 21 and every b∈ℬb , write uσ:=uhσu_σ:=u_h^σ and let r be as in Lemma 20. Then 0≤uσ(b)−D(b)≤(2ceSσ−/μ)⌈b/r⌉0≤ u_σ(b)-D(b)≤(2c_eS^-_σ/μ) b/r . Proof. Let φ0 ^0 be the stationary policy that always accepts and τ0 _0 its first hitting time of queue state zero. By equation 103, bφ0[τ0]≤2⌈b/r⌉/μE_b ^0[ _0]≤ 2 b/r /μ. Moreover, equation 104 gives uσ(b)≤bφ0[∑t=1τ0(ℓhσ(Bt,Jt,0)−ghσ)],D(b)=bφ0[∑t=1τ0(ℓh0(Bt,Jt,0)−gh0)],u_σ(b) _b ^0\! [ _t=1 _0( _h^σ(B_t,J_t,0)-g_h^σ) ], D(b)=E_b ^0\! [ _t=1 _0( _h^0(B_t,J_t,0)-g_h^0) ], where the second identity uses Lemma 18, since φ0 ^0 is a stationary minimizing policy at h0h^0. Moreover, equation 109 gives, at every queue state b, ∑j=0Kωj[ℓhσ(b,j,0)−gσ−(ℓh0(b,j,0)−gh0)]=ceSσ−. _j=0^K _j [ _h^σ(b,j,0)-g_σ- ( _h^0(b,j,0)-g_h^0 ) ]=c_eS^-_σ. Subtracting the two stopped-cost representations and using the last display gives uσ(b)−D(b)≤ceSσ−bφ0[τ0]u_σ(b)-D(b)≤ c_eS^-_σ\,E_b ^0[ _0]; together with the bound on bφ0[τ0]E_b ^0[ _0] and uσ≥Du_σ≥ D from Lemma 16, this proves the claim. ∎ C.6.3 Lemmas for the instance-dependent lower bound Lemma 24 (Transformed disadvantages). For h∈[0,1]Kh∈[0,1]^K, set kh(b,j,a):=choldb+acexp−Lh(j)k_h(b,j,a):=c_ holdb+a\c_ exp-L^h(j)\ and g¯h:=gh−∑k=0KωkLh(k) g_h:=g_h- _k=0^K _kL^h(k). Then (g¯h,uh)( g_h,u_h) solves equation 64 with ℓh _h replaced by khk_h. The disadvantages values in equation 65 are unchanged, and equation 71 holds with the transformed gains. Define vh(b,j):=mina∈(b,j)kh(b,j,a)+(Pauh)(b)−g¯hv_h(b,j):= _a (b,j)\k_h(b,j,a)+(P_au_h)(b)\- g_h. For a state s=(b,k)s=(b,k) and an action a∈(b,k)a (b,k), h(s,a)=kh(s,a)−g¯h+[vh(St+1)∣St=s,At=a]−vh(s). d_h(s,a)=k_h(s,a)- g_h+E[v_h(S_t+1) S_t=s,A_t=a]-v_h(s). (111) Moreover, for any instance, any algorithm, and any bounded w:ℬ×0,…,K→ℝw:B×\0,…,K\ , with JT+1∼ωJ_T+1 ω independently after the last decision and ST+1:=(BT+1,JT+1)S_T+1:=(B_T+1,J_T+1), the process ℳt:=∑s=1t(w(Ss+1)−[w(Ss+1)∣Ss,As])M_t:= _s=1^t(w(S_s+1)-E[w(S_s+1) S_s,A_s]), 0≤t≤T0≤ t≤ T, is a martingale with respect to ℋt:=ℱt∨σ(Jt+1)H_t:=F_t σ(J_t+1), with |ℳt−ℳt−1|≤sp(w)|M_t-M_t-1| (w) for 1≤t≤T1≤ t≤ T. Proof. Since ℓh(b,j,a)=kh(b,j,a)+Lh(j) _h(b,j,a)=k_h(b,j,a)+L^h(j) and Lh(j)L^h(j) does not depend on the action, subtracting ∑k=0KωkLh(k) _k=0^K _kL^h(k) from equation 64 gives the transformed ACOE for (g¯h,uh)( g_h,u_h). The minimizing actions and disadvantages are unchanged, and equation 71 holds with the transformed gains. For the martingale property, ℳtM_t is ℋtH_t-measurable. Since Bt+1=(Bt+At−Ct)+B_t+1=(B_t+A_t-C_t)^+ with CtC_t independent of (ℋt−1,At)(H_t-1,A_t), and Jt+1∼ωJ_t+1 ω is independent of (ℋt−1,At,Ct)(H_t-1,A_t,C_t), the conditional law of St+1S_t+1 given ℋt−1H_t-1 and AtA_t depends only on (St,At)(S_t,A_t); the tower property therefore gives [ℳt−ℳt−1∣ℋt−1]=0E[M_t-M_t-1 _t-1]=0. The increment bound follows from |w(St+1)−[w(St+1)∣St,At]|≤sp(w)|w(S_t+1)-E[w(S_t+1) S_t,A_t]| (w). ∎ Appendix D Additional related work D.1 Alternative self-confidence signals Our experiments use the self-consistency score, i.e. the empirical mass of the largest answer class. This choice is natural when multiple responses are sampled, and answer consistency has previously been used as a signal to decide whether to escalate queries in LLM cascades (Yue et al., 2024). However, the deferral and learning components of our model do not depend on this particular definition of the score. They only require that queries be partitioned into levels Jt∈[K]J_t∈[K], each with an unknown conditional accuracy hj⋆=Pr(Yt=1∣Jt=j)h_j = (Y_t=1 J_t=j) that can be learned from selectively observed expert feedback. Thus, other scalar self-confidence measures can be substituted for self-consistency and discretized into levels before applying the same deferral framework. In particular, neither the deferral rule nor our analysis requires the self-confidence signal itself to be calibrated or the conditional accuracy to be monotone in that signal. Sample-based self-confidence. Let tC_t denote the set of answer classes represented among the G sampled responses, with empirical frequencies qt,c:=1G∑g=1GR^t,g∈c,c∈t.q_t,c:= 1G _g=1^G1\ R_t,g∈ c\, c _t. Writing qt,(1)≥qt,(2)≥⋯q_t,(1)≥ q_t,(2)≥·s for the ordered frequencies, with qt,(2)=0q_t,(2)=0 if |t|=1|C_t|=1, the self-consistency score is simply qt,(1).q_t,(1). It therefore retains only the empirical mass of the most frequent answer class. A natural alternative is to use the entropy of the empirical answer distribution, Ht:=−∑c∈tqt,clogqt,c.H_t:=- _c _tq_t,c q_t,c. which takes all sampled answer classes into account. For tasks with canonical answers, the classes can be defined directly by the extracted answer, yielding an entropy over answer labels. For open-ended tasks, responses can instead be grouped according to semantic equivalence, giving the discrete variant of semantic entropy (Kuhn et al., 2023; Farquhar et al., 2024). Semantic entropy has also been considered directly as an uncertainty signal for LLM routing (Zhang et al., 2025). Another simple alternative is the margin between the two most frequent answer classes, qt,(1)−qt,(2),q_t,(1)-q_t,(2), which measures how strongly the dominant answer is separated from its closest competitor. Margin-based self-confidence scores have been used to control escalation in language-model cascades (Ramírez et al., 2024). The statistic above applies the same principle directly to the empirical distribution of sampled answer classes. For open-ended generation, self-confidence can also be estimated without first assigning responses to hard equivalence classes. Kernel Language Entropy (KLE) (Nikitin et al., 2024), for example, constructs a kernel from pairwise semantic similarities between sampled responses and measures uncertainty through its von Neumann entropy. Semantic voting (Jiang et al., 2026) also aggregates pairwise semantic similarities, using them to select responses for model self-improvement. Model-internal self-confidence. When model logits are available, self-confidence can instead be estimated from the predictive distributions produced during generation. For a generated response R^t=(r^t,1,…,r^t,Lt) R_t=( r_t,1,…, r_t,L_t), let pt,ℓ:=Pr(r^t,ℓ∣Xt,r^t,<ℓ)p_t, := ( r_t, X_t, r_t,< ) denote the probability assigned by the model to the generated token at position ℓ . For example, Gupta et al. (2024) consider the sequence log-probability, which they refer to as the Chow-Sum, Stsum:=∑ℓ=1Ltlogpt,ℓ,S_t^sum:= _ =1^L_t p_t, , as a measure of model confidence. Since this score is affected by sequence length, they also consider the length-normalized Chow-Average score Stavg:=1Lt∑ℓ=1Ltlogpt,ℓ.S_t^avg:= 1L_t _ =1^L_t p_t, . These and other model-internal scores can be incorporated into our model by discretizing the resulting scalar self-confidence signal into levels and learning the corresponding conditional accuracies online. These alternatives modify the self-confidence signal supplied to the deferral layer, but not the central learning problem studied in this work. In all cases, the system must learn from delayed and selectively observed feedback which score levels are sufficiently reliable to accept, while simultaneously accounting for the state-dependent cost of expert intervention. D.2 Probabilistic calibration Calibration is the property that characterizes the relationship between a model’s confidence scores and the probability that its predictions are correct: a confidence score is calibrated if predictions assigned confidence p are correct with probability p. Classical post-hoc calibration methods include histogram binning (Zadrozny and Elkan, 2001), Platt scaling (Platt, 1999), isotonic regression (Robertson et al., 1988; Zadrozny and Elkan, 2002). In our problem, the conditional accuracies are learned from delayed, selectively observed human feedback, and learning is evaluated through deferral regret rather than calibration error. D.3 Additional learning-to-defer formulations Charusaie et al. (2022) study learning predictors that complement humans and develop active learning schemes to reduce the number of expert predictions required during training. In our model, expert feedback is instead acquired during deployment to learn the conditional accuracies of a fixed LLM. Learning to defer with multiple experts. Building on the single-expert setting, learning to defer has been generalized to multiple heterogeneous experts whose competencies may vary across inputs. In this setting, the learner must determine not only whether to defer, but also which expert should handle each instance. Early multi-expert formulations jointly learned a predictive model and an allocation rule while accounting for heterogeneous expert competencies and, in some cases, expert biases and fairness considerations (Keswani et al., 2021; Hemmer et al., 2022). Subsequent work developed more principled formulations for expert selection, including calibrated and consistent surrogate objectives (Verma et al., 2023). A complementary line of work has focused on the theoretical foundations of multi-expert deferral. Mao et al. (2023) study the two-stage setting, in which a predictor is fixed and a deferral function is learned to select among experts, while Mao et al. (2024a) develop surrogate losses with realizable H-consistency, H-consistency, and Bayes-consistency guarantees. More recently, Mao et al. (2025) provide a unified treatment of single-stage and two-stage multi-expert deferral and establish stronger consistency guarantees for both settings. Other extensions consider learning with limited expert annotations (Hemmer et al., 2023), adaptation to previously unseen experts (Tailor et al., 2024), regression (Mao et al., 2024b), imbalanced expert data (Cortes et al., 2026), and Top-k deferral, where multiple experts may be selected for the same instance (Montreuil et al., 2026). D.4 LLM routing FrugalGPT constructs LLM cascades to reduce monetary cost while maintaining predictive quality (Chen et al., 2024). RouteLLM learns routers between stronger and weaker models using preference data (Ong et al., 2025). Jitkrittum et al. (2026) introduce UniRoute, a model-routing framework that represents LLMs through their prediction-error profiles on representative prompts, thus enabling cost-aware routing to previously unseen models without retraining the router. These methods allocate computation among automated models. We first sample from a fixed LLM, then decide whether to defer to a capacity-constrained human expert whose completed reviews supply ground-truth labels. D.5 Structured admission control Queue-aware learning has also been studied through structured admission-control policies. In particular, UCRL-AC (Weber et al., 2024) exploits the structure of multi-class admission control to obtain sharper worst-case regret guarantees than generic MDP approaches, while SALMUT (Roy et al., 2021) assumes that the optimal policy has an ordered multi-threshold structure and directly learns the corresponding vector of class-specific thresholds using stochastic approximation. The learning problems differ substantially from ours. In Weber et al. (2024), arrival rates are unknown and are learned passively from exogenously observed arrivals, while the class-dependent reward model is known. Roy et al. (2021) allows the transition dynamics to be unknown, but learns from state, reward, and transition observations generated by interaction with the system. In our model, by contrast, the confidence level-dependent cost of accepting the LLM answer is itself unknown since it depends on the unknown conditional accuracy vector, and it is learned only through queries deferred to the expert. Thus, obtaining information about prediction quality requires consuming expert capacity and increasing queue congestion, with the resulting feedback additionally delayed by expert service. D.6 Active learning and oracle queries Active learning studies how a learner can selectively acquire costly labels while maintaining predictive performance (Dasgupta, 2011; Hanneke, 2014). Online variants require these query decisions to be made sequentially under partial feedback (Hao et al., 2018). Réveillard et al. (2026) study online classification in which costly expert consultations provide labeled examples used to improve subsequent predictions. In our model, the LLM remains fixed, and expert feedback instead estimates conditional accuracies to guide deferral. Russo et al. (2024) study a limited budget of best-action queries, which reveal the identity of the best action before the current decision. This information differs from our completed-review labels, which reveal whether the LLM response was correct without revealing the optimal deferral decision. D.7 Human–AI decision support Related work studies how the relationship between human and AI confidence affects the usefulness of AI advice (Corvelo Benz and Rodriguez, 2023; Corvelo Benz and Gomez Rodriguez, 2025). Benz et al. (2026) study learning decision policies through repeated interactions under human-alignment and full feedback. In our model, an accepted LLM response provides no correctness feedback, and expert review is limited by queue capacity. Appendix E Datasets E.1 Synthetic datasets In this section, we describe the two synthetic instances used in Appendix G. Both have K=10K=10 self-confidence levels and are shown in Figure 5. Figure 5: Top: two-point instance with δ=0.08δ=0.08. Bottom: monotone instance (dashed line: perfect calibration, i.e., hj⋆=pjh _j=p_j with pj=(j−1)/9p_j=(j-1)/9). Two-point instance. The first instance uses the two-point family hj⋆∈0.5−δ,0.5+δh _j∈\0.5-δ,0.5+δ\, j∈[K]j∈[K], with uniform arrival rates ωj=1/K _j=1/K (top panel of Figure 5). This construction is motivated by the hard instances used in the proof of our minimax lower bound (Theorem 4): under immediate service (μ=1μ=1), uniform arrivals and cexp/ce=1/2c_ exp/c_e=1/2, the oracle threshold is τ=1/2τ=1/2, and if K≤TK≤ T, these instances take hj⋆∈τ−δ,τ+δh _j∈\τ-δ,τ+δ\ with δ=18K/Tδ= 18 K/T. Appendix G.1 uses this instance in that setting only, with δ as a free parameter. Monotone instance. The second instance combines increasing conditional accuracies h⋆=(.04,.13,.23,.33,.43,.54,.64,.75,.86,.94)h =(.04,.13,.23,.33,.43,.54,.64,.75,.86,.94) with arrival rates concentrated on the high levels (bottom panel of Figure 5). Since the scores are pj=(j−1)/9p_j=(j-1)/9, they are nearly calibrated: this instance models an LLM that is slightly underconfident at the three lowest levels and overconfident at the others, by at most 0.060.06. Since the conditional accuracies span most of [0,1][0,1], the oracle threshold falls between the accuracies of two levels for a wide range of service rates, costs and queue sizes. This ensures the optimal actions do not become trivial to learn when we vary these parameters, unless deferral is never worthwhile (cexp≥cec_ exp≥ c_e). We use this instance in Appendix G.2, where we vary the service rate μ, the expert cost cexpc_ exp and the holding cost choldc_ hold, with the conditional accuracies held fixed. E.2 Real-world datasets In this section, we describe the datasets used in our real-world experiments. We start from three question-answering datasets with verifiable answers: • GSM8K (Cobbe et al., 2021) contains grade-school math problems with numerical answers. We use all 8,7928,792 questions from its train and test splits. • NuminaMath-TIR (Li et al., 2024) contains competition-style mathematics problems. We use its first 10,00010,000 training questions, from which we remove 157157 exact duplicates and 659659 questions that also appear in GSM8K, leaving 9,1849,184. • MedMCQA is a large-scale medical dataset derived primarily from medical entrance and professional examinations (Pal et al., 2022). It contains multiple-choice questions with four options. We use all 187,005187,005 questions of its train and validation splits. We use Qwen3-4B-Instruct-2507, Qwen3-0.6B, and Qwen2.5-0.5B-Instruct (Yang et al., 2025a; Qwen et al., 2025), and sample 2020 answers per question. Answers are sampled with vLLM at temperature 0.60.6 and top-p 0.950.95, without top-k truncation, with up to 2,0482,048 generated tokens per answer and a maximum model sequence length (prompt plus output) of 4,0964,096 tokens. Answer extraction and grading. A final answer is extracted from each generation by taking the last complete <answer> block. If none is present, we use a terminal expression or an explicit final-answer declaration. Otherwise, no answer is returned. Answers are grouped into equivalence classes, either by mathematical equivalence with Math-Verify (Hugging Face, 2025) for the math datasets, which handles fractions, radicals, tuples, sets and symbolic expressions, or by option letter for MedMCQA. The same equivalence decides whether Rt≡Rt⋆R_t≡ R _t. Ties between the largest classes are broken by first occurrence. Generations without an extractable answer count in the denominator of the score but cast no vote, so a question on which no generation yields an answer has score 00 and Yt=0Y_t=0. Questions whose ground-truth answer could not be parsed, or whose grading timed out, are removed: 5858 NuminaMath questions for the Qwen3 models, and 8585 NuminaMath and 1212 GSM8K questions for Qwen2.5-0.5B. Instances. Figure 6 shows h⋆h and ω for all (dataset, LLM) pairs, from which the instances are built as in Section 6. The regimes of Figure 2 extend to Qwen2.5-0.5B: its conditional accuracy is above the score on the math datasets, especially on GSM8K, and varies little with the score on MedMCQA. Across all three datasets, the self-consistency score is generally not calibrated, i.e. h⋆≠ph ≠ p. Moreover, h⋆h generally is not monotone in the self-consistency level j. This motivates learning h⋆h rather than assuming the score is calibrated. Figure 6: Conditional accuracy (top) and arrival rates (bottom) of the self-consistency score. The dashed line represents a perfectly calibrated score. Appendix F Baselines and implementation details The costs (ce,cexp,chold)(c_e,c_ exp,c_ hold), the service distribution ρC _C and the queue capacity B¯ B are known to all algorithms. In the synthetic experiments, ω is known. In the real-world experiments it is unknown, and every learning algorithm that needs it uses the empirical frequencies ω^j,t=Oj,t/t ω_j,t=O_j,t/t, where Oj,t=∑s≤tJs=jO_j,t= _s≤ t1\J_s=j\ counts the arrivals at self-confidence level j up to round t. Let Nj,tN_j,t and Cj,tC_j,t be the number of level-j labels received before the decision of round t and the number of correct ones among them, and h^j,Nj,t=(Cj,t/Nj,t)Nj,t>0 h_j,N_j,t=(C_j,t/N_j,t)1\N_j,t>0\ their empirical mean. Plug-in algorithms (KL-PBD, KL-PBD+, GBD, NBD). These four algorithms share the decision rule of KL-PBD, At=ce(1−hJt,t)>Mtht(Bt)A_t=1\c_e(1-h_J_t,t)>M^h_t_t(B_t)\, and differ only in the conditional accuracy vector hth_t plugged into the marginal cost of deferral MthtM^h_t_t: • KL-PBD: the KL lower confidence bound h¯j,t h_j,t with confidence budget ϕ(T)=logT+loglogTφ(T)= T+ T; • KL-PBD+: the same bound with the level-dependent budget log(Oj,t/Nj,t) (O_j,t/N_j,t) in place of ϕ(T)φ(T), in the spirit of the KL-UCB+ family of algorithms (Garivier and Cappé, 2011; Achab et al., 2018); • GBD (Greedy Bellman Deferral): the empirical mean h^j,Nj,t h_j,N_j,t; • NBD (Naive Bellman Deferral): the surrogate h=ph=p, i.e. the score is taken as calibrated and nothing is learned about h⋆h . With known ω, the finite-horizon recursion is re-solved whenever a new label arrives (never for NBD). With unknown ω, it is re-solved at every round with the current (ht,ω^t)(h_t, ω_t), for NBD included. The marginal costs are computed exactly by backward induction over the remaining horizon, with the terminal draining cost D(b)D(b) of Section 3. Fixed-threshold policies (FCT, FQT). Fixed Confidence Threshold defers iff pJt<τconfp_J_t<τ conf; Fixed Queue Threshold defers iff Bt<τJtqueueB_t<τ queue_J_t for a threshold vector τqueue∈0,…,B¯Kτ queue∈\0,…, B\^K. Both are tuned once before the run to minimize the expected cost under the surrogate h=ph=p, computed exactly by dynamic programming. When ω is unknown the uniform distribution ωj=1/K _j=1/K is used in place of ω. Since only the ordering of p1,…,pK\p_1,…,p_K\ matters for FCT, its K+1K+1 candidate policies are enumerated exhaustively. For FQT, exhaustive search over 0,…,B¯K\0,…, B\^K is infeasible, so we use coordinate descent: each threshold τjqueue _j queue is optimized over 0,…,B¯\0,…, B\ with the others fixed, cycling over j=1,…,Kj=1,…,K until the vector no longer changes. We run this from several deterministic initializations and keep the best result, which need not be the global optimum. SALMUT-SF. We adapt SALMUT (Roy et al., 2021), which defers a level-j query iff Bt<τjB_t< _j, with one queue threshold τj∈[0,B¯] _j∈[0, B] per level, initialized at τj=0 _j=0. In the setting of Roy et al. (2021), h would be known. In our case, we estimate it with h^j,Nj,t h_j,N_j,t. Let b=Btb=B_t, j=Jtj=J_t, a=Ata=A_t and b′=Bt+1b =B_t+1. At the end of round t, after service and label processing, the relative-value critic, initialized at V(b)=0V(b)=0, is updated at the pre-decision queue state b, including when b=B¯b= B, with step size αnb=(⌊nb/100⌋+2)−0.6 _n_b=( n_b/100 +2)^-0.6 where nbn_b counts the updates at state b: V(b)←V(b)+αnb[choldb+cexpa+ce(1−h^j,Nj,t+1)(1−a)+V(b′)−V(0)−V(b)].V(b)← V(b)+ _n_b [c_ holdb+c_ expa+c_e(1- h_j,N_j,t+1)(1-a)+V(b )-V(0)-V(b) ]. For the actor, we retain SALMUT’s sigmoid interpolation f(b,τ)=1/(1+e−(b−τ−1/2))f(b,τ)=1/(1+e^-(b-τ-1/2)). Note that unlike Roy et al. (2021), the transition probabilities are assumed known in our setting, so they do not need to be estimated. Therefore, after the critic update, for b<B¯b< B, we estimate the advantage function via the updated critic, Q^1−Q^0=cexp−ce(1−h^j,Nj,t+1)+(P1V)(b)−(P0V)(b) Q_1- Q_0=c_ exp-c_e(1- h_j,N_j,t+1)+(P_1V)(b)-(P_0V)(b), where (PaV)(b)=∑b′Pa(b,b′)V(b′)(P_aV)(b)= _b P_a(b,b )V(b ), and for b<B¯b< B we update the threshold of the arriving level as τj←Π[0,B¯][τj−10t+100f(b,τj)(1−f(b,τj))(Q^1−Q^0)]. _j← _[0, B] [ _j- 10t+100\,f(b, _j) (1-f(b, _j) )( Q_1- Q_0) ]. We do not use SALMUT’s ordering constraint on the thresholds, since conditional accuracies need not be ordered in the score. When the threshold rule accepts, we force a feasible deferral with probability εt=min.5, 2/t _t= \.5,\,2/ t\; without it a level can stop producing the labels needed to revise an optimistic threshold due to the selective nature of the feedback in our problem (this failure mode is explored in Appendix H.5). The same hyperparameters are used in every instance. Queue-unaware bandit algorithms (KL-UCB-4P, KL-UCB+-4P). Under immediate service (μ=1μ=1), the queue stays empty and Mt(0)=cexpM_t(0)=c_ exp. The problem then reduces to a separate learning task for each self-confidence level: at each round the level Jt=jJ_t=j is revealed and the learner faces a two-armed bandit with one known arm. Indeed, one can equivalently subtract the cost ce(1−Yt)c_e(1-Y_t) of accepting from both actions, so that accepting has reward 00, while deferring has reward ce(1−Yt)−cexpc_e(1-Y_t)-c_ exp, with unknown mean ce(1−hj⋆)−cexpc_e(1-h _j)-c_ exp, observed only when the query is deferred. This is close to the profitable-bandit problem of Achab et al. (2018), played independently at each level, which motivates a comparison with their algorithms. We adapt their algorithms to our setting, so that they defer iff ceq¯j,t≥cexpc_e q_j,t≥ c_ exp, ignoring the queue except when it is full, where q¯j,t q_j,t is the KL upper confidence bound on the LLM error probability 1−hj⋆1-h _j, computed from 1−h^j,Nj,t1- h_j,N_j,t with budget logOj,t+3loglogOj,t O_j,t+3 O_j,t for KL-UCB-4P and log(Oj,t/Nj,t) (O_j,t/N_j,t) for KL-UCB+-4P, using the level clock Oj,tO_j,t in place of t, with unlabeled levels having q¯j,t=1 q_j,t=1 so that they are deferred. Equivalently, up to ties, they follow the decision rule of KL-PBD with Mtht(Bt)M^h_t_t(B_t) replaced by cexpc_ exp, its value under immediate service: KL-UCB+-4P is then KL-PBD+, and KL-UCB-4P is KL-PBD with budget logOj,t+3loglogOj,t O_j,t+3 O_j,t in place of ϕ(T)φ(T). When μ<1μ<1, labels arrive only once a deferred query is served, and the algorithms use the labels available at each round. Simulation protocol. For every run, all algorithms see the same arrivals (Jt)(J_t), latent labels (Yt)(Y_t) and service capacities (Ct)(C_t), and keep separate first-in, first-out (FIFO) queues. A label completed after the decision of round t is available from round t+1t+1. We compute the regret as R^T=∑t≤T,Bt<B¯|L⋆(Jt)−Mt⋆(Bt)|At≠At⋆ R_T= _t≤ T,\,B_t< B|L (J_t)-M _t(B_t)|1\A_t≠ A _t\ (Lemma 2), averaged over independent runs with pointwise 95%95\% normal confidence intervals. The code used for our experiments is provided in the supplementary files. Appendix G Synthetic experiments G.1 Effect of K and of the oracle gap on KL-PBD regret In this experiment, we examine how the number of levels, the separation between the accuracies and the oracle threshold, and the arrival distribution ω affect KL-PBD’s regret. We use the two-point instance of Appendix E.1 (top row of Figure 5) with μ=1μ=1, ce=1c_e=1 and cexp=0.5c_ exp=0.5, so that the oracle threshold is τ=1/2τ=1/2: the oracle defers the levels j with hj⋆=0.5−δh _j=0.5-δ and accepts those with hj⋆=0.5+δh _j=0.5+δ. All experiments use T=2⋅105T=2· 10^5 and 100100 runs. Figure 7: Regret of KL-PBD on the two-point instance for different K with δ=0.08δ=0.08 (left, shown up to t=105t=10^5) and different δ with K=10K=10 (center), and for different arrival distributions with K=10K=10 and hj⋆=0.5+ϕ(T)/(2ωjT)h _j=0.5+ φ(T)/(2 _jT) (right). 95%95\% confidence intervals over 100 runs are displayed as bands but are too small to be visible. Figure 7 shows that regret increases with K, since the learner must acquire more expert feedback to identify the correct action in a larger number of levels. The dependence on δ reflects the usual gap-dependent learning trade-off. Larger gaps make an incorrect deferral more costly initially, but they are statistically easier to distinguish from the oracle threshold, so the learner identifies the optimal action sooner and the regret curves flatten earlier. Conversely, when δ is small, the conditional accuracies lie closer to the threshold and require substantially more feedback to distinguish, leading to larger regret over the range of the figure. Under immediate service, KL-PBD defers level j until Nj,tkl(h^j,Nj,t,1/2)≥ϕ(T)N_j,t\,kl( h_j,N_j,t,1/2)≥φ(T). A level with hj⋆=0.5+δh _j=0.5+δ therefore needs about ϕ(T)/(2δ2)φ(T)/(2δ^2) labels before it is accepted, each deferral costing δ, while a level with hj⋆=0.5−δh _j=0.5-δ is deferred, as by the oracle. Each level above the threshold thus contributes about minδωjT,ϕ(T)/(2δ) \δ\, _jT,\ φ(T)/(2δ)\ to the regret. Table 1 complements Figure 7 by reporting the final regret at T=2⋅105T=2· 10^5. The results show that regret increases significantly with K, confirming the additional learning burden associated with a larger number of levels. At δ=0.08δ=0.08, every oracle decision is learned well before T, and each level above the threshold adds about ϕ(T)/(2δ)≈90φ(T)/(2δ)≈ 90 to the regret. For every value of K except K=50K=50, regret decreases as δ increases; we explain the exception below. K δ=0.02δ=0.02 δ=0.04δ=0.04 δ=0.06δ=0.06 δ=0.08δ=0.08 δ=0.10δ=0.10 δ=0.15δ=0.15 2 334±23334± 23 179±13179± 13 121±9121± 9 87±787± 7 71±571± 5 50±350± 3 5 1085±511085± 51 555±21555± 21 357±13357± 13 267±12267± 12 209±9209± 9 138±6138± 6 10 1592±361592± 36 878±28878± 28 589±20589± 20 439±13439± 13 353±12353± 12 235±7235± 7 20 1948±111948± 11 1757±441757± 44 1181±291181± 29 889±17889± 17 706±15706± 15 462±11462± 11 50 1997±21997± 2 3564±303564± 30 2966±422966± 42 2225±342225± 34 1789±271789± 27 1168±141168± 14 Table 1: Final regret RTR_T of KL-PBD at T=2⋅105T=2· 10^5 for different K and δ, mean ± 95% confidence interval half-width over 100 runs. δ 0.0025 0.005 0.01 0.015 0.0192†0.0192 0.025 RTR_T 250±0250± 0 500±0500± 0 998±3998± 3 1442±161442± 16 1596±321596± 32 1396±431396± 43 δ 0.03 0.04 0.06 0.08 0.10 0.15 RTR_T 1189±411189± 41 878±28878± 28 589±20589± 20 439±13439± 13 353±12353± 12 235±7235± 7 Table 2: Final regret RTR_T of KL-PBD at T=2⋅105T=2· 10^5 for K=10K=10 and different δ, mean ± 95% confidence interval half-width over 100 runs. † : δ⋆=ϕ(T)K/(2T)≈0.019δ = φ(T)K/(2T)≈ 0.019. Table 2 further illustrates the non-monotone dependence of finite-horizon regret on the oracle gap δ. For very small δ, distinguishing which side of the oracle threshold the accuracy is requires many observations, but the regret incurred by an incorrect decision is itself only O(δ)O(δ), so the regret is at most of order δTδ T. In this case, KL-PBD does not have enough time to learn the oracle decisions, and RT≈δT/2R_T≈δ T/2. For larger δ, the levels are statistically easier to distinguish and the regret decreases roughly as Kϕ(T)/(4δ)Kφ(T)/(4δ). Consequently, the largest regret is observed near the crossover between these two regimes, δ⋆=ϕ(T)K/(2T)≈0.019δ = φ(T)K/(2T)≈ 0.019 for K=10K=10. Since δ⋆δ grows with K (δ⋆≈0.043δ ≈ 0.043 for K=50K=50), the regret for K=50K=50 is lower at δ=0.02δ=0.02 than at δ=0.04δ=0.04 (Table 1). The right panel examines the effect of non-uniform arrivals with K=10K=10. We parameterize ω1=1K+ρ(1−1K),ωj=1−ρK,j=2,…,K, _1= 1K+ρ (1- 1K ), _j= 1-ρK, j=2,…,K, so that ρ=0ρ=0 corresponds to uniform arrivals and increasing ρ concentrates probability mass on a single level. We set hj⋆=0.5+δjh _j=0.5+ _j with δj=ϕ(T)/(2ωjT) _j= φ(T)/(2 _jT) for every level. Unlike in the two-point instance, all levels lie above the threshold: a level below it is deferred by both KL-PBD and the oracle and adds no regret, so this makes every level contribute. Level j receives about ωjT _jT arrivals and needs about ϕ(T)/(2δj2)=ωjTφ(T)/(2 _j^2)= _jT labels before it is accepted, so with this gap KL-PBD starts accepting level j at about the horizon. For uniform arrivals, δj=δ⋆ _j=δ . Each level then contributes about δjωjT=ϕ(T)ωjT/2 _j\, _jT= φ(T)\, _jT/2 to the regret. Regret is largest under the uniform distribution and decreases as the arrival mass becomes more concentrated. This ordering is consistent with the dependence on ω in Theorem 2, whose leading term is O~(T∑j=1Kωj) O ( T _j=1^K _j ) and is maximized when ωj=1/K _j=1/K for j∈[K]j∈[K]. Quantitatively, RT/(T∑j=1Kωj)R_T/( T _j=1^K _j) stays between 2.212.21 and 2.292.29 for all ρ. G.2 Effect of service rate and cost parameters In this experiment we use the monotone instance (bottom row of Figure 5) in order to evaluate how KL-PBD is affected by the expert service rate. We set the queue capacity to B¯=20 B=20, the LLM error cost to ce=2c_e=2, the expert-review cost to cexp=0.25c_ exp=0.25, and the holding cost to chold=0.1c_ hold=0.1. We vary the expert service rate over μ∈0.2,0.4,0.5,0.6,0.8,1.0μ∈\0.2,0.4,0.5,0.6,0.8,1.0\, where in each round the expert completes a pending query with probability μ. For each service rate, we run the system for T=50,000T=50,000 rounds and average results over 20 independent runs. We report the average queue size, the cumulative regret relative to the oracle, the response accuracy, i.e., the fraction of queries answered correctly, counting deferred queries as correct, and the average threshold, i.e., the time average of τ^t(Bt) τ_t(B_t). Table 3: Finite-horizon KL-PBD performance across service rates μ with B¯=20 B=20. Results are mean ± 95% confidence interval half-width over 20 runs. Service rate μ Average queue size Cumulative regret RTR_T Response accuracy Average threshold 0.20 0.74±0.010.74± 0.01 194.06±14.96194.06± 14.96 0.7710±0.00100.7710± 0.0010 0.2663±0.00240.2663± 0.0024 0.40 0.70±0.010.70± 0.01 233.59±30.80233.59± 30.80 0.8445±0.00080.8445± 0.0008 0.5130±0.00300.5130± 0.0030 0.50 0.74±0.000.74± 0.00 115.49±7.42115.49± 7.42 0.8856±0.00050.8856± 0.0005 0.5716±0.00130.5716± 0.0013 0.60 0.55±0.010.55± 0.01 135.36±11.75135.36± 11.75 0.9070±0.00060.9070± 0.0006 0.6579±0.00180.6579± 0.0018 0.80 0.37±0.000.37± 0.00 301.17±19.26301.17± 19.26 0.9603±0.00110.9603± 0.0011 0.7632±0.00100.7632± 0.0010 1.00 0.00±0.000.00± 0.00 80.61±10.9180.61± 10.91 0.9937±0.00010.9937± 0.0001 0.8750±0.00000.8750± 0.0000 Table 3 shows that increasing the expert service rate generally makes deferral more attractive. As μ increases, deferred queries are processed more quickly, reducing the congestion cost associated with sending additional jobs to the expert. Consistent with this behavior, the average threshold increases from 0.26630.2663 at μ=0.2μ=0.2 to 0.8750=1−cexp/ce0.8750=1-c_ exp/c_e at μ=1.0μ=1.0, while response accuracy improves substantially from 0.77100.7710 to 0.99370.9937. The average queue size remains below one across all service rates, falls from 0.740.74 at μ=0.5μ=0.5 to 0.370.37 at μ=0.8μ=0.8, and reaches zero at μ=1.0μ=1.0, where deferred jobs are cleared immediately. Cumulative regret remains non-monotone in μ, ranging from 80.6180.61 to 301.17301.17 across the tested service rates. In particular, regret is relatively high at μ=0.4μ=0.4 and reaches its maximum at μ=0.8μ=0.8, despite the corresponding improvements in response accuracy and queue length. This reflects the fact that each service rate induces a different control problem and oracle policy. At μ=1.0μ=1.0, the experiment attains both its highest response accuracy, 0.99370.9937, and its lowest cumulative regret, 80.6180.61. Figure 8: KL-PBD threshold τ^T/2(b) τ_T/2(b) as a function of queue size b, averaged over the runs (left), and learned KL-PBD decisions, most frequent over the runs (right), at t=T/2t=T/2 for different service rates μ. Figure 8 (right) shows the policy learned by KL-PBD at t=T/2t=T/2. It exhibits a clear threshold structure: within each level, the action changes from defer to accept once the queue size exceeds a level-dependent boundary, producing the staircase-shaped frontier visible across all panels. As the expert service rate μ increases, the defer region expands toward larger queue sizes, since a faster expert clears deferred requests more quickly. Figure 8 (left) shows how the KL-PBD threshold τ^t(b) τ_t(b) varies with the current queue size for different expert service rates. For every value of μ, the threshold decreases as the queue grows, reflecting the increasing congestion cost of sending additional queries to the expert. Once the queue becomes sufficiently large, the threshold reaches zero, and no query is deferred. Increasing the service rate shifts the threshold curve upward and extends it over larger queue sizes. When the expert is slow, as for μ=0.2μ=0.2, the threshold falls rapidly and reaches zero at a small queue size. In contrast, faster service allows the system to tolerate substantially more queued queries before the threshold becomes small. We then fix μ=0.5μ=0.5 and vary the holding cost chold∈0.005,0.01,0.05,0.1,0.5c_ hold∈\0.005,0.01,0.05,0.1,0.5\ with cexp=0.25c_ exp=0.25, and the expert-review cost cexp∈0,0.25,1,3c_ exp∈\0,0.25,1,3\ with chold=0.05c_ hold=0.05, over 10 independent runs of T=50,000T=50,000 rounds. In these sweeps, the thresholds of KL-PBD are recomputed after every 50 changes of the lower bounds. Table 4: KL-PBD performance across holding and expert-review costs with μ=0.5μ=0.5. Results are mean ± 95% confidence interval half-width over 10 runs. choldc_hold cexpc_exp Mean queue Defer (%) Accuracy (%) Average threshold Regret 0.005 0.25 2.840±0.0572.840± 0.057 49.62±0.1849.62± 0.18 91.27±0.1191.27± 0.11 0.635±0.0030.635± 0.003 138.20±8.81138.20± 8.81 0.010 0.25 2.106±0.0592.106± 0.059 48.68±0.2948.68± 0.29 90.87±0.1390.87± 0.13 0.638±0.0050.638± 0.005 183.55±14.20183.55± 14.20 0.050 0.25 1.042±0.0271.042± 0.027 45.20±0.2445.20± 0.24 89.48±0.1289.48± 0.12 0.604±0.0040.604± 0.004 145.86±18.98145.86± 18.98 0.100 0.25 0.751±0.0070.751± 0.007 43.52±0.1743.52± 0.17 88.56±0.1088.56± 0.10 0.571±0.0030.571± 0.003 118.58±13.78118.58± 13.78 0.500 0.25 0.229±0.0050.229± 0.005 22.90±0.3922.90± 0.39 81.06±0.1181.06± 0.11 0.429±0.0030.429± 0.003 222.55±33.59222.55± 33.59 0.050 0.00 1.221±0.0101.221± 0.010 48.72±0.1848.72± 0.18 89.97±0.1089.97± 0.10 0.611±0.0030.611± 0.003 71.57±10.9771.57± 10.97 0.050 1.00 0.264±0.0110.264± 0.011 18.71±0.7918.71± 0.79 80.20±0.3680.20± 0.36 0.441±0.0010.441± 0.001 432.32±57.30432.32± 57.30 0.050 3.00 0.000±0.0000.000± 0.000 0.00±0.000.00± 0.00 67.83±0.1267.83± 0.12 0.000±0.0000.000± 0.000 0.00±0.000.00± 0.00 Table 4 shows that the holding cost and expert-review cost affect KL-PBD in distinct ways. With cexp=0.25c_exp=0.25, increasing choldc_hold from 0.0050.005 to 0.50.5 reduces the mean queue from 2.842.84 to 0.230.23 and the deferral rate from 49.62%49.62\% to 22.90%22.90\%. As fewer queries are routed to the expert, response accuracy decreases from 91.27%91.27\% to 81.06%81.06\%. The average threshold decreases overall from 0.6350.635 to 0.4290.429. Regret is non-monotone because changing choldc_hold also changes the oracle policy. The expert-review-cost sweep exhibits a sharper transition. With chold=0.05c_hold=0.05, increasing cexpc_exp from 00 to 11 reduces the mean queue from 1.221.22 to 0.260.26, the deferral rate from 48.72%48.72\% to 18.71%18.71\%, and response accuracy from 89.97%89.97\% to 80.20%80.20\%; the average threshold correspondingly decreases from 0.6110.611 to 0.4410.441. For cexp≥3c_exp≥ 3, KL-PBD does not defer, and the accuracy of 67.83%67.83\% is the LLM-only accuracy. This is due to the choice of cost parameters: the expected cost ce(1−hJt⋆)≤ce=2c_e(1-h _J_t)≤ c_e=2 of accepting the LLM response is below the expert-review cost, so both KL-PBD and the oracle accept every query, yielding zero regret. Figure 9: Effect of the expert-review cost on finite-horizon KL-PBD. We vary cexp∈0,0.25,1,3c_exp∈\0,0.25,1,3\, with T=50,000T=50,000, μ=0.5μ=0.5, B¯=20 B=20, ce=2c_e=2, and chold=0.05c_hold=0.05. From the upper-left to the lower-right, the panels show accepted-LLM accuracy, cumulative regret, empirical deferral rate, cumulative completed expert labels, average queue size, and the deferral threshold τ^T/2(b) τ_T/2(b) at t=T/2t=T/2 as a function of queue size b. Figure 9 demonstrates how the expert-review cost cexpc_exp governs the operating regime of KL-PBD. For cexp∈0,0.25c_exp∈\0,0.25\, expert review remains sufficiently inexpensive that the policy defers a large fraction of arriving queries, thereby maintaining a high accepted-LLM accuracy. As cexpc_exp increases to 11, the policy becomes substantially more selective: the deferral rate decreases, more queries are answered directly by the LLM, and the accuracy of the accepted LLM responses decreases because LLM answers are also accepted at lower confidence levels. For cexp≥3c_exp≥ 3, expert review is no longer used, and the accepted-LLM accuracy equals the LLM accuracy. The cumulative-regret trajectories show that the intermediate-cost regime is the most challenging. At cexp=1c_exp=1, expert review is still optimal in a substantial set of level–queue states, but less frequent deferrals slow the acquisition of informative expert feedback. Consequently, regret is substantially larger than for cexp∈0,0.25c_exp∈\0,0.25\. The second row makes the underlying mechanism explicit. Increasing cexpc_exp reduces the number of completed expert labels, lowers the average queue size, and shifts the threshold τ^T/2(b) τ_T/2(b) downward. Figure 10: Effect of the holding cost on KL-PBD. We vary chold∈0.005,0.01,0.05,0.1,0.5c_hold∈\0.005,0.01,0.05,0.1,0.5\ with T=50,000T=50,000, μ=0.5μ=0.5, B¯=20 B=20, ce=2c_e=2, and cexp=0.25c_exp=0.25. From the upper-left to the lower-right, the panels show accepted-LLM accuracy, cumulative regret, empirical deferral rate, cumulative completed expert labels, average queue size, and the deferral threshold τ^T/2(b) τ_T/2(b) at t=T/2t=T/2 as a function of queue size b. Figure 10 illustrates the behavior shown in Table 4 over time: as choldc_hold increases, maintaining backlog becomes more expensive, so the policy becomes increasingly selective and a larger fraction of queries is answered directly by the LLM. It also shows that the reduction in expert use limits the feedback available for learning: at low holding costs, roughly half of all queries, about 2.5×1042.5× 10^4, yield completed expert labels, compared with only about 1.1×1041.1× 10^4 at chold=0.5c_hold=0.5. The threshold curves provide the state-level view: for every choldc_hold, τ^T/2(b) τ_T/2(b) decreases with backlog, with the decline becoming markedly steeper as the holding cost increases. In particular, the threshold reaches zero at moderate backlog for chold=0.1c_hold=0.1 and almost immediately for chold=0.5c_hold=0.5, showing that higher holding costs make the deferral rule increasingly sensitive to congestion. Figure 11: Learned KL-PBD decisions for varying cexpc_ exp and choldc_ hold, for t=T/2t=T/2 (most frequent decision over the runs). Figure 11 shows how the learned KL-PBD deferral policy changes as the expert-review and holding costs increase. In both sweeps, higher costs make deferral less attractive, so the deferral region progressively shrinks: the yellow defer region contracts while the purple accept region expands, indicating that the policy relies increasingly on the LLM and sends fewer queries to the expert. Appendix H Additional real-world experiments and details This section extends the experiments of Section 6. We first report the regret of all algorithms on the instances of Section 6 and on additional Qwen2.5-0.5B instances, and in a separate experiment, we report how the final regret grows with the horizon. We then vary the service rate to show the benefit of queue awareness, and compare the policy learned by KL-PBD with the oracle policy. Finally, we show how selective feedback makes GBD fail, which illustrates the need for pessimism. Unless stated otherwise, the setting is that of Section 6. Implementation details are given in Appendix F. H.1 Regret of all algorithms Table 5 and Figure 12 report the regret of all algorithms described in Appendix F on all instances. KL-PBD+ has the lowest regret in 88 of the 99 instances. The exception is Qwen2.5-0.5B on NuminaMath, where the score is nearly calibrated and NBD (102102) is still below KL-PBD+ (115115) at T=105T=10^5; a longer horizon is needed for KL-PBD+ to overtake NBD in this instance, as we show in Table 6. The queue-unaware algorithms KL-UCB-4P and KL-UCB+-4P incur 2424 to 450450 times the regret of KL-PBD+ on eight instances. KL-UCB+-4P is only competitive on Qwen3-4B GSM8K, where the LLM accuracy averages 95%95\%, which makes deferral rare and the queue almost always empty. FCT is the worst of the three h=ph=p surrogate algorithms on every instance. Table 5: Final regret at T=105T=10^5, mean ± 95% confidence half-width over 100 runs. Model Dataset KL-PBD KL-PBD+ GBD SALMUT-SF KL-UCB-4P KL-UCB+-4P NBD FCT FQT Qwen3-4B GSM8K 100.4 ± 3.3 35.4 ± 2.7 425.5 ± 20.5 531.0 ± 26.7 223.6 ± 5.9 71.3 ± 3.8 255.6 ± 1.3 756.3 ± 2.6 477.2 ± 1.9 Qwen3-4B NuminaMath 424.4 ± 6.6 108.0 ± 4.2 1057.3 ± 82.0 1460.2 ± 41.0 7646.6 ± 101.8 4985.1 ± 156.7 231.9 ± 0.9 542.3 ± 1.2 195.4 ± 0.6 Qwen3-4B MedMCQA 99.9 ± 2.9 40.2 ± 2.5 2236.7 ± 128.8 678.8 ± 13.8 7610.5 ± 191.7 5247.8 ± 87.7 2579.0 ± 4.0 5307.1 ± 5.4 4349.8 ± 5.3 Qwen3-0.6B GSM8K 287.9 ± 7.0 93.5 ± 3.5 1468.3 ± 110.7 1173.0 ± 61.8 4654.6 ± 60.3 2227.5 ± 63.9 146.1 ± 2.6 2228.8 ± 3.8 185.5 ± 1.0 Qwen3-0.6B NuminaMath 451.9 ± 7.3 159.4 ± 5.0 1264.1 ± 287.7 3007.0 ± 68.9 23600.1 ± 40.1 21539.7 ± 57.8 528.7 ± 1.3 8414.0 ± 27.7 1040.3 ± 2.3 Qwen3-0.6B MedMCQA 106.0 ± 4.5 75.0 ± 5.8 2443.2 ± 218.0 306.0 ± 15.4 21834.8 ± 18.9 21831.9 ± 18.8 1501.1 ± 3.2 10439.5 ± 4.1 4070.2 ± 4.7 Qwen2.5-0.5B GSM8K 314.1 ± 6.3 116.0 ± 3.8 1931.4 ± 317.9 1961.7 ± 43.2 20567.7 ± 109.6 17884.8 ± 107.5 1108.8 ± 2.4 3349.7 ± 20.2 1423.1 ± 2.4 Qwen2.5-0.5B NuminaMath 298.5 ± 5.3 115.4 ± 3.6 1141.0 ± 296.9 1973.4 ± 34.0 23363.2 ± 34.0 22291.6 ± 54.5 102.3 ± 0.3 9785.0 ± 24.3 588.7 ± 1.7 Qwen2.5-0.5B MedMCQA 100.6 ± 1.9 46.5 ± 2.2 2520.2 ± 168.7 161.6 ± 9.0 21041.6 ± 18.2 21039.9 ± 18.2 1365.7 ± 1.7 10027.7 ± 7.3 1003.3 ± 1.6 Figure 12: Regret of all algorithms, T=105T=10^5, 100 runs. Truncated curves are listed with their final regret. H.2 Regret as a function of the horizon Figure 13 and Table 6 report the final regret when the horizon varies in T∈104,5⋅104,105,2⋅105T∈\10^4,5· 10^4,10^5,2· 10^5\, averaged over 24 runs per horizon. The regret of GBD and of the algorithms using the h=ph=p surrogate grows linearly in T, by a factor of about 1010 from T=104T=10^4 to 10510^5, whereas that of KL-PBD and KL-PBD+ grows by a factor between 1.61.6 and 4.44.4. SALMUT-SF lies in between, with a factor between 3.43.4 and 6.86.8. Figure 13: Final regret versus horizon T, 24 runs per point. Truncated curves are listed with their final regret. Table 6: Final regret of the horizon sweep, mean ± 95% confidence half-width over 24 runs. Model Dataset T KL-PBD KL-PBD+ GBD SALMUT-SF KL-UCB-4P KL-UCB+-4P NBD FCT FQT Qwen3-4B GSM8K 10k 43.4 ± 3.0 17.1 ± 1.8 39.9 ± 3.8 152.8 ± 5.5 97.6 ± 5.6 35.0 ± 3.9 26.4 ± 0.8 76.0 ± 1.6 48.9 ± 0.9 50k 77.5 ± 4.8 28.7 ± 3.4 206.3 ± 20.2 357.4 ± 20.8 168.8 ± 10.8 60.4 ± 5.0 128.9 ± 1.9 379.2 ± 3.7 240.8 ± 2.9 100k 98.2 ± 6.9 34.2 ± 4.3 414.6 ± 40.4 513.6 ± 33.7 221.6 ± 10.9 79.1 ± 6.7 255.7 ± 2.8 754.8 ± 4.1 478.5 ± 3.9 200k 110.8 ± 9.6 38.3 ± 5.2 831.4 ± 81.0 746.8 ± 52.9 312.0 ± 8.9 99.4 ± 10.0 510.0 ± 3.3 1510.5 ± 5.7 954.4 ± 5.2 Qwen3-4B NuminaMath 10k 130.4 ± 4.3 48.8 ± 3.5 108.0 ± 13.6 250.6 ± 18.7 1201.9 ± 25.1 560.0 ± 38.4 22.2 ± 0.8 53.7 ± 0.8 19.3 ± 0.4 50k 314.2 ± 9.8 88.6 ± 7.5 530.3 ± 69.4 813.0 ± 63.7 4416.2 ± 167.6 2504.3 ± 183.6 115.2 ± 1.2 270.2 ± 1.9 97.0 ± 0.8 100k 420.9 ± 14.4 105.2 ± 9.5 1058.1 ± 138.5 1423.0 ± 112.6 7610.0 ± 187.2 4847.4 ± 311.0 231.9 ± 2.1 541.8 ± 2.6 195.3 ± 1.2 200k 545.1 ± 17.4 120.2 ± 12.2 2113.5 ± 278.3 2554.9 ± 210.0 13963.5 ± 193.3 9537.6 ± 470.1 465.0 ± 3.1 1085.5 ± 2.8 391.6 ± 1.8 Qwen3-4B MedMCQA 10k 52.1 ± 3.6 27.8 ± 3.7 233.5 ± 23.9 144.4 ± 17.1 2512.0 ± 96.9 1000.8 ± 137.4 257.9 ± 2.9 531.2 ± 4.1 434.8 ± 3.5 50k 83.5 ± 5.4 38.8 ± 4.5 1180.6 ± 123.7 407.1 ± 20.3 5484.8 ± 371.2 3073.6 ± 185.5 1290.9 ± 5.0 2656.6 ± 8.3 2176.5 ± 9.0 100k 100.2 ± 5.6 43.5 ± 5.5 2360.2 ± 247.8 680.5 ± 20.9 7911.0 ± 395.9 5541.9 ± 205.0 2579.9 ± 8.1 5311.9 ± 13.6 4352.6 ± 13.7 200k 115.2 ± 7.5 46.8 ± 6.4 4717.8 ± 492.6 1190.8 ± 20.8 12638.8 ± 424.3 10127.5 ± 206.9 5155.7 ± 11.9 10622.1 ± 16.4 8703.8 ± 18.7 Qwen3-0.6B GSM8K 10k 148.9 ± 4.4 51.6 ± 4.1 161.7 ± 22.6 235.9 ± 16.0 1272.3 ± 56.5 367.5 ± 32.6 10.8 ± 1.7 225.2 ± 2.8 18.6 ± 0.5 50k 243.6 ± 9.7 77.3 ± 5.2 796.3 ± 117.2 645.9 ± 50.8 2951.3 ± 93.1 1217.7 ± 86.3 68.5 ± 4.7 1115.9 ± 5.6 93.0 ± 1.6 100k 281.0 ± 13.3 92.2 ± 6.4 1583.5 ± 235.0 1053.5 ± 81.6 4689.9 ± 144.5 2174.1 ± 138.4 144.4 ± 6.2 2227.4 ± 5.5 184.7 ± 1.8 200k 317.6 ± 13.1 113.6 ± 8.6 3153.0 ± 471.6 1755.3 ± 130.5 7439.0 ± 290.3 4039.5 ± 220.3 297.8 ± 6.7 4453.4 ± 9.6 369.6 ± 3.1 Qwen3-0.6B NuminaMath 10k 188.2 ± 5.3 80.1 ± 4.5 140.7 ± 56.9 429.8 ± 17.2 2638.2 ± 21.5 2297.8 ± 31.9 54.4 ± 1.4 840.0 ± 19.2 104.1 ± 1.5 50k 372.5 ± 10.8 130.1 ± 8.3 657.6 ± 284.0 1600.3 ± 75.5 12100.9 ± 62.0 10916.9 ± 95.9 265.6 ± 2.5 4212.3 ± 49.2 518.8 ± 3.4 100k 451.3 ± 14.7 154.9 ± 9.5 1302.7 ± 568.8 2901.5 ± 121.0 23607.2 ± 76.8 21467.3 ± 128.5 529.2 ± 2.5 8430.3 ± 72.0 1039.3 ± 4.3 200k 533.0 ± 16.7 179.5 ± 8.9 2583.8 ± 1132.9 5352.0 ± 140.2 45998.7 ± 119.2 42326.4 ± 161.1 1052.1 ± 3.6 16824.7 ± 83.6 2075.5 ± 4.8 Qwen3-0.6B MedMCQA 10k 33.0 ± 2.2 31.8 ± 3.1 254.1 ± 52.3 75.7 ± 5.8 2177.1 ± 14.4 2176.2 ± 14.5 151.1 ± 2.2 1042.4 ± 3.3 407.0 ± 3.8 50k 77.3 ± 7.8 58.6 ± 5.7 1251.6 ± 263.9 194.0 ± 23.8 10903.5 ± 32.8 10902.6 ± 33.0 746.2 ± 5.1 5214.8 ± 7.0 2028.9 ± 8.6 100k 100.0 ± 7.3 71.9 ± 7.9 2500.1 ± 527.2 310.2 ± 37.4 21829.0 ± 43.5 21828.1 ± 43.8 1499.8 ± 6.2 10436.3 ± 10.7 4068.7 ± 11.0 200k 137.5 ± 8.0 88.3 ± 9.6 4986.9 ± 1060.1 511.4 ± 60.3 43671.7 ± 54.5 43670.7 ± 54.7 3004.1 ± 9.2 20877.6 ± 15.3 8144.1 ± 14.6 Qwen2.5-0.5B GSM8K 10k 179.5 ± 4.3 72.3 ± 3.8 181.7 ± 55.0 339.6 ± 25.5 2576.5 ± 21.3 2008.0 ± 37.7 111.4 ± 1.6 334.3 ± 13.2 141.2 ± 2.3 50k 288.3 ± 13.4 108.2 ± 9.1 867.6 ± 270.1 1113.6 ± 70.8 11065.3 ± 71.7 9117.7 ± 159.8 552.6 ± 4.0 1660.7 ± 32.3 708.4 ± 4.6 100k 324.3 ± 15.4 122.4 ± 10.3 1727.4 ± 539.4 1949.2 ± 104.2 20651.6 ± 189.5 17814.5 ± 290.7 1108.4 ± 6.0 3339.7 ± 40.0 1422.7 ± 5.7 200k 351.4 ± 16.3 137.1 ± 11.0 3442.9 ± 1078.1 3515.2 ± 144.5 38398.9 ± 281.5 35172.1 ± 529.0 2218.8 ± 7.5 6670.6 ± 57.6 2850.4 ± 7.4 Qwen2.5-0.5B NuminaMath 10k 156.4 ± 6.3 62.3 ± 3.5 111.8 ± 34.8 292.8 ± 11.0 2574.9 ± 18.1 2321.4 ± 20.7 10.2 ± 0.2 992.4 ± 9.1 58.6 ± 1.1 50k 251.7 ± 9.8 96.0 ± 4.8 497.9 ± 178.8 1081.2 ± 33.7 11909.0 ± 44.4 11218.1 ± 64.4 51.2 ± 0.4 4919.7 ± 37.2 293.1 ± 2.9 100k 301.2 ± 9.5 115.3 ± 6.9 983.2 ± 357.0 1956.1 ± 47.5 23287.7 ± 72.8 22275.3 ± 117.0 102.3 ± 0.5 9813.7 ± 47.3 589.1 ± 4.1 200k 363.9 ± 10.4 138.4 ± 9.6 1954.2 ± 715.0 3648.0 ± 66.5 45885.3 ± 140.9 44312.7 ± 181.2 204.3 ± 0.8 19570.7 ± 60.9 1177.4 ± 5.0 Qwen2.5-0.5B MedMCQA 10k 23.9 ± 2.3 21.0 ± 3.2 256.9 ± 34.9 51.9 ± 4.3 2096.5 ± 13.5 2096.1 ± 13.5 133.7 ± 1.4 1002.7 ± 4.0 99.8 ± 0.9 50k 65.2 ± 3.2 37.5 ± 4.9 1230.6 ± 176.7 119.6 ± 11.9 10502.6 ± 32.6 10501.6 ± 32.8 679.3 ± 2.8 5012.3 ± 10.4 500.6 ± 2.1 100k 104.3 ± 3.6 49.5 ± 5.8 2444.1 ± 351.8 178.9 ± 18.1 21033.0 ± 41.9 21031.6 ± 42.3 1362.7 ± 3.8 10027.7 ± 15.4 1001.6 ± 3.4 200k 160.5 ± 6.2 63.2 ± 6.9 4870.1 ± 706.8 275.3 ± 27.7 42084.4 ± 54.6 42082.6 ± 55.1 2732.4 ± 6.3 20040.8 ± 19.8 2008.3 ± 6.4 H.3 Service rate and queue-unaware algorithms We rerun the Qwen3-4B MedMCQA instance with service rates μ∈.2,.35,.5,.7,1μ∈\.2,.35,.5,.7,1\, T=5⋅104T=5· 10^4 and 2424 runs, with FCT and FQT retuned for each μ (Figure 14 and Table 7). At μ=1μ=1 the queue stays empty and Mt(0)=cexpM_t(0)=c_ exp. The regret of queue-aware and queue-unaware algorithms then coincides pairwise, KL-PBD+ with KL-UCB+-4P (41.941.9) and NBD with FCT and FQT (2,5112,511), while KL-PBD and KL-UCB-4P differ in regret only due to the difference in their confidence budgets (113113 against 158158). For μ<1μ<1, KL-UCB-4P and KL-UCB+-4P keep deferring regardless of the queue: at μ=.2μ=.2 it is full in 4141 to 56%56\% of the rounds, and their regret exceeds 200200 times that of KL-PBD+. The regret of KL-PBD and KL-PBD+ varies by less than a factor 22 across μ. The regret of NBD grows with μ, since a faster expert makes deferral optimal at more levels, which the surrogate h=ph=p misses. FCT and FQT follow the same trend up to μ=.7μ=.7, with an additional cost from their restricted form, which vanishes at μ=1μ=1: the optimal rule under h=ph=p is then a fixed confidence cutoff, which both can represent, so they recover the decisions of NBD. Figure 14: Service-rate sweep, T=5⋅104T=5· 10^4, 24 runs: final regret (left) and mean queue length (right). Table 7: Final regret of the service-rate sweep, mean ± 95% confidence half-width over 24 runs. Algorithm μ=.2μ=.2 μ=.35μ=.35 μ=.5μ=.5 μ=.7μ=.7 μ=1μ=1 KL-PBD 66.4 ± 3.9 83.5 ± 5.4 69.7 ± 7.0 80.9 ± 12.2 113.0 ± 17.2 KL-PBD+ 36.2 ± 4.9 38.8 ± 4.5 31.2 ± 6.5 29.8 ± 6.4 41.9 ± 10.9 GBD 595.6 ± 57.9 1180.6 ± 123.7 1788.5 ± 207.5 2347.8 ± 249.4 2584.3 ± 259.7 SALMUT-SF 316.4 ± 13.8 407.1 ± 20.3 597.1 ± 58.6 528.1 ± 100.4 185.4 ± 148.1 KL-UCB-4P 10060.2 ± 228.5 5484.8 ± 371.2 2679.8 ± 302.3 1754.2 ± 252.2 158.2 ± 16.9 KL-UCB+-4P 8605.3 ± 110.2 3073.6 ± 185.5 584.5 ± 109.6 285.3 ± 50.8 41.9 ± 10.9 NBD 531.8 ± 3.7 1290.9 ± 5.0 1767.2 ± 5.5 2268.8 ± 6.6 2510.5 ± 7.2 FCT 1285.3 ± 4.7 2656.6 ± 8.3 3604.9 ± 11.1 3793.1 ± 13.9 2510.6 ± 7.2 FQT 1230.9 ± 4.7 2176.5 ± 9.0 2444.1 ± 8.7 2671.8 ± 9.1 2510.6 ± 7.2 H.4 Learned threshold and decisions The top row of Figure 15 shows the threshold and decisions learned by KL-PBD for Qwen3-0.6B at t=T/2t=T/2, from one run for each μ, in the format of Figure 4: the threshold τ^t(b)=1−Mth¯t(b)/ce τ_t(b)=1-M h_t_t(b)/c_e (top) and the decisions h¯j,t<τ^t(b)1\ h_j,t< τ_t(b)\ for μ=0.35μ=0.35 (bottom). The learned thresholds are within 0.090.09 of the oracle ones for every μ. The learned decisions differ from the oracle’s due to extra deferrals caused by KL-PBD’s pessimism. For example, at b=2b=2 on MedMCQA, KL-PBD defers queries with scores up to 30%30\% whereas the oracle accepts those with scores higher than 5%5\%. The difference is larger for Qwen3-4B (middle and bottom rows of Figure 15). Since this LLM is more accurate, it is deferred less, so by t=T/2t=T/2 KL-PBD has collected fewer labels (2,6492,649 on GSM8K against 9,9019,901 for Qwen3-0.6B). Its lower confidence bounds are consequently looser, and it keeps deferring levels at intermediate queue sizes where the oracle no longer does (b=3b=3 to 66 on GSM8K, b=2b=2 to 55 on MedMCQA). On GSM8K, the levels still deferred at b=7b=7 are those with at most 2020 labels, which produces the isolated cells in the bottom row. Figure 15: Thresholds for varying μ and decisions for μ=0.35μ=0.35 at t=T/2t=T/2: KL-PBD for Qwen3-0.6B (top), oracle (middle) and KL-PBD (bottom) for Qwen3-4B, one run per μ. H.5 The importance of pessimism GBD uses the same decision rule as KL-PBD, except that it uses the empirical mean h^j,Nj,t h_j,N_j,t instead of the lower confidence bound h¯j,t h_j,t. Figure 16 shows, for every instance, the final regret of each of the 100100 runs against the run’s average deferral rate. For GBD the two are strongly negatively correlated, with correlation ρ∈[−.95,−.85]ρ∈[-.95,-.85] across the nine instances: runs that defer less accumulate more regret. This is the selective-feedback failure of greedy learning: an early optimistic estimate of the conditional accuracy of a level stops its deferrals, hence no more feedback for this level is received, and the estimate is never corrected. On Qwen2.5-0.5B NuminaMath, for instance, the deferral rate of GBD ranges from .15.15 to .31.31 and its regret from 1111 to 10,65810,658. KL-PBD+, which keeps deferring a level while its lower confidence bound is below the current threshold, shows neither the spread nor the correlation. Figure 16: Final regret against average deferral rate, one point per run, T=105T=10^5, and ρ the correlation for GBD.