Paper deep dive
Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Yunxiao Zhao, Changxiao Cai
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/13/2026, 3:14:49 AM
Summary
This paper introduces the Expected Decoding Rounds (EDR) objective for training parallel and semi-autoregressive speculative draft models. By formulating speculative decoding as a Markov reward process, the authors derive an exact objective that minimizes the expected number of decoding rounds, addressing the cross-round dependencies ignored by previous block-local surrogate objectives. The method includes an exact temporal-difference gradient for training and an offline evaluator for comparison. Empirical results show that finetuning state-of-the-art drafters (DSpark, DFly) with EDR improves mean accepted length across nine benchmarks.
Entities (7)
Relation Signals (6)
EDR → basedon → Markov Reward Process
confidence 95% · we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective
EDR → optimizes → Speculative Decoding
confidence 95% · This formulation yields the Expected Decoding Rounds (EDR) objective... which exactly equals the expected number of decoding rounds.
DSpark → trainedwith → EDR
confidence 92% · Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length
DFly → trainedwith → EDR
confidence 92% · Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length
Temporal-Difference Gradient → enables → EDR
confidence 90% · We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts.
EDR → improves → Mean Accepted Length
confidence 90% · Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the- art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat.
Tags
Links
- Source: https://arxiv.org/abs/2610.10411v1
- Canonical: https://arxiv.org/abs/2610.10411v1
Trouble viewing inline? Open PDF directly →
Full Text
95,756 characters extracted from source content.
Expand or collapse full text
Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds Yunxiao Zhao †thanks: Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA. Email: zhyx,cxcai@umich.edu. Changxiao Cai11footnotemark: 1 October 7, 2026 Abstract Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi-AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the-art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat. https://github.com/y-x-zhao/AngelSpec-EDR Contents 1 Introduction 2 Preliminaries 3 Main Results 3.1 Speculative Decoding as a Markov Reward Process 3.2 The EDR Objective 3.3 TD-Form Gradient of EDR 3.4 Exact Offline Evaluation 3.5 Suboptimality of Block-Local Surrogates 4 Related Work 4.1 Literature Review 4.2 Comparison of Training Objectives 5 Experiments 5.1 Experimental Setup 5.2 Experimental Results 6 Conclusion References A Speculative Decoding Algorithm B Proofs B.1 Markov Process Formulation in Section B.2 Proofs of Proposition and Proposition B.3 Proof of Theorem B.4 Proof of Theorem B.5 Proof of Theorem C Additional Details for Expected Decoding Rounds C.1 Draft EOS Probability Does Not Reduce EDR C.2 Fixed-Size Anchor Sampling D Additional Details for Experiments 1 Introduction Autoregressive (AR) generation is a central bottleneck in large language model (LLM) inference. Since each token can be produced only after its preceding tokens are available, the number of function evaluations required to generate a text sequence grows linearly with its length. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) reduces this sequential cost by using a lightweight draft model to propose multiple candidate tokens, which the full-size target model verifies in parallel in a single forward pass. With rejection-sampling verification, the generated sequence follows the target model’s distribution exactly. Therefore, speculative decoding accelerates inference without sacrificing quality, and the goal is to reduce expensive target model verifications—equivalently, decoding rounds—while keeping drafting itself inexpensive. Early work employed AR drafters (Li et al., 2024b) that predict draft tokens sequentially, so the latency contributed in the drafting phase grows linearly with the draft block size. To reduce the sequential work of drafting, parallel drafters (Cai et al., 2024; Chen et al., 2026a) have emerged to propose multiple draft tokens in one forward pass, typically built on masked diffusion language models (Sahoo et al., 2024; Shi et al., 2024). Their semi-AR variants (Cheng et al., 2026; Liu et al., 2026) add limited sequential dependence within each draft block to improve the draft quality. Parallel drafting supports longer draft blocks and deeper drafter networks, with potential to significantly accelerate the inference of a large target model. Challenges in parallel drafter training: cross-round dependence. For a fixed architecture, a drafter’s quality affects inference speed primarily through the number of decoding rounds, so the natural training objective is to minimize the expected number of rounds. For an AR drafter, the proposal at position t is conditioned on the full realized prefix x<tx_<t regardless of where the current round started, so its acceptance probability depends only on this prefix. As a result, the standard local training objective for AR drafters—matching the target model’s next-token distribution at every prefix—is directly aligned with this goal (Yin et al., 2024a). In contrast, for a parallel or semi-AR drafter, the proposal can instead depend on an earlier prefix x≤nx_≤ n with n<tn<t (in a round starting at n), with access to only some or none of the intermediate tokens. Consequently, the same output position can have different draft distributions depending on where the round starts. This additional dependence breaks the direct link between local distribution matching and global decoding efficiency. The draft for position t from a round starting at n matters only if the decoding trajectory first reaches a round starting at n and then accepts all preceding draft tokens in that round up to position t−1t-1. Both events depend on earlier acceptance outcomes induced by the drafter itself. Thus, changing the drafter affects both acceptance within a round and which draft distributions are encountered in subsequent rounds, making the training objective inherently coupled across rounds. However, existing objectives for parallel and semi-AR drafters do not account for these cross-round dependencies and instead rely on local surrogates. Examples include the expected number of tokens accepted within a single round (Li et al., 2026c) and draft-target distributional discrepancies under hand-crafted weights (Chen et al., 2026a; Wu et al., 2026). This motivates the central question: Can we train parallel and semi-AR drafters by directly minimizing the expected number of decoding rounds? Our contributions. We answer this question affirmatively. Our approach relies on two key observations. First, conditioned on the output sequence, speculative decoding with a parallel or semi-AR drafter can be formulated as a Markov reward process (MRP) (Howard, 1971): each state records where the current round started and which position is being verified, and each rejection costs one round. Second, since rejection-sampling verification ensures that the output follows the same target distribution for every drafter, averaging over target rollouts recovers the expected number of rounds of actual decoding. Together, these insights allow us to capture the dependencies across decoding rounds explicitly and characterize the expected number of decoding rounds exactly, yielding a unified framework for training and evaluating drafters. Building on this framework, we introduce the Expected Decoding Rounds (EDR) objective, whose minimization is equivalent to minimizing the expected number of decoding rounds. We then derive a temporal-difference gradient estimator, enabling drafter training from target-model rollouts. EDR applies across drafter architectures and introduces no hyperparameters. We further prove that drafters maximizing the expected accepted length for each round can be strictly suboptimal for EDR, theoretically justifying the advantages of our global objective. The framework also yields an exact offline evaluator of expected round counts, enabling comparisons of drafters on the same target-model rollouts without running online speculative decoding. Empirically, we finetune two state-of-the-art (SOTA) drafters, DSpark (Cheng et al., 2026) and DFly (Liu et al., 2026), with EDR while keeping their architectures and inference procedures unchanged. Across nine benchmarks spanning mathematical reasoning, code generation, and chat, a single epoch of EDR finetuning consistently improves mean accepted length over the original drafters and existing training objectives. Our main contributions are summarized as follows: • A Markov reward process formulation. We provide a theoretical framework that exactly characterizes expected speculative decoding rounds for parallel and semi-AR drafters. • An exact and trainable global objective. We introduce EDR, establish its equivalence to minimizing the expected number of decoding rounds, derive a tractable gradient for drafter training, and prove its advantage over local surrogates. • An exact offline evaluator. We provide an offline evaluator of the expected number of decoding rounds, enabling paired evaluation of drafters on the same target-model rollouts. • Consistent empirical gains. EDR finetuning improves two SOTA drafters across nine benchmarks, outperforming existing training objectives. 2 Preliminaries Let V be a vocabulary of ordinary tokens and ¯:=∪<EOS> V:=V∪\ <EOS>\, where <EOS> denotes the end-of-sequence token. For two distributions p,qp,q, define the total variation (TV) distance (p,q):=∑v∈¯[p(v)−q(v)]+ TV(p,q):= _v∈ V [p(v)-q(v) ]_+, where [z]+:=max(0,z)[z]_+:= (0,z). For any sequence x=(x1,x2,…)x=(x_1,x_2,…), we denote x≤a=(x1,x2,…,xa)x_≤ a=(x_1,x_2,…,x_a), x<a:=x1:a−1x_<a:=x_1:a-1, and xa:b:=(xa,xa+1,…,xb)x_a:b:=(x_a,x_a+1,…,x_b) for 1≤a≤b1≤ a≤ b. For a>ba>b, xa:b:=∅x_a:b:= . Target distribution. Let p denote the distribution of the target model. It generates tokens autoregressively as xt∼pt(⋅∣x<t)x_t p_t(· x_<t) for t≥1t≥ 1 and stops upon sampling <EOS>. Denote by L:=infℓ≥0:xℓ+1=<EOS>L:= \ ≥ 0:x_ +1= <EOS>\ the sequence length excluding <EOS>. We assume L has a universal constant upper bound for x∼px p. A complete sequence x=(x1:L,<EOS>)x=(x_1:L, <EOS>) has probability ∏t=1Lpt(xt∣x<t)pL+1(<EOS>∣x1:L), _t=1^Lp_t(x_t x_<t)p_L+1( <EOS> x_1:L), and we call x target-supported if p(x)>0p(x)>0. Speculative decoding. Speculative decoding generates a token sequence x in rounds. Consider a round that starts from a committed prefix x1:nx_1:n. A drafter q first proposes a block of B draft tokens x~n+1:n+B x_n+1:n+B with x~t∼qt x_t q_t. For simplicity, we write qtq_t for the draft distribution at position t in the current round, suppressing its dependence on the round start n, committed prefix x1:nx_1:n, and preceding within-block proposals x~n+1:t−1 x_n+1:t-1. We will make this explicit in the next paragraph. The target model then verifies the draft block from left to right in a single forward pass. When position t is verified, all earlier drafts in the block have been accepted, so the current prefix is x<t=(x1:n,x~n+1:t−1)x_<t=(x_1:n, x_n+1:t-1). Write pt=pt(⋅∣x<t)p_t=p_t(· x_<t). The draft x~t x_t is accepted with probability min1,pt(x~t)/qt(x~t) \1,p_t( x_t)/q_t( x_t)\, in which case it is committed as xt=x~tx_t= x_t. Otherwise, it is replaced with a correction token xt∼rtx_t r_t drawn from the residual distribution rt(v):=[pt(v)−qt(v)]+/(pt,qt)r_t(v):=[p_t(v)-q_t(v)]_+/ TV(p_t,q_t), v∈¯v∈ V. The remaining drafts in the block are discarded, and the next round begins from the updated prefix x1:tx_1:t. If all B draft tokens are accepted, the same forward pass also yields one additional bonus token xn+B+1∼pn+B+1(⋅∣x≤n+B)x_n+B+1 p_n+B+1(· x_≤ n+B), and the next round starts after it. Generation stops when <EOS> is committed. This rejection-sampling verification ensures that the output sequence satisfies x∼px p for any drafter q (Leviathan et al., 2023). Draft distribution. Drafters differ in what qtq_t conditions on. An AR drafter proposes each token from the full prefix, whereas parallel and semi-AR drafters propose the whole block from the prefix committed at the start of the round. We now refine the shorthand qtq_t to make this explicit. For a round starting from a committed prefix x1:nx_1:n, let qn,t(⋅∣x1:n;x~n+1:t−1)q_n,t(· x_1:n; x_n+1:t-1) denote the draft distribution for position t=n+1,…,n+Bt=n+1,…,n+B. The subscript t identifies the position being predicted, and n records the position where the round starts. The semicolon separates the committed prefix x1:nx_1:n from the preceding within-block proposals x~n+1:t−1 x_n+1:t-1, which the drafter may use in whole or in part. This notation covers AR, parallel, and semi-AR drafters. An AR drafter is the special case qn,t(⋅∣x1:n;x~n+1:t−1)=qt(⋅∣x1:n,x~n+1:t−1)q_n,t(· x_1:n; x_n+1:t-1)=q_t(· x_1:n, x_n+1:t-1), which conditions on the concatenated prefix and has no further dependence on the round start n. A parallel drafter samples the block independently given the committed prefix x1:nx_1:n, i.e., qn,t(⋅∣x1:n;x~n+1:t−1)=qn,t(⋅∣x1:n)q_n,t(· x_1:n; x_n+1:t-1)=q_n,t(· x_1:n). A semi-AR drafter allows partial dependence on preceding within-block drafts while maintaining parallel prediction across some positions. For parallel and semi-AR drafters, the same current prefix can therefore induce different draft distributions depending on the round start n. This dependence is the main difference from AR drafters and the central challenge this work addresses. We give the Pseudocode for speculative decoding process we considered in Algorithm 2 in Appendix A. Decoding rounds and mean accepted length. Each decoding round requires one forward pass of the target model. Let qT^q denote the number of decoding rounds needed to produce a complete output under drafter q, including the one that produces <EOS>. A common metric is the mean accepted length (MAL), the ratio of the expected output length to the expected number of rounds, (q):=[L+1][q], MAL(q):= E[L+1]E[T^q], (1) where L+1L+1 is the length of the complete output counting all committed tokens, including correction tokens, bonus tokens, and the terminal <EOS> token. As the output sequence follows the target distribution, [L]E[L] is independent of the drafter q for a fixed target model and evaluation corpus. Consequently, maximizing (q) MAL(q) is equivalent to minimizing [q]E[T^q]. 3 Main Results This section presents our main results. Section 3.1 presents a Markov reward process (MRP) formulation for speculative decoding. Section 3.2 introduces the EDR objective and Section 3.3 presents a temporal-difference form of its gradient. Section 3.4 gives offline estimators for the decoding statistics, and suboptimality of block-local objectives is discussed in Section 3.5. 3.1 Speculative Decoding as a Markov Reward Process Recall that qT^q denotes the number of decoding rounds in a speculative decoding run with drafter q, and let X=(X1,…,XL,<EOS>)X=(X_1,…,X_L, <EOS>) be the output sequence of the same run. Since speculative decoding ensures X∼pX p for any drafter q, the overall expected number of decoding rounds satisfies [q]=X∼p[τq(X)], with τq(x):=[q∣X=x], [T^q]=E_X p[τ^q(X)], with τ^q(x):=E[T^q X=x], (2) where τq(x)τ^q(x) is the expected number of rounds conditioned on producing the sequence x. Therefore, it suffices to characterize τq(x)τ^q(x) for any target-supported sequence x. Markov reward process. In light of this, we fix a target-supported sequence x=(x1,…,xL,<EOS>)x=(x_1,…,x_L, <EOS>). Conditioned on the output x of a speculative decoding run, the remaining randomness is which positions are filled by accepted drafts and which by the target model after a rejection, and this determines the number of decoding rounds q(x)T^q(x). This can be described as a Markov chain on states (n,t)(n,t) for 0≤n<t≤L+10≤ n<t≤ L+1. (see Appendix B.1 for a detailed derivation). State (n,t)(n,t) indicates that the current round started after n (i.e., x1:nx_1:n was committed) and is verifying the draft for position t. Decoding starts at (0,1)(0,1) and terminates when <EOS> is produced at position L+1L+1. Note that at state (n,t)(n,t) with t≤n+Bt≤ n+B, the draft for position t is drawn from qn,t(⋅∣x1:n;xn+1:t−1)q_n,t(· x_1:n;x_n+1:t-1). By Bayes’ rule, given that position t outputs xtx_t, the probablity that it was produced by an accepted draft is An,tq(x≤t):=min1,qn,t(xt∣x1:n;xn+1:t−1)pt(xt∣x<t).A_n,t^q(x_≤ t):= \1, q_n,t(x_t x_1:n;x_n+1:t-1)p_t(x_t x_<t) \. (3) Denote by Rn,tq:=1−An,tqR_n,t^q:=1-A_n,t^q the rejection probability. Acceptance continues the current round, whereas rejection starts a new round after position t, provided that xt≠<EOS>x_t≠ <EOS>. Therefore, the Markov chain has two transitions: (n,t)⟶(n,t+1),with probabilityAn,tq(x≤t),(t,t+1),with probabilityRn,tq(x≤t).(n,t) cases(n,t+1),&with probability~~A_n,t^q(x_≤ t),\\ (t,t+1),&with probability~~R_n,t^q(x_≤ t). cases (4) We set An,tq=0A_n,t^q=0 and Rn,tq=1R_n,t^q=1 for t>n+Bt>n+B, so the same transition rule covers the case when all tokens in a draft block are accepted and a new round starts after that. Let ωn,tq(x) _n,t^q(x) denote the occupancy of state (n,t)(n,t): the probability that this state is visited, conditioned on the output sequence being x. It is not hard to see that ωn,tq(x) _n,t^q(x) depends on x only through x<tx_<t and hence we write ωn,tq(x<t)=ωn,tq(x) _n,t^q(x_<t)= _n,t^q(x). Starting from ω0,1q(x)=1 _0,1^q(x)=1, the occupancies satisfy the forward recursion: ωn,t+1q(x≤t) _n,t+1^q(x_≤ t) =ωn,tq(x<t)An,tq(x≤t),n<t≤L, = _n,t^q(x_<t)A_n,t^q(x_≤ t), n<t≤ L, (5a) ωt,t+1q(x≤t) _t,t+1^q(x_≤ t) =∑n=0t−1ωn,tq(x<t)Rn,tq(x≤t),1≤t≤L. = _n=0^t-1 _n,t^q(x_<t)R_n,t^q(x_≤ t), 1≤ t≤ L. (5b) Since every round begins at a diagonal state (t,t+1)(t,t+1), the number of rounds equals the number of diagonal states visited, yielding the following exact expression (see Appendix B.2 for proof). Proposition 1. For any target-supported sequence x=(x1:L,<EOS>)x=(x_1:L, <EOS>) and any drafter q, the expected number of decoding rounds to produce x is τq(x)=∑t=0Lωt,t+1q(x≤t)=1+∑t=1L∑n=0t−1ωn,tq(x<t)Rn,tq(x≤t).τ^q(x)= _t=0^L _t,t+1^q(x_≤ t)=1+ _t=1^L _n=0^t-1 _n,t^q(x_<t)R_n,t^q(x_≤ t). (6) The second identity has a direct MRP interpretation. Rejection starts a new round and incurs unit cost, while acceptance incurs zero cost. Hence, the rejection probability Rn,tqR_n,t^q is the expected cost of visiting state (n,t)(n,t), and the occupancy-weighted sum of costs gives the round count. 3.2 The EDR Objective By (2), averaging (6) over x∼px p provides an exact expression for the overall expected number of decoding rounds [q]E[T^q]. However, each rejection probability Rn,tq(x≤t)R_n,t^q(x_≤ t) uses the target only through the single realized token xtx_t, even though the full distribution pt(⋅∣x<t)p_t(· x_<t) is available. Hence, we average over this token under the target distribution ptp_t, conditioned on xtx_t being a non-EOS token. Definition 1. Define the local rejection cost of state (n,t)(n,t), for 0≤n<t≤L0≤ n<t≤ L, as cn,tq(x<t):=Xt∼p′t(⋅∣x<t)[Rn,tq(x<t,Xt)] c_n,t^q(x_<t):= E_X_t p _t(· x_<t)[R_n,t^q(x_<t,X_t)] (7) where pt′(v∣x<t):=pt(v∣x<t)/(1−pt(<EOS>∣x<t))p _t(v x_<t):=p_t(v x_<t)/(1-p_t( <EOS> x_<t)) for v≠<EOS>v≠ <EOS> is the target distribution normalized over non-EOS tokens. For t≤n+Bt≤ n+B, its value is given by cn,tq(x<t)=11−pt(<EOS>∣x<t)∑v∈[pt(v∣x<t)−qn,t(v∣x1:n;xn+1:t−1)]+, c_n,t^q(x_<t)= 11-p_t( <EOS> x_<t) _v [p_t(v x_<t)-q_n,t(v x_1:n;x_n+1:t-1)]_+, (8) and for t>n+Bt>n+B, it is equal to 11. Note that cn,tqc_n,t^q is well-defined because every position t≤Lt≤ L outputs a non-EOS token so pt(<EOS>∣x<t)<1p_t( <EOS> x_<t)<1. We denote cn,L+1q(x)=0c_n,L+1^q(x)=0 for notational convenience, since the EOS token incurs no cost. In words, the cost cn,tq(x<t)c_n,t^q(x_<t) is the rejection probability associated with state (n,t)(n,t), given that the committed output token xtx_t is non-EOS. It measures the local distributional discrepancy between the drafter qn,tq_n,t and target ptp_t for position t, given by the TV distance between the draft and target distributions for position t, excluding EOS and normalized by the target’s non-EOS mass. Excluding EOS reflects the fact that when the target model produces EOS, it never starts another decoding round and incurs no penalty for draft rejection. We now introduce the Expected Decoding Rounds (EDR) objective, which exactly equals the overall expected number of decoding rounds, as formalized in Theorem 1 below. The proof can be found in Appendix B.3. Theorem 1. For any drafter q, the EDR objective equals the expected number of decoding rounds: ℒEDR(q):=1+X∼p[∑t=1L∑n=0t−1ωn,tq(X<t)cn,tq(X<t)]=X∼p[τq(X)]=[q].L_EDR(q):=1+E_X p\! [ _t=1^L _n=0^t-1 _n,t^q(X_<t)c_n,t^q(X_<t) ]=E_X p[τ^q(X)]=E[T^q]. (9) The EDR objective is a sum of local rejection costs cn,tqc_n,t^q, measuring the local distributional discrepancy between the drafter qn,tq_n,t and target ptp_t, each weighted by the corresponding occupancy ωn,tq _n,t^q. The quantity inside the expectation, which we call the per-sequence EDR loss, ℒ^EDR(q,x) L_EDR(q;x) :=1+∑t=1L∑n=0t−1ωn,tq(x<t)cn,tq(x<t) :=1+ _t=1^L _n=0^t-1ω^q_n,t(x_<t)\,c^q_n,t(x_<t) (10) can be computed exactly given a target-sampled sequence x via the forward occupancy recursion (5) and the local rejection cost (7). Therefore, averaging over a batch of target rollouts provides an empirical EDR loss for drafters that directly targets minimizing the expected number of rounds. We make several important remarks. • Replacement of Rn,tqR_n,t^q with cn,tqc_n,t^q. The EDR loss (10) replaces Rn,tq(x≤t)R_n,t^q(x_≤ t) in (6) with its conditional expectation cn,tq(x<t)c_n,t^q(x_<t). For an individual rollout, it generally differs from τq(x)τ^q(x), but they agree after averaging over x∼px p. In return, the EDR loss uses the next-token distribution rather than one sampled token, so its gradient aggregates information across the vocabulary. • Exact occupancy weighting. Prior surrogate objectives (Chen et al., 2026a; Wu et al., 2026) do not account for EOS termination in their local discrepancy terms, and more importantly, weight them in a heuristic way. These weights can be interpreted as fixed surrogates for the drafter-dependent occupancies ωn,tq _n,t^q. In contrast, EDR combines EOS-aware local costs with the actual occupancies, yielding an exact objective for the expected number of decoding rounds. • Parallel vs. AR drafters. For an AR drafter, qn,t(⋅∣x1:n;xn+1:t−1)=qt(⋅∣x<t)q_n,t(· x_1:n;x_n+1:t-1)=q_t(· x_<t) for all n<tn<t. Without a block cap (B=∞B=∞), EDR reduces to ℒEDR(q)=1+X∼p[∑t=1Lct(X<t)]withct(x<t):=∑v∈[pt(v∣x<t)−qt(v∣x<t)]+1−pt(<EOS>∣x<t). _EDR(q)=1+E_X p [ _t=1^Lc_t(X_<t) ] c_t(x_<t):= _v [p_t(v x_<t)-q_t(v x_<t)]_+1-p_t( <EOS> x_<t). (11) EDR hence decouples into local prefix costs, each minimized by matching qtq_t to ptp_t. This recovers the result of Yin et al. (2024a, Theorem 1) established for AR drafters and extends it to sequences that terminate with EOS. A finite block size adds only a mild dependence on n through the boundary states. Therefore, Theorem 1 highlights the key difference between parallel and AR drafters, and EDR captures the cross-round dependencies exactly. 3.3 TD-Form Gradient of EDR The per-sequence EDR loss in (10) depends on the drafter q through both the local costs cn,tqc_n,t^q and the occupancies ωn,tq _n,t^q. Differentiating it with respect to q directly is expensive because it must backpropagate through the entire forward recursion for occupancy ωn,tq _n,t^q (5). Fortunately, the MRP formulation gives the gradient in a temporal-difference (TD) (Sutton, 1988) form that uses only local derivatives. For a fixed target-supported x, consider the MRP with transitions (4) and costs cn,tq(x<t)c_n,t^q(x_<t) from (7). Define the value function Vn,tq(x)V_n,t^q(x), the expected cumulative cost starting from state (n,t)(n,t). The value function satisfies the Bellman equation (Bellman, 1966) Vn,tq(x) V_n,t^q(x) =cn,tq(x<t)+An,tq(x≤t)Vn,t+1q(x)+Rn,tq(x≤t)Vt,t+1q(x),0≤n<t≤L, =c_n,t^q(x_<t)+A_n,t^q(x_≤ t)V_n,t+1^q(x)+R_n,t^q(x_≤ t)V_t,t+1^q(x), 0≤ n<t≤ L, (12) with the terminal condition Vn,L+1q(x)=0V_n,L+1^q(x)=0 for all 0≤n≤L0≤ n≤ L. Unlike the cost and occupancy, the value function Vn,tq(x)V_n,t^q(x) depends on the whole sequence x. The value function of the initial state gives the per-sequence EDR loss: V0,1q(x)=ℒ^EDR(q,x)−1.V_0,1^q(x)= L_EDR(q;x)-1. (13) Algorithm 1 One EDR training step (sg(⋅)sg(·) denotes the stop-gradient operator) 1: drafter qθq_θ, target model p, block size B 2: For each prompt i=1,…,Mi=1,…,M, sample an independent target rollout x(i)=(x1:Li(i),<EOS>)x^(i)=(x^(i)_1:L_i, <EOS>) from the target model p 3: Compute pt(⋅∣x<t(i))p_t(· x^(i)_<t) for all i and t in one batched target forward pass 4: for i=1,…,Mi=1,…,M (in parallel) do 5: Compute An,t(i)A^(i)_n,t and cn,t(i)c^(i)_n,t from qθq_θ at all states (n,t)(n,t) 6: Compute ωn,t(i)ω^(i)_n,t by (5) and Vn,t(i)V^(i)_n,t by (12), without gradients 7: end for 8: ℓ(θ)←M−1∑i=1M∑t=1Li∑n=0t−1sg(ωn,t(i))[cn,t(i)+An,t(i)sg(Vn,t+1(i)−Vt,t+1(i))] (θ)← M^-1 _i=1^M _t=1^L_i _n=0^t-1sg(ω^(i)_n,t) [c^(i)_n,t+A^(i)_n,tsg(V^(i)_n,t+1-V^(i)_t,t+1) ] 9: Update θ using ∇θℓ(θ) _θ (θ) We now present the gradient of the per-sequence EDR loss in the following theorem. The proofs of (13) and Theorem 2 are given in Appendix B.4. Theorem 2 (EDR gradient). We parametrize the drafter q=qθq=q_θ by neural network parameters θ. For any target-supported sequence x, the gradient of the per-sequence EDR loss (10) is given by ∇θℒ^EDR(qθ,x)=∑t=1L∑n=0t−1ωn,tq(x<t)[∇θcn,tq(x<t)+∇θAn,tq(x≤t)(Vn,t+1q(x)−Vt,t+1q(x))]. _θ L_EDR(q_θ;x)= _t=1^L _n=0^t-1 _n,t^q(x_<t)\, [ _θc_n,t^q(x_<t)+ _θA_n,t^q(x_≤ t) (V_n,t+1^q(x)-V_t,t+1^q(x) ) ]. (14) As a result, one has X∼p[∇θℒ^EDR(qθ,X)]=∇θℒEDR(qθ).E_X p [ _θ L_EDR(q_θ;X) ]= _θL_EDR(q_θ). (15) The gradient has a standard one-step TD form, separated into two contributions: the immediate effect on the local rejection cost, and the downstream effect of changing the acceptance probability. The latter is weighted by the value gap, which measures the difference between the cumulative costs of the two possible subsequent states. The occupancies account for how often each state is reached. Implementation. We compute the occupancies ωn,tqω^q_n,t by the forward recursion (5) and the value functions Vn,tqV^q_n,t by the backward Bellman recursion (12). Since they enter the gradient expression (14) only as weights, we hold them fixed during differentiation. Gradients are required only through the local quantities cn,tqc_n,t^q and An,tqA_n,t^q. By (15), averaging (14) over a batch of i.i.d. target rollouts gives an unbiased stochastic gradient of EDR, which directly optimizes the expected number of decoding rounds. One EDR training step is summarized in Algorithm 1. 3.4 Exact Offline Evaluation The MRP formulation also yields exact offline evaluators for any drafter, computed from target-model rollouts alone without running speculative decoding. We first give an unbiased estimator of the expected number of decoding rounds, from which we obtain an estimator of the MAL. We then extend it to a finer-grained statistic, the expected number of rounds that commit at least k tokens, which yields an estimator of the position-wise acceptance rate. 3.4.1 Expected Decoding Rounds and MAL By Proposition 1, the expected number of decoding rounds conditioned on the output X=xX=x equals the sum of the diagonal occupancies. For a target-supported sequence x=(x1:L,<EOS>)x=(x_1:L, <EOS>), define τoffq(x):=∑t=0Lωt,t+1q(x≤t). _off^q(x):= _t=0^L _t,t+1^q(x_≤ t). (16) This quantity averages out all randomness of speculative decoding given X=xX=x, namely draft sampling and verifier randomness, and is computed exactly by a single pass of the forward occupancy recursion. Given a dataset of target rollouts =x(i)=(x1:Li(i),<EOS>)i=1ND=\x^(i)=(x_1:L_i^(i), <EOS>)\_i=1^N independently sampled from x(i)∼px^(i) p, we estimate the expected number of rounds and the MAL by ^(q):=1N∑i=1Nτoffq(x(i))and^(q):=N−1∑i=1N(Li+1)^(q). T_D(q):= 1N _i=1^N _off^q(x^(i)) MAL_D(q):= N^-1 _i=1^N(L_i+1) T_D(q). (17) Since [q]=X∼p[τq(X)]E[T^q]=E_X p[τ^q(X)], ^(q) T_D(q) is an unbiased estimator of [q]E[T^q], and ^(q) MAL_D(q), a ratio of sample means, is a consistent estimator of (q) MAL(q). Moreover, since the rollouts do not depend on the drafter, all drafters can be evaluated on the same D. This yields paired comparisons free of the rollout variation that arises when each drafter is evaluated on independently generated outputs. 3.4.2 Rounds Committing at Least k Tokens and Acceptance Rate A finer-grained statistic is the number of rounds whose committed length, i.e., the number of tokens committed in the round including the correction or bonus token, is at least k. Let ≥kqT^q_≥ k denote this number in a speculative decoding run. Since a round commits at most B+1B+1 tokens, we consider 1≤k≤B+11≤ k≤ B+1. A round starting after position t commits at least k tokens if and only if its first k−1k-1 drafts are accepted, i.e., it visits state (t,t+k)(t,t+k). This observation generalizes Proposition 1 as follows; see Appendix B.2 for the proof. Proposition 2. For any target-supported sequence x=(x1:L,<EOS>)x=(x_1:L, <EOS>), any drafter q, and any 1≤k≤B+11≤ k≤ B+1, the expected number of rounds committing at least k tokens to produce x is τ≥kq(x):=[≥kq|X=x]=∑n=0L+1−kωn,n+kq(x<n+k). _≥ k^q(x):=E [T^q_≥ k\, |\,X=x ]= _n=0^L+1-k _n,n+k^q(x_<n+k). (18) Taking k=1k=1 recovers Proposition 1, so that τoffq=τ≥1q _off^q= _≥ 1^q. For 1≤k≤B1≤ k≤ B, we define the acceptance rate at draft position k as ARk(q):=[≥k+1q][≥kq],AR_k(q):= E[T^q_≥ k+1]E[T^q_≥ k], (19) the fraction of rounds reaching draft position k in which the k-th draft is also accepted.11 1 A round whose k-th position produces the terminal <EOS> is counted in ≥kqT^q_≥ k but not in ≥k+1qT^q_≥ k+1, since generation stops there. Using the same dataset D, we estimate these quantities by ^≥k,(q):=1N∑i=1Nτ≥kq(x(i))andAR^k,(q):=^≥k+1,(q)^≥k,(q). T_≥ k,D(q):= 1N _i=1^N _≥ k^q(x^(i)) AR_k,D(q):= T_≥ k+1,D(q) T_≥ k,D(q). (20) As before, ^≥k,(q) T_≥ k,D(q) is unbiased for [≥kq]E[T^q_≥ k], and AR^k,(q) AR_k,D(q) is consistent for ARk(q)AR_k(q). 3.5 Suboptimality of Block-Local Surrogates Finally, we show that maximizing accepted length independently within each block can be strictly suboptimal for the global MAL. Since maximizing MAL is equivalent to minimizing the expected number of decoding rounds (see Section 2), which EDR equals exactly (Theorem 1), this theoretically justifies optimizing the EDR objective rather than block-local surrogates. We consider semi-AR drafters that condition on earlier drafts in the block only through the immediately preceding token, i.e., qn,t(⋅∣x1:n;xn+1:t−1)=qn,t(⋅∣x1:n;xt−1)q_n,t(· x_1:n;x_n+1:t-1)=q_n,t(· x_1:n;x_t-1), referred to as first-order drafters. This class includes existing semi-AR drafters such as DSpark (Cheng et al., 2026) and DFly (Liu et al., 2026). The following theorem shows that even when the expected accepted length is optimized in every round, the resulting MAL can be a constant factor below the best achievable within the same drafter class. The construction and proof are deferred to Appendix B.5. Theorem 3. For any block size B≥3B≥ 3, there exists a target distribution p such that, for any first-order drafter q q that maximizes the expected accepted length in every round, there exists another first-order drafter q∗q^* satisfying (q∗)(q^)>1+13B+9. MAL(q^*) MAL( q)>1+ 13B+9. The gap arises from the cross-round interaction ignored by block-local objectives. The draft distribution in the current round determines not only how many tokens are accepted in that round, but also where the next round starts. Consequently, a drafter that is locally optimal for the current round can induce a less favorable distribution of future round starts and result in a lower global MAL. 4 Related Work 4.1 Literature Review Drafter architectures for speculative decoding. The latency per generated token depends on the accepted length per round as well as the costs of drafting and verification. Architectural work therefore seeks to increase accepted length while keeping drafting inexpensive. Medusa (Cai et al., 2024) attaches multiple prediction heads to target-model features, while Hydra (Ankner et al., 2024) introduces sequential dependencies between these heads to improve draft accuracy. EAGLE-3 (Li et al., 2026d) combines features from multiple target layers. To reduce sequential drafting overhead, parallel and diffusion-based drafters generate multiple proposals in a single forward pass. DFlash (Chen et al., 2026a) uses a lightweight block-diffusion drafter conditioned on target-model context features. DiffuSpec (Li et al., 2026a) repurposes a pretrained diffusion language model as a parallel drafter and uses causal-consistency path search to construct proposals for autoregressive verification. To improve within-block consistency without returning to fully autoregressive drafting, DSpark (Cheng et al., 2026) and DFly (Liu et al., 2026) combine parallel backbones with lightweight sequential heads that condition on preceding draft tokens. Training objectives for speculative drafters. Drafter training commonly uses cross-entropy or KL-divergence objectives to align draft predictions with the target distribution (Li et al., 2026d; Chen et al., 2026a). However, distributional matching under these losses does not necessarily maximize token acceptance when drafter capacity is limited. LK losses (Samarin et al., 2026) address this distinction through acceptance-oriented objectives. For multi-token drafting, the first rejection invalidates all subsequent proposals in the block, making the contributions of different positions unequal. DFlash (Chen et al., 2026a) accounts for this asymmetry using fixed exponentially decaying position weights. D-PACE (Wu et al., 2026) replaces this fixed schedule with adaptive weights derived from a differentiable accepted-length surrogate based on draft confidences. The end-to-end (E2E) multi-step TV loss (Li et al., 2026c) moves beyond independent token-level matching by incorporating the multiplicative structure of acceptance across positions within a block. Recent work addresses the mismatch between drafter training and inference by incorporating verification feedback. Draft-OPD (Lei et al., 2026) replays draft proposals from inference-time anchors and applies acceptance-dependent distillation losses. Verification-Aware Training (Gu et al., 2026) uses simulated verification outcomes to supervise an auxiliary head and adapt position weights. Although these methods incorporate inference-time behavior into training, their objectives remain surrogates for MAL. Diffusion language models. Diffusion language models (DLMs) (Nie et al., 2025; Inception Labs et al., 2025; Ye et al., 2025) offer an alternative to AR generation by iteratively refining sequences with bidirectional context, enabling parallel token generation. Continuous DLMs (Li et al., 2022; Chen et al., 2026b; Li et al., 2026b) extend continuous diffusion models (Ho et al., 2020; Song et al., 2020b; Song et al., 2020a) and flow matching (Lipman et al., 2022) to token embedding spaces. Starting from Gaussian noise, they generate continuous token representations through learned denoising or flow dynamics, then map these representations to discrete tokens. This framework facilitates adapting established techniques for controlling and accelerating diffusion sampling, including classifier-free guidance (Ho and Salimans, 2022), self-conditioning (Chen et al., 2022), few-step solvers (Lu et al., 2022; Li and Cai, 2024; Jiao et al., 2026), and distillation (Song and Dhariwal, 2024; Yin et al., 2024b). Discrete DLMs (Lou et al., 2023; Sahoo et al., 2024; Shi et al., 2024) operate directly in token space using categorical corruption. Masked DLMs belong to this category and corrupt sequences by replacing tokens with a special mask token. Then it generate text by progressively unmasking multiple positions in parallel. This formulation underlies many parallel and semi-AR drafters. However, conditionally independent sampling within an update can overlook dependencies among simultaneously generated tokens, creating a trade-off between generation quality and sampling efficiency (Li and Cai, 2025; Chen et al., 2025; Dmitriev et al., 2026; Zhao and Cai, 2026; Cai and Li, 2026). 4.2 Comparison of Training Objectives In this section, we present a unified summary of existing drafter training objectives in the literature, together with our proposed EDR objective. Let X=(X1,…,XL,<EOS>)X=(X_1,…,X_L, <EOS>) be a rollout sequence sampled from the target distribution, and let B be the number of proposed draft tokens in a block. The block is drafted from the committed prefix X1:nX_1:n; qn,tq_n,t and ptp_t denote the draft and target distributions at position t, respectively. We suppress their dependence on the sampled sequence for brevity. Up to additive constants and positive scaling factors, the objectives considered here can be written as ℒ(q)=X∼p[∑n=0Lan∑t=n+1minn+B+1,Lbn,tℓn,t].L(q)=E_X p [ _n=0^La_n _t=n+1 \n+B+1,L\b_n,t\, _n,t ]. (21) This template separates each objective into three components: • Per-position loss ℓn,t _n,t specifies the local training signal at position t for a block drafted from anchor n. For token-level alignment surrogates, ℓn,t _n,t measures the discrepancy between the draft distribution qn,tq_n,t and the target ptp_t, whereas for acceptance-aware objectives, ℓn,t _n,t is the surrogate for the rejection probability at position t. All objectives except EDR set ℓn,n+B+1=0 _n,n+B+1=0, since no draft token is proposed at the bonus position. EDR keeps this term because a round that reaches the bonus position always ends there (An,n+B+1q=0A^q_n,n+B+1=0). • Block position weight bn,tb_n,t sets how much position t counts relative to the other positions in the same block. Later draft tokens are verified only if all earlier draft tokens in the block are accepted, so their contributions depend on acceptance at preceding positions. Non-uniform position weights bn,tb_n,t reflect this asymmetry between positions. • Anchor weight ana_n sets how much the block drafted from anchor n counts relative to the blocks drafted from other anchors. All objectives except EDR weight anchors equally, i.e., an=1a_n=1. Table 1 summarizes the three components of each objective. Here, sg(⋅)sg(·) denotes the stop-gradient operator: sg(z)sg(z) has the same value as z, but it is treated as a constant when taking gradients with respect to the drafter parameters. We discuss these three groups below. Table 1: Drafter training objectives as instances of the template in (21). Objective Per-position loss ℓn,t _n,t Block position weight bn,tb_n,t Anchor weight ana_n Token-level surrogates CE −logqn,t(xt)- q_n,t(x_t) 11 11 KL (pt∥qn,t) KL(p_t\,\|\,q_n,t) 11 11 TV (pt,qn,t) TV(p_t,q_n,t) 11 11 LK (Samarin et al., 2026) (λ+(1−λ)) (λ\, KL+(1-λ)\, TV ), λ=exp(−κsg(1−))λ= (- (1- TV) ) 11 11 Block-level surrogates Exponential decay (Chen et al., 2026a) CE / KL / TV / LK or a weighted combination exp(−(t−n−1)/η) (-(t-n-1)/η ) 11 D-PACE (Wu et al., 2026) CE / KL / TV / LK or a weighted combination sg(∑m=t−nB∏s=n+1n+mq~n,s(xs)) ( _m=t-n^B\, _s=n+1^n+m q_n,s(x_s) ), q~n,s=(1−α)qn,s+α q_n,s= (1-α )q_n,s+α 11 E2E-TV (Li et al., 2026c) ((pt,qn,t)−1)( TV(p_t,q_n,t)-1) ∏s=n+1t−1(1−(ps,qn,s)) _s=n+1^t-1 (1- TV(p_s,q_n,s) ) 11 Exact objective EDR (ours) cn,tqc^q_n,t ∏s=n+1t−1An,sq _s=n+1^t-1A^q_n,s ωn,n+1qω^q_n,n+1 EDR (TD form) (ours) cn,tq+An,tqsg(Vn,t+1q−Vt,t+1q)c^q_n,t+A^q_n,t\,sg (V^q_n,t+1-V^q_t,t+1 ) sg(∏s=n+1t−1An,sq) ( _s=n+1^t-1A^q_n,s ) sg(ωn,n+1q)sg(ω^q_n,n+1) Token-level surrogates. Token-level surrogates set bn,t=an=1b_n,t=a_n=1 and align the draft distribution qn,tq_n,t with the target ptp_t at each position t independently. CE and KL both minimize a KL divergence between a reference distribution and qn,tq_n,t: the reference is the one-hot distribution at the sampled target token xtx_t for CE and the full target distribution ptp_t for KL. TV uses the TV distance, which equals the rejection probability at position t. LK interpolates between KL and TV using an acceptance-dependent coefficient λ that balances acceptance awareness against ease of optimization. As the acceptance probability 1−(pt,qn,t)1- TV(p_t,q_n,t) increases, λ decreases, so LK relies mainly on the KL term that is easier to optimize when acceptance is low and moves weight to the TV term as acceptance improves. Block-level surrogates. Block-level surrogates extend token-level surrogates with nonuniform block position weights bn,tb_n,t, which reflect the asymmetry between positions within a block. Later positions receive smaller weights because they are verified less often—a position is verified only if all earlier draft tokens in its block are accepted. Exponential decay and D-PACE multiply a token-level loss by a weight under stop-gradient. The weight is a fixed schedule in exponential decay and a confidence-based surrogate of the accepted length in D-PACE. E2E uses ℓn,t=(pt,qn,t)−1 _n,t= TV(p_t,q_n,t)-1 at draft positions and bn,t=∏s=n+1t−1(1−(ps,qn,s))b_n,t= _s=n+1^t-1(1- TV(p_s,q_n,s)) without stop-gradient. Therefore, the inner sum in (21) is given by −∑k=1minB,L+1−n∏s=n+1n+k(1−(ps,qn,s)),- _k=1 \B,L+1-n\ _s=n+1^n+k (1- TV(p_s,q_n,s) ), the negative of a surrogate for the expected accepted length of the block drafted from anchor n. In contrast, the block position weight of EDR is the conditional survival probability bn,t=∏s=n+1t−1An,sqb_n,t= _s=n+1^t-1A^q_n,s along the target rollout. It is exact in expectation, whereas the product of 1−1- TV terms in E2E is not. Like token-level surrogates, block-level surrogates ignore cross-round coupling, treating each block separately and assigning the same weight an=1a_n=1 to all anchors. EDR. For EDR, the three components are the EOS-aware local rejection cost ℓn,t=cn,tq _n,t=c^q_n,t, the within-round survival probability bn,t=∏s=n+1t−1An,sqb_n,t= _s=n+1^t-1A^q_n,s, and the anchor weight an=ωn,n+1qa_n=ω^q_n,n+1. By the recursion in (5a), we have anbn,t=ωn,n+1q∏s=n+1t−1An,sq=ωn,tq,a_n\,b_n,t=ω^q_n,n+1 _s=n+1^t-1A^q_n,s=ω^q_n,t, which is the probability, conditioned on the target rollout, that decoding visits state (n,t)(n,t). Thus, with these choices, the weighted sum inside the expectation in (21), plus the constant 11, equals the per-sequence EDR loss (10). Taking the expectation over target rollouts gives the expected number of decoding rounds. Among the objectives in Table 1, only EDR has drafter-dependent anchor weights. These weights capture how acceptance decisions change where later rounds start, that is, the cross-round coupling that the surrogates above ignore. EDR in TD form. The TD form of EDR is a different scalar loss, but it has the same gradient with respect to the drafter parameters as the EDR loss above (see Theorem 2). It applies stop-gradient to both the block position weights and the anchor weights, and it adds the term An,tqsg(Vn,t+1q−Vt,t+1q)A^q_n,t\,sg (V^q_n,t+1-V^q_t,t+1 ) to the per-position loss. Gradients therefore flow only through cn,tqc^q_n,t and An,tqA^q_n,t, not through ωqω^q or VqV^q, so training with the TD form is more efficient. We use this form in training (see Algorithm 1). 5 Experiments We evaluate whether directly optimizing EDR improves self-speculative decoding over existing block-local training objectives. We consider two SOTA self-speculative drafters, DSpark (Cheng et al., 2026) and DFly (Liu et al., 2026), paired with Qwen3 (Yang et al., 2025) target models of different scales. 5.1 Experimental Setup Models and sampling. We evaluate two target–drafter pairs: Qwen3-4B with DSpark and Qwen3-8B with DFly. We set block size B=7B=7 for both drafters. For Qwen3-4B, target trajectories are sampled with temperature 0.70.7, top-p=0.8p=0.8, and top-k=20k=20, while the DSpark proposal distribution uses temperature 0.70.7 without top-p or top-k truncation. For Qwen3-8B, both the target and DFly drafter use temperature 1.01.0 without top-p or top-k truncation. Thinking mode is disable in all settings. Training objectives. Starting from the corresponding publicly released draft-model checkpoint, we finetune only the drafter parameters. We compare our exact EDR objective against the end-to-end TV (E2E) block-local objective (Li et al., 2026c), using the same model architecture, training data, optimizer configuration, and number of update steps for both objectives. For all methods, we use at most 512 training anchors per trajectory. For EDR, we use importance sampling to select training anchors. For details, see Appendix C.2. Training data. We train on Open-PerfectBlend (Xu et al., 2024). Rather than using reference answers, we generate a fresh trajectory for each prompt from the target model using the same target sampling configuration as evaluation. Each drafter is finetuned for one epoch with batch size 9696. Evaluation. We use the same evaluation prompts as DSpark (Cheng et al., 2026) on nine benchmarks spanning math (Cobbe et al., 2021; Hendrycks et al., 2021; Zhang and Math-AI, 2025), code (Austin et al., 2021; Chen et al., 2021; Jain et al., 2025), and chat (Zheng et al., 2023; Taori et al., 2023; Li et al., 2024a). We generate at least 1,5001,500 target trajectories per benchmark (multiple trajectories for each prompt) under the same sampling configurations as in training. We compute MAL using the offline estimator introduced in Section 3.4. All methods for the same target model and sampling configurations are evaluated on exactly the same target trajectories. This provides paired comparisons across training objectives and removes additional variance from independently sampled evaluation outputs. All training and evaluation are performed on a single NVIDIA H200 GPU or H100 GPU. For more experimental details, see Appendix D. 5.2 Experimental Results Table 2: One-epoch finetuning results. We report mean accepted length (MAL; higher is better) after one epoch of draft-model finetuning. Target and draft sampling configurations in evaluation are the same as in training. Block size B=7B=7 for all drafters. Bold marks the best finetuned result. Target Drafter (B=7B=7) Math Code Chat GSM8K MATH AIME25 MBPP HumanEval LCB MT-Bench Alpaca Arena-Hard Qwen3-4B (T=0.7T=0.7 top-p=0.8p=0.8 top-k=20k=20) DSpark 6.23 6.12 5.46 5.28 5.52 5.46 3.70 3.62 3.59 E2E 6.32 6.20 5.57 5.34 5.58 5.54 3.75 3.66 3.67 EDR (Ours) 6.32 6.20 5.59 5.34 5.59 5.56 3.78 3.70 3.70 Qwen3-8B (T=1.0T=1.0 no top-p/top-k) DFly 6.30 5.94 5.15 5.26 5.58 5.23 3.57 3.45 3.18 E2E 6.34 5.98 5.18 5.32 5.62 5.26 3.61 3.49 3.21 EDR (Ours) 6.35 6.00 5.22 5.32 5.63 5.29 3.65 3.52 3.25 Table 2 shows that EDR consistently improves speculative decoding across both DSpark and DFly without modifying the drafter architecture. Compared with E2E finetuning, EDR achieves higher or equal MAL across all benchmarks, with larger gains on AIME25, LiveCodeBench, and the chat benchmarks. These results suggest that directly optimizing the global expected number of decoding rounds provides additional gains beyond block-local end-to-end surrogates, and that the improvement transfers across different semi-AR drafter architectures and target-model scales. 6 Conclusion We introduced Expected Decoding Rounds (EDR), an exact global training objective for parallel and semi-AR speculative drafters. By formulating speculative decoding as a Markov reward process conditioned on target-model outputs, we derived a tractable gradient for directly minimizing the expected number of verification rounds, together with an exact offline evaluator for paired draft-model comparison. Empirically, EDR consistently improves two strong drafters, DSpark and DFly, across nine benchmarks without modifying their architectures or inference procedures, and outperforms the E2E training objective in overall mean accepted length. These results suggest that explicitly accounting for cross-round dependencies provides a principled and practical direction for training more efficient parallel speculative draft models. We conclude with several directions for future work. First, our current framework focuses on a fixed draft block size. However, under high-concurrency serving, verifying long draft blocks can become inefficient, especially when many later proposals are likely to be rejected. Adaptive block sizing can reduce unnecessary verification by shortening the draft block when confidence is low, thereby further reducing inference latency (Hu et al., 2026; Huang et al., 2024). An important direction is to extend our framework to adaptive block sizes and to study whether the optimal block size can be selected dynamically for each decoding round. Second, EDR requires computing occupancy weights ω and value functions V for multiple anchor positions. This is more computationally expensive than block-local objectives. It would be interesting to explore more efficient ways to utilize the occupancy and value computations. Finally, as shown in Table 2, MAL varies substantially across datasets, suggesting that some target distributions are intrinsically more difficult to draft than others. Our current work focuses on optimizing the drafter given a target distribution, but does not provide a complete theoretical characterization of this difficulty. A promising direction is to identify properties of the target distribution that determine the best achievable speculative decoding efficiency. Acknowledgements C. Cai is supported by the NSF CAREER Award CCF-2541600 and the NSF grant DMS-2515333. References Ankner et al. (2024) Z. Ankner, R. Parthasarathy, A. Nrusimha, C. Rinard, J. Ragan-Kelley, and W. Brandon Hydra: sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109. Cited by: §4.1. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §5.1. Bellman (1966) R. Bellman Dynamic programming. science 153 (3731), p. 34–37. Cited by: §3.3. Cai and Li (2026) C. Cai and G. Li Confidence-based decoding is provably efficient for diffusion language models. arXiv preprint arXiv:2603.22248. Cited by: §4.1. Cai et al. (2024) T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao Medusa: simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774. Cited by: §1, §4.1. Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: §1. Chen et al. (2026a) J. Chen, Y. Liang, and Z. Liu Dflash: block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.06036. Cited by: §1, §1, 2nd item, §4.1, §4.1, Table 1. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §5.1. Chen et al. (2025) S. Chen, K. Cong, and J. Li Optimal inference schedules for masked diffusion models. arXiv preprint arXiv:2511.04647. Cited by: §4.1. Chen et al. (2022) T. Chen, R. Zhang, and G. Hinton Analog bits: generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202. Cited by: §4.1. Chen et al. (2026b) Y. Chen, C. Liang, H. Sui, R. Guo, C. Cheng, J. You, and G. Liu Langflow: continuous diffusion rivals discrete in language modeling. arXiv preprint arXiv:2604.11748. Cited by: §4.1. Cheng et al. (2026) X. Cheng, X. Yu, C. Shao, J. Li, Y. Xiong, Y. Qian, J. Zhu, S. Ma, X. Zhang, J. Ye, et al. DSpark: confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147. Cited by: §1, §1, §3.5, §4.1, §5.1, §5. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §5.1. Dmitriev et al. (2026) D. Dmitriev, Z. Huang, and Y. Wei Efficient sampling with discrete diffusion models: sharp and adaptive guarantees. arXiv preprint arXiv:2602.15008. Cited by: §4.1. Gu et al. (2026) G. Gu, B. Heo, H. Jun, Y. Kang, S. Lee, S. Yun, and D. Han Verification-aware training for speculative decoding. arXiv preprint arXiv:2608.30135. Cited by: §4.1. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §5.1. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §4.1. Ho and Salimans (2022) J. Ho and T. Salimans Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. Cited by: §4.1. Howard (1971) R.A. Howard Dynamic probabilistic systems: semi-markov and decision processes. Decision and Control Series, Wiley. External Links: ISBN 9780471416661, LCCN 74112593, Link Cited by: §1. Hu et al. (2026) X. Hu, Y. Shen, B. Zhang, H. Zhang, J. Dai, S. Ge, L. Chen, Y. Li, and M. Wan Echo: elastic speculative decoding with sparse gating for high-concurrency scenarios. arXiv preprint arXiv:2604.09603. Cited by: §6. Huang et al. (2024) K. Huang, X. Guo, and M. Wang Specdec++: boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715. Cited by: §6. Inception Labs et al. (2025) Inception Labs, S. Khanna, S. Kharbanda, S. Li, H. Varma, E. Wang, S. Birnbaum, Z. Luo, Y. Miraoui, A. Palrecha, S. Ermon, A. Grover, and V. Kuleshov Mercury: ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298. Cited by: §4.1. Jain et al. (2025) N. Jain, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025, p. 58791–58831. Cited by: §5.1. Jiao et al. (2026) Y. Jiao, N. Li, C. Cai, and G. Li Are first-order diffusion samplers really slower? a fast forward-value approach. In International Conference on Machine Learning, Cited by: §4.1. Lei et al. (2026) H. Lei, Y. Li, H. Zhang, S. Zhang, Q. Cheng, X. Qu, G. Cui, B. Zhou, N. Ding, Y. Luo, et al. Draft-opd: on-policy distillation for speculative draft models. arXiv preprint arXiv:2605.29343. Cited by: §4.1. Leviathan et al. (2023) Y. Leviathan, M. Kalman, and Y. Matias Fast inference from transformers via speculative decoding. In International conference on machine learning, p. 19274–19286. Cited by: §1, §2. Li and Cai (2024) G. Li and C. Cai Provable acceleration for diffusion models under minimal assumptions. arXiv preprint arXiv:2410.23285. Cited by: §4.1. Li and Cai (2025) G. Li and C. Cai Breaking AR’s sampling bottleneck: provable acceleration via diffusion language models. In Advances in Neural Information Processing Systems, Vol. 38, p. 11700–11725. Cited by: §4.1. Li et al. (2026a) G. Li, Z. Fu, M. Fang, Q. Zhao, M. Tang, C. Yuan, and J. Wang Diffuspec: unlocking diffusion language models for speculative decoding. In Findings of the Association for Computational Linguistics: ACL 2026, p. 20896–20910. Cited by: §4.1. Li et al. (2026b) N. Li, Y. Jiao, C. Cai, and G. Li ConvergeFlow: language flow with provable convergence to token embeddings. arXiv preprint arXiv:2608.23551. Cited by: §4.1. Li et al. (2024a) T. Li, W. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica From crowdsourced data to high-quality benchmarks: arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939. Cited by: §5.1. Li et al. (2022) X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems, Vol. 35, p. 4328–4343. Cited by: §4.1. Li et al. (2026c) Y. Li, H. Jiang, Y. Xu, J. Yang, Y. Zhang, Y. Cao, Y. Shen, F. Zhou, R. Men, J. Zhang, et al. Breaking entropy bounds: accelerating rl training via mtp with rejection sampling. arXiv preprint arXiv:2606.12370. Cited by: §1, §4.1, Table 1, §5.1. Li et al. (2024b) Y. Li, F. Wei, C. Zhang, and H. Zhang Eagle: speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077. Cited by: §1. Li et al. (2026d) Y. Li, F. Wei, C. Zhang, and H. Zhang Eagle-3: scaling up inference acceleration of large language models via training-time test. Advances in Neural Information Processing Systems 38, p. 136737–136756. Cited by: §4.1, §4.1. Lipman et al. (2022) Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §4.1. Liu et al. (2026) H. Liu, R. Cen, J. Shi, G. Qin, J. Zhang, T. Liu, R. Fan, G. Zhao, R. Xie, K. Zhang, et al. AngelSpec: towards real-world high performance inference with speculative decoding. arXiv preprint arXiv:2607.25852. Cited by: §1, §1, §3.5, §4.1, §5. Lou et al. (2023) A. Lou, C. Meng, and S. Ermon Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834. Cited by: §4.1. Lu et al. (2022) C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu Dpm-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems 35, p. 5775–5787. Cited by: §4.1. Nie et al. (2025) S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li Large language diffusion models. arXiv preprint arXiv:2502.09992. Cited by: §4.1. Sahoo et al. (2024) S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. Chiu, A. Rush, and V. Kuleshov Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems 37, p. 130136–130184. Cited by: §1, §4.1. Samarin et al. (2026) A. Samarin, S. Krutikov, A. Shevtsov, S. Skvortsov, F. Fisin, and A. Golubev LK losses: direct acceptance rate optimization for speculative decoding. arXiv preprint arXiv:2602.23881. Cited by: §4.1, Table 1. Shi et al. (2024) J. Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems 37, p. 103131–103167. Cited by: §1, §4.1. Song et al. (2020a) J. Song, C. Meng, and S. Ermon Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Cited by: §4.1. Song and Dhariwal (2024) Y. Song and P. Dhariwal Improved techniques for training consistency models. In International Conference on Learning Representations, Vol. 2024, p. 15078–15097. Cited by: §4.1. Song et al. (2020b) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. Cited by: §4.1. Sutton (1988) R. S. Sutton Learning to predict by the methods of temporal differences. Machine learning 3 (1), p. 9–44. Cited by: §3.3. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Stanford alpaca: an instruction-following llama model. GitHub. Note: https://github.com/tatsu-lab/stanford_alpaca Cited by: §5.1. Wu et al. (2026) T. Wu, Y. Yao, Z. Qi, H. Zheng, Z. Wang, H. Ma, L. Liao, H. Lakkaraju, J. Li, and Y. Du D-pace: dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810. Cited by: §1, 2nd item, §4.1, Table 1. Xu et al. (2024) T. Xu, E. Helenowski, K. A. Sankararaman, D. Jin, K. Peng, E. Han, S. Nie, C. Zhu, H. Zhang, W. Zhou, et al. The perfect blend: redefining rlhf with mixture of judges. arXiv preprint arXiv:2409.20370. Cited by: §5.1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5. Ye et al. (2025) J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong Dream 7b: diffusion large language models. arXiv preprint arXiv:2508.15487. Cited by: §4.1. Yin et al. (2024a) M. Yin, M. Chen, K. Huang, and M. Wang A theoretical perspective for speculative decoding algorithm. Advances in Neural Information Processing Systems 37, p. 128082–128117. Cited by: §1, 3rd item. Yin et al. (2024b) T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park One-step diffusion with distribution matching distillation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6613–6623. Cited by: §4.1. Zhang and Math-AI (2025) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2025. Cited by: §5.1. Zhao and Cai (2026) Y. Zhao and C. Cai Adaptation to intrinsic dependence in diffusion language models. arXiv preprint arXiv:2602.20126. Cited by: §4.1. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: §5.1. Appendix A Speculative Decoding Algorithm Algorithm 2 summarizes the speculative decoding procedure we consider. Algorithm 2 Speculative decoding with a semi-autoregressive drafter 1: Block size B, target distributions ptp_t, draft distributions qn,tq_n,t 2: n←0n← 0, ←0T← 0 ⊳ n: number of committed ordinary tokens 3: while true do 4: Draw a draft block x~n+1:n+B∼qn,n+1(⋅∣x1:n)⋯qn,n+B(⋅∣x1:n;x~n+1:n+B−1) x_n+1:n+B q_n,n+1(· x_1:n)·s q_n,n+B(· x_1:n; x_n+1:n+B-1) 5: Evaluate pt(⋅∣x1:n,x~n+1:t−1)p_t(· x_1:n, x_n+1:t-1) for t=n+1,…,n+B+1t=n+1,…,n+B+1 in one target-model pass 6: ←+1T +1 7: rejected←falserejected 8: for t=n+1,…,n+Bt=n+1,…,n+B do 9: p←pt(⋅∣x1:n,x~n+1:t−1)p← p_t(· x_1:n, x_n+1:t-1) 10: q←qn,t(⋅∣x1:n;x~n+1:t−1)q← q_n,t(· x_1:n; x_n+1:t-1) 11: Draw Ut∼Unif[0,1]U_t [0,1] 12: if Ut≤min1,p(x~t)/q(x~t)U_t≤ \1,p( x_t)/q( x_t)\ then 13: xt←x~tx_t← x_t 14: else 15: Draw xt∼[p−q]+/(p,q)x_t [p-q]_+/ TV(p,q) 16: rejected←truerejected 17: end if 18: if xt=<EOS>x_t= <EOS> then 19: return (x1:t−1,<EOS>)(x_1:t-1, <EOS>) and T 20: end if 21: if rejectedrejected then 22: n←tn← t 23: break 24: end if 25: end for 26: if not rejectedrejected then 27: Draw x~n+B+1∼pn+B+1(⋅∣x1:n+B) x_n+B+1 p_n+B+1(· x_1:n+B) ⊳ bonus token 28: if x~n+B+1=<EOS> x_n+B+1= <EOS> then 29: return (x1:n+B,<EOS>)(x_1:n+B, <EOS>) and T 30: else 31: xn+B+1←x~n+B+1x_n+B+1← x_n+B+1 32: n←n+B+1n← n+B+1 33: end if 34: end if 35: end while Appendix B Proofs B.1 Markov Process Formulation in Section 3.1 In this section, we give a detailed derivation of the rejection-sampling transition dynamics conditioned on a fixed target-supported output sequence x=(x1:L,<EOS>)x=(x_1:L, <EOS>). Specifically, we define the Markov chain states, derive the conditional transition probabilities, and prove that the occupancy ωn,tq(x) _n,t^q(x) depends on x only through x<tx_<t. Throughout, all probabilities refer to speculative decoding with a fixed drafter q. States and decoding histories. For each position t reached by decoding, let NtN_t denote the round-start index of the round that produces XtX_t. Thus, immediately before processing position t, the state is St=(Nt,t),S1=(0,1).S_t=(N_t,t), S_1=(0,1). If Nt=nN_t=n, then the current round started from the committed prefix X1:nX_1:n, and all preceding drafts in this round have been accepted. Consequently, their values equal Xn+1:t−1X_n+1:t-1. For t≤n+Bt≤ n+B, position t is processed by draft verification; for t=n+B+1t=n+B+1, it is processed by drawing a target bonus token. States with t>n+B+1t>n+B+1 are unreachable. Let ℋt:=σ(X<t,N1,…,Nt)H_t:=σ(X_<t,N_1,…,N_t) denote the history of committed tokens and round-start indices before processing position t. We first analyze the next output token conditional on this history, without conditioning on the complete output sequence. The local rejection-sampling law. Consider a history with X<t=x<tX_<t=x_<t and Nt=nN_t=n at a draft position t≤n+Bt≤ n+B. For brevity, write pt(v):=pt(v∣x<t),qt(v):=qn,t(v∣x1:n;xn+1:t−1),v∈¯.p_t(v):=p_t(v x_<t), q_t(v):=q_n,t(v x_1:n;x_n+1:t-1), v∈ V. Let acctacc_t and rejtrej_t denote the acceptance and rejection events at this position. The verification rule gives Pr(Xt=v,acct∣ℋt) (X_t=v,acc_t _t) =minpt(v),qt(v), = \p_t(v),q_t(v)\, Pr(Xt=v,rejt∣ℋt) (X_t=v,rej_t _t) =[pt(v)−qt(v)]+. =[p_t(v)-q_t(v)]_+. The first identity follows by multiplying the proposal probability by the verifier’s acceptance probability. For the second identity, rejection has total probability TV(pt,qt)TV(p_t,q_t), and the correction token is drawn from the residual distribution [pt−qt]+/TV(pt,qt)[p_t-q_t]_+/TV(p_t,q_t). When TV(pt,qt)=0TV(p_t,q_t)=0, rejection has probability zero and no residual draw is needed. Adding the two identities yields Pr(Xt=v∣ℋt)=minpt(v),qt(v)+[pt(v)−qt(v)]+=pt(v∣X<t). (X_t=v _t)= \p_t(v),q_t(v)\+[p_t(v)-q_t(v)]_+=p_t(v X_<t). (22) At a bonus-token position, the same identity holds because the token is drawn directly from the target distribution. Therefore, conditional on any decoding history, the next committed token has the target conditional distribution, regardless of the current round-start index. Transition kernels conditioning on the complete output. Equation (22) implies that, on any history with X<t=x<tX_<t=x_<t, Pr(X=x∣ℋt)=p(xt:L+1∣x<t):=∏s=tL+1ps(xs∣x<s). (X=x _t)=p(x_t:L+1 x_<t):= _s=t^L+1p_s(x_s x_<s). (23) To see this rigorously, use backward induction along the fixed sequence x. At t=L+1t=L+1, the required remaining output is just <EOS>, so the identity follows directly from (22). Suppose it holds at t+1t+1 for every possible decoding history. After committing Xt=xtX_t=x_t, every possible resulting history has the same remaining-output probability p(xt+1:L+1∣x≤t)p(x_t+1:L+1 x_≤ t). Multiplying this probability by Pr(Xt=xt∣ℋt)=pt(xt∣x<t) (X_t=x_t _t)=p_t(x_t x_<t) proves the identity at t. In particular, after committing the same non-EOS token xtx_t, the future suffix has the same probability whether the token was accepted or produced by residual sampling. Thus, for t≤Lt≤ L at a draft position, Bayes’ rule gives Pr(acct∣ℋt,X=x) (acc_t _t,X=x) =minpt(xt),qt(xt)p(xt+1:L+1∣x≤t)pt(xt)p(xt+1:L+1∣x≤t) = \p_t(x_t),q_t(x_t)\\,p(x_t+1:L+1 x_≤ t)p_t(x_t)\,p(x_t+1:L+1 x_≤ t) =minpt(xt),qt(xt)pt(xt) = \p_t(x_t),q_t(x_t)\p_t(x_t) =min1,qn,t(xt∣x1:n;xn+1:t−1)pt(xt∣x<t) = \1, q_n,t(x_t x_1:n;x_n+1:t-1)p_t(x_t x_<t) \ =An,tq(x≤t). =A_n,t^q(x_≤ t). All target-probability factors being cancelled are positive because x is target-supported. Similarly, Pr(rejt∣ℋt,X=x)=1−An,tq(x≤t)=Rn,tq(x≤t). (rej_t _t,X=x)=1-A_n,t^q(x_≤ t)=R_n,t^q(x_≤ t). Notice that the ratio here is qt(xt)/pt(xt)q_t(x_t)/p_t(x_t), rather than the verifier’s ratio pt(X~t)/qt(X~t)p_t( X_t)/q_t( X_t): we are conditioning on the committed output token, not on the proposed token. If the token is accepted, the current round continues. If it is rejected, the next round starts after position t. At a bonus-token position t=n+B+1t=n+B+1, the next round also starts after position t, provided that the bonus token is not EOS. Accordingly, setting An,tq=0,Rn,tq=1,t>n+B,A_n,t^q=0, R_n,t^q=1, t>n+B, incorporates the bonus-token case into the same transition rule. At such a position, Rn,tq=1R_n,t^q=1 represents a forced round transition, not an actual draft rejection. It follows that, under Pr(⋅∣X=x) (\,· X=x), the transitions for t≤Lt≤ L are (n,t)⟶(n,t+1),with probability An,tq(x≤t),(t,t+1),with probability Rn,tq(x≤t).(n,t) cases(n,t+1),&with probability A_n,t^q(x_≤ t),\\ (t,t+1),&with probability R_n,t^q(x_≤ t). cases With x fixed, these probabilities depend on the past history only through the current state (n,t)(n,t). This establishes the Markov property. The conditional chain has states 0≤n<t≤L+10≤ n<t≤ L+1 and is stopped at position L+1L+1. All states (n,L+1)(n,L+1) are terminal for the round-count process: the final EOS is produced in the current round and does not start another round, regardless of whether it is accepted, sampled as a correction, or sampled as a bonus token. Occupancy depends only on the committed prefix. Since the position index increases by one at every transition, state (n,t)(n,t) is visited exactly when Nt=nN_t=n. Hence, ωn,tq(x)=Pr(Nt=n∣X=x). _n,t^q(x)= (N_t=n X=x). By (23), conditional on the committed prefix x<tx_<t, the probability of the remaining output xt:L+1x_t:L+1 is the same for every possible round-start history. Therefore, ωn,tq(x) _n,t^q(x) =Pr(Nt=n,X<t=x<t)p(xt:L+1∣x<t)Pr(X<t=x<t)p(xt:L+1∣x<t) = (N_t=n,X_<t=x_<t)\,p(x_t:L+1 x_<t) (X_<t=x_<t)\,p(x_t:L+1 x_<t) =Pr(Nt=n∣X<t=x<t) = (N_t=n X_<t=x_<t) =:ωn,tq(x<t). =: _n,t^q(x_<t). This proves that the occupancy depends only on x<tx_<t, even though it was originally defined by conditioning on the complete output sequence. Starting from ω0,1q=1 _0,1^q=1, the conditional Markov transitions give the forward recursions ωn,t+1q(x≤t) _n,t+1^q(x_≤ t) =ωn,tq(x<t)An,tq(x≤t), = _n,t^q(x_<t)A_n,t^q(x_≤ t), 0≤n<t≤L, 0≤ n<t≤ L, ωt,t+1q(x≤t) _t,t+1^q(x_≤ t) =∑n=0t−1ωn,tq(x<t)Rn,tq(x≤t), = _n=0^t-1 _n,t^q(x_<t)R_n,t^q(x_≤ t), 1≤t≤L. 1≤ t≤ L. The first recursion accounts for continuing the current round. The second sums over all states from which a new round can start after position t. These are precisely the occupancy recursions used in Section 3.1. B.2 Proofs of Proposition 1 and Proposition 2 We first prove Proposition 2, and then obtain Proposition 1 as its special case k=1k=1. Proof of Proposition 2. Fix 1≤k≤B+11≤ k≤ B+1. The random variable ≥kqT^q_≥ k counts the decoding rounds that commit at least k tokens. Conditioned on a fixed output sequence x, consider the round that starts after position n, i.e., from the committed prefix x1:nx_1:n. By the transition rule (4), a rejection transition always leads to a diagonal state, so a non-diagonal state (n,t)(n,t) with t>n+1t>n+1 can only be reached from (n,t−1)(n,t-1) by an acceptance. Hence state (n,n+k)(n,n+k) is visited exactly when this round has started and its drafts at positions n+1,…,n+k−1n+1,…,n+k-1 have all been accepted. In that case, position n+kn+k is processed in the same round and commits one more token, which is an accepted draft, a correction token, a bonus token, or the terminal <EOS>. Conversely, if the round commits at least k tokens, its first k−1k-1 committed tokens must be accepted drafts: a correction token or <EOS> ends the round, and a bonus token can only be the (B+1)(B+1)-th committed token. Therefore, the round starting after position n commits at least k tokens exactly when state (n,n+k)(n,n+k) is visited. It follows that ≥kq=∑n=0L+1−kstate (n,n+k) is visited. ^q_≥ k= _n=0^L+1-k1\state (n,n+k) is visited\. Conditioned on X=xX=x, the event that state (n,n+k)(n,n+k) is visited has probability ωn,n+kq(x<n+k) _n,n+k^q(x_<n+k). Therefore, by linearity of expectation, τ≥kq(x)=[≥kq∣X=x]=∑n=0L+1−kωn,n+kq(x<n+k). _≥ k^q(x)=E[T^q_≥ k X=x]= _n=0^L+1-k _n,n+k^q(x_<n+k). Proof of Proposition 1. Every round commits at least one token, so q=≥1qT^q=T^q_≥ 1. Equivalently, a round starts after position t exactly when state (t,t+1)(t,t+1) is visited. Taking k=1k=1 in Proposition 2 gives τq(x)=[q∣X=x]=∑t=0Lωt,t+1q(x≤t). τ^q(x)=E[T^q X=x]= _t=0^L _t,t+1^q(x_≤ t). Separating the term t=0t=0, using ω0,1q=1 _0,1^q=1, and substituting the occupancy recursion (5b) for 1≤t≤L1≤ t≤ L gives τq(x)=ω0,1q+∑t=1Lωt,t+1q(x≤t)=1+∑t=1L∑n=0t−1ωn,tq(x<t)Rn,tq(x≤t). τ^q(x)= _0,1^q+ _t=1^L _t,t+1^q(x_≤ t)=1+ _t=1^L _n=0^t-1 _n,t^q(x_<t)R_n,t^q(x_≤ t). B.3 Proof of Theorem 1 To handle the randomness of the sequence length L, we rewrite Proposition 1 and take the expectation over X∼pX p, [τq(X)]=1+∑t≥1∑n=0t−1[t≤Lωn,tq(X<t)Rn,tq(X≤t)].E[τ^q(X)]=1+ _t≥ 1 _n=0^t-1E\! [1_\t≤ L\ _n,t^q(X_<t)R_n,t^q(X_≤ t) ]. (24) Before XtX_t is revealed, ωn,tq(X<t) _n,t^q(X_<t) is determined by X<tX_<t, and t≤Lt≤ L is exactly the event Xt≠<EOS>X_t≠ <EOS>. Hence, conditioning on X<tX_<t and using the definition of cn,tqc_n,t^q, [t≤Lωn,tq(X<t)Rn,tq(X≤t)∣X<t] \! [1_\t≤ L\ _n,t^q(X_<t)R_n,t^q(X_≤ t) X_<t ] =ωn,tq(X<t)Pr(Xt≠<EOS>∣X<t)cn,tq(X<t) = _n,t^q(X_<t) (X_t≠ <EOS> X_<t)c_n,t^q(X_<t) =[t≤Lωn,tq(X<t)cn,tq(X<t)∣X<t]. =E\! [1_\t≤ L\ _n,t^q(X_<t)c_n,t^q(X_<t) X_<t ]. (25) Taking expectations and summing over n and t proves (9). B.4 Proof of Theorem 2 We first prove (13), which relates the initial-state value to the per-sequence EDR loss. V0,1q(x) V_0,1^q(x) =∑t=1L[∑n=0t−1ωn,tq(x<t)Vn,tq(x)−∑n=0tωn,t+1q(x≤t)Vn,t+1q(x)] = _t=1^L [ _n=0^t-1 _n,t^q(x_<t)V_n,t^q(x)- _n=0^t _n,t+1^q(x_≤ t)V_n,t+1^q(x) ] =∑t=1L∑n=0t−1ωn,tq(x<t)[Vn,tq(x)−An,tq(x≤t)Vn,t+1q(x)−Rn,tq(x≤t)Vt,t+1q(x)] = _t=1^L _n=0^t-1 _n,t^q(x_<t) [V_n,t^q(x)-A_n,t^q(x_≤ t)V_n,t+1^q(x)-R_n,t^q(x_≤ t)V_t,t+1^q(x) ] =∑t=1L∑n=0t−1ωn,tq(x<t)cn,tq(x<t) = _t=1^L _n=0^t-1 _n,t^q(x_<t)c_n,t^q(x_<t) =ℒ^EDR(qθ,x)−1. = L_EDR(q_θ;x)-1. Now we prove Theorem 2. Differentiating the Bellman equation (12) at one state gives ∇θVn,tq _θV_n,t^q =∇θcn,tq+∇θAn,tq(Vn,t+1q−Vt,t+1q)⏟local TD gradient = _θc_n,t^q+ _θA_n,t^q (V_n,t+1^q-V_t,t+1^q )_local TD gradient +An,tq∇θVn,t+1q+Rn,tq∇θVt,t+1q, +A_n,t^q _θV_n,t+1^q+R_n,t^q _θV_t,t+1^q, (26) where we used Rn,tq=1−An,tqR_n,t^q=1-A_n,t^q. Unrolling the last two terms from the initial state weights each local TD gradient by the probability of visiting that state, namely ωn,tq _n,t^q. Apply (26) recursively from (0,1)(0,1). Every time a state (n,t)(n,t) is reached, its local term is accumulated once, while the two recursive terms propagate with transition probabilities An,tqA_n,t^q and Rn,tqR_n,t^q. The total coefficient on the local term at (n,t)(n,t) is therefore exactly its state occupancy ωn,tq _n,t^q. Using (13) yields (14). Rigorously, ∇θℒ^EDR(qθ,x) _θ L_EDR(q_θ;x) =∇θV0,1q(x) = _θV_0,1^q(x) =∑t=1L[∑n=0t−1ωn,tq(x<t)∇θVn,tq(x)−∑n=0tωn,t+1q(x≤t)∇θVn,t+1q(x)] = _t=1^L [ _n=0^t-1 _n,t^q(x_<t) _θV_n,t^q(x)- _n=0^t _n,t+1^q(x_≤ t) _θV_n,t+1^q(x) ] =∑t=1L∑n=0t−1ωn,tq(x<t)[∇θVn,tq(x)−An,tq(x≤t)∇θVn,t+1q(x)−Rn,tq(x≤t)∇θVt,t+1q(x)] = _t=1^L _n=0^t-1 _n,t^q(x_<t) [ _θV_n,t^q(x)-A_n,t^q(x_≤ t) _θV_n,t+1^q(x)-R_n,t^q(x_≤ t) _θV_t,t+1^q(x) ] =∑t=1L∑n=0t−1ωn,tq(x<t)[∇θcn,tq(x<t)+∇θAn,tq(x≤t)(Vn,t+1q(x)−Vt,t+1q(x))]. = _t=1^L _n=0^t-1 _n,t^q(x_<t) [ _θc_n,t^q(x_<t)+ _θA_n,t^q(x_≤ t) (V_n,t+1^q(x)-V_t,t+1^q(x) ) ]. B.5 Proof of Theorem 3 Let B≥3B≥ 3 and let the vocabulary be =0,1,aV=\0,1,a\. We independently generate m subsequences W1,…,WmW_1,…,W_m, where Wi=0a⋯a⏟B−2 times0a,with probability 2/3,1a⋯a⏟B−2 times1,with probability 1/3.W_i= cases0 a·s a_B-2 times0a,&with probability 2/3,\\ 1 a·s a_B-2 times1,&with probability 1/3. cases We refer to these as type-00 and type-11 subsequences, respectively. Let the target distribution be X=W1W2⋯Wm<EOS>X=W_1W_2·s W_m <EOS>, so that [L]=(B+23)mE[L]= (B+ 23 )m. The integer m will be chosen at the end of the proof. We consider first-order drafters of the form qn,t(⋅∣x1:n;xn+1:t−1)=qn,t(⋅∣x1:n;xt−1).q_n,t (· x_1:n;x_n+1:t-1 )=q_n,t (· x_1:n;x_t-1 ). The block-local optimum. Consider a round that starts exactly at the beginning of a subsequence, following a committed prefix x1:nx_1:n. The first B−1B-1 draft positions can match the target distribution exactly: q^n,n+1=23δ0+13δ1,q^n,n+j=δa,2≤j≤B−1. q_n,n+1= 23 _0+ 13 _1, q_n,n+j= _a, 2≤ j≤ B-1. Here and below, these equalities are understood on target-supported within-block prefixes. At the final draft position n+Bn+B, the target repeats the first bit of the subsequence. Since B≥3B≥ 3, the immediately preceding token is a for both types. A first-order drafter must therefore use the same distribution at this position for both types. To maximize the expected accepted length, q q should maximize probablity of accepting the last token in the round, i.e, q^n,n+B=δ0. q_n,n+B= _0. If Wi=0a⋯a⏟B−2 times0aW_i=0 a·s a_B-2 times0a, all B draft tokens are accepted, and the target bonus token commits the final a. If Wi=1a⋯a⏟B−2 times1W_i=1 a·s a_B-2 times1, the proposal is rejected at the final draft position, and residual sampling commits the final 11. In either case, the round commits exactly the entire subsequence, and the next round starts at the beginning of Wi+1W_i+1. After the m subsequences have been generated, one additional round generates <EOS>. Therefore, every block-local optimal drafter q q satisfies [q^]=m+1.E[T q]=m+1. A continuation-aware drafter. We next construct another first-order drafter q∗q^* that intentionally sacrifices accepted length in some rounds to obtain more favorable future round starts. Whenever a round starts at the beginning of a subsequence, let q∗q^* match the target distribution in the first B−1B-1 positions: qn,n+1∗=23δ0+13δ1,qn,n+j∗=δa,2≤j≤B−1,q^*_n,n+1= 23 _0+ 13 _1, q^*_n,n+j= _a, 2≤ j≤ B-1, but choose qn,n+B∗=δ1.q^*_n,n+B= _1. If Wi=0a⋯a⏟B−2 times0aW_i=0 a·s a_B-2 times0a, the final proposal is rejected. The round commits the corrected 00 and advances by B tokens, leaving the final a for the next round. If Wi=1a⋯a⏟B−2 times1W_i=1 a·s a_B-2 times1, all B draft tokens are accepted. The bonus token then commits the first token of the next subsequence, or <EOS> if i=mi=m. Whenever a round starts inside a subsequence, let q∗q^* match the target conditionals at every draft position. This is possible within the first-order drafter class: the bit defining the current subsequence is already contained in the committed prefix. Any fresh bit from the next subsequence appears at draft position j≥2j≥ 2, and its repeated occurrence is at position j+B−1≥B+1j+B-1≥ B+1, outside the draft block. Before that repeated occurrence, the intervening tokens are all a. Thus no draft position needs access to an earlier within-block bit beyond the immediately preceding token. Every such round therefore advances by B+1B+1 tokens, unless generation terminates earlier at <EOS>. To count the decoding rounds, extend W1,W2,…W_1,W_2,… to an infinite independent sequence for this calculation. Group consecutive subsequences between successive rounds that start at the beginning of a subsequence. If a group starts with a type-00 subsequence, its first round leaves the final a uncommitted. Each subsequent round either leaves the final a of another type-00 subsequence or commits an entire type-11 subsequence and returns to a subsequence boundary. The group therefore ends at the first type-11 subsequence after its initial type-00 subsequence. Its expected number of subsequences is 1+3=41+3=4, and it uses exactly one round per subsequence. If a group starts with a type-11 subsequence, every round advances by B+1B+1 tokens until a subsequence boundary is reached again. For any initial segment of this group, its total length is B+1B+1 times its number of subsequences, minus its number of type-11 subsequences. Consequently, the first return to a subsequence boundary occurs upon completing the (B+1)(B+1)-st type-11 subsequence, counting the initial one. The expected number of subsequences in this group is 1+3B1+3B. If the group contains ℓ subsequences, its total length is (ℓ−1)(B+1)( -1)(B+1), so it uses exactly ℓ−1 -1 rounds and saves one round. Let ℓj _j denote the number of subsequences in group j. The groups are independent and identically distributed, with [ℓj]=23⋅4+13(1+3B)=B+3.E[ _j]= 23· 4+ 13(1+3B)=B+3. Let N:=mink≥1:∑j=1kℓj≥m,N:= \k≥ 1: _j=1^k _j≥ m \, and let H be the number of these N groups that start with a type-11 subsequence. Since N≥j\N≥ j\ depends only on the preceding groups and N≤mN≤ m, summing expectations gives m m ≤[∑j=1Nℓj]=∑j=1m[N≥jℓj]=(B+3)[N], \! [ _j=1^N _j ]= _j=1^mE\! [1_\N≥ j\ _j ]=(B+3)E[N], [H] [H] =∑j=1mPr(N≥j,group j starts with type 1)=13[N]. = _j=1^m \! (N≥ j,\,group $j$ starts with type $1$ )= 13E[N]. All groups except possibly the last are fully generated before termination, and each completed type-11 group saves one round. For the last group, suppose that r of its subsequences remain before <EOS>. These subsequences contain at most r(B+1)r(B+1) tokens, and only the first round of the group can lose one token to rejection. Thus generating these subsequences together with <EOS> requires at most ⌈r(B+1)+2B+1⌉=r+1 r(B+1)+2B+1 =r+1 rounds. At most one of the H type-11 groups is the last group, so q∗≤m+2−H.T^q^*≤ m+2-H. Taking expectations yields [q∗] [T^q^*] ≤m+2−[H]=m+2−13[N] ≤ m+2-E[H]=m+2- 13E[N] ≤(1−13(B+3))m+2. ≤ (1- 13(B+3) )m+2. Finally, choose any integer m>(3B+9)(3B+11)m>(3B+9)(3B+11). Since both drafters preserve the same target distribution, (q∗)(q^) MAL(q^*) MAL( q) =[q^][q∗] = E[T q]E[T^q^*] ≥m+1(1−13(B+3))m+2 ≥ m+1 (1- 13(B+3) )m+2 >1+13B+9. >1+ 13B+9. This proves the theorem. Appendix C Additional Details for Expected Decoding Rounds C.1 Draft EOS Probability Does Not Reduce EDR For the EDR objective, a correctly drafted terminal <EOS> does not save a target-model verification round. Conditional on reaching the terminal position, accepting a drafted <EOS> terminates in the current verification pass. If the draft instead misses <EOS>, the correction branch can generate <EOS> in that same pass and terminates before another round starts. The same holds when the terminal token is produced as the target bonus token after block exhaustion. Accordingly, the EDR sum runs only over ordinary-token positions t≤Lt≤ L. This is also visible in the dense local cost cn,tc_n,t . Its numerator is ∑v∈[pt(v∣x<t)−qn,t(v∣x1:n;xn+1:t−1)]+, _v [p_t(v x_<t)-q_n,t(v x_1:n;x_n+1:t-1) ]_+, which excludes <EOS>. Thus matching target EOS mass is not directly rewarded by EDR. C.2 Fixed-Size Anchor Sampling In this section, we explain how to use importance sampling to select a fixed number of anchors in EDR training. We apply anchor sampling separately to each target rollout in Algorithm 1. For a rollout x=(x1:L,<EOS>)x=(x_1:L, <EOS>), define +(x):=0≤n<L:ωn,n+1qθ(x1:n)>0.N_+(x):= \0≤ n<L: _n,n+1^q_θ(x_1:n)>0 \. An anchor outside +(x)N_+(x) contributes zero to (14), since its occupancy ωn,tqθ(x<t) _n,t^q_θ(x_<t) is zero for every t>nt>n. Let ρ be the maximum number of training anchors per rollout, set to 512512 in our experiments. If ρ≥|+(x)|ρ≥|N_+(x)|, select ℐ=+(x)I=N_+(x) and set πn(x)=1 _n(x)=1 for every n∈+(x)n _+(x). Otherwise, choose λ(x)>0λ(x)>0 such that ∑n∈+(x)min1,λ(x)ωn,n+1qθ(x1:n)=ρ, _n _+(x) \1,λ(x) _n,n+1^q_θ(x_1:n) \=ρ, (27) and set πn(x):=min1,λ(x)ωn,n+1qθ(x1:n). _n(x):= \1,λ(x) _n,n+1^q_θ(x_1:n) \. (28) We draw a set ℐ⊆+(x)I _+(x) of exactly ρ anchors by random-start systematic sampling, with Pr(n∈ℐ∣x)=πn(x). (n x)= _n(x). For rollout x(i)x^(i), denote the sampled anchor set by ℐ(i)I^(i) and write πn(i):=πn(x(i)) _n^(i):= _n(x^(i)). Using the same notation as Algorithm 1, replace its loss by ℓ(θ)=1M∑i=1M∑n∈ℐ(i)∑t=n+1Lisg(ωn,t(i)πn(i))[cn,t(i)+An,t(i)sg(Vn,t+1(i)−Vt,t+1(i))]. (θ)= 1M _i=1^M _n ^(i) _t=n+1^L_isg\! ( ω^(i)_n,t _n^(i) ) [c^(i)_n,t+A^(i)_n,tsg\! (V^(i)_n,t+1-V^(i)_t,t+1 ) ]. (29) The occupancies and value functions are still computed by the full recursions in Algorithm 1, without gradients; sampling restricts only which anchor contributions are differentiated. The sampled anchor sets and inclusion probabilities are also held fixed during differentiation. Conditioned on the target rollouts, the inclusion probabilities cancel the inverse-probability weights. Hence, ℐ(i)i=1M[∇θℓ(θ)|x(i)i=1M] _\I^(i)\_i=1^M\! [ _θ (θ)\, |\,\x^(i)\_i=1^M ] =1M∑i=1M∑n∈+(x(i))∑t=n+1Liωn,t(i)[∇θcn,t(i)+∇θAn,t(i)(Vn,t+1(i)−Vt,t+1(i))] = 1M _i=1^M _n _+(x^(i)) _t=n+1^L_iω^(i)_n,t [ _θc^(i)_n,t+ _θA^(i)_n,t (V^(i)_n,t+1-V^(i)_t,t+1 ) ] =1M∑i=1M∇θℒ^EDR(qθ,x(i)), = 1M _i=1^M _θ L_EDR(q_θ;x^(i)), (30) where the last equality follows from (14). Taking the expectation over the target rollouts and applying (15) gives [∇θℓ(θ)]=∇θℒEDR(qθ).E\! [ _θ (θ) ]= _θL_EDR(q_θ). Thus anchor sampling preserves preserves the unbiasedness of the stochastic gradient. Appendix D Additional Details for Experiments For all training settings and methods, we use the hyperparameters summarized in Table 3. Table 3: Training hyperparameters used in all experiments. Hyperparameter Value Learning rate 3×10−53× 10^-5 Optimizer AdamW AdamW β1 _1 0.90.9 AdamW β2 _2 0.9990.999 Learning-rate schedule 4%4\% linear warmup of the first epoch, then constant Gradient clipping Global norm ≤1≤ 1 Global batch size 9696