Paper deep dive

Cost-Aware Speculative Execution for LLM-Agent Workflows: An Integrated Five-Dimension Method

Faisal Fareed

Year: 2026Venue: arXiv preprintArea: cs.DCType: PreprintEmbeddings: 84

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 7/9/2026, 3:11:42 AM

Summary

The paper introduces a cost-aware speculative execution framework for LLM-agent workflows that optimizes the latency-cost trade-off through five integrated design dimensions. It proposes launching downstream operations before upstream completion using predicted inputs, pricing speculations via separate input/output token rates, exposing a user preference dial (α), applying an expected-value decision rule with failure-weighted costs, and estimating success probabilities using a Bayesian Beta-Binomial posterior. The method enforces admissibility preconditions to prevent irreversible side effects, includes a five-stage calibration pipeline, and demonstrates clear differentiators over existing systems like DSP, Speculative Actions v2, Sherlock, and B-PASTE.

Entities (13)

Speculative Execution · technique · 98%LLM-Agent Workflows · systemdomain · 97%Design Decision 1 (D1) · method-component · 96%Design Decision 2 (D2) · method-component · 96%Expected Value (EV) Rule · algorithm · 96%Design Decision 3 (D3) · method-component · 95%Design Decision 4 (D4) · method-component · 95%Design Decision 5 (D5) · method-component · 95%Admissibility Precondition · constraint · 94%Beta-Binomial Posterior · algorithm · 94%Calibration Pipeline · process · 93%B-PASTE · baseline-system · 92%DSP · baseline-system · 92%

Relation Signals (10)

Design Decision 1 (D1) → enables → Speculative Execution

confidence 96% · D1. Pre-upstream-completion speculation over arbitrary DAGs of LLM operations.

Design Decision 4 (D4) → implements → Expected Value (EV) Rule

confidence 96% · D4. Expected-value decision rule with failure-weighted waste term and α-scaled threshold.

Design Decision 5 (D5) → estimates → Beta-Binomial Posterior

confidence 95% · D5. Bayesian Beta-Binomial posterior over success probability, with structural priors keyed to an LLM dependency-type taxonomy.

Speculative Execution → optimizes → LLM-Agent Workflows

confidence 95% · Speculative execution attacks the latency by starting a downstream step before its upstream has finished, working from a best-guess input in place of the output the upstream will eventually produce.

Design Decision 2 (D2) → defines → Expected Value (EV) Rule

confidence 94% · D2. Two-rate per-token monetary cost, plugged into the decision rule at runtime.

Admissibility Precondition → restricts → Speculative Execution

confidence 94% · The rule fires only on edges passing an admissibility precondition (side-effect-free, idempotent, or stageable behind a commit barrier), since a wrong speculation is rolled back by re-execution.

Design Decision 3 (D3) → exposes → Design Decision 4 (D4)

confidence 93% · D3. A user-facing α ∈ [0,1] preference dial, separated from a deployment-configured λ($/second) latency-value conversion.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:LLM-agent workflows chain model calls and tool invocations, and spend most of their wall-clock time waiting on upstream operations before downstream ones can start. Speculative execution can reclaim that idle time by launching a downstream operation with a predicted upstream input, but here each speculation costs real money (per-token billing) and its success probability is hard to estimate and drifts over time. This paper presents a method organized around five design decisions: (D1) start a downstream operation before its upstream completes; (D2) price each speculation in real dollars at separate input and output rates; (D3) expose a single operator dial for latency versus cost; (D4) decide via an expected-value rule with a failure-weighted cost term and a preference-adjusted threshold; and (D5) estimate the success probability with a Bayesian Beta-Binomial posterior whose prior is keyed to a dependency-type taxonomy. Variants of these ideas appear in recent work; the combination, with every decision logged in dollars, is what is new. The rule fires only on edges passing an admissibility precondition (side-effect-free, idempotent, or stageable behind a commit barrier), since a wrong speculation is rolled back by re-execution, which refunds tokens but cannot un-send an irreversible side effect. We specify the runtime mechanics, a closed-form result that the rule self-limits as the upstream branching factor grows, a five-stage calibration pipeline (offline replay, shadow, canary, online calibration, drift-triggered kill-switch), and a workload-fit rubric over eight production archetypes. Contrast tables against the four closest published systems (DSP, Speculative Actions v2, Sherlock, B-PASTE) show differentiators on every dimension, and a synthetic validation suite confirms the predicted decision boundary, probability threshold, posterior recovery, and streaming-cancellation behavior.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,703 characters extracted from source content.

Expand or collapse full text

Cost-Aware Speculative Execution for LLM-Agent Workflows: An Integrated Five-Dimension Method Faisal Fareed, AWS June 5, 2026 Abstract LLM-agent workflows chain large-language-model calls and tool invocations, and they spend most of their wall-clock time waiting on upstream operations before downstream ones can start. Speculative execution can reclaim that idle time by launching a downstream operation with a predicted upstream input, but in this setting each speculation costs real money (per-token API billing) and the success prob- ability is hard to estimate and changes over time. This paper describes a method organized around five design decisions: (D1) start a downstream operation before its upstream completes, (D2) price each speculation in real dollars at separate input and output rates, (D3) expose a single operator dial for latency versus cost, (D4) decide using an expected-value rule with a failure-weighted cost term and a preference-adjustedthreshold, and(D5)estimatethesuccessprobabilitywithaBayesianBeta-Binomial posterior whose prior is keyed to a dependency-type taxonomy. Variants of these ideas appear in recent work; the combination, with every decision logged in dollars, is what is new. The decision rule fires only on edges that pass an admissibility precondition (the speculated downstream must be side-effect- free, idempotent, or stageable behind a commit barrier), because a wrong speculation is rolled back by re-execution, which refunds wasted tokens but cannot un-send an irreversible side effect. The pa- per also specifies the runtime mechanics: a two-phase plan-plus-runtime decision model with bidirec- tional override, and streaming re-estimation of the predicted input with mid-stream cancellation and fractional-waste accounting. It gives a closed-form derivation that the decision rule self-limits as the upstream branching factor grows. For deployment it specifies a five-stage calibration pipeline (offline replay→shadow→canary with recovery of the implied dollar-value-of-latency→online calibration →drift-triggered kill-switch) and a workload-fit rubric covering eight production archetypes. Contrast tables against the four closest published systems, namely Dynamic Speculative Agent Planning (DSP), Speculative Actions v2, Sherlock, and B-PASTE, show clear differentiators on every dimension. A syn- thetic numerical validation suite (Appendix D) confirms, at the canonical AutoReply parameters, that the speculate-or-wait boundary matches the closed-form critical branching factor, that the expected value crosses zero at the predicted success-probability threshold, that the Beta-Binomial posterior re- covers the true success rate from a uniform prior, that streaming cancellation reduces per-failure waste as predicted, and that the implied-value audit catches the divergent operating point used as the running example. Keywords:speculative execution, LLM agents, agent workflows, cost-aware scheduling, Bayesian in- ference, dependency types, multi-agent orchestration. 1 arXiv:2606.07846v1 [cs.DC] 5 Jun 2026 1. Introduction 1.1 Setting An LLM-agent workflow is a chain of steps in which each step feeds the next — formally, a directed acyclic graph (DAG) of operations. Each step (vertex) is an LLM call or a tool invocation, and each dependency (edge) means one step consumes another’s output. Two things make these workflows expensive. First, theyareslow: LLMlatencydominatesexecutiontime, withasingleoperationtypically taking hundreds of milliseconds to tens of seconds, and end-to-end latency is the critical-path sum of thoseoperationlatenciesminuswhateverparallelismtheruntimeextracts. Second, theyarecostly: cost is the sum of per-token billing, which for major commercial APIs is asymmetric (output tokens billed at 3–8× the input-token rate) and varies substantially across providers and model tiers. Speculativeexecutionattacksthelatencybystartingadownstreamstepbeforeitsupstreamhasfinished, working from a best-guess input in place of the output the upstream will eventually produce. Formally, it starts a downstream operationvbefore its upstreamuhas produced output, using a predicted inputî in place ofu’s eventual outputi. When the guess is good enough — the downstream result computed fromîis acceptable giveni— end-to-end wall-clock time decreases. When it is not, the downstream is re-executed withiand the speculative cost is wasted. 1.2 The research question When should an LLM-agent runtime launch a downstream operation before its upstream has finished, given that each speculation costs real money, the chance the guess pays off is hard to estimate and drifts over time, and operators need to trade speed against cost differ- ently at different times? 1.3 Contributions The method has five design dimensions and seven auxiliary mechanisms: 1.D1.Pre-upstream-completion speculation over arbitrary DAGs of LLM operations (Section 3). 2.D2.Two-rate per-token monetary cost, plugged into the decision rule at runtime (Section 4). 3.D3.A user-facingα ∈ [0,1]preference dial, separated from a deployment-configuredλ($/sec- ond) latency-value conversion (Section 5). 4.D4.Expected-value decision rule with failure-weighted waste term andα-scaled threshold (Sec- tion 6). 5.D5.Bayesian Beta-Binomial posterior over success probability, with structural priors keyed to an LLM dependency-type taxonomy (Section 7). 6.Two-phase plan-time + runtime decision model with bidirectional override (Section 8). 7.Streaming partial-output re-estimation with mid-stream cancellation and fractional waste accounting (Section 9). 8.Self-limiting behavior under branching factor with closed-form critical-kderivation and effective- kaccounting under skew (Section 7.6). 9.Five-stage calibration-and-evaluation pipeline, including implied-λrecovery from observed oper- ator preferences (Section 12). 10.Four-point workload-fit rubric with eight production archetypes and four explicit non-fit shapes (Section 13). 11.Per-decision telemetry schema sufficient for every calibration signal (Appendix C). 2 12.Synthetic numerical validation suite covering decision boundary, P-threshold, posterior conver- gence, streaming cancellation, and implied-λrecovery, each fully specified from the equations under a single fixed seed (Appendix D). Each item is a specific design choice with a real alternative the paper rejects. Section 11 shows that no subset of the audited prior art (DSP, Speculative Actions v2, Sherlock, B-PASTE) discloses the combina- tion. Appendices A–D give the posterior-update mechanics, a worked router-dependency example, the per-decision telemetry schema, and a synthetic numerical validation suite. 1.4 Scope In scope.Cost-aware speculative execution for LLM-agent workflows expressed as a DAG of LLM calls and tool invocations, billed per token or per GPU-hour. Out of scope.Token-level speculative decoding inside a single LLM forward pass (a different layer of the stack); dynamic workflows whose topology is determined at runtime (loops, reflection, agent spawning); RLHF and alignment questions; settings where the runtime cannot produce a predicted in- putîat the moment a speculation could fire; and downstream operations whose external side effects are not side-effect-free, idempotent, or stageable behind a commit barrier (the admissibility precondi- tion of Section 3.3, since a speculation that sends an email or charges a card cannot be rolled back by re-execution). 2. Problem and notation 2.1 Setting AnLLM-agentworkflowisaDAGW = (V, E)whereeachvertexv ∈ VisanLLMcallortoolinvocation and each edge(u, v) ∈ Eindicates thatvconsumes output fromu. A speculative-execution runtime decides, for each downstream operationvwhose upstreamuis currently executing, whether to launch vbeforeucompletes (using a predicted upstream inputî) or wait foruto finish. edge predicted input speculative launch using î u completes, emits i Tier 1/2/3 match mismatch Upstream u executing, output i not yet emitted Downstream v ready to launch v′ result under î Check: i == î or equiv? Commit v′ Re-execute v with i Figure 1. DAG speculation geometry. The runtime launchesvagainst a predicted inputîwhileuis still generating. Onu’s completion, a three-tier check (Section 7.4) decides whetherv’s speculative result is reusable. 2.2 Notation 3 SymbolMeaningUnits Pprobability that a speculation is useful dimensionless,[0,1] C_specmonetary cost of speculatively executing the downstream USD L_valuedollar-value of latency saved on speculation success USD Lestimated latency savingsseconds αuser preference (0 = cost-sensitive, 1 = latency-sensitive) dimensionless,[0,1] λlatency value ratio (dollars per second saved) USD/s input_tokens,output_tokensestimated token counts for the speculative operation tokens input_price,output_priceAPI billing ratesUSD/token î,ipredicted upstream input, actual upstream input n/a By convention all money is in US dollars; currency conversion is a deployment detail. 3. Dimension 1: Pre-upstream-completion speculation 3.1 Definition For a downstream operationvthat depends on an upstreamu, the runtime may launchvbeforeuhas produced its output, using a predicted inputîin place of the real inputithatuwill eventually emit. Whenucompletes and emitsi, the runtime checks whetherv’s speculative result (computed usingî) is acceptable per the three-tier criterion of Section 7.4. If acceptable, the speculative result is committed; if not,vis re-executed withi. This is mechanically distinct from three adjacent patterns: •Token-level speculative decoding[Leviathan et al., 2023] operates within a single LLM for- ward pass, not across agent operations. •Post-output, pre-verification speculation[Ro et al., 2025] uses the actual upstream outputi and speculates thatverificationofiwill pass. •Reasoning-timetoolpre-launch[Suietal., 2026; Song, 2026]speculatesthenexttoolcallwhile the agent LLM is still generating its next plan step. D1 is specifically“upstream has not yet producedi; we guessînow.” 3.2 Where the predicted input comes from Three sources, in preference order: 4 1.Context-conditioned prediction.A cheap auxiliary model or template predictsîfrom the up- stream’s input and partial state. If the upstream is a topic-extraction agent known to produce 3–5 topics,îmay be“the top-ranked candidate topic from the upstream’s partial state.” 2.Most-likely historical input.From logged(upstream_input, upstream_output)pairs, the modal output for similar inputs. 3.Streaming partial output(Section 9). If the upstream streams tokens, re-estimateîas tokens arrive, updating the speculation decision in flight. îis a design choice; thecorrectnessof the rest of the method does not depend onhowîwas produced, onlythat(a)thereisapredictedinputatspeculation-launchtimeand(b)thesuccesscriterionofSection 7.4labels each trial. The method’slatencyeconomics do dependon the predictor’sowncost: a predictor that is itself an expensive model call can erode or erase the net latency saving (Section 14.2). 3.3 Admissibility: speculation requires a side-effect-free or compensable downstream A speculation may turn out to be wrong: whenuemitsiand the tier-1/2 check (Section 7.4) fails, the method re-executesvwith the correct input. This rollback is sound only if launchingvagainst the wrongîleft no observable trace outside the runtime. The cost model (Section 4) accounts for the wasted tokensof a failed speculation, but it does not account for a failed speculation’sexternal effects. Re-execution refunds neither. A downstream operationvis thereforeadmissible for speculationonly if at least one of the following holds: 1.Side-effect-free.vis a pure LLM generation or a read-only tool call (retrieval, aGET, a lookup). Discarding its result on tier-failure costs only tokens. This is the default and covers every archetype in Section 13. 2.Idempotent under the natural key.v’s effect is keyed such that the speculative invocation and the corrected re-execution collapse to the same final state (e.g., an upsert keyed on a deterministic id). The speculative write is overwritten, not duplicated. 3.Staged behind a commit barrier.v’s externally-visible effect is buffered (a draft, a transaction not yet committed, an outbound message held in a queue) and released only after the tier-1/2 check passes. On failure the staged effect is dropped andvre-runs before anything is released. Operations that fail all three (a tool call that sends an email, charges a card, posts an irreversible muta- tion, or triggers a downstream actuator the moment it is invoked)must not be speculated, regardless of EV. Re-execution cannot un-send the message, and(1−P) · C_specdoes not price the cost of the erroneous side effect. Such edges are taggednon_speculableand the per-edge enable bit (Section 12) is held off for them at deployment time, independent of the decision rule. The EV gate of Section 6 runs only on edges that have already passed this admissibility test. This is a hard precondition, not a tuning knob: the failure-weighted cost model is correct precisely because the only thing wasted on a failed speculation is compute. That assumption holds exactly when vis admissible by one of the three routes above. 4. Dimension 2: Per-token monetary cost in decisions 4.1 Two-rate per-token pricing For an LLM operation billed per token at distinct input and output rates, 5 C_spec = input_tokens · input_price + output_tokens · output_price . input_tokensandoutput_tokensare estimates available at speculation time. Commercial APIs (An- thropic, OpenAI, Google, Mistral) bill input and output tokens at distinct rates differing by 3–8× as of 2026, so conflating them into a single rate materially distorts the decision at high output-to-input ratios (the common case for generation-heavy agents). A pricing-map data structure records per-(provider, model)rates: @dataclass classPricingEntry: provider:str# e.g. "anthropic", "openai" model:str# e.g. "claude-opus-4-7" input_price_per_token:float# USD per input token output_price_per_token:float# USD per output token PRICING_MAP:dict[tuple[str,str], PricingEntry]=... 4.2 Token estimation ThespeculativeinputîisknownbyconstructionofD1,soinput_tokens = tokenize(prompt(î)).length is computable exactly. Output-token estimation is harder because LLM output length is non- deterministic. Three acceptable approaches: •EMA over historical output lengthsfor the same(agent, input-shape)pair with decay constantα_EMA = 0.2(default). •Fixed-ceiling policywithmax_tokens = estimated + 2σ. •Deployment overridewhere operators supply a conservative point estimate per agent. The method is robust to moderate estimation error (Section 10.2 sensitivity analysis), but the decision rule’s sensitivity toC_specgrows asPdecreases, so low-P/ high-variance agents should be taggedun- certain_costand excluded from speculation until history stabilizes. 4.3 Other cost models C_specis intentionally pluggable. For self-hosted open-source models,C_specmay be computed via GPU-hour amortization: C_spec = (unit_price · num_gpus · output_tokens) / (throughput · utilization) . This reduces to a linear-per-token form, so the decision rule is unchanged. Thedistinctivechoice is the two-rate (input≠output) form at API-billing granularity; single-rate GPU-hour forms do not fully exploit the billing asymmetry. 5. Dimension 3: User-facingαpreference dial 5.1 Semantics αis a dimensionless scalar on[0,1]exposed to the operator (or end user) that expresses the relative preference between minimizing latency and minimizing cost. •α = 0: fully cost-sensitive. Speculate only whenEV≥C_spec(full-cost break-even). 6 •α = 1: fully latency-sensitive. Speculate wheneverEV≥0. •α = 0.5: balanced. αdrives the decision threshold viathreshold = (1−α) · C_spec(Section 6). 5.2 Runtime mutability αmay be changed at any time during workflow execution. The runtime recomputes the threshold for every speculation decision that has not yet fired. This lets an operator tighten cost controls mid- execution (e.g., when an SLO budget is being consumed faster than expected) or relax them (e.g., when an interactive user indicates impatience). 5.3λ(time-to-dollars) specification The latency value ratioλ(USD/second) is a separate input fromα.λconverts latency into monetary value so both sides of the decision rule live in the same units. Four standard derivations: SourceFormulaExample User value-of-timeoperator sets directly“1 minute saved = $1”→λ = $0.0167/s Labor costλ = hourly_wage / 3600$100/hr analyst→λ = $0.0278/s Workflow valueλ = value / expected_duration $10 per 100-s workflow→λ = $0.10/s Budget-deadlineλ = (B − C₀) / (T₀ − T)derived from willingness to spendBto hit deadlineT αis a dimensionless preference;λis a units-bearing conversion. Keeping them separate is deliberate: αcan change frequently as a UX knob;λis a deployment-level setting that rarely changes. Conflating them into one parameter makes cost-preference updates accidentally re-calibrate the time-to-dollars conversion. 6. Dimension 4: EV decision rule with failure-weighted cost andα-threshold 6.1 The formula Latency value:L_value = L · λ Speculation cost: C_spec = input_tokens · input_price + output_tokens · output_price Expected value:EV= P · L_value − (1 − P) · C_spec Threshold:threshold = (1 − α) · C_spec Decision:speculate iff EV≥threshold . On a tie (EV == threshold), the default is to speculate: speculation has potential upside, waiting has none. 7 6.2 Failure-weighted cost: why(1−P) Several published systems charge speculation cost unconditionally (B-PASTE’sμ · ΔI, Speculative Actions v2’sc · min Theorem 4). Our form charges cost only on failure: •Success (probabilityP).Speculative result is reused; the operation would have been paid either way; the incremental cost attributable tospeculation(as opposed to the work that would have happened anyway) is zero. This holds under the elastic-capacity assumption of Section 4, where a speculative call consumes only its own tokens; under a fixed serving budget there is also an opportunity cost on success, treated in Section 14.2. •Failure (probability1−P).The speculative cost is wasted because the operation must be re- executed with the correct input. Incremental cost= C_spec. Weighted expectation:EV = P · (+L_value) + (1−P) · (−C_spec) = P · L_value − (1−P) · C_spec. AsP→1, unconditional-cost formulations diverge materially from the failure-weighted form. Failure- weighting is the principled form under pay-per-use API billing, where the cost ofre-execution, not the cost of running once, is the relevant waste. 6.3 Why the threshold is(1−α) · C_spec Settingthreshold = (1−α) · C_specgivesαa clean interpretation: •α = 0→threshold = C_spec: speculate only if expected value exceeds the full cost of specula- tion (i.e., the expected gain covers the full worst-case waste). •α = 1→threshold = 0: speculate whenever expected value is non-negative. •Intermediate: linear interpolation. The form also scales with the cost magnitude (C_spec), so cheap speculations have a low bar and expen- sive speculations have a proportionally higher bar. This is the behavior deployments want: the sameα should produce more-aggressive speculation on cheap operations and more-conservative speculation on expensive ones. 8 6.4 Decision-rule flow yesno Speculation candidate u → v Estimate: P, input_tokens, output_tokens, L, input_price, output_price Read α, λ C_spec ← input_tokens · input_price + output_tokens · output_price L_value ← L · λ EV ← P · L_value − (1−P) · C_spec threshold ← (1−α) · C_spec EV ≥ threshold ? SPECULATEWAIT 9 Figure 2. Decision-rule flow. The same formula runs at plan time and at runtime (Section 8); only the input parameters differ. 6.5 Decision-rule pseudocode defspeculation_decision( P:float,# posterior-mean success probability alpha:float,# in [0, 1] lambda_dollars_per_sec:float,# deployment-configured input_tokens:int, output_tokens:int, input_price:float, output_price:float, latency_seconds:float, )->str: C_spec=input_tokens*input_price+output_tokens*output_price L_value=latency_seconds*lambda_dollars_per_sec EV=P*L_value-(1-P)*C_spec threshold=(1-alpha)*C_spec return"SPECULATE"ifEV>=thresholdelse"WAIT" 6.6 Multi-provider routing underα An orthogonal consequence: an operation may be routed to different provider/model tiers based onα. Cost-sensitive preferences (α→0) favor cheaper models; latency-sensitive preferences (α→1) favor faster models. Routing is implemented by evaluating the decision rule independently per(operation, provider, model)candidate and selecting the best perα. This sits at the boundary of D2 (pricing) and D3 (α) and is a dependent feature of the integrated method. 7. Dimension 5: BayesianPwith structural priors Appendix A covers the full posterior-update mechanics and extended taxonomy entries. 7.1 The cold-start and small-sample problem Pis the probability that a speculation is useful; it is the central quantity in the EV decision rule. Esti- mating it has three failure modes: 1.Cold start.When a(u, v)pair has never been speculatively executed, there is no history to estimate from. 2.Small-sample regime. Early in the history, a point estimate p̂ = successes / trials is high variance. A single failure in the first two trials givesp̂ = 0.5, which may or may not reflect the true rate. 3.Distribution shift.Agent behavior changes as prompts are edited, models are updated, or up- stream operations change their output distribution. Old history should be weighted but not dom- inant. A Bayesian approach with an informative structural prior and conjugate updating handles all three. 10 7.2 Dependency-type taxonomy The prior onPis selected from a small taxonomy ofdependency types, each capturing a qualitative structural relationship between upstream output and downstream usability. Dependency typePrior onPRationale always_produces_output0.9Upstream always emits; downstream robust to minor variation. list_output_variable_length0.7Upstream emits a list; first-item speculation usually useful. conditional_output0.5Upstream output highly variable; maximum-entropy baseline. router_k_way1/kUpstream selects 1 ofkdownstream paths. rare_event_trigger0.1–0.2Downstream fires only on a rare upstream signal. Thepriorvaluesforrouter_k_wayarederived(1/k,notempirical); thevaluesforrare_event_trigger are a narrow range to be pinned per deployment. 7.3 Beta-Binomial posterior n₀ = 2# prior strength (effective sample size) α₀ = n₀ · p_structural# Beta shape 1 β₀ = n₀ · (1 − p_structural)# Beta shape 2 After s successes and f failures on a given (u, v) pair: P | data ~ Beta(α₀ + s, β₀ + f) E[P | data] = (α₀ + s) / (α₀ + β₀ + s + f) . Prior mean equalsp_structuralby construction. Withn₀ = 2, after roughly 10 observations the posterior mean is ~82% data-weighted and ~18% prior-weighted. Appendix A.2 motivatesn₀ = 2 against larger alternatives. 7.4 “Speculation useful”: three-tier success criterion A speculation with predicted inputîand actual inputiis labelled successful if any of: •Tier 1, Exact match:i == î. •Tier 2, Semantic equivalence:equiv(i, î) == Trueper a domain predicate. Default: normalized-embedding cosine similarity≥0.95 for text; AST equality modulo formatting for code;semantic_json_equalfor structured outputs. •Tier 3, Downstream-output validation(opt-in, offline): the downstream output computed fromîis accepted giveni. DefaultpolicyisTier1+Tier2withtheembedding-similaritythreshold. Tier3isopt-inperdependency because it requires running the actual downstream and comparing results post-hoc, which defeats the latency benefit on that specific trial (but is fine for offline calibration). 11 7.5 Credible-interval gating The EV decision rule (Section 6) uses a point estimateP. With a posterior distribution available, the rule can be tightened to the(1−γ)one-sided lower credible bound: P_lower = Beta⁻¹(γ; α₀ + s, β₀ + f)# e.g., γ = 0.1 speculate iff P_lower · L_value − (1 − P_lower) · C_spec≥(1 − α) · C_spec . This reduces cold-start over-eagerness: two speculations with identical posterior means can have very differentlowerbounds(e.g., 0.80after100trialsvs.0.33after2trials), andonlytheconfidentoneshould fire. 7.6 Self-limiting behavior under branching factork Anaturalquestionaboutanysingle-shotspeculationschemeishowitbehavesastheupstream’sbranch- ingfactorkgrows. Thesingle-shotrulecommitsonepredictedinputî(themodeoftheupstreamoutput distribution)andpaysawastedC_specwhenevertheactualifallsoutsidethetier-1/2equivalenceclass ofî. Intuition suggests the rule must break down for largek; the question isat what kandwhether the rule self-limits or silently degrades. Claim.The D4 decision rule is self-limiting: askgrows under a uniform upstream distribution,Pfalls, EVcollapses below the(1−α) · C_specthreshold, and the rule correctly produces WAIT without any additional machinery. Critical-k derivation.SubstitutingP = 1/k(uniform-mode prior, no learned skew) into the D4 rule gives EV= (1/k) · L_value − (1 − 1/k) · C_spec threshold = (1 − α) · C_spec SolvingEV≥thresholdfork: (1/k) · L_value − (1 − 1/k) · C_spec≥(1 − α) · C_spec (1/k) · (L_value + C_spec)≥(2 − α) · C_spec k≤(L_value + C_spec) / ((2 − α) · C_spec) . Call thisk_crit(α). For anyk > k_crit(α), the uniform-case rule WAITs. Critically, this happens beforeEVgoes negative; the(1−α)·C_specthreshold gates off low-but-positive-EV speculations in pro- portion to cost-sensitivity. Numerical table at AutoReply parameters.WithL_value = $0.064andC_spec = $0.0135: k(uniform)P = 1/kEVα = 0α = 0.5α = 1 20.500 +$0.0253 SPECULATE SPECULATE SPECULATE 30.333 +$0.0123 WAITSPECULATE SPECULATE 50.200 +$0.0020 WAITWAITSPECULATE 100.100 −$0.0058 WAITWAITWAIT 200.050 −$0.0096 WAITWAITWAIT k_crit(α=1)≈5.7,k_crit(α=0.5)≈3.8,k_crit(α=0)≈2.9at these parameters. 12 Effective k under skewed distributions.Raw branch count overstates the adverse case. For a skewed distribution with a dominant mode of probabilityp_mode, the relevant quantity is theeffec- tivebranching factork_eff = 1 / p_mode, which may be much smaller thankitself. For example, a 5-way classifier whose output is 62% “billing” has k_eff ≈ 1.6 , and the EV calculation uses P = 0.62 , notP = 0.2: EV = 0.62 · 0.064 − 0.38 · 0.0135 = +$0.0346→SPECULATE at all α . D5 bridges the two regimes.The Bayesian posterior (Section 7.3) begins at the structural priorp = 1/k(therouter_k_wayrow of the taxonomy, Section 7.2) and converges to the empirical mode rate as trials accumulate. Cold-start behavior is governed by Case A (uniform); production steady-state behavior is governed by Case B (learned). The same mechanism handles both without case-by-case logic. Remedies whenk_effis genuinely large.When the upstream output distribution is both high-k and flat, single-shot speculation is not the right abstraction and the EV rule correctly refuses to fire. Three escape hatches are compatible with the rest of the method: 1.Richer conditioning onî.Condition the predicted input on side-features (e.g., subject-line pattern, time-of-day, tenantclass)thatcollapsek_efflocally. Therouter_k_waytaxonomyentry of Section 7.2 admits per-context variants; a single dependency can host multiple posterior cells. 2.Top-mmulti-shot speculation.Launchm > 1speculations covering the top-mmodes, payingm · C_specon all failures but hitting on the union of the top-mbranches with probabilityΣ_i≤m p_i. This is a different decision regime (combinatorial overm); B-PASTE’s beam admission [Song, 2026, §3] and Speculative Actions’ integer breadth [Ye et al., 2025, Thm. 4] both live in this space. The single-shot EV rule is a degenerate case atm = 1. 3.Don’t speculate.The rule already says WAIT. No additional guardrail needed. Theself-limitingpropertymeansthatdeployingthemethodonaworkloadwithunknownk_effissafe: under-estimation ofPleaves money on the table (missed latency wins), but it does not cause run-away waste. 8. Two-phase decision model with bidirectional override 8.1 Phase 1: planning Before execution, the runtime enumerates candidate parallelization plans and, for each plan, makes a SPECULATE/WAIT decision per candidate downstream using the Section 6 rule. The planner’s objec- tive (combined with D2/D3/D4) is: minimize α · ( Latency(plan) · λ ) + (1 − α) · MonetaryCost(plan) subject to: MonetaryCost(plan)≤max_budget(if specified) Latency(plan)≤max_latency(if specified) |wave|≤max_concurrency MonetaryCost(plan) = Σ_v cost(v) + Σ_spec v (1 − P_v) · cost_actual(v) base costexpected speculation waste Latency(plan)= Σ_waves max_v ∈ wave latency(v) . 13 Candidate plans are generated over discrete concurrency settings (sequential / maximally parallel / intermediate levels). For small DAGs (5–20 operations) enumeration is tractable; for larger DAGs, list- scheduling, ILP, or constraint programming substitute without changing the rest of the method. Phase 1 outputs:(plan, per-candidate SPECULATE/WAIT decisions, expected latency, ex- pected cost). 8.2 Phase 2: runtime re-evaluation Immediately before launching any operation marked SPECULATE (and also before operations marked WAIT that are about to become ready), the runtime re-runs the Section 6 decision rule withcurrent parameters: •Posterior-updatedP(D5 / Section 7.3), which may be higher or lower than at planning time. •Updated latency estimates from recent executions (EMA). •Possibly-changedα(D3 / Section 5.2). •C_specrecomputed with current token estimates. The runtime decision can differ from the planning decision ineither direction: •Planning SPECULATE→Runtime WAIT (downgrade; e.g.,Phas dropped after recent failures). •Planning WAIT→Runtime SPECULATE (upgrade; e.g.,αwas raised to favor latency). Phase 1: Planning Phase 2: Runtime upgrade WAIT → SPECULATE downgrade SPECULATE → WAIT no change Event: operation becomes ready Re-read current P, α, C_spec, L Re-run Section 6 rule Override plan tag? Launch speculativeWait for upstreamExecute as planned Enumerate candidate plans Apply Section 6 rule per candidate edge Select plan minimizing α · L · λ + (1−α) · Cost Output: plan + per-edge SPECULATE/WAIT tags Figure 3. Two-phase decision model. Phase 1 commits a plan and cost/latency estimate. Phase 2 can override in either direction when current parameters differ from plan-time parameters. 8.3 Why two phases with bidirectional override Three reasons: 1.Visibility.Phase 1 produces a cost/latency estimate the user can see before execution starts. 2.Adaptivity.Phase 2 handles distribution shift, estimate drift, andαchanges. 3.Consistency.Same formula in both phases; only the parameters differ. 14 Bidirectional override matters. One-way “can only cancel” is not enough: a plan-time WAIT may be wrong if the upstream agent has had a run of successes since planning and itsPhas risen above the threshold. Allowing upgrades preserves that option. 9. Streaming re-estimation and mid-stream cancellation with waste refinement 9.1 StreamingPre-estimation Many modern LLM APIs stream output tokens as they are generated. If the upstreamustreams, the runtime can re-estimate the predicted inputî(and thereforeP) as tokens arrive, without waiting foru to finish. At each streamed chunk k of u's output: î_k←predict_input(partial_output_k) P_k←P( î_k matches eventual i | u-partial-so-far ) re-run Section 6 decision rule with P_k . This gives the speculation decision the benefit of upstream partial evidence, at the cost of repeated decision-rule evaluation. The decision rule itself is a handful of multiplies and a comparison (Section 6.5) and is negligible. The non-trivial per-chunk cost is the input re-predictionpre- dict_input(partial_output_k)and, at commit time, the tier-2 equivalence check (Section 7.4), a normalized-embedding cosine similarity that runs on the critical path. Both must be cheap relative to the latency being reclaimed for streaming re-estimation to pay for itself: re-estimation should be throttled (e.g., everyNchunks or on sentence boundaries, not every token) and the tier-2 embedding model should be small. Section 14.2 (runtime overhead of the speculation machinery) lists overhead accounting as an open limitation; a deployment whose tier-2 check orî-predictor is itself an expensive LLM call may find the machinery costs more latency than it saves, which the calibration pipeline’s offline replay (Section 12.1) will surface as a negative net-latency result before any traffic is exposed. 9.2 Mid-stream cancellation IfatchunkktheupdatedP_kfallsbelowthespeculationthreshold, theruntimecancelsthespeculative downstream mid-execution. Cancellation matters for billing: the downstream has consumed some fraction of its tokens already. 9.3 Waste refinement below fullC_spec NaiveaccountingchargesthefullC_speconcancellation. Thisover-estimateswaste,becausethedown- stream may have completed only a fractionf ∈ [0,1]of its planned generation before cancellation. Waste refinement tracks the actual token spend: C_spec_actual(t) = input_tokens · input_price# paid once at launch + tokens_generated_so_far(t) · output_price . On cancellation at time t: C_waste = C_spec_actual(t) # not full C_spec . Updating Phase 1’sExpected_Speculation_Wasteto reflect this reduces the planner’s pessimism about speculation: Expected_Speculation_Waste_v = (1 − P_v) · E[ C_spec_actual(v) ] 15 = (1 − P_v) · ( C_input + ρ_v · C_output ) , where ρ_v ∈ [0,1] is the expected fraction of output generated before cancellation (estimated from streaming-history EMA; default ρ = 0.5 in the absence of history) . yes no yes no u begins streamingChunk 1 arrivesRe-estimate î_1, P_1P_1 ≥ threshold? Keep speculating v Cancel v; charge C_spec_actual(t) only Chunk k arrivesRe-estimate î_k, P_kP_k ≥ threshold? Continue Cancel mid-stream; waste = ρ · C_output Figure4. Streamingre-estimationandmid-streamcancellation. Oneachnewupstreamchunk,îisrefined, Pis re-estimated, and the decision rule re-evaluated. On cancellation, only the actually-emitted output tokens are billed, not the full planned budget. The combination of(streaming re-estimation + mid-stream cancellation + fractional waste refinement) is one of the method’s differentiators against the four audited systems (Section 11). 10. Worked examples 10.1 Single-decision example Setup. •Upstream: document-analyzer (producing a list of topics). •Downstream: topic-researcher. •Dependency type:list_output_variable_length, priorp = 0.7, 3 successes + 1 failure ob- served→posterior mean= 4.4 / 6.0≈0.733. •Pricing (two-rate example at typical frontier-API prices): input $3/M tokens =0.000003 USD/- token, output $15/M tokens =0.000015 USD/token. •Token estimates: input 500, output 1000. •Estimated latency savings: 5 seconds. •λ = $0.01/s;α = 0.5. Compute. C_spec = 500 · 0.000003 + 1000 · 0.000015 = 0.0015 + 0.015 = $0.0165 L_value = 5 · 0.01= $0.05 EV = 0.733 · 0.05 − 0.267 · 0.0165 = 0.03665 − 0.00440 = $0.03225 threshold = (1 − 0.5) · 0.0165= $0.00825 Decision: EV ($0.03225)≥threshold ($0.00825)→SPECULATE . Sensitivity toα(atP = 0.733): αthreshold decision 0 $0.01650 SPECULATE (EV > threshold) 0.2 $0.01320 SPECULATE 0.5 $0.00825 SPECULATE 0.8 $0.00330 SPECULATE 16 αthreshold decision 1.0 $0SPECULATE Allαvalues speculate because EV comfortably exceeds the full cost. Now lowerPto 0.4: αEVthreshold decision 00.4·0.05 − 0.6·0.0165 = 0.01010.01650 WAIT 0.2 0.01010.01320 WAIT 0.5 0.01010.00825 SPECULATE 0.8 0.01010.00330 SPECULATE 1.0 0.01010SPECULATE AtP = 0.4the decision flips atα≈0.4. This is the behavior theα-threshold is designed to produce: cost-sensitive operators wait, latency-sensitive operators speculate, and the transition point scales with C_spec. 10.2 Two-phase with bidirectional override Phase 1 (planning). •Same agents as Section 10.1. •Pat planning time: 0.733 (from history). •Decision: SPECULATE. •Plan commits to speculative launch. Between planning and runtime. •Two more trials complete: both failures. •Posterior updates:α = 4.4, β = 1.6 + 2 = 3.6, mean= 4.4/8.0 = 0.55. Phase 2 (runtime). •Re-evaluate:EV = 0.55 · 0.05 − 0.45 · 0.0165 = 0.0275 − 0.0074 = $0.0201. •Threshold atα = 0.5: $0.00825. •Still SPECULATE (margin narrowed from $0.0240 to $0.0119). Alternative: operator raisedαto 0.9 between phases. •Threshold: $0.00165. Still SPECULATE (more aggressive, appropriate for latency-sensitive pref- erence). Alternative: operator loweredαto 0.1 between phases. •Threshold: $0.01485. WAIT. •Plan said SPECULATE; runtime says WAIT. Bidirectional downgrade in action. 17 10.3 Streaming cancellation Setup.Topic-researcher speculating on an assumed topicî. Output-generation progress is streamed. Partway through generation (300 tokens out of estimated 1000 produced), the upstream completes and produces actual topic i . Tier-1 and Tier-2 checks on i vs. î both fail. Waste accounting. C_spec_planned = 500 · 0.000003 + 1000 · 0.000015 = $0.0165 C_spec_actual = 500 · 0.000003 + 300 · 0.000015 = 0.0015 + 0.0045 = $0.0060 Waste refinement saves: $0.0165 − $0.0060 = $0.0105 (64% reduction) . Posterior update.One failure.α = 4.4, β = 1.6 + 1 = 2.6, mean= 0.629. Downward revision is appropriate: failures under streaming cancellation are still real failures forP-estimation purposes, even though the waste was less thanC_spec. 11. Contrast with prior art and limitations 11.1 Five-dimension contrast The four closest audited systems as of May 2026 are: •DSP:Dynamic Speculative Agent Planning [Guan et al., 2025]. •SA:Speculative Actions [Ye et al., 2025]. •SH:Sherlock [Ro et al., 2025]. •BP:B-PASTE [Song, 2026]. Each cell below is anchored to a specific section or equation in the cited paper; the compact tags above (DSP, SA, SH, BP) key the table columns and the per-cell anchors that follow. DimDSP Spec Actions v2SherlockB-PASTEThis method D1Pre- completion; linear planning chain ofksteps [DSP §3, Eq. 1] Pre-response; linear MDP,k future actions [SA Alg. 1, lines 10–15] Post-output, pre-verify; structural, not content- predictive [SH §7 Fig. 9] Reasoning- time, pattern-guided tool subgraphs [BP §3 Eq. 1; Alg. 1] Pre- completion, arbitrary DAG, LLM-op targets D2Dollars in post-hoc eval only; loss uses token counts [DSP §5; Eqs. 6–9]¹ Abstract scalar c(cost-per- unit-time proxy) [SA §5; Thm. 5] Single-rate GPU-hour (unit_price · num_gpus · num_tokens) / throughput [SH App. A.2 Eq. 9] Multi-resource compute profileρ_i; no dollars [BP Eqs. 1, 4] Two-rate per-token USD (input≠ output) in the decision rule 18 DimDSP Spec Actions v2SherlockB-PASTEThis method D3τ ∈ (0,1)+β (asymmetric- loss params); 3 named modes [DSP §4.3, §5.3] (r, c)offline hyperparame- ters; integerk [SA §5.2] λ on the wrong axis (accuracy-vs- cost, not latency-vs-cost) +B(USD cap, not normalized) [SH §6.1 Eq. 1; §7.1 Eq. 5] Multi-resource budgetB; internalλ,μhy- perparameters [BP Eqs. 3, 6] α ∈ [0,1] user-facing, runtime- mutable;λ (USD/s) separated D4TD(λ) value regression with asymmetric loss [DSP Eq. 1, §4.1]; noP, no cost term in loss arg max_m q(m; p) · Δ − c · m; greedyΔ · δq ≥c[SA Thm. 4]; costuncon- ditional Hard feasibility: N_spec = j : Σ lat_exec < lat_vrf ∧C_spec≤ B[SH §7.1 Eqs. 4–5]; Eq. 6 (1−m) · ΣCis closest structural waste mirror but used for budget check, not decision gate EU(H_i) = q_i · (ΔO + λ · ΔU − μ · ΔI)[BP Eq. 3], closest structural EV form;μ · ΔI unconditional, not(1−P)- weighted; combinatorial over a beam P · L_value − (1−P) · C_spec≥ (1−α) · C_spec; binary per-candidate gate D5DistilBERT regressor, random init, predictsk(not P) [DSP §5, §6.1]; paper explicitly rejects a BO baseline Model logits or auxiliary classifier; constant 0.5 cutoff[SA §5.2] Empirical match ratem_i × node-position / fan-in policy [SH §3.3 Fig. 5; §5.4 Alg. 1], not dependency type q_ifrom offline PrefixSpan frequency counts [BP §3]; no runtime Bayesian update Beta-Binomial posterior, structural prior keyed to dependency- type taxonomy Notes. 1.DSP abstract vs. mechanism.DSP’s abstract claims the system“explicitly optimizes a joint ob- jective balancing end-to-end latency against dollar cost.”The TD(λ) training loss [DSP Eq. 1] uses token counts, not billing rates, and dollars appear only in post-hoc evaluation metrics [DSP §5, Eqs. 6–9]. The abstract is framing; the mechanism does not depend on dollar units. 2.Sherlock D2 proximity.The GPU-hour form is linear per-token and could be substituted into a 19 decisionrule. ThesurvivingD2differentiatoristwo-rate(input≠output)API-billinggranularity: single-rate reductions miss the 3–8× input/output rate asymmetry (Section 4.1). 3.Sherlock D3 axis.Sherlock’sλtrades verifier accuracy against cost, not latency against cost. A cost-vs-latencydialsteersspeculationaggressiveness; acost-vs-accuracydialsteersmodel/verifier tier selection. They are different axes. 4.B-PASTE D4 credit.EU(H_i)is the closest structural EV form in the audited corpus. Three differences from this method: (a)μ · ΔIis unconditional interference, not failure-weighted waste; (b) the optimization is combinatorial over an admission beam, not a binary gate; (c) it is time-denominated, not dollar-denominated. No paper matches this method on any single dimension, and no subset of the four matches on the conjunction(D2 two-rate∧D4 failure-weighted +(1−α) · Cthreshold∧D5 dependency-type prior). 11.2 Auxiliary-dimension contrast Thefourneighborsalsodifferfromthismethodonimplementationchoicesbeyondthefiveprincipaldi- mensions. DSP and Speculative Actions v2 schedule speculation by tuningk-step or breadth-mintegers; Sherlock places per-node binary speculations under a topology-driven FSM; B-PASTE runs preemption- first greedy admission over a subgraph beam. This method runs a per-candidate binary EV gate inside a two-phase plan-plus-runtime model with bidirectional override (Section 8). On learning: DSP is online from random init [Guan et al., 2025, §6.1], Speculative Actions is offline only [Ye et al., 2025, §5.1], Sher- lock and B-PASTE are hybrid; this method is online per-(u, v)from a structural cold-start prior. The most informative axis is cancellation behavior: DSP cancels on upstream-target mismatch [Guan et al., 2025, Fig. 1] but does not combine cancellation with streaming re-estimation or fractional-completion accounting; Speculative Actions discards on cache miss (correctness, not waste); Sherlock rolls back at node granularity after the verifier runs; B-PASTE preempts on resource contention, not input mis- match. The integrated triple (streamingPre-estimation, mid-stream cancel on input mismatch, and fractional-waste refinement) appears in no audited system. 11.3 Adjacent contributions to credit Two ideas from B-PASTE are close enough to credit explicitly: •Downstream-unlock-value decomposition(ΔO + λ · ΔU). TheL_value = L · λterm in this paper is end-to-end latency savings and implicitly absorbs unlock value for downstream operations on the critical path. For workflows where unlock effects propagate multiple hops, the explicit decomposition is the right refinement and the decomposed form is a strict generalization of what this paper specifies. •Interference termμ · ΔI. In edge or shared-compute regimes, co-run interference is a real cost that(1−P) · C_specdoes not capture (the latter assumes pay-per-call API billing). The unified formEV = P · L − (1−P) · C − μ · ΔIcovers both regimes; this paper specifies the API-billing case. Two position papers frame, at the level of general principle, the design stance this method makes oper- ational: •Bayes-consistent agentic orchestration[Papamarkou et al., 2026]. Argues that the control layer of an agentic system should be Bayes-consistent, maintaining beliefs over task-relevant la- tent quantities and updating them from observed interactions. The position is generic: it specifies 20 no posterior, no taxonomy, and no speculation mechanism. The method here is one concrete in- stance of that stance, with a Beta-Binomial posterior keyed to the Section 7.2 dependency-type taxonomy and a closed-form EV gate (Section 6). •Marginal token allocators[Zhu, 2026]. Argues that agentic systems should be designed as marginal token-allocation economies solvingmarginal_benefit = marginal_cost + la- tency_cost + risk_cost. Withαheld fixed, this is the same first-order condition as the EV ruleEV = P · L · λ − (1−α) · C_spec − (1−P) · C_spec. That paper is descriptive (no algo- rithm, no calibration); this method is the operational instance, withα-controlled risk weighting, a Bayesian estimate ofP, and the calibration pipeline of Section 12. 11.4 Explicit differentiators Beyond the per-dimension contrast of Section 11.1, five combinations are individually uncontested across DSP, Speculative Actions v2, Sherlock, and B-PASTE: (1) a failure-weighted(1−P) · Crule paired with two-rate per-token billing (Sections 4.1, 6); (2) a runtime-mutableαdial that drives the (1−α) · Cthreshold rather than the cost (Section 5); (3) a Beta-Binomial posterior keyed to the dependency-type taxonomy (Section 7.2); (4) two-phase plan-plus-runtime control with bidirectional override (Section 8); and (5) the integrated triple of streamingPre-estimation, mid-stream cancellation, and fractional-waste refinement (Section 9). No neighbor discloses any of the five, and none discloses their conjunction. 12. Calibration and evaluation pipeline Sections 3–11 specify the decision mechanism; this section specifies how to tune its free parameters safely and how to decide, per edge, whether speculation should be enabled at all. Each of the method’s tunable knobs (the dependency-type tag per edge, the priorp_structural, the posterior update rate n₀ , the user-facing preference α , the deployment constant λ , the tier-2 equivalence threshold, token estimators, and the per-edge enable/disable bit) is set or kept honest by one of the five lifecycle stages below. The pipeline is staged in order of increasing exposure: offline replay touches no production traffic, shadow mode serves a decision but discards it, canary serves a decision to a fraction of traffic, online calibration runs forever in steady state, and drift detection closes the loop by flipping the enable bit when any earlier assumption breaks. 12.1 Offline replay on sequential logs A strictly-sequential deployment of the workflow produces logs of(upstream_input, up- stream_output, downstream_input, downstream_output, latency, cost)tuples. Every calibration step below can be bootstrapped from these logs before any speculation is enabled. What offline replay computes. •Effective branching factor.Fit the empirical upstream-output distribution per(agent, ten- ant). Recordp_modeandk_eff = 1 / p_mode(Section 7.6). •Dependency-type auto-assignment.Assign a type from the Section 7.2 taxonomy by rule:p_mode≥0.8→always_produces_output; upstream emits a list→ list_output_variable_length;k≤5with flat distribution→router_k_way;p_mode ≤0.2→rare_event_trigger; otherwiseconditional_output. This replaces hand- authored tags for edges with sufficient history. 21 •Data-seeded prior.For each candidate predictor ofî(modal, regex-extracted, historical- majority, small auxiliary model), compute the empirical tier-1 and tier-2 match rate over the logs. Use(s, f)from the log as(s_0, f_0)to start the posterior, so the edge opens production with P already close to truth rather than at the structural prior. •Counterfactual EV grid.Replay the D4 rule over the logs at a grid of(α, λ)values and report for each grid point: expected latency, expected cost, expected waste, and the realized decisions per edge. This is thego/no-go per edgebefore a single dollar of speculative waste is spent. Whatofflinereplaytunes.Dependency-typetag; initialPprior; per-edgeenable/disable; deployment- time defaultα. 12.2 Shadow mode Launchthespeculativedownstreamalongsidethesequentialexecution,butcommitonlythesequential result. Log the full per-decision row (Appendix C) for every trial. Whyitmatters.Shadowmodeistheonlystagewheretheruntimegetsposteriordata, cost-actualdata, tier-2 outcomes, and fractional-completion evidence without exposing users to any speculative output. The posterior moves from the cold-start prior to a data-dominated regime in the same time it would take a live-rolled speculation to do so, but with zero downside if the tier-2 predicate is miscalibrated or if the token estimator is badly wrong. What shadow mode tunes. •Pposterior per edge (to convergence, per Appendix A). •Tier-2 predicate threshold (defaultembedding_similarity≥0.95from Section 7.4). Run a grid sweep on the shadow logs; select the threshold that maximizes F1 against a human-graded subset. •Token estimators (EMA per(agent, tenant); flaguncertain_costfor edges with CoV > threshold). •Fractional-completion rateρfor the planner’s waste term (Section 9.3). •Credible-boundγ(Section 7.5), if the deployment opts into credible-bound gating. Exit criterion: each edge accumulates at leastN_shadowtrials (default 100) with stable posterior mean over the last 50. 12.3 Canary rollout withαsweep and implied-λrecovery Percentage rollout (1%→5%→25%→100%) of live speculation with a held-out sequential control. The canary structure: ArmPrimary metricGuardrail metrics Control (sequential)latency, user-facing CSATcost/request, tier-2 false-accept Speculation atα = α_defaultΔlatency, ΔCSAT, Δcost vs. control posterior drift, waste $/hr αsweep0.1, 0.3, 0.5, 0.7, 0.9 pareto (latency, cost)same 22 Theαsweep traces the empirical(latency, cost)Pareto frontier per edge. The canary is success- ful when the rollout-rate arm matches or beats the sequential control on latency while staying within budget guardrails, and when theαsweep’s selected operating point Pareto-dominates sequential. Implied-λrecovery.A novel observation enabled by theα-sweep arm: if operators or automated con- trol loops consistently prefer anαthat the modeledλwould price out, the modeledλis under-valued. Concretely, at the chosen operating point α* , the D4 rule equates P · L · λ_implied − (1 − P) · C_spec = (1 − α*) · C_spec , giving a closed-form recoverableλ_impliedper edge as a function of observed(P, C_spec, α*)and knownL. Averaging over edges and comparing to the deployment’s declaredλyields an audit signal: •λ_implied>λ_declaredby a stable margin→operators value latency more than the deploy- ment’s pricing assumes. Refreshλ. •λ_implied≈λ_declared→the pricing and preferences are consistent. •λ_implied<λ_declared→the pricing over-values latency. Inspect whether the declaredλis based on stale CSAT elasticity or stale churn economics. This converts what is otherwise an opaque preference drift into a dollar-denominated, auditable quan- tity. What canary tunes.αoperating point per deployment context (peak vs. off-hours vs. incident);λ refresh via the implied-λaudit; credible-boundγ(final check); go/no-go to full rollout. 12.4 Online calibration in steady state Four continuous checks, ideally on a single dashboard: Posterior calibration curve.Bucket decisions by predictedPin width-0.1 buckets. Within each bucket, the empirical success rate (tier-1 or tier-2 match on the realizedi) should match the bucket midpoint within confidence intervals. Miscalibration diagnoses: •Monotonic over-prediction ofP→the priorp_structuralis too high or the posterior update has missed a regime shift. Re-run dependency-type auto-assignment (§12.1) or lowern₀. •Over-confident lower bound→credible-boundγtoo low; raise it. •Structural deviation for a specific edge→the dependency-type assignment was wrong; re-tag. Tier-2 sampling audit.Sample a fraction (default 1%) of committed speculations. Submit to an offline reviewer (human or LLM-judge) with bothîandiavailable; record whether the downstream output computed fromîwould have been acceptable underi(tier-3 check). False-accept rate above tolerance (e.g., 5%)→tighten the tier-2 threshold. Token-estimate monitoring.Track EMA and CoV ofoutput_tokens_actual / out- put_tokens_estper(agent, tenant). High CoV→flaguncertain_cost, disable speculation on that edge until CoV drops below threshold. λrefresh.Quarterly, re-run the CSAT-to-churn regression (or equivalent source-of-truth for latency valuation) on fresh data; cross-check against the implied-λaudit from §12.3; update the deployment constant. 23 12.5 Drift detection and kill-switch Automatedtriggersthatfliptheper-edgeorglobalenablebitwithouthuman-in-the-loopapproval. This is the mechanism by which non-stationarity (Section 14.3, non-stationarity response) is handled oper- ationally. TriggerScopeAction Posterior mean drops > 20% over a 100-trial window vs. the prior 500 EdgeAutomatically lowerα_edgeby 0.2 for the next hour. Credible lower boundP_lower < (1 − α) · C_spec / (L · λ + C_spec)forNconsecutive decisions EdgeDisable speculation on the edge. Require a fresh §12.2 shadow-mode run to re-enable. Tier-2 false-accept rate (§12.4) exceeds tolerance EdgeDisable speculation. Page on-call. Monthly cost SLO guardrail tripped GlobalSetα←0for all edges until next billing cycle. New model version deployed for any agent All edges using that modelFlip back to shadow mode for 24 hours; re-run §12.1 auto-assignment on the shadow logs. Token-estimate CoV > threshold EdgeDisable speculation on the edge until CoV drops. The per-edge enable bit is the method’s most consequential operational knob. Prior sections treat it as given; §12.1 sets it at deployment time, §12.5 flips it at runtime in response to evidence. 12.6 Knob-to-stage map KnobPrimary stageSecondary Dependency-type tag per edge §12.1 offline replay§12.5 drift-triggered re-tag p_structuralprior§12.1§12.2 shadow refinement PosteriorPper edge§12.2 shadow; §12.4 onlineper-decision log α(operating point)§12.3 canary sweeppost-incident review λ(time-to-dollars)§12.3 implied-λrecovery§12.4 quarterly refresh Tier-2 threshold§12.2§12.4 audit Token estimators§12.2; §12.4§12.5 drift Per-edge enable/disable§12.1 go/no-go§12.5 kill-switch Credible-boundγ§12.3§12.4 Without the full telemetry row of Appendix C, none of the five stages run. 24 13. Workload fit and archetype catalog The method targets a specific workload shape. Section 1.4 stated scope at the abstraction level (static DAG, pay-per-token, predictedîavailable); this section restates fit in workload terms suitable for pick- ing a pilot and for excluding workloads where single-shot speculation is the wrong tool. 13.1 Four-point fit rubric A workload is a good fit for the method when all four of the following hold: 1.Multi-stage workflow with upstream latency to reclaim.Two or more LLM or tool calls with a real upstream wait. Single-call workloads have no upstream to speculate against. 2.Small effective branching factor.Either rawkis low (≤5) or the real-world mix is strongly skewed (p_mode≥0.5, equivalentlyk_eff≤2). See Section 7.6; the method is self-limiting when this fails, but a pilot will surface few positive-EV decisions. 3.Output-heavy downstream.The speculative downstream generates enough tokens for the two- rate pricing (Section 4) to matter. For input-heavy, output-light operations, the blended-rate ap- proximation error is small and the decision is dominated by latency alone. 4.Defensibleλ.Someone in the organization can defend a USD/second-of-latency-saved figure, typically via CSAT elasticity, operator-time value, or an SLA-penalty derivation (Section 5.3). Ifλ cannot be defended, the method collapses to “always speculate” (α = 1) or “never speculate” (α = 0) under the operator’s implicit preference. A workload failing any one of these four is not disqualified, but the expected yield is low; §12.1 offline replay will show a counterfactual EV grid dominated by WAIT decisions. 13.2 Eight archetypes Production workloads that fit the rubric span four domains. For each archetype, the entry states the speculation point, the branching characteristic (rawkork_effunder skew), the business stakes, and the principal watch-out. Customer-facing, real-time. •Voice bot / IVR response generation.Shape:STT→intent classifier→response synthesizer →TTS.Speculate:responsesynthesizerwiththemodalintent’stemplatewhiletheclassifierruns. Branching:per-tenant call mix is Zipfian; top 3 intents cover 60–80% (k_eff≈1.5–2).Stakes: each additional 400 ms raises call abandonment; telcos pay per minute.Watch-out:tier-2 equiva- lence must accept paraphrases (invest in the semantic-match predicate). •IDE code autocomplete agent.Shape:context classifier (continue-line / complete-block / suggest-test / import-fix)→generator.Speculate:the generator with the modal intent while the classifier inspects surrounding code. Branching: ~70% of invocations are “finish this line” in most repos (k_eff≈1.4).Stakes:sub-200 ms feel is the product; aggregate GPU hours are real. Watch-out:cost-sensitivity is high (operators runαnear 1 and rely on streaming cancellation, §9). High-volume enterprise workflows. •Insurance claims triage.Shape:OCR + claim-type classifier→next-action drafter.Speculate: drafter for the modal next-action per claim type.Branching:3–4 claim archetypes cover most volumeperinsurer(k_eff≈2–3).Stakes:adjustertimeat$50–100/hr; 20%cycle-timereduction 25 scalestoseven-figureannualsavings.Watch-out:tier-3offlinevalidationismandatory(regulatory risk); credible-bound gating (Section 7.5) from day one. •Content-moderation triage.Shape:safety classifier→action drafter (allow / warn / remove / escalate).Speculate:the “allow” path with its user-facing message.Branching:extreme skew; “allow” wins atp_mode > 0.95(k_eff≈1.05).Stakes:platforms process billions of items/day; unit wins compound.Watch-out:the rare non-allow paths are where quality matters most; tier-2 must never be softened for them. •Medical prior-authorization drafting.Shape:document extraction→procedure-code classi- fier→policy retrieval→approval/denial drafter.Speculate:the retrieval + drafter path for the modal code.Branching:per payer-specialty, top 5–10 codes cover most volume (k_eff≈3– 5).Stakes:prior-auth backlogs delay hospital revenue; each day shaved is directly monetizable. Watch-out:cold-start on new payers is high-risk (wrong denial draft); credible-bound gating from day one plus shadow-mode runway per new payer. Developer-tooling workflows. •PR / code-review bot.Shape:diff analyzer→change-type classifier (feature / bugfix / refactor / config / docs)→review-strategy selector→reviewer prompt.Speculate:the reviewer prompt for themodalchangetypeperrepo.Branching:per-reposkewisstrong(frontendrepos~60%features, infra repos ~50% config), sok_eff≈2per repo even withk = 5raw.Stakes:reviewer wait time is engineering velocity; at organization scale this is a multi-million-dollar lever.Watch-out: cross-repo generalization is weak; rely on per-repo posteriors (the method does this by default). •RAG query-answering pipeline.Shape:intent classifier (factual / multi-hop / comparison / procedural)→retriever strategy→answer synthesizer.Speculate:the synthesizer with the most- likely intent’s retrieval path.Branching:factual lookup dominates at 60–70% for most products (k_eff≈1.5–2).Stakes:user-facing latency drives engagement; output-heavy synthesis is the expensive stage.Watch-out:the retriever is itself a tool call and may be slow; consider speculation at the retriever level separately. High-stakes, low-volume. •Security alert / incident triage.Shape:alert enricher→alert-type classifier→runbook se- lector→remediation-plan drafter.Speculate:remediation drafter for the most-likely runbook. Branching:top 5 alert types dominate most SOCs; time-of-incident tightens further (k_eff≈ 2–3).Stakes:MTTR has dollar value in breach exposure; incident-minutes of operator time are expensive.Watch-out:low volume per unique alert→posterior converges slowly; lean on the structural prior (router_k_way) longer than in a high-volume setting. 13.3 Where the method does not fit Four workload shapes where single-shot speculation is the wrong tool and no amount of tuning helps: •Open-ended creative generation(prompt→long-form essay, one call). The downstreamis the workflow; there is no upstream to speculate against. •Runtime-determined topology(reflection loops, dynamic spawning, recursive planning). Sec- tion 1.4 scopes these out. Each expansion requires re-planning, and the planner assumptions (Section 8.1) do not hold. •Highk_effwithflatdistribution.Single-shotspeculation’sEVcollapsesbelowthreshold(Sec- tion 7.6); remedies are richer conditioning on î , top- m multi-shot speculation, or declining to 26 speculate. •Cheap-downstream workloads.WhenC_specandL · λare both small, EV is small by con- struction and rarely clears the(1 − α) · C_specthreshold. The decision rule correctly says WAIT, but the effort of instrumentation has no payoff. 13.4 Pilot-picking rubric Score candidate workloads on: 1.Is there latency pain a user or operator actually feels? 2.Is there a single upstream-output mode hitting above 50% (k_eff≤2)? 3.Is the downstream output-heavy (so D2’s two-rate pricing moves the decision)? 4.CanPbe instrumented observationally (§12.1 offline replay, §12.2 shadow) before turning specu- lation on? Workloads scoring high on all four (voice-bot, claims triage, moderation, code-review bot among §13.2) are the best first pilots. Workloads that need streaming cancellation (§9) to be economical (IDE auto- complete, RAG) are second-tier. Workloads that need credible-bound gating from day one (prior-auth, security triage) require a longer shadow-mode runway. 14. Limitations and open problems The limitations below fall into two kinds: those addressed operationally by the calibration pipeline (Sec- tions 12–13) but lacking a closed-form solution, and those that are genuinely open with no mitigation yet. Each item states which. They are grouped into three categories — scope boundaries, cost-model completeness, and estimation and calibration — with the cost-model items placed first because they bear most directly on the central expected-value claim. 14.1 Scope boundaries •Static DAG assumption.The planner (Section 8.1) assumes a fixed topology. Dynamic work- flows (loops, reflection, dynamic spawning) would require treating each runtime expansion as its own planning sub-problem against the Section 8.1 rule; this extension is open and unaddressed. •Streaming availability.Section 9’s waste refinement requires APIs that stream output tokens andsupport mid-stream cancellation. Most major providers support both as of 2026; some tool- call APIs do not, in which case the method falls back to full- C_spec waste accounting (Section 6) with no loss of correctness, only of the streaming saving. 14.2 Cost-model completeness These two items bear directly on the EV accounting and are the most consequential limitations in this section. •Capacity contention and opportunity cost.The cost model (Section 4) assumes pay-per-call API billing with effectively elastic capacity: a speculative call’s only cost is its own tokens. Under a fixed serving budget (a rate-limit ceiling, a reserved GPU fleet, or a self-hosted endpoint at high utilization), this assumption fails. A speculation then consumes capacity that would otherwise serve live, non-speculative requests, so an aggressiveαcanraisetail latency for the rest of the workload even as it lowers latency for the speculated edge. The(1−P) · C_specterm prices 27 the wasted tokens but not this opportunity cost. The interference-augmented EV form credited to B-PASTE in Section 11.3 is the right hook for the contended-capacity regime; this paper speci- fies only the elastic-API case. A deployment near its capacity ceiling should treat speculation as drawing from a shared budget and gate it accordingly (the §12.5 global cost-SLO trigger is a coarse version of this); a principled per-decision opportunity-cost term is open. •Runtimeoverheadofthespeculationmachinery.Themethod’sownbookkeepingisnotfree, and a latency-reclaiming method must show its overhead is dominated by the latency it reclaims. The EV evaluation is negligible (Section 6.5), but three components are not: per-chunk input re-prediction under streaming (Section 9.1), the tier-2 embedding-similarity check on the critical pathatcommittime(Section7.4), andtheposteriorread/updateperdecision(Section7.3). When theî-predictor or the tier-2 check is itself a non-trivial model call, the machinery can cost more latency than it saves. The method mitigates this by throttling re-estimation and recommending small tier-2 models (Section 9.1), and the offline-replay stage (Section 12.1) flags a net-negative- latency edge before any traffic is exposed — but flagging is not measuring. A measured overhead characterization on a live deployment, and a closed-form overhead budget folded into the EV rule rather than caught empirically downstream, are both open; Appendix D is synthetic and does not measure machinery overhead. 14.3 Estimation and calibration •Token-estimation variance.Sensitivity toC_specerror rises asPdecreases. Agents with high output-length variance should be taggeduncertain_costper §12.2 and §12.5 and excluded until history stabilizes (operationally mitigated). •λelicitation and the two-parameter interface.The four methods in Section 5.3 cover most cases, but genuinely cost-insensitive settings (e.g., research exploration) are better served byα = 1than by a largeλ; §12.3’s implied-λrecovery is an audit, not an independent source of truth. Exposing bothα(preference) andλ(conversion) is also a UX burden: a single-dial interface is im- plementable by fixingλfrom deployment config and surfacing onlyα, but the underlying method requires both. •Prior-value provenance.The specific values 0.9 / 0.7 / 0.5 in Section 7.2 are design choices motivated by the semantic taxonomy. §12.1’s data-seeded prior replaces the taxonomy default for any edge with sufficient history; large-scale empirical calibration across public workflow corpora nonetheless remains open. •Non-stationarity response.The default prior strengthn₀ = 2favors responsiveness over stabil- ity. §12.5 drift triggers handle non-stationarity operationally (by flipping the enable bit or lower- ingα); a closed-form decayed-observation scheme that down-weights older trials (e.g., exponen- tial forgetting / a discounted Beta update) is a natural complement rather than a replacement, and is open. •Joint estimation.Two dependencies of the same type in the same workflow may share infor- mation; a hierarchical Bayesian model could pool evidence. Each(u, v)pair currently gets an independent belief (open). •Tier-2 threshold calibration.The defaultembedding_similarity≥0.95is a reasonable starting value. §12.2 and §12.4 sampled audits provide per-deployment tuning; cross-domain benchmarks are open. References •[Guan et al., 2025] Guan, Y., Lan, Q., Sun, F., Ding, D., Acharya, D., Wang, C., Wang, W. Y., Hua, 28 W.Dynamic Speculative Agent Planning(DSP). arXiv:2509.01920 [cs.AI], 2025. •[Leviathan et al., 2023] Leviathan, Y., Kalman, M., Matias, Y.Fast Inference from Transformers via Speculative Decoding.arXiv:2211.17192 [cs.LG], 2022. •[Papamarkou et al., 2026] Papamarkou, T., Alquier, P., Bauer, M., Buntine, W., Davison, A., Dzi- ugaite, G. K., Filippone, M., Foong, A. Y. K., Fortuin, V., Fouskakis, D., Frellsen, J., Hüllermeier, E., Karaletsos, T., Khan, M. E., Kotelevskii, N., Lahlou, S., Li, Y., Liu, F., Lyle, C., Möllenhoff, T., Palla, K., Panov, M., Sale, Y., Schweighofer, K., Shelmanov, A., Swaroop, S., Trapp, M., Waege- man, W., Wilson, A. G., Zaytsev, A.Position: Agentic AI Orchestration Should Be Bayes-Consistent. arXiv:2605.00742 [cs.AI], 2026. •[Ro et al., 2025] Ro, Y., Qiu, H., Goiri, Í., Fonseca, R., Bianchini, R., Akella, A., Wang, Z., Erez, M., Choukse, E.Sherlock: Reliable and Efficient Agentic Workflow Execution.arXiv:2511.00330 [cs.MA], 2025. •[Song, 2026] Song, Y.B-PASTE: Beam-Aware Pattern-Guided Speculative Execution for Resource- Constrained LLM Agents.arXiv:2604.16469 [cs.DC], 2026. •[Sui et al., 2026] Sui, Y., Zhao, H., Ma, R., He, Z., Wang, H., Li, J., Yang, Y.Act While Thinking: Accelerating LLM Agents via Pattern-Aware Speculative Tool Execution(PASTE). arXiv:2603.18897 [cs.AI], 2026. •[Ye et al., 2025] Ye, N., Ahuja, A., Liargkovas, G., Lu, Y., Kaffes, K., Peng, T.Speculative Actions: A Lossless Framework for Faster Agentic Systems.arXiv:2510.04371 [cs.AI], 2025. •[Zhu, 2026] Zhu, S.Agentic AI Systems Should Be Designed as Marginal Token Allocators. arXiv:2605.01214 [cs.AI], 2026. Appendix A: Bayesian posterior mechanics A.1 Conjugate pair Pis modeled as a random variable in[0,1]. The prior is Beta; each speculation outcome is a Bernoulli trial (success = “speculation useful” per Section 7.4); by conjugacy the posterior is Beta. Prior:P ~ Beta(α₀, β₀) Observation: for each trial i ∈ 1..n, X_i ~ Bernoulli(P) s = Σ X_i (successes), f = n − s (failures) Posterior: P | data ~ Beta(α₀ + s, β₀ + f) . A.2 Whyn₀ = 2and not larger or smaller A degenerate prior withα₀ = β₀ = 0(improper Beta) gives the maximum-likelihood estimate directly after one observation. Two reasons not to use it: 1.Withn₀ = 0, a single failure on trial 1 gives posterior mean 0, which collapses the decision rule to “never speculate again” for that dependency, since a single noisy outcome has disproportionate influence. 2.The structural prior carries real information (the expected average success rate for this depen- dency type). Discarding it forfeits the benefit of the taxonomy. At the other extreme,n₀ = 100gives a very stiff prior; it takes ~100 observations to move the posterior mean materially. Given the non-stationarity of LLM-agent workflows (model updates, prompt edits), responsivenesstorecentevidencemattersmorethanstability.n₀ = 2isthesmallestintegerthatretains the structural prior as a tie-breaker without overwhelming early observations. 29 A.3 Verification table Withn₀ = 2andα₀ + β₀ = 2in every row below: Dependency typep_struct (α₀, β₀)Prior mean Prior mode always_produces_output0.9(1.8, 0.2)0.9001.0 (boundary) list_output_variable_length0.7(1.4, 0.6)0.7001.0 (boundary) conditional_output0.5(1.0, 1.0)0.500undefined (uniform) router_k_way(k=3)0.333(0.667, 1.333) 0.3330.0 (boundary) Whenα₀ < 1orβ₀ < 1, the Beta density goes to infinity at the corresponding boundary. This is statistically correct (the prior is pulling toward that boundary) and is harmless for the decision rule: the posterior mean, not the mode, drives decisions, and after one interior observation the density is bounded. A.4 Posterior update worked example AgentA(documentanalyzer)→AgentB(topicresearcher). Dependencytaggedlist_output_variable_length (p = 0.7,n₀ = 2, soα₀ = 1.4, β₀ = 0.6). Step EventSuccesses Failuresα βPosterior mean 0 Initial001.4 0.6 0.700 1 Success102.4 0.6 0.800 2 Success203.4 0.6 0.850 3 Failure213.4 1.6 0.680 4 Success314.4 1.6 0.733 5–10 Mixed: 5 successes, 0 failures 819.4 1.6 0.855 After 10 observations the posterior mean has converged to 0.855, weighted ~82% by data (8/9 empirical success) and ~18% by prior (0.700). The intended balance: responsiveness with a sanity-check prior. A.5 Credible-bound example: cold-start vs. mature Two dependencies, both with posterior mean 0.85: Scenarioα βMean 10% lower bound Mature 85 15 0.850 0.803 Cold-start 1.7 0.3 0.850 0.325 The mature dependency is confidently ~0.85; the cold-start dependency (1 success after anal- ways_produces_outputprior) has wide uncertainty. UnderP_min = 0.5: •Mean-based rule: both speculate. •Lower-bound rule: only the mature one speculates; the cold-start one waits for more data. 30 Appendix B: Router-dependency example Agent A routes to exactly one of three downstream agents (B, C, D). Speculation candidate: Agent B. Dependency taggedrouter_k_waywithk=3(priorp = 1/3,α₀ = 2/3, β₀ = 4/3). TrialActual route Speculation of B useful?αβPosterior mean 0n/an/a0.6671.3330.333 1BSuccess1.6671.3330.556 2CFailure1.6672.3330.417 3BSuccess2.6672.3330.533 4DFailure2.6673.3330.444 5BSuccess3.6673.3330.524 After 5 trials with 3 successes (60% empirical) and the 33% prior, the posterior mean is 0.524, data- weighted toward the empirical rate but anchored by the prior. If the router has genuine skew (e.g., B is selected 60% of the time), the posterior will converge to roughly 0.6 with more data; if the router is uniform overk=3, the posterior will regress toward 0.33. Appendix C: Telemetry schema Every calibration and evaluation stage of Section 12 consumes the same per-decision log row. Without it, none of the stages run. This appendix specifies the schema. C.1 Per-decision row fromdataclassesimportdataclass fromtypingimportLiteral, Optional @dataclass classSpeculationDecision: # identity decision_id:str# UUID, unique per candidate edge event trace_id:str# workflow execution id (joins across decisions) edge:tuple[str,str]# (upstream_agent, downstream_agent) dep_type: Literal[ "always_produces_output", "list_output_variable_length", "conditional_output", "router_k_way", "rare_event_trigger", ] tenant: str# per-tenant posteriors require this key model_version:tuple[str,str]# (agent, version) for drift-triggered re-tag # decision inputs (at evaluation time) 31 alpha:float# in [0, 1] lambda_usd_per_s:float P_mean:float# Beta posterior mean P_lower_bound: Optional[float]# γ-credible lower bound, if gating C_spec_est_usd:float L_est_s:float# estimated latency savings on success input_tokens_est:int output_tokens_est:int input_price:float# USD/token output_price:float# USD/token # decision outputs EV_usd:float threshold_usd:float decision: Literal["SPECULATE","WAIT"] phase: Literal["plan","runtime"]# Section 8 two-phase model overrode: Literal["none","upgrade","downgrade"]# runtime vs. plan i_hat_source: Literal[ "modal","regex","historical","stream_k","auxiliary_model" ] # guardrails / audit (set at decision time) uncertain_cost_flag:bool# set by §12.4 EMA monitor enabled:bool# §12.5 kill-switch state at decision time budget_remaining_usd: Optional[float]# for cost SLO triggers # realized outcomes (filled in after upstream completes; default None) i_actual: Optional[object]=None# full upstream output for replay tier1_match: Optional[bool]=None tier2_match: Optional[bool]=None tier3_accept: Optional[bool]=None# filled offline, sampled (§12.4) committed_speculative:bool=False# True→kept; False→re-ran with i C_spec_actual_usd: Optional[float]=None# Section 9.3 fractional waste tokens_generated_before_cancel: Optional[int]=None latency_actual_s: Optional[float]=None Thedataclassaboveisacompletespecification; realized-outcomefieldsdefaulttoNone(orFalseforthe boolean) so that a row can be emitted at decision time and filled in later when the upstream operation completes. C.2 Deriving every calibration signal from one row SignalDerivation Posterior update (§7.3)(s, f)←(s, f) + (tier1_match ∨ tier2_match, ¬(tier1_match ∨ tier2_match))per edge 32 SignalDerivation Effectivek(§7.6)empirical distribution ofi_actualper(edge, tenant) Counterfactual EV grid (§12.1)replay D4 over the row with varied(α, λ) Tier-2 false-accept rate (§12.4)fraction ofcommitted_speculative∧ ¬tier3_acceptover sampled rows Token-estimate CoV (§12.4)std(tokens_generated_before_cancel / output_tokens_est)over committed rows; on full-completion rows, tokens_generated_before_cancelequals the actual output count Implied-λ(§12.3)solveP · L · λ − (1−P) · C_spec = (1−α) · C_specforλat observedα* Waste per failed speculation (§9.3)C_spec_actual_usdwhen ¬committed_speculative Cost SLO burnΣ C_spec_actual_usdover budget window Drift trigger (§12.5)posterior-mean delta over rolling windows per edge C.3 Retention and sampling policy Per-decision rows are small (< 1 KB serialized). A reasonable default policy: •Retainallrows for 30 days (on-call debugging, drift attribution). •Retainaggregated(edge, tenant, day)posterior updates indefinitely (the Bayesian state). •Retainsampledrows (1%) indefinitely for long-term calibration studies. •tier3_acceptisfilledofflinefromthesampledpool; humansorLLM-judgesconsumeaworking queue populated from those samples. The schema is intentionally flat. Join keys(decision_id, trace_id, edge, tenant, model_version)allow cross-cutting queries without requiring a specific data-warehouse shape. Appendix D: Synthetic numerical validation This appendix validatesthe method’sclosed-form behaviorat the canonical AutoReplyparameters (Sec- tion 10) through five seeded synthetic experiments. Each experiment is fully specified below as a direct evaluation of an equation stated in Sections 4–9, driven only bynumpyunder a single fixed seed (seed = 20260531); the reported numbers are stable across runs and reproducible from the specification alone. These experiments validate thedecisionruleanditssupportingmechanismsagainst their own equations. They arenotmeasurements of any deployed LLM workload. The calibration pipeline in Section 12 specifies how to extend this validation to a real deployment. No code, data, or other files accompany this submission. Each experiment is reconstructible from the equation it evaluates and the AutoReply parameters, under the single fixed seed above for the synthetic Bernoulli draws; no proprietary dependency, dataset, or LLM call is involved. 33 D.1 Decision-boundary validation The grid(k, α) ∈ 1, ..., 10 × 0, 0.25, 0.5, 0.75, 1.0is swept at AutoReply parameters withP = 1/k(Section 7.6 Case A, uniform prior overkbranches). Each cell applies the EV decision rule and records SPECULATE or WAIT. The closed-form critical- k curve is overlaid in red. 12345678910 branching factor k (uniform prior, P = 1/k) 0.00 0.25 0.50 0.75 1.00 operator preference α SSSSWWWWWW SSSSWWWWWW SSSSSWWWWW SSSSSWWWWW SSSSSWWWWW D.1 SPECULATE/WAIT decision grid vs. closed-form boundary SPECULATE (EV > 0) WAIT (EV ≤ 0) closed-form k crit (α) The empirical SPECULATE/WAIT boundary lies exactly alongk_crit(α) = (L_value + C_spec) / ((2 - α) · C_spec). AtAutoReplyparametersthisgivesk_crit(0)approximately2.87,k_crit(0.5) approximately 3.83, andk_crit(1.0)approximately 5.74, matching Section 7.6’s table. The validation confirms that the rule self-limits as the upstream branching factor grows: under uniform prior, noαin [0, 1] makes the rule SPECULATE forkat least 6 at this scenario’s economics. D.2 P-threshold validation Withα = 0.5fixed at AutoReply parameters,Pis swept over[0.05, 0.95]and EV is plotted. The closed-form break-even isP* = C_spec / (L_value + α · C_spec). 34 0.20.40.60.8 success probability P −0.01 0.00 0.01 0.02 0.03 0.04 0.05 EV (USD per decision) cold-start P=0.20 EV=$0.0007 drifted P=0.47 EV=$0.0198 learned P=0.62 EV=$0.0304 D.2 EV(P) at α= 0.5, AutoReply parameters EV at α= 0.5 P * = 0.191 (break-even) At AutoReply parameters,P*is approximately 0.19 forα = 0.5. The cold-start uniform priorP = 0.20 sits just above break-even (EV approximately +$0.0007, borderline SPECULATE), the post-drift poste- riorP = 0.47clears it comfortably (EV approximately +$0.020), and the learned steady stateP = 0.62 is well above (EV approximately +$0.030). The same EV rule produces qualitatively different decisions across the three regimes without any parameter tuning, exactly the Section 10 worked example. D.3 Bayesian posterior convergence Starting from theconditional_outputstructural priorBeta(1, 1)(Section 7.2,n₀ = 2), 200 sequen- tial observations are drawn from a Bernoulli withP_true = 0.62and the Beta-Binomial posterior is updated. Posterior mean and 95% credible interval are tracked over time. 35 0255075100125150175200 number of observations 0.0 0.2 0.4 0.6 0.8 1.0 estimated P D.3 Beta-Binomial posterior convergence (Beta(1,1) prior, P true = 0.62) 95% credible interval posterior mean P_true = 0.62 The posterior mean enters the neighborhood ofP_truewithin roughly 30 observations and the 95% credible interval narrows continuously. After 200 observations the interval is[0.53, 0.67](per-seed), consistent with the standard1/√nshrinkage from Beta posteriors. This validates the §7 claim that production traffic supplies sufficient signal to recover the operativePwithout tuned hyperparameters. D.4 Streaming cancellation waste reduction The validationsimulates10,000 speculativeattemptsat AutoReplyparameterswithP_success = 0.62. Failed speculations are aborted mid-stream after a fractionfof Agent B’s output tokens have been emitted, payingC_input + f · C_outputinstead of the fullC_spec. Three policies are compared: no streaming (every failure costs fullC_spec), mean-cancel atf = 0.37, and per-attempt random cancel f ∼ Unif[0.10, 0.60]. 36 no_streaming mean_cancel_f=0.37 per_attempt_unif[0.1,0.6] 0 20 40 60 80 100 120 140 total cost over 10,000 attempts (USD) $135.00 $106.62 $105.69 D.4 Streaming cancellation cost over 10,000 attempts (P_success=0.62) Mean-cancel reduces total cost from $135.00 to $106.62 (a 21% saving) with per-failure waste dropping from $0.0135 to $0.0059 (a 56% drop, matching the headline number in Section 9). The per-attempt random cancel produces a similar saving ($105.69) at this distribution of cancel fractions. The §9.3 fractional-waste accounting is therefore not a marginal optimization: it is the difference between losing the full speculative cost on every failure and recovering most of it. Telemetry-schema conformance.This evaluation is structured so that each simulated decision car- ries the full field set of the Appendix CSpeculationDecisionschema (33 fields), and the aggregate plot and cost summary are derived only from those per-decision records, demonstrating, on synthetic data, the same discipline §C.2 claims is sufficient to reconstruct every calibration signal in a real de- ployment. The other four experiments exercise their closed-form equations directly over the quantities each plot needs; a real-deployment harness would record the full schema for every decision regardless of which signal it serves. D.5 Implied-λrecovery (audit signal) For eachα* ∈ [0, 1], the EV equation is solved backwards for theλthat would makeα*the rational operating point at AutoReply(P, C_spec, L_upstream): λ_implied = [(1 − α*) · C_spec + (1 − P) · C_spec] / (P · L_upstream). This is the §12.3 audit signal: if operators settle at anα*whose impliedλis far from the declaredλ, calibration must refresh. 37 0.00.20.40.60.81.0 operating α * 10 −2 2 × 10 −2 3 × 10 −2 4 × 10 −2 6 × 10 −2 λ (USD/s of latency saved) consistent alpha*=0.5 lambda_imp=$0.024/s divergent (audit-flag) alpha*=0.9 lambda_imp=$0.013/s D.5 Implied-λ recovery vs. declared λ (Section 12.3 audit) implied λ(α * ) declared lambda = $0.08/s Atα* = 0.5the impliedλis approximately $0.024/s, broadly comparable to the declared $0.08/s, consistent with operators behaving as the latency-dollars conversion suggests. Atα* = 0.9(the paper’s runningexample)theimpliedλcollapsestoroughly$0.013/s, anorderofmagnitudebelowthedeclared value. Operators driving the dial that aggressively are revealing a far lower marginal value of latency than the deployment-configuredλclaims, the audit-flag scenario in Section 12.3. D.6 Threats to validity These experiments validate the method against its own equations and assumptions at the AutoReply parameters. They do not establish that: •Real LLM workloads have stationaryP. The calibration pipeline in Section 12 exists precisely becausePdrifts; D.5 is the audit signal. •Latency and token cost are deterministic point estimates. Real deployments see distributions; the streaming re-estimation in Section 9 is the runtime mechanism for this. •The AutoReply parameters generalize. Section 13’s workload-fit rubric is the way to evaluate a new scenario. A real-deployment evaluation following Section 12.1→12.5 (offline replay→shadow→canary with αsweep→online calibration→drift-triggered kill-switch) is the natural next step and the explicit goal of the calibration pipeline. 38

Instant research discovery