Paper deep dive
SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding
Zhi-Kai Chen, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/7/2026, 8:38:40 AM
Summary
SchemaFill is a framework that improves the efficiency of Large Language Model (LLM) tool calling by using slot-parallel speculative decoding. It addresses the latency issues of standard autoregressive decoding by concurrently generating candidate values for multiple argument slots and tool calls. These candidates are verified by the target model under the actual output prefix, ensuring correctness while achieving up to a 4.05x improvement in end-to-end throughput on benchmarks like Glaive and BFCL.
Entities (7)
Relation Signals (6)
SchemaFill → uses → Slot-Parallel Speculative Decoding
confidence 95% · We present SchemaFill, a framework for efficient LLM tool calling through slot-parallel speculative decoding.
SchemaFill → outperforms → Autoregressive Decoding
confidence 92% · SchemaFill achieves up to a 4.05× improvement in end-to-end throughput over autoregressive decoding.
SchemaFill → evaluatedon → Glaive
confidence 90% · On Glaive and BFCL, SchemaFill achieves up to a 4.05× improvement...
SchemaFill → evaluatedon → BFCL
confidence 90% · On Glaive and BFCL, SchemaFill achieves up to a 4.05× improvement...
SchemaFill → uses → BM25
confidence 88% · our default runtime forms a lexical query and ranks the available tools with BM25
SchemaFill → uses → EAGLE-3
confidence 85% · We instantiate the drafter Dϕ with EAGLE-3
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM agents interact with external systems by generating structured tool calls. Given a user request, conversational context, and a catalog of tool schemas, a tool-calling model must select tools and generate their arguments, potentially producing multiple calls in a single response. Standard autoregressive decoding generates these calls token by token, incurring substantial latency for requests involving multiple calls or many argument fields. The explicit argument structure offers opportunities for parallel generation, but later argument values may depend on preceding fields and calls, so independently generated values can differ from the target model's output. We present SchemaFill, a framework for efficient LLM tool calling through slot-parallel speculative decoding. SchemaFill generates future slot values concurrently as candidates, without requiring advance knowledge of the actual call sequence or argument values. Candidates spanning multiple fields and calls are concatenated for verification by the target model under the actual output prefix. Only verified tokens are committed, and the target supplies corrections when candidates disagree. This applies target verification while exploiting parallelism across slots and calls. On Glaive and BFCL, SchemaFill achieves up to a 4.05$\times$ improvement in end-to-end throughput over autoregressive decoding. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.07086v1
- Canonical: https://arxiv.org/abs/2610.07086v1
Trouble viewing inline? Open PDF directly →
Full Text
66,164 characters extracted from source content.
Expand or collapse full text
SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding Zhi-Kai Chen Affiliation: School of Artificial Intelligence, Nanjing University, China Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University, China Song-Yan Li Affiliation: Nanjing University, China De-Chuan Zhan Affiliation: School of Artificial Intelligence, Nanjing University, China Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University, China Han-Jia Ye †thanks: Corresponding author. Affiliation: School of Artificial Intelligence, Nanjing University, China Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University, China Abstract LLM agents interact with external systems by generating structured tool calls. Given a user request, conversational context, and a catalog of tool schemas, a tool-calling model must select tools and generate their arguments, potentially producing multiple calls in a single response. Standard autoregressive decoding generates these calls token by token, incurring substantial latency for requests involving multiple calls or many argument fields. The explicit argument structure offers opportunities for parallel generation, but later argument values may depend on preceding fields and calls, so independently generated values can differ from the target model’s output. We present SchemaFill, a framework for efficient LLM tool calling through slot-parallel speculative decoding. SchemaFill generates future slot values concurrently as candidates, without requiring advance knowledge of the actual call sequence or argument values. Candidates spanning multiple fields and calls are concatenated for verification by the target model under the actual output prefix. Only verified tokens are committed, and the target supplies corrections when candidates disagree. This applies target verification while exploiting parallelism across slots and calls. On Glaive and BFCL, SchemaFill achieves up to a 4.05× improvement in end-to-end throughput over autoregressive decoding. Code is available at https://github.com/Czzzk/SchemaFill. 1 Introduction LLM agents extend language models from producing text to acting on external systems (Yao et al., 2023; Schick et al., 2023). Given a user request, conversation history, and a catalog of available tools, an agent may need to query a database, invoke a software service, or coordinate several actions. Tool calling connects the model’s decision to such operations: the model selects an available function and supplies the information required to invoke it (Liu et al., 2024). Function-calling evaluations therefore test whether an agent can choose and parameterize actions from natural-language requests (Patil et al., 2025; Li et al., 2023). To make those actions executable, the agent cannot communicate them only as free-form text. A tool runtime expects a machine-readable object whose tool names, argument keys, value types, and nesting satisfy predefined interface schemas (Willard & Louf, 2023; Dong et al., 2025; Geng et al., 2025). This structured object is the contract between the model and the runtime: it can be parsed, validated, routed to the selected tools, and executed without another semantic interpretation step. Generating it requires the model to decide which tools to invoke, how many calls to emit, their order, and the exact argument values; one response may contain different tools or repeat the same tool several times (Qin et al., 2024; Patil et al., 2025). Current tool-calling models usually generate the entire response autoregressively, one token at a time (Xia et al., 2026). Much of that response has explicit structure, including call wrappers, argument keys, and typed value slots, yet AR decoding treats it as one flat token sequence (Zheng et al., 2024). The model must finish earlier fields and calls before reaching later slots, so requests with many fields or multiple calls incur substantial decoding latency (Chen et al., 2023; Xia et al., 2026). Using the structure to draft values for possible future slots concurrently can shorten this serial path and achieve substantially higher throughput than AR decoding. Figure 1: SD denotes speculative decoding. Direct slot-parallel generation can raise throughput but may change the target response. SchemaFill combines parallel proposals with target verification. The direct-parallel baseline follows the skeleton-then-parallel-fill idea of Skeleton-of-Thought (Ning et al., 2024), with the correct skeleton supplied in advance. Direct slot-parallel generation faces two problems: the realized tool-call sequence and argument slots are unknown when drafting begins, and independently drafted values may differ from what the target would generate under the actual preceding output. Figure 2 shows the second problem even with an oracle skeleton: deviations from target AR accumulate as more fields and calls are generated independently. Keeping the speedup while committing only target-consistent tokens is the central challenge. These disagreements become output errors if independently drafted slot values are committed without checking them against the target under the preceding fields and calls. To retain parallel generation while preserving the target response, the drafted values must remain candidates until the target decides which tokens to accept. Speculative decoding provides this accept-or-correct mechanism: the target verifies proposed tokens under the actual output prefix, accepts matches, and supplies a correction at the first mismatch (Leviathan et al., 2023; Chen et al., 2023). Conventional speculative decoders draft a continuation of the current prefix (Li et al., 2025); applying the same verification rule to future slots would let their values be drafted concurrently without committing disagreements to the response. We introduce SchemaFill, which combines parallel drafting of future slots with target verification across fields and calls (Figure 1). From the request and tool catalog, it forecasts possible call positions and slots without assuming the realized calls or values. Isolated branches draft slot values in parallel while the committed output advances. The runtime packs ready values with proposed tool names, field keys, and structural text for shared target verification. All packed tokens remain speculative until accepted by the target under the actual output prefix. Our contributions are threefold: • We present SchemaFill, a framework for slot-parallel speculative decoding of structured tool calls. It drafts future argument values concurrently and packs them with structural text across fields and calls, while the target verifies every token under the actual output prefix. • We design a shared verification runtime that evaluates concurrent drafts in one target forward pass while isolating branch attention and reusing verified KV states. • We achieve up to a 4.05× mean end-to-end speedup over autoregressive decoding and outperform existing speculative decoding baselines on Glaive and BFCL. 2 Related Work Tool Use, Function Calling, and Structured Generation. Tool-use research studies how language models select APIs, decide when to invoke them, and produce executable arguments: Toolformer learns these decisions from self-supervised API annotations (Schick et al., 2023), Gorilla adds retrieval-aware training for large and changing API collections (Patil et al., 2024), and ToolLLM/ToolBench scales to thousands of real-world APIs and multi-tool solution paths (Qin et al., 2024). API-Bank and BFCL evaluate tool selection and structured arguments in tool-augmented and single-/multi-call settings (Li et al., 2023; Patil et al., 2025), while grammar-constrained methods enforce valid output languages or schemas (Geng et al., 2023; Dong et al., 2025). These works improve tool-use capability or structural validity; SchemaFill addresses the inference cost of generating an already structured response, using public schema metadata to propose slots while retaining target verification and target control over tools, fields, ordering, and values. Speculative Decoding. Early methods use a lightweight drafter to propose future tokens for target verification (Leviathan et al., 2023; Chen et al., 2023). SpecInfer verifies draft trees (Miao et al., 2024), EAGLE and EAGLE-3 draft from target features (Li et al., 2024; Li et al., 2025), and Medusa uses multiple decoding heads to propose candidate continuations (Cai et al., 2024). Lookahead Decoding generates and verifies parallel token continuations without an auxiliary drafter (Fu et al., 2024). Despite different proposal mechanisms, these methods work on continuations of the current serialized prefix. SchemaFill applies target verification to a different unit of parallel work: values for possible fields and tool calls beyond that prefix. Schema-Aware Tool-Call Acceleration. ToolSpec is the closest prior work to our setting: it combines schema proposals, retrieval, and token recycling to accelerate tool calls (Xia et al., 2026). SchemaFill instead prepares argument values for future fields and forecast call positions, and packs those candidates with proposed structural syntax across call boundaries. Branches can prepare candidates early, but their local acceptance never commits text to the final response; every proposed value is checked again under the actual output prefix. This distinction lets us evaluate slot-parallel drafting while preserving the target’s freedom to select tools, fields, ordering, and values. Together, these lines of work leave open how to apply target-verified speculation when the candidate units are noncontiguous slots in a structured multi-call response. SchemaFill addresses this gap by drafting possible argument slots across fields and calls, packing them for shared verification, and extending the response only with target-consistent tokens. The next section defines the tool-call setting and target verification rule, then examines why independently generated slot values cannot be committed directly. 3 Setting and Motivation 3.1 Structured Multi-Call Generation An LLM agent processes a user request x in conversational context c, which may include earlier messages and tool results. Some requests can be handled directly; others require information from, or actions by, an external system. For the latter, the agent uses tool calling: it expresses the needed operation as a structured call that a runtime can execute. The runtime supplies a catalog T of available tools, each with a name, description, and argument schema StS_t defining how it can be called. The model receives the request, context, and full catalog in the prompt P=P(x,c,)P=P(x,c,T). Given P, the model generates an ordered call sequence. For each call, it chooses a tool, selects its argument fields, and supplies their values from x and c. The sequence may contain multiple calls or repeated use of one tool: (x,c,)⟼C=((t1,a1),…,(tK,aK)).(x,c,T) C=((t_1,a_1),…,(t_K,a_K)). (1) Here tj∈t_j is the selected tool and aja_j is its generated argument object. The model determines the number, order, and contents of the calls. Only after C is complete does the runtime dispatch each (tj,aj)(t_j,a_j) to its tool for execution. Each tool schema declares possible argument fields, their types, and whether they are required. For a realized call (tj,aj)(t_j,a_j), we can view its arguments as field–value pairs aj=fj,k:vj,kk=1Mja_j=\f_j,k:v_j,k\_k=1^M_j in the order they are emitted. The target must decide which optional fields appear and generate the value vj,kv_j,k of every included field using the request and context. Thus the schema exposes possible value positions, but neither the realized field set nor its values. In a multi-call response, different calls may use different schemas, and later values may depend on fields or calls generated earlier. The target emits C as a serialized token sequence y1:Ly_1:L, interleaving tool names, argument keys, values, punctuation, and enclosing structure. Standard autoregressive (AR) decoding generates all of these tokens in order under the full prompt P, even when the schema makes parts of the structure predictable. Generation stops at the first closed outer structured object or end-of-sequence token. 3.2 Speculative Decoding on Serialized Tool Calls Speculative decoding proposes several tokens before committing them, then uses the target to verify the proposal (Leviathan et al., 2023; Chen et al., 2023). Let m be the committed output prefix and z=(z1,…,zn)z=(z_1,…,z_n) a proposed continuation, and let ∥\| denote sequence concatenation. At candidate position i, the verifier evaluates pθ(⋅∣P,m∥z<i)p_θ(· P,m\|z_<i): causal target evaluation can score the proposed positions together, while each score depends only on the actual prefix and preceding candidate tokens. The prefix or proposal may cross a tool name, argument key, value, or structural delimiter; schema constraints do not make later fields or calls committed decisions. For the greedy target policy used in our main experiments at temperature 00, define gθ(P,u)=argmaxv∈pθ(v∣P,u)g_θ(P,u)= _v p_θ(v P,u). Serial AR decoding chooses yiAR=gθ(P,y<iAR)y_i^AR=g_θ(P,y_<i^AR) after the complete preceding output. Whether the drafter selects z deterministically or stochastically, greedy verification accepts the longest candidate prefix that follows the same target decisions: r⋆(m,z)=maxr∈0,…,n:zi=gθ(P,m∥z<i)for every 1≤i≤r.r (m,z)= \r∈\0,…,n\:z_i=g_θ(P,m\|z_<i)\ for every 1≤ i≤ r \. (2) If r⋆<nr <n, the verifier commits z≤r⋆z_≤ r and the target correction gθ(P,m∥z≤r⋆)g_θ(P,m\|z_≤ r ), discards the rejected suffix, and resumes from the corrected prefix. If r⋆=nr =n, it commits all of z and continues from the next target position. This checks structural and value tokens under the same serialized prefix and can accept several tokens per target pass. Under a stochastic target policy, standard speculative sampling instead accepts a proposed token v with probability min1,pθ(v∣P,u)/qu(v) \1,p_θ(v P,u)/q_u(v)\, where quq_u is its proposal distribution, and samples a correction from the residual distribution on rejection. At temperature 00, our runtime uses the gθg_θ verification rule. Appendix B proves consistency with greedy AR for this rule and gives the full stochastic rule and its distributional-consistency proof. 3.3 Accumulated AR Deviation in Direct Parallel Slot Generation Figure 2: AR deviation from independently generated slot values. Left: BFCL responses with at least eight parameters; right: at least three calls. Each curve follows a fixed cohort as more slots or calls are considered. Shading: 95% Wilson intervals (Wilson, 1927). We isolate the effect of missing inter-slot context in direct parallel generation with a controlled BFCL diagnostic. For each response, we take the call and field layout from an offline greedy autoregressive (AR) output and replace every argument value with a distinct slot marker. The target model then generates each value independently from the public prompt, the complete marked layout, and that slot’s identity, without seeing values generated for other slots. We compare each value with greedy AR at the corresponding position, where AR conditions on the full preceding output. This oracle layout controls for tool and field selection in the diagnostic; SchemaFill does not receive it at inference. Figure 2 reports the fraction of responses with at least one independently generated value differing from AR. Within the fixed 200-response parameter cohort, this fraction changes from 3.0% after the first slot to 49.5% after eight slots. Within the fixed 270-response multi-call cohort, it changes from 6.7% after the first call to 43.3% after the third. Thus, direct parallel generation can change the target’s serialized output as deviations accumulate across values. These curves measure cumulative AR deviation, not downstream execution success or the difficulty of an individual slot. They motivate treating parallel values as provisional candidates and verifying them under the target’s actual serialized prefix before committing them. 4 Method: SchemaFill 4.1 Method Overview SchemaFill follows a forecast–draft–verify loop. From the request, dialogue context, and full tool catalog, the method identifies possible argument slots and begins drafting later values before their serialized positions are reached. These drafts remain provisional. When a candidate becomes relevant, SchemaFill packs it with the required structural text after the current output prefix and lets the target accept or correct the resulting sequence in the same order as ordinary autoregressive decoding. We call the authoritative output path the committed stream and the provisional value paths speculative branches; the implementation refers to them as MAIN and SIDE, respectively. The full schema always remains in the target prompt, and the target still determines the realized tools, fields, order, and values. Forecasting changes only where draft computation is spent, while target verification determines every emitted token. Figure 3 previews the target-KV paths, and Algorithm 1 summarizes the complete forecast–draft–verify loop. The following subsections explain the forecast, slot drafting and verification, cross-call packing, and target-KV management. Figure 3: Target-KV cache lifecycle in SchemaFill. The public prefix and verified rows occupy a shared arena, while attention masks restrict branches to valid context and their own candidate paths. One target pass verifies branch scratch rows. Accepted rows extend their cache paths; rejected suffixes are discarded. Only verified tokens on the committed branch become output; a side-branch value must be verified again under the actual output prefix before it can be committed. 4.2 Slot-Parallel Speculative Decoding During initialization, candidate key prefixes are verified and an admissible MAIN path is selected by accepted key-prefix length, then total accepted tokens and forecast order. For each branch b, aba_b counts matching proposal tokens, cbc_b is the first target correction (ϵε if none), and Kb′K _b contains its target-cache rows. Trimming bounds a draft to a candidate value literal; it does not accept that literal. The stopping test uses only verified MAIN tokens, including closing text. Metadata-Guided Tool and Slot Forecasting. A forecasted slot s=(j,t,f)s=(j,t,f) specifies a possible call position j, candidate tool t∈t , and field f from its public schema. The slot’s type and optionality guide drafting, but neither the forecasted call nor field is a commitment. Given the request and dialogue context, our default runtime forms a lexical query and ranks the available tools with BM25 (Robertson & Zaragoza, 2009). Let q denote the tokenized request and context, and let dtd_t denote the tokenized public metadata of tool t (its name, description, and schema field names and descriptions). The ranking is π=rankt∈BM25(q,dt),π=rank_t BM25(q,d_t), (3) where tools with equal scores keep their original catalog order. We cycle this ranked list and truncate it to the first B=3B=3 call groups, ℱB=prefixB(cycle(π)),F_B=prefix_B\! (cycle(π) ), (4) then expand each forecasted tool into slots in rendered-schema key order, recording its declared type and requiredness. The scheduler activates these slots near the current committed position. If verified structure rules out a forecast slot, the scheduler retires it; an unused producer group may be rebound when its tool matches a verified call header. Forecasts allocate draft computation, while the target determines the realized tools, fields, and values (Algorithm 1, lines 4–6). SIDE speculative drafting. For an active slot si=(ji,ti,fi)s_i=(j_i,t_i,f_i), let HiH_i contain its call-position hint, tool/field prefix, and branch-private draft history. We instantiate the drafter DϕD_φ with EAGLE-3 (Li et al., 2025). It produces a provisional value from the common prompt and this local context, v^i=Dϕ(P⊕Hi), v_i=D_φ\! (P H_i ), (5) so branches share the public prompt, while the attention mask excludes every other branch’s private history HℓH_ from branch i for ℓ≠i ≠ i. MAIN continues along the committed stream. The target checks v^i v_i under the local context P⊕HiP H_i; accepted tokens and corrections advance only that provisional SIDE branch. On a local mismatch, the rejected suffix is discarded, while the locally accepted prefix and correction remain provisional. A packed MAIN proposal is checked separately under the actual prefix P⊕CtP C_t. These paths can share one target tree forward, but a SIDE value completed during that forward can enter a packed MAIN proposal only in a later round. Local acceptance never appends text to the response (Algorithm 1, lines 8–12 and 16–19). Cross-call packing. When the committed stream reaches a field for which a SIDE branch has prepared a completed or partial value, the packer places that value after the actual MAIN prefix. It also proposes the intervening tool names, field keys, punctuation, and closing text, so that one proposal can continue across fields and tool-call boundaries. If CtC_t is the target-verified MAIN prefix, v^i v_i is a prepared SIDE value, and SiS_i is a proposed structural bridge, the packed suffix is Zt=S0⊕v^1⊕S1⊕⋯⊕v^k⊕Sk.Z_t=S_0 v_1 S_1 ·s v_k S_k. The target checks ZtZ_t as one causal MAIN path after P⊕CtP C_t: each token attends to earlier proposed tokens on that path, including across field and call boundaries, but never to a SIDE branch’s private history. For greedy target decoding, Equation 2 accepts only the longest prefix matching the target’s decisions. At a mismatch, MAIN commits that prefix and the target correction, discarding the remaining packed suffix and its target-KV rows. A prepared SIDE value can remain a private candidate if its forecast slot is still possible, but it can be used only after the realized call and field match; its local KV is never transferred to MAIN, and the value must be checked again under the corrected MAIN prefix (Algorithm 1, lines 14–17). Algorithm 1 SchemaFill: slot-parallel speculative decoding 1: Request x, context c, public tool catalog T, forecast budget B; target fθf_θ, drafter DϕD_φ 2: Target-verified structured output y 3: P←BuildPrompt(x,c,)P (x,c,T) 4: π←RankMetadata(x,c,)π (x,c,T); G←ForecastGroups(π,B)G (π,B) 5: ←ExpandSlots(G)S (G) 6: (y,K,MAIN,Hi)←InitVerifyθ(P,)(y,K,MAIN,\H_i\) _θ(P,S) ⊳ verified MAIN prefix, provisional SIDE states 7: while ¬Stop(y) (y) do 8: A←MAIN∪ActiveSIDE(MAIN,)A←\MAIN\ (MAIN,S) 9: L←PreparedSIDE()L (S) ⊳ snapshot before this round 10: HMAIN←yH_MAIN← y 11: for all b∈Ab∈ A in parallel do 12: Rb←TrimToLiteralBoundary(Dϕ(P⊕Hb))R_b \! (D_φ(P H_b) ) 13: end for 14: Z←PackMAIN(y,RMAIN,L,)Z (y,R_MAIN,L,S) ⊳ proposes structural text 15: ℬ←(MAIN,Z)∪(i,Ri):i∈A∖MAINB←\(MAIN,Z)\∪\(i,R_i):i∈ A \MAIN\\ 16: (ab,cb,Kb′)b∈ℬ←VerifyTreeθ(ℬ,K)(a_b,c_b,K _b)_b _θ(B,K) 17: y←y⊕Z1:aMAIN⊕cMAINy← y Z_1:a_MAIN c_MAIN 18: for all i∈A∖MAINi∈ A \MAIN\ do 19: Hi←Hi⊕(Ri)1:ai⊕ciH_i← H_i (R_i)_1:a_i c_i ⊳ SIDE remains provisional 20: end for 21: K←RetainAcceptedRows(K,Kb′,abb∈ℬ)K (K,\K _b,a_b\_b ) ⊳ discard rejected scratch rows 22: (MAIN,)←AlignForecast(MAIN,,y)(MAIN,S) (MAIN,S,y) 23: end while 24: return y 4.3 KV-Cache Management Building on tree-based speculative verification (Miao et al., 2024), SchemaFill manages the target-KV lifecycle shown in Figure 3. The physical arena is shared by MAIN and SIDE branches, but logical ownership and attention visibility remain branch-specific. The public-prefix region stores the common prompt context; the persistent verified region stores the authoritative target-accepted history accumulated across verification rounds; and the scratch region holds the current batch of unverified candidate rows. Free capacity is managed separately from these logical regions so that temporary tree paths can be created without making them part of the committed stream. The lifecycle follows the stages in the figure. (1) Read shared context: each branch attends to the public prefix and verified history with only its visible ancestors and private tokens; shared physical storage does not merge branch histories. (2) Batched target verification: MAIN and active SIDE candidates are placed in one structured verification tree. The MAIN path is causal across its packed fields and calls, while the visibility mask prevents it from attending to private SIDE paths and masks sibling SIDE paths from one another. Candidate rows remain scratch until the target accepts them. (3) Resolve and update branch paths: accepted rows extend only their owning path. A MAIN correction extends the committed output, whereas a SIDE correction remains provisional; rejected suffix rows are discarded. A SIDE value is never grafted directly into MAIN state: when used as a MAIN proposal, it is verified again under the true prefix. This reuse of verified history avoids repeated re-prefill while preserving the target’s serialized acceptance rule. Drafter KV and captured target features remain separate branch-owned state rather than additional authoritative target history. 5 Experiments 5.1 Experimental Setup Datasets. Our Glaive test set consists of 650 examples sampled from the full Glaive Function Calling dataset (Glaive AI, 2023). For the Berkeley Function Calling Leaderboard (BFCL) v3 (Patil et al., 2025), we select the three categories most suited to evaluating concurrent tool calling from the 25 categories in the release used here. Our BFCL test set contains all 450 requests from these three categories. Preprocessing normalizes tool definitions and extracts the user request and available preceding context at the selected decision point. The main and appendix comparisons use these fixed test sets; Section 5.4 reports a separate call-count diagnostic. Results characterize the selected examples and categories. Task and input. All datasets use the same plain-text instruction template, and all methods share the same base task prompt for each request, containing its prior context and full tool catalog. The target generates one JSON object with a tool_calls list; each entry contains a tool name and an arguments object. We use Qwen2.5-7B-Instruct (Yang et al., 2024) as the target model. Appendix A gives the model, training, runtime, and hardware settings. Metrics. For request i, let LiL_i count committed output tokens, including values, tool names, keys, punctuation, and target corrections; let TiT_i be elapsed time in seconds and ViV_i its actual target verification calls, including initialization. We report End-to-end throughput as TPSE2E=∑iLi/∑iTiTPS_E2E= _iL_i/ _iT_i and Acceptance length as AL=∑iLi/∑iViAL= _iL_i/ _iV_i. A shared tree forward counts as one verification even when it advances MAIN and SIDE branches. Timing uses CUDA synchronization at request boundaries and covers input preparation, initialization, drafting, target verification, and stopping. Model loading, CUDA graph construction, and warmup precede measurement; trace serialization follows it. Failed requests retain their tokens, target calls, and elapsed time in pooled metrics. 5.2 Main Results We compare decoding methods that represent different sources of acceleration. Autoregressive decoding (Leviathan et al., 2023) provides the target’s one-token reference rate, while EAGLE-3 (Li et al., 2025) measures learned token-tree speculation along the emitted prefix without SchemaFill’s slot forecasting or cross-call packing. ToolSpec (Xia et al., 2026) represents schema-aware acceleration based on proposed and retrieved continuations. Parallel Fill follows the skeleton-then-fill idea (Ning et al., 2024), independently generating slot values with the target given an AR-derived call/field skeleton; its throughput measures filling and reconstruction with this skeleton supplied, and AL is not applicable. SchemaFill prepares argument slots from public tool metadata and verifies slot candidates together with structural text across calls. Table 1 summarizes matched requests under the same target decoding policy; all methods use temperature 00. Results for AR, EAGLE-3, ToolSpec, and SchemaFill at target temperature 11 appear in Appendix C.5. The component studies in Sections 5.3 and 5.4 use temperature 00. Table 1: End-to-end results on the Glaive (Glaive AI, 2023) and BFCL v3 (Patil et al., 2025) test sets with Qwen2.5-7B-Instruct. All methods use temperature 00. Acceptance length (AL) is committed tokens per target verification call; E2E TPS is committed tokens per second, shown as mean ± sample standard deviation over three launches. Speedup is the TPS ratio to AR within each dataset. Dataset Method AL E2E TPS Speedup AR agreement (%) Glaive AR 1.001.00 91.2191.210.18 1.00× 100.00 EAGLE-3 (Li et al., 2025) 3.243.24 225.04225.040.91 2.47× 100.00 ToolSpec (Xia et al., 2026) 2.642.64 194.47194.471.04 2.13× 100.00 Parallel Fill — 219.79219.790.72 2.41× 82.62 SchemaFill 6.216.21 340.60 2.96 3.73×3.73 × 100.00 BFCL v3 AR 1.001.00 92.6892.680.14 1.00× 100.00 EAGLE-3 2.842.84 200.99200.990.99 2.17× 100.00 ToolSpec 3.273.27 241.06241.061.05 2.60× 100.00 Parallel Fill — 233.35233.350.18 2.52× 62.44 SchemaFill 6.426.42 375.71 3.66 4.05×4.05 × 100.00 SchemaFill achieves the highest throughput on both datasets: 340.60±2.96340.60± 2.96 TPS on Glaive and 375.71±3.66375.71± 3.66 TPS on BFCL. These correspond to 3.73× and 4.05× speedups over AR, 1.51× and 1.87× over EAGLE-3, and 1.75× and 1.56× over ToolSpec. The Glaive test requests average 1.00 reference call and 1.93 top-level argument fields; BFCL averages 2.96 calls and 7.46 fields. Appendix C.1 gives the counting scope. On single-call Glaive requests, SchemaFill can prepare argument values within a call; on BFCL, it can also prepare later calls and verify candidates across call boundaries. Combining prepared values with structural text yields 2.35× and 1.96× ToolSpec’s AL, respectively, reducing target verification frequency on both workloads. Additional Experiments. We report a BFCL runtime breakdown including BM25 forecasting but excluding request initialization (Appendix C.3), KV-cache overhead (Appendix C.2), and run-to-run variance (Appendix C.4). Temperature-one results and slot-order diagnostics appear in Appendices C.5 and C.6, respectively; cross-target evaluations on Qwen3 (Yang et al., 2025), Mistral (Jiang et al., 2023), and Hermes (Teknium et al., 2024) appear in Appendix D. 5.3 Ablation Study We use four cumulative stages in Table 2. Stage A is the EAGLE-3 baseline. Stage B uses SchemaFill’s schema-guided MAIN path, including tool and slot forecasting and structural/slot concatenation, without SIDE branches. Stage C enables SIDE speculative drafting for future slots. Stage D enables cross-call packing, which combines prepared candidates and intervening structural text into one proposal for target verification under the current MAIN prefix. Stages B–D share the same SchemaFill runtime. Table 2: Cumulative ablation on the BFCL test set. Each stage retains all mechanisms from the preceding stage. A is the EAGLE-3 baseline. B uses SchemaFill’s schema-guided MAIN path to forecast tool and slot positions and concatenate structural text with drafted values, without SIDE branches. C adds SIDE drafting of future slot values. D adds cross-call packing of prepared values and intervening structural text into one target-verification proposal, yielding full SchemaFill. Speedup is the TPS ratio to the EAGLE-3 baseline (A). Stage Configuration Accept Length E2E TPS Speedup A EAGLE-3 2.84 200.99200.99 1.00× B A + schema-guided MAIN path 4.57 297.61297.61 1.48× C B + SIDE speculative drafting 5.48 330.46330.46 1.64× D C + cross-call packing (full SchemaFill) 6.42 375.71375.71 1.87× Relative to EAGLE-3, Stage B raises TPS by 48.07% and AL by 61.18%. This comparison measures the combined effect of SchemaFill’s MAIN runtime, metadata-guided forecasting, and structural/slot concatenation. Within the shared runtime, enabling SIDE drafting in Stage C changes TPS by +11.04% and AL by +19.95% by preparing later values before MAIN reaches them. Cross-call packing in Stage D changes TPS by +13.69% and AL by +17.07% over Stage C. This final comparison isolates the benefit of proposing prepared values and structural bridges across call boundaries in one target verification. 5.4 Scaling with Call Count Table 3: Separate BFCL v3 call-count diagnostic (100 requests/group), using B=CB=C. AL is committed tokens per verification; E2E TPS is tokens/s; Nodes/Verify is pooled tree size. All requests and initialization are included. Ref. Calls (C) Accept Length E2E TPS Nodes/ Verify 1 5.14 265.95265.95 23.4323.43 2 7.98 438.62438.62 35.7635.76 3 8.06 443.54443.54 39.2939.29 4 7.54 410.72410.72 42.7142.71 We study how SchemaFill’s performance varies with task size, measured by reference call count. For this separate diagnostic, we randomly sample 100 natural requests from the broader BFCL v3 release (Patil et al., 2025) for each count from one to four (400 requests in total). All groups use the common target prompt. In this count-informed diagnostic, SchemaFill receives the reference call count C to set its forecast budget B=CB=C, with other production settings unchanged. The groups are independently sampled rather than paired prefixes. We measure mean verification-tree size as N¯=∑r∑v=1VrNr,v/∑rVr N= _r _v=1^V_rN_r,v/ _rV_r, where VrV_r counts actual target verification calls, including initialization, and Nr,vN_r,v counts distinct candidate-token nodes processed in that call. MAIN and SIDE nodes count whether accepted or rejected; shared nodes count once per verification, and prompt prefill rows are excluded. Figure 4: Target-verification workload during SchemaFill decoding. Candidate-node counts peak at slot initialization and drop sharply in subsequent rounds. Curves average requests reaching each round in the Table 3 diagnostic, excluding prompt tokens. Rounds 6–11 are compressed; endpoint labels mark the last rounds with at least 10 contributing requests. Figure 4 shows how SchemaFill prepares more slots as the number of tasks within a request increases. With B=CB=C, mean initialization-tree size grows from 61.78 nodes at one call to 151.98 at four. These additional nodes initialize more speculative slot contexts together, allowing their candidate values to be generated concurrently in the early decoding rounds. Slot preparation is therefore concentrated at the beginning of generation, so candidates can become available early for packing. The curves drop sharply after initialization, with subsequent rounds using much smaller verification trees. Once slot candidates are ready, further acceleration depends on their quality: the target must accept them under the committed prefix to increase AL. Table 3 shows that performance reaches its peak around two to three calls, with 438.62–443.54 TPS and AL 7.98–8.06, up from 265.95 TPS and AL 5.14 at one call. At four calls, mean verification-tree size grows to 42.71 nodes, while AL drops to 7.54 and throughput to 410.72 TPS. This pattern is consistent with slot-drafter quality limiting further acceptance gains despite the early preparation of more slot candidates. 6 Conclusion SchemaFill uses tool schemas to draft future argument values concurrently and packs candidates across fields and calls for verification under the actual output prefix. On the Glaive and BFCL test sets, it achieves 3.73× and 4.05× end-to-end throughput over AR, respectively, outperforming EAGLE-3 and ToolSpec. Its central insight is that tool-call structure enables parallel speculation without assuming independent arguments, since the target verifies every token before commitment. References Cai et al. (2024) Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024. Chen et al. (2023) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. Chen et al. (2025) Zhi-Kai Chen, Jun-Peng Jiang, Han-Jia Ye, and De-Chuan Zhan. Hawk: Leveraging spatial context for faster autoregressive text-to-image generation. arXiv preprint arXiv:2510.25739, 2025. URL https://arxiv.org/abs/2510.25739v1. Chen et al. (2026a) Zhi-Kai Chen, Jun-Jie Tao, Wei-Xiang Mao, De-Chuan Zhan, and Han-Jia Ye. ResiSpec: Enhancing multi-candidate speculative sampling via residual distribution shaping. arXiv preprint arXiv:2608.24411, 2026a. URL https://arxiv.org/abs/2608.24411. Chen et al. (2026b) Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li, De-Chuan Zhan, and Han-Jia Ye. PeakBench: Benchmarking resource-aware tool invocation in LLM agents. arXiv preprint arXiv:2608.24509, 2026b. URL https://arxiv.org/abs/2608.24509. Dong et al. (2025) Yixin Dong, Charlie F Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. Xgrammar: Flexible and efficient structured generation engine for large language models. Proceedings of Machine Learning and Systems, 7, 2025. Efron (1979) Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1–26, 1979. doi: 10.1214/aos/1176344552. URL https://doi.org/10.1214/aos/1176344552. Fu et al. (2024) Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 14060–14079, 2024. URL https://proceedings.mlr.press/v235/fu24a.html. Geng et al. (2023) Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar-constrained decoding for structured nlp tasks without finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 10932–10952, 2023. Geng et al. (2025) Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. Jsonschemabench: A rigorous benchmark of structured outputs for language models. arXiv preprint arXiv:2501.10868, 2025. Glaive AI (2023) Glaive AI. Glaive function calling dataset. Hugging Face dataset, 2023. URL https://huggingface.co/datasets/glaiveai/glaive-function-calling. Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International conference on machine learning, p. 19274–19286. PMLR, 2023. Li et al. (2023) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 3102–3116, 2023. Li et al. (2024) Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024. Li et al. (2025) Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, p. 136737–136756. Curran Associates, Inc., 2025. doi: 10.52202/085713-4562. URL https://proceedings.neurips.c/paper_files/paper/2025/file/c7b5a35ea98b62512a869c19ea7b03cb-Paper-Conference.pdf. Liu et al. (2025) Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. ToolACE: Winning the points of LLM function calling. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, p. 41359–41381, 2025. URL https://proceedings.iclr.c/paper_files/paper/2025/file/663865ea167425c6c562cb0b6bcf76c7-Paper-Conference.pdf. Liu et al. (2024) Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems, 37:54463–54482, 2024. Miao et al. (2024) Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, p. 932–949, 2024. Mistral AI (2024) Mistral AI. Model card for Mistral-7B-Instruct-v0.3. Hugging Face model card, 2024. URL https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3. Ning et al. (2024) Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of-thought: Prompting llms for efficient parallel generation. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/hash/03d7e13f0092405804f3a381ade8f3f0-Abstract-Conference.html. Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.c/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html. Patil et al. (2024) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. In Advances in Neural Information Processing Systems, volume 37, 2024. Patil et al. (2025) Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. Qin et al. (2024) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations, 2024. Robertson & Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. doi: 10.1561/1500000019. URL https://doi.org/10.1561/1500000019. Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023. Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023. Teknium et al. (2024) Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes 3 technical report, 2024. Willard & Louf (2023) Brandon T Willard and Rémi Louf. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702, 2023. Wilson (1927) Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927. doi: 10.1080/01621459.1927.10502953. URL https://doi.org/10.1080/01621459.1927.10502953. Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 38–45. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6/. Xia et al. (2026) Heming Xia, Yongqi Li, Cunxiao Du, Mingbo Song, and Wenjie Li. Toolspec: Accelerating tool calling via schema-aware and retrieval-augmented speculative decoding. arXiv preprint arXiv:2604.13519, 2026. Yang et al. (2024) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2210.03629. Zheng et al. (2024) Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.c/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html. Appendix A Implementation Details Target and drafter. The main experiments use Qwen2.5-7B-Instruct (Yang et al., 2024), temperature 00 (greedy decoding), repetition penalty 1.0, BF16 target weights and computation, and FP32 target logits. RoPE (Su et al., 2024) frequency and sine/cosine computations use FP32, with BF16 rounding preserved in rotary application. Reduced-precision BF16 reduction is disabled within target forwards. EAGLE-3 and our method share the same locally trained EAGLE-3 head (Li et al., 2025): epoch 6, a 14,136-token draft vocabulary, and target hidden-state indices [2,14,25][2,14,25]. In the temperature-one experiments in Appendices C.5 and D, all four methods use target temperature 11, while learned drafters retain temperature 00. The drafter was trained on fresh target-generated continuations from separate public tool-calling requests drawn from BFCL and additional public corpora including Glaive, API-Bank, ToolACE, and ToolAlpaca (Glaive AI, 2023; Li et al., 2023; Liu et al., 2025; Tang et al., 2023). Training uses 11,517 positive-mask records constructed from 9,216 public requests, six epochs, learning rate 8×10−58× 10^-5, batch size four with two accumulation steps, and two rollout horizons with loss decay 0.80.8. Runtime configuration. The production runtime uses a three-call forecast budget. Candidate tools are ranked by BM25 lexical relevance: the query contains the request and dialogue context, and each tool document contains its name, description, and schema field metadata; ties retain catalog order, and the ranked list is cycled to allow repeated calls. The scheduler admits the current forecast group and the next group, and can rebind an unused matching producer after observed call-order drift. It uses the hint Select the second request. only inside the corresponding branch prefix. Multiple active branches share a segmented draft tree with branch-isolated attention; a single active branch uses the standard segmented drafting path. The committed stream and speculative branches share one target tree verification. Target KV allocation capacity, exact prompt-tokenization prefixes, and exact public prompt KV prefixes are reused when valid. Initialization combines causal attention for uncached prompt tokens with masked MAIN/SIDE tree attention in one target forward. Cached prompt features are retained, and branch visibility remains isolated. The measured draft budget is k=3k=3, committed-stream depth 22, branch depth 11, and total draft tokens 88. Hardware and measurement. Experiments run on one NVIDIA RTX PRO 6000 Blackwell Workstation GPU (96 GB), using PyTorch 2.11.0 (Paszke et al., 2019), CUDA 13.0, Transformers 5.5.4 (Wolf et al., 2020), and batch size one. GPU timing runs are serialized. All methods share SDPA, fused/ordered normalization, exact-shape CUDA graphs, and the target numerical settings. AR, EAGLE-3, and ToolSpec use matching prompts and target settings from same-machine evaluations. Model load, graph setup, warmup, and trace serialization are excluded from E2E time. Appendix B Speculative-Sampling Consistency in the Tool-Call Setting The target next-token distribution pθ(⋅∣P,u)p_θ(· P,u) is defined at every prefix u of the serialized tool-call response. Tool names, field keys, values, punctuation, and closing tokens are all members of the same token sequence. Forecasted slots, SIDE values, and structural bridges are proposals; they do not restrict the target vocabulary or determine the realized call layout. Greedy consistency at temperature zero. Fix a prompt P and a deterministic tie-breaking rule for gθg_θ. Assume that every MAIN verification evaluates the same conditional target logits as serial AR decoding at the same token prefix, that the verification tree masks SIDE histories from MAIN, and that only the matching MAIN prefix and the target’s first correction are committed. Both decoders use the same stopping rule and, if needed, the same finite length cap. Under these assumptions, SchemaFill emits exactly the greedy AR token sequence. Proof. Induct on the length of the committed prefix m. Initially m is empty, hence agrees with the AR output. Suppose m agrees with AR and let z=(z1,…,zn)z=(z_1,…,z_n) be any packed MAIN proposal, including possible tokens from future slots or calls. By Equation 2, each accepted ziz_i equals gθ(P,m∥z<i)g_θ(P,m\|z_<i). Therefore appending the accepted prefix follows AR token by token. If r⋆<nr <n, the next committed token is gθ(P,m∥z≤r⋆)g_θ(P,m\|z_≤ r ), which is precisely AR’s next token; the remaining proposal is discarded. If r⋆=nr =n, the entire accepted proposal is an AR prefix and the next round starts from that prefix. SIDE acceptance cannot alter this argument because a SIDE token is never committed without a fresh MAIN verification under the actual prefix. The induction continues through proposed closing tokens and ends at the common stopping condition. □ Distributional consistency under exact speculative sampling. We apply the standard speculative-sampling rule (Leviathan et al., 2023; Chen et al., 2023) to the serialized tool-call setting. The following result concerns distributional consistency under a stochastic target policy. At any MAIN prefix u, let pu(v)=pθ(v∣P,u)p_u(v)=p_θ(v P,u) be the target next-token distribution at the selected temperature. Conditional on the proposal state available before the next acceptance decision, let qu(v)q_u(v) be the distribution of the next proposed MAIN token. This state may include public metadata and provisional SIDE values, but must not change pup_u; the acceptance draw must be fresh and independent of proposal randomness. A fixed proposed structural or slot token corresponds to a point-mass quq_u. For a proposed token v with qu(v)>0q_u(v)>0, use the speculative-sampling acceptance probability αu(v)=min(1,pu(v)qu(v)). _u(v)= \! (1, p_u(v)q_u(v) ). (6) On rejection, draw a correction from the residual distribution ρu(w)=pu(w)−minpu(w),qu(w)1−∑v∈minpu(v),qu(v), _u(w)= p_u(w)- \p_u(w),q_u(w)\1- _v \p_u(v),q_u(v)\, (7) whenever its denominator is positive. If the denominator is zero, rejection has probability zero and ρu _u is not used. The accepted token or residual correction is committed; after a rejection, all later tokens in that proposal are discarded. After a fully accepted proposal, decoding resumes at the next target position. Under exact target conditionals and a common stopping rule with almost-sure termination or a common finite length cap, this rule produces the same distribution of stopped tool-call responses as serial sampling from pθp_θ. Proof. Fix u and the pre-decision proposal state. The probability of proposing and accepting w is qu(w)αu(w)=minpu(w),qu(w)q_u(w) _u(w)= \p_u(w),q_u(w)\. The rejection probability is Ru=1−∑vminpu(v),qu(v)R_u=1- _v \p_u(v),q_u(v)\. For Ru>0R_u>0, the probability that a rejection emits w is Ruρu(w)=pu(w)−minpu(w),qu(w)R_u _u(w)=p_u(w)- \p_u(w),q_u(w)\. Hence the total probability of the next committed token is pu(w)p_u(w). When Ru=0R_u=0, the accepted-token term alone equals pu(w)p_u(w) for every w. This calculation applies again at each new MAIN prefix, including prefixes that cross field or call boundaries: the target conditional is always evaluated under the full serialized prefix, while a SIDE value remains a proposal until checked on MAIN. By induction on output position, every finite committed prefix has the same probability as under serial target sampling. Applying the same deterministic stopping rule to these prefixes gives the same distribution of complete responses. □ The greedy result assumes identical target conditionals at each prefix under exact arithmetic. The stochastic extension additionally uses the acceptance and residual-correction rules above. Appendix C Additional Measurement Details C.1 Evaluation Sets and Reference Workload Counts The evaluation uses the Glaive650 and BFCL450 test sets described in Section 5. Glaive is sampled by source conversation, then by eligible decision point, from 27,278 automatically eligible decisions in 20,670 conversations after structural checks and removal of training/development overlaps. The resulting 650 requests contain 650 reference calls and 1,255 top-level argument fields. BFCL contains all 50 executable-parallel, 200 parallel, and 200 parallel-multiple requests in the three selected source categories, totaling 1,333 reference calls and 3,356 fields. Executable-parallel references are parsed from the raw ground-truth call expressions. For the other categories, field counts use the first official allowed argument value, omitting fields whose representative value denotes absence. References are stored separately from the inference inputs and used only for offline workload statistics. C.2 Per-Request KV-Cache Overhead Table 4: SchemaFill’s per-request effective KV-cache overhead with an EAGLE-3 drafter, BF16 weights, and batch size one. KV-cache growth is relative to greedy AR with the same target-KV layout. The model-and-cache comparison holds target and drafter weights fixed and measures the memory increase due to extra KV. Target model KV-cache increase Model-and-cache increase from extra KV Qwen2.5-7B-Instruct +24.65%+24.65\% +0.28%+0.28\% Across BFCL450, effective KV averages 182.93 MiB for AR and 228.02 MiB for SchemaFill. Effective KV counts retained target rows and drafter KV, using the same expanded target layout; reserved spare capacity is excluded. The separate reusable public-prefix snapshot averages 100.68 MiB of process-level storage. Target and drafter peaks are measured separately, so the effective-KV sum is a conservative upper bound. C.3 BFCL Runtime Time Breakdown Table 5: SchemaFill time breakdown on the BFCL test set. Percentages use the sum of the four profiled phases. Phase Total (s) Mean (ms/request) Share (%) BM25 tool forecast 0.032 0.070 0.027 EAGLE-3 drafting 21.14 46.98 18.28 Target verification 86.94 193.19 75.17 Other generation 7.54 16.76 6.52 Total 115.65 256.99 100.00 We profiled all BFCL test requests with the production configuration of Table 1. BM25 time covers lexical ranking of public tool metadata during forecast preparation. Drafting covers EAGLE-3 candidate generation; target verification covers the cached shared-tree verifier, including tree construction and result evaluation. The remaining generation time covers scheduling, candidate packing, acceptance, KV updates, and stopping. Timed draft and verification calls use CUDA synchronization at their boundaries, so the four rows partition the profiled time. Prompt construction, tokenization, scheduler initialization, target prefill, model loading, graph setup, and warmup are excluded from this breakdown. The instrumented run took 129.23 seconds end to end, compared with 127.51 seconds in the corresponding uninstrumented launch; Table 1 uses uninstrumented throughput. BM25 scoring is bypassed when only one tool is available. C.4 Run-to-Run Variance For the repeated launches in Table 1, sample variance uses denominator n−1=2n-1=2. TPS variances for Glaive/BFCL are: AR 0.03/0.020.03/0.02, EAGLE-3 0.82/0.970.82/0.97, ToolSpec 1.07/1.101.07/1.10, SchemaFill 8.78/13.408.78/13.40. SchemaFill’s output-token and target-call counts are identical across its three runs on each dataset, giving AL variance 0.000.00. In Table 2, A and D reuse the three-launch means from Table 1; B and C each use one BFCL450 launch. All other refreshed experiments use one launch per configuration. C.5 Experiments at Temperature One We evaluate AR, EAGLE-3, ToolSpec, and SchemaFill at target temperature 11 on the same fixed cohorts of 650 Glaive and 450 BFCL requests. Our ToolSpec adapter retains its candidate construction and uses categorical target draws to select accepted tree paths and supply corrections. Table 6 reports the results. Table 6: End-to-end results on Glaive650 and BFCL450 at target temperature 11, with one run per method and dataset. Learned drafters use temperature 00. Speedup is the TPS ratio to AR at target temperature 11. Dataset Method Accept Length E2E TPS Speedup Glaive AR 1.001.00 91.0191.01 1.00× EAGLE-3 (Li et al., 2025) 3.123.12 203.27203.27 2.23× ToolSpec (Xia et al., 2026) 2.622.62 190.64190.64 2.09× SchemaFill 5.995.99 326.13 3.58×3.58 × BFCL v3 AR 1.001.00 92.2292.22 1.00× EAGLE-3 2.752.75 188.13188.13 2.04× ToolSpec 3.283.28 238.57238.57 2.59× SchemaFill 6.166.16 356.61 3.87×3.87 × At temperature 11, SchemaFill achieves speedups of 3.58×3.58× on Glaive and 3.87×3.87× on BFCL over AR at the same temperature. C.6 Slot-Order Diagnostic Model The BFCL450 slot-order diagnostic in Section 3.3 analyzes 450 responses. A source- and call-count-adjusted response-level logistic model associates each additional parameter with 1.25× the odds of at least one AR deviation (95% interval: 1.14–1.38, estimated by request-level bootstrap resampling (Efron, 1979)). This descriptive diagnostic uses AR-selected layouts and characterizes candidate behavior, rather than held-out end-to-end performance. Appendix D Cross-Target Evaluation We additionally evaluate Qwen3-8B (Yang et al., 2025), Mistral-7B-Instruct-v0.3 (Jiang et al., 2023; Mistral AI, 2024), and Hermes-3-Llama-3.1-8B (Teknium et al., 2024) on the 450-request BFCL cohort used in Appendix C.5. Mistral and the Llama-based Hermes provide non-Qwen targets. Each target has its own EAGLE-3 head, shared by EAGLE-3 and SchemaFill. The heads use target hidden-state indices [2,18,33][2,18,33] for Qwen3, and [2,16,29][2,16,29] for Mistral and Hermes. Training uses fresh target-generated continuations from separate public requests. The Mistral head was continued for four epochs from its epoch-two weights with a reset optimizer; the other new heads ran for six consecutive epochs. Hermes uses 11,609 positive-mask training records and 553 development records, with a 15,611-token draft vocabulary. Qwen3 uses its non-thinking chat template; Mistral and Hermes use their native chat templates; Qwen2.5 uses the plain-text task prompt. All four decoders receive identical prompt tokens within each target. All targets use the Table 1 base settings: forecast budget three, k=3k=3, MAIN depth two, SIDE depth one, and total draft budget eight. All four methods are evaluated at target temperatures 00 and 11, using greedy and stochastic target decoding, respectively; learned drafters retain temperature 00. The optimized backends enable CUDA graphs, ordered fused normalization, and expanded rotary/KV execution; SchemaFill also uses shared target-prefix reuse, hybrid prompt attention, and multi-branch tree drafting. Each drafter retains its training-time rotary configuration. Table 7 reports one full launch per method, target, and temperature, including request initialization and failed attempts. Model loading and one-time graph setup are outside request time. These single-run measurements are separate from the repeated-run estimates in Table 1. Table 7: Cross-target evaluation on BFCL450 at target temperatures 00 and 11. TPS is end-to-end throughput, and AL includes initialization. Speedup is the TPS ratio to AR at the corresponding temperature. Model Method Temperature =0=0 Temperature =1=1 TPS AL Speedup TPS AL Speedup Qwen2.5-7B AR 92.51 1.00 1.00× 92.22 1.00 1.00× EAGLE-3 199.89 2.84 2.16× 188.13 2.75 2.04× ToolSpec 239.87 3.27 2.59× 238.57 3.28 2.59× SchemaFill 371.48 6.42 4.02×4.02 × 356.61 6.16 3.87×3.87 × Qwen3-8B AR 79.78 1.00 1.00× 79.33 1.00 1.00× EAGLE-3 178.43 2.77 2.24× 174.45 2.77 2.20× ToolSpec 212.05 3.35 2.66× 209.24 3.34 2.64× SchemaFill 306.36 6.04 3.84×3.84 × 299.98 6.00 3.78×3.78 × Mistral-7B AR 86.83 1.00 1.00× 86.64 1.00 1.00× EAGLE-3 230.72 3.32 2.66× 227.80 3.32 2.63× ToolSpec 228.58 3.30 2.63× 231.13 3.37 2.67× SchemaFill 328.24 5.98 3.78×3.78 × 309.59 5.59 3.57×3.57 × Hermes-3-8B AR 83.50 1.00 1.00× 83.23 1.00 1.00× EAGLE-3 203.51 3.04 2.44× 196.91 3.01 2.37× ToolSpec 231.66 3.38 2.77× 229.31 3.40 2.76× SchemaFill 337.42 6.33 4.04×4.04 × 307.82 5.81 3.70×3.70 × Across the four targets, SchemaFill achieves 1.44×–1.55× ToolSpec’s throughput at temperature 00. At temperature 11, its throughput is 1.36×–1.90× that of EAGLE-3 and 3.57×–3.87× that of AR. These measurements use a separately trained drafter for each target and the same public-input forecasting and packed-verification procedure. 3, 5, 4