Paper deep dive

Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents

Abolfazl Younesi

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 76

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 10/13/2026, 4:38:21 AM

Summary

The paper introduces STEPGATE, an uncertainty-aware handoff framework that enables small language models (SLMs) to act as local agent controllers by selectively escalating challenging steps to stronger cloud models. By scoring each local action based on structured uncertainty signals (token confidence, schema validity, risk tiers), STEPGATE reduces remote inference costs and latency while maintaining high task success rates. Evaluations on Qwen2.5 models show that STEPGATE achieves performance comparable to strong-only models but with significantly fewer cloud actions (30% vs 100%) and lower token transmission.

Entities (9)

STEPGATE · framework · 98%Qwen2.5-1.5B · model · 95%Qwen2.5-7B · model · 95%BFCL · benchmark · 90%Uncertainty · concept · 90%FrugalGPT · method · 85%RouteLLM · method · 85%Self-Consistency · method · 85%Tool Errors · problem · 85%

Relation Signals (7)

STEPGATE → outperforms → Local-Only

confidence 95% · STEPGATE achieves 69.0% trajectory success... compared with 48.0% local-only

STEPGATE → uses → Uncertainty

confidence 95% · STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action

STEPGATE → escalatesto → Qwen2.5-7B

confidence 90% · selectively escalates challenging steps to a stronger model... Qwen2.5-7B backend

Qwen2.5-1.5B → islocalcontrollerfor → STEPGATE

confidence 90% · Qwen2.5-1.5B/7B pair... local SLM action

STEPGATE → reducescloudusage → Strong-Only

confidence 90% · using only 30.0% cloud actions, compared with... 100% cloud actions

BFCL → evaluates → STEPGATE

confidence 85% · On a 52-task held-out single-step BFCL-derived test split

FrugalGPT → issimilarto → STEPGATE

confidence 80% · FrugalGPT... show that selectively invoking stronger models can reduce cost

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail. Existing routers typically select a model once per query. However, agents expose sequential decision points whose difficulty dynamically changes based on intermediate observations. We propose STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action and selectively escalates challenging steps to a stronger model. On a 52-task held-out single-step BFCL-derived test split, the Qwen2.5-1.5B/7B pair attains 82.7% task success with 30.8% escalation, versus 67.3% local-only and 75.4% random escalation (which uses 33.8% escalation). In a separate multi-turn evaluation, STEPGATE achieves 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions, compared with 48.0%/70.5% local-only, 60.0%/78.2% random escalation, and 57.0%/77.1% query-level routing (strong-only achieves 82.0% trajectory success at 100% cloud actions). These results suggest that step-level escalation recovers a large share of the performance gap to the stronger Qwen2.5-7B backend at a matched cloud-action rate while transmitting fewer tokens remotely. However, our evaluation is limited to one model family, a single stronger backend, and scripted tasks. Furthermore, the test sets are small, multi-turn comparisons rely on paired intervals and statistical tests, and our risk tiers serve as research annotations rather than formal safety guarantees.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

72,565 characters extracted from source content.

Expand or collapse full text

Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents Abolfazl Younesi Department of Computer Engineering Sharif University of Technology Tehran, Iran abolfazl.younesi@ieee.org Abstract Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail. Existing routers typically select a model once per query. However, agents expose sequential decision points whose difficulty dynamically changes based on intermediate observations. We propose STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action and selectively escalates challenging steps to a stronger model. On a 52-task held- out single-step BFCL-derived test split, the Qwen2.5-1.5B/7B pair attains 82.7% task success with 30.8% escalation, versus 67.3% local-only and 75.4% random escalation (which uses 33.8% escalation). In a separate multi-turn evaluation, STEPGATE achieves 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions, compared with 48.0%/70.5% local-only, 60.0%/78.2% random escalation, and 57.0%/77.1% query-level routing (strong-only achieves 82.0% trajectory success at 100% cloud actions). These results suggest that step- level escalation recovers a large share of the performance gap to the stronger Qwen2.5-7B backend at a matched cloud-action rate while transmitting fewer tokens remotely. However, our evaluation is limited to one model family, a single stronger backend, and scripted tasks. Furthermore, the test sets are small, multi-turn comparisons rely on paired intervals and statistical tests, and our risk tiers serve as research annotations rather than formal safety guarantees. 1 Introduction Agentic systems repeatedly invoke language models for narrow decisions: selecting a tool, filling arguments, interpreting an observation, or choosing the next action. This execution pattern strengthens the case for SLMs: many calls are repetitive and specialized, and heterogeneous systems can reserve large models for the minority of steps that truly require them (Belcak et al., 2025). The difficulty is reliability. Function calling is structurally unforgiving: a nearly correct natural-language answer may still be useful, whereas one wrong function name, entity identifier, enum value, or destructive argument can fail an entire trajectory. BFCL consequently evaluates serial/parallel calls, abstention, stateful multi-turn behavior, and increasingly holistic agentic behavior rather than text quality alone (Patil et al., 2025). Model routing is a natural response to heterogeneous capabilities. FrugalGPT, RouteLLM, Hybrid LLM, AutoMix, and related cascades show that selectively invoking stronger models can reduce cost while retaining quality (Chen et al., 2024; Ong et al., 2025; Ding et al., 2024; Aggarwal et al., 2024; Yue et al., 2024). However, these systems mostly route at query level and optimize answer quality/cost. An agent instead exposes a sequence of decisions whose difficulty can change after every tool result. A request such as “find my meeting and move it if the room is 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: SLMs for Agentic Systems, Paris, France, 2026. arXiv:2610.07816v1 [cs.AI] 6 Oct 2026 Agent state s t user goal + history + tools Local SLM propose a t Action uncertainty φ(s t ,a t ) ˆp fail + risk Execute locally Local verify or retry Strong LLM handoff Tool / environment observation o t+1 low risk medium high risk next step Figure 1: STEPGATE routes each agent step. The local SLM first proposes an action. A lightweight gate estimates failure probability from action-structured uncertainty and risk. Only uncertain/high-risk steps disclose the required context to a stronger model. unavailable” may require an easy local search, a difficult disambiguation after observing multiple events, and a high-risk calendar mutation. Routing the entire request to the cloud wastes local capability. Committing the entire trajectory to an SLM compounds errors. We therefore ask: Can an SLM predict, before execution, when its next agent action is likely to fail and use that uncertainty to escalate only the necessary step? This question differs from generic LLM routing in three ways. First, the decision unit is an action, not a user query. Second, uncertainty is action-structured: tool identity, arguments, schema validity, execution state, and action risk provide signals unavailable in ordinary QA. Third, the objective is multi-dimensional: cloud calls and monetary cost matter, but so do local latency/energy, end-to-end task success, and the amount of private context disclosed remotely. Our contributions are fourfold. (1) We formulate action-level SLM–LLM handoff as a constrained risk minimization problem for tool-augmented agents, accounting for task failure, cloud invocation, latency/energy, and remote-context exposure. (2) We design an action-aware uncertainty representa- tion that fuses token confidence, schema and argument structure, sample consistency, and external risk tiers. (3) We implement a calibrated, adaptive gate that improves on local-only and query-level routing at matched cloud-action rates, is positive but not clearly separated from random escalation, and matches self-consistency routing with fewer cloud actions in our multi-turn setting. (4) We evaluate both single-step and multi-turn agent trajectories: at a 30.0% cloud-action budget, STEP- GATE 1 improves trajectory success from 48.0% (local-only) to 69.0% (versus 60.0% random and 57.0% query-level routing), with 55.0% of successful trajectories executing via mixed local/cloud collaboration . We report paired bootstrap intervals and McNemar tests for these comparisons and evaluate the risk features at an equal cloud budget (Sec. 4). 2 Related work Efficient heterogeneous inference. LLM cascades exploit heterogeneous cost and capability. FrugalGPT learns cascades over commercial models (Chen et al., 2024). RouteLLM learns strong- versus-weak routing from preference data (Ong et al., 2025). A hybrid LLM predicts query difficulty and enables quality/cost control (Ding et al., 2024). AutoMix combines self-verification with sequential routing (Aggarwal et al., 2024), and mixture-of-thought cascades use answer consistency for cost-efficient reasoning (Yue et al., 2024). Complementary systems work reduces inference cost through memory-aware serving and resource disaggregation: vLLM improves KV-cache management (Kwon et al., 2023), FlexGen targets constrained-memory inference (Sheng et al., 2023), while Splitwise and DistServe separate inference phases across heterogeneous resources (Patel et al., 2024; Zhong et al., 2024). Beyond server-side disaggregation, collaborative edge–cloud frameworks partition LLM execution under dynamic network and workload conditions (Younesi et al., 2025). These systems optimize where and how inference executes. STEPGATE instead decides which model should execute the next structured agent action. Its routing unit is the state-action pair(s t ,a t )rather than the full query. 1 https://zenodo.org/records/22308389 2 Table 1: Positioning against representative recent work. “Step” means routing can change after each tool observation. “Structured UQ” uses tool/argument/state signals. “Risk” accounts for action consequence. “Guarantee” denotes a finite-sample/distribution-free risk-control guarantee. MethodSmall→largeStepStructured UQ Tool risk Agent metric Guarantee FrugalGPT (Chen et al., 2024)✓– RouteLLM (Ong et al., 2025)✓– Hybrid LLM (Ding et al., 2024)✓– AutoMix (Aggarwal et al., 2024)✓–partial– UALA (Han et al., 2024)–partial– RACER (Hao et al., 2026)✓–✓ BFCL (Patil et al., 2025)–✓– STEPGATE (ours)✓– Uncertainty and selective prediction. Calibration is central when confidence controls deferral: modern neural predictors can be miscalibrated (Guo et al., 2017), while selective prediction formalizes risk-coverage trade-offs under abstention (Geifman and El-Yaniv, 2019). For language models, sampling disagreement provides a practical uncertainty signal: self-consistency aggregates multiple reasoning paths (Wang et al., 2023), SelfCheckGPT detects inconsistent generations (Manakul et al., 2023), and semantic entropy targets meaning-level uncertainty (Farquhar et al., 2024). Uncertainty has also guided information-seeking plans (Hu et al., 2024). Three further lines are closely related (Appendix B discusses each in depth): TokUR estimates token-level epistemic uncertainty for free- form reasoning via weight-perturbation ensembles (Zhang et al., 2026), motivating our simpler decode- time intrinsic features. UALA gates whether a ReAct-style agent consults external tools (Han et al., 2024), a decision complementary to STEPGATE’s choice of which executor runs an already-selected call; and RACER casts model selection asα-Valid Optimal Routing with a finite-sample, distribution- free misrouting-risk guarantee (Hao et al., 2026), which STEPGATE’s empirically calibrated gate does not yet provide (Appendix C). We adapt these principles to tool actions, where similar generations may differ by a single executable argument, leading to different external consequences. Agents and tool use. ReAct interleaves reasoning with environment actions (Yao et al., 2023), Toolformer studies learned API invocation (Schick et al., 2023), and Gorilla targets accurate selection and invocation from large API collections (Patil et al., 2024). TinyAgent demonstrates function calling with small models at the edge (Erdogan et al., 2024), while AgentDojo highlights that tool- using agents face reliability and security failures in realistic environments (Debenedetti et al., 2024). BFCL provides standardized function-calling evaluation spanning serial/parallel calls, abstention, and stateful agent behavior (Patil et al., 2025). MCP supplies an interoperable JSON-RPC/tool-schema protocol (MCP Contributors, 2025). These works motivate reliable tool execution. Our focus is calibrated pre-execution handoff from a local SLM to a stronger model. 3 Uncertainty-aware step-level handoff Let an agent interact with an environment forTsteps. At stept, local modelM s observes state s t = (x,h t ,T t )containing the user goalx, interaction historyh t , and available toolsT t , then proposes structured actiona t = (z t ,v t )with toolz t and argumentsv t . A routing policyπchooses r t ∈L,V,C: execute locally (L), perform a bounded local verification/retry (V), or escalate the step to strong cloud modelM l (C). The environment returns observationso t+1 and updates the state. Unlike query-level routing, π is re-evaluated after every observation. Action-aware uncertainty.We constructφ t = φ(s t ,a t )from six signal families. Token uncertainty summarizes length-normalized negative log-likelihood and entropy over the generated function name/arguments when logits are available. Tool margin measures separation between the chosen tool and plausible alternatives, using normalized log probability or a lightweight scorer. Schema signals include parse validity, required-field coverage, enum/type constraints, argument count, and schema complexity. Consistency comparesKlow-cost stochastic proposals by exact tool agreement and argument-level normalized agreement. A black-box variant can operate without logits, analogous in spirit to sampling-based uncertainty (Manakul et al., 2023). State signals encode trajectory length, previous tool errors, repeated actions, empty/ambiguous observations, and remaining retry budget. Finally, risk signals mark read-only versus state-changing tools, destructive/open-world hints when 3 SLM traces (s t ,a t ,o t+1 ) Counterfactual label would a t succeed? Features φ t UQ + schema + state + risk Calibrated gate g θ (φ t ) = ˆp fail Budget/risk policy L if p < τ L ; V if τ L ≤ p < τ C ; C otherwise Deployment constraints cloud budget B C latency/energy/privacy Pareto frontier success vs. cloud/latency/energy Figure 2: Training and deployment. Failure labels supervise only the gate. Thresholds are selected on validation data to satisfy a cloud-call/risk budget, enabling deployment-time trade-offs without retraining either language model. available, and a deployment-defined severity class. Appendix A.5 gives concrete worked examples for each risk tier. MCP annotations can provide interoperability hints, but because such annotations need not be trustworthy, the implementation must combine them with a local allowlist/policy rather than treating remote metadata as authoritative (MCP Contributors, 2025). A lightweight gateg θ (φ t ) ∈ [0, 1]estimatesp t = Pr(y t = 1 | φ t ), wherey t = 1denotes that executing the SLM proposal would be judged incorrect by the benchmark/environment. We fit logistic regression, gradient-boosted trees, and a small MLP; the primary model is selected based on validation AUROC and calibration error, not on downstream test performance. Temperature or isotonic calibration is fitted on a disjoint calibration split. This separation matters because a router that ranks failures well but is miscalibrated can violate an escalation budget. This calibration is purely empirical and carries no finite-sample misrouting bound, unlike distribution-free conformal-style calibration (Appendix C). Budget- and risk-aware decision rule.A binary threshold is insufficient because local verification may be much cheaper than cloud escalation. We minimize expected deployment loss L(π) =E[ℓ task + λ C C + λ L D + λ E E + λ P P],(1) whereCis cloud usage/cost,Dlatency,Elocal energy, andPprivacy exposure (e.g., normalized remote-context tokens). In a budgeted form, maximize task success subject toE[C] ≤ B C and optional latency/energy/privacy constraints. For an interpretable default policy, choose thresholds τ L < τ C : low estimated failure probability executes locally; the middle region requests one bounded local verification/retry; the high region escalates. Risk shifts thresholds monotonically: state- changing/destructive actions require greater confidence for local execution. Thresholds are chosen only on validation data and are empirical rather than backed by a distribution-free coverage guarantee. Figure 2 shows the training/deployment separation. Handoff context minimization. Escalation should not automatically transmit the entire local history. The cloud request includes the user subgoal, the current tool schemas needed for the decision, a compact state summary, the proposed SLM action and uncertainty evidence, and only the tool observations needed to resolve the step. We evaluate full-context and minimized-context handoffs. This turns remote-context exposure into a measurable system variable rather than a qualitative claim. Algorithm (see Algorithm 1). For each step, (i) obtain one SLM proposal and cheap intrinsic features, (i) if the gate is clearly belowτ L , execute, (i) otherwise acquire optionalK− 1samples and recompute consistency features, (iv) select local verification or cloud handoff using calibrated risk-adjusted thresholds, and (v) execute exactly one selected action and continue. Adaptive feature acquisition avoids paying for self-consistency on obviously easy steps. A maximum retry count prevents loops. The policy abstains or requests user clarification if neither the local nor the cloud proposal satisfies a deployment safety predicate. Local self-verification here is limited to sample- based consistency plus one bounded revision or retry. 4 Experimental results Setup.We evaluate Qwen2.5-0.5B, 1.5B, and 3B local models against a Qwen2.5-7B 4-bit strong model , which stands in for the cloud LLM. All results therefore concern one model family and one stronger backend. The original BFCL-derived function-calling experiment uses 100 tasks with a held-out 52-task test partition. We separate gate training/calibration and threshold selection from 4 Table 2: Main results: qwen2.5-1.5b (local) / qwen2.5-7b 4-bit (strong), real 100-task BFCL subset, 52 test tasks. Cloud rate budget-matched at 30% target (threshold/model chosen on calibration split only, applied once to test). “Wrong risky” = % of executed test actions that were both incorrect and risk-tier≥R2 (our taxonomy has no R3 action type in this task set). Realized cloud rates differ across routers and rate-matched variants are in Appendix A.1. SystemTask succ. ↑ Cloud %↓ Wrong risky↓ Latency↓ Remote tok. ↓ SLM only67.3015.40.7020 Strong only92.31003.82.255475.0 Random @ budget75.433.810.80.823158.5 Query-level router73.142.315.41.052266.2 Self-consistency82.738.57.70.829175.2 STEPGATE82.730.83.80.777134.5 that test split, and select the 30% cloud target before test evaluation. We additionally evaluate the 1.5B/7B configuration on 100 held-out multi-turn trajectories (mean 4.1 agent steps, up to 8) in a sandboxed calendar/email/file/search tool environment with scripted observations, where routing is reconsidered after every step. Appendix A.6 gives the full task composition, tool inventory, and exact baseline configurations (including the query-level router) needed to reproduce this split. Our multi-turn evaluation reports trajectory success, action success, cloud-action rate, and successful mixed local/cloud trajectories. Risk tiers remain research annotations. We do not assume the strong model is correct. All methods are run on the same tasks, so comparisons are paired. We report paired bootstrap 95% intervals (10,000 resamples over task IDs) and exact McNemar tests on binary trajectory success. Baselines and metrics. We compare local-only, strong-only, and random escalation matched to the realized cloud budget, a query-level router, self-consistency routing, and STEPGATE. Task success and cloud escalation are the primary focus. We additionally report wrong risk-tier-≥R2 actions, measured wall-clock latency, remote tokens, and failure-predictor discrimination/calibration. Random routing is averaged over five seeds where reported in the routing sweep. The oracle in supplementary results is diagnostic and non-deployable. We do not compare against risk-controlled routers with formal guarantees (e.g., RACER (Hao et al., 2026)), since adapting its single-turn, fixed-pool formulation into a step-level baseline is nontrivial (Appendix C). In the single-step test, realized escalation differs across routers (30.8% STEPGATE, 33.8% random, 42.3% query-level, 38.5% self-consistency). Appendix A.1 lists realized rates next to every result and adds rate-matched variants of random, query-level, and self-consistency routing at 30.8%. In the multi-turn test, all methods except self-consistency (38.0%) operate at 30.0% cloud actions. Main result.At the preselected 30% cloud budget for the 1.5B/7B pair, STEPGATE reaches 82.7% task success with 30.8% escalation (Table 2). This represents a +15.4 percentage point improvement over local-only and +7.3 points over random escalation (which uses more cloud steps: 33.8%), while reducing cloud escalations by 69.2% compared to strong-only. While strong-only reaches 92.3% task success at 100% remote execution, STEPGATE closes over 60% of this headroom while running nearly 70% of actions locally. Compared with the query-level router, STEPGATE achieves +9.6 points higher task success while escalating fewer actions (30.8% versus 42.3%) . This baseline also escalates more, so the comparison favors it rather than STEPGATE. Self-consistency matches STEPGATE’s 82.7% success rate but requires 38.5% escalation and has a higher wrong-risky rate (7.7% vs. 3.8%). On this 52-task split one task equals 1.9 points, so the +7.3 and +9.6 gaps correspond to about four and five tasks, and the individual 95% Wilson intervals overlap (Appendix A.1). We read the single-step result as supporting evidence and rest our main claim on the multi-turn comparison. STEPGATE executed a risky wrong action in only 3.8% of cases, compared to 15.4% for local-only and 10.8% for random escalation (2, 8, and 6 out of 52 steps). While the small sample size limits strong conclusions, this suggests uncertainty gating helps prevent high-consequence mistakes on this benchmark. Multi-turn handoff. Table 3 directly tests the step-level motivation. At 30.0% cloud actions, STEPGATE reaches 69.0% trajectory success, +21.0 points over local-only, +9.0 over random escalation at the same cloud-action rate, and +12.0 over query-level routing. Action success reaches 84.0%, with 55.0% of tasks successfully completed through collaborative local/cloud execution. By 5 Table 3: Multi-turn results for Qwen2.5-1.5B local and Qwen2.5-7B strong models, evaluated on the 100-task multi-turn split described in Appendix A.6. “Mixed success” is the share of trajectories that combined at least one local and one cloud action and still succeeded. ‡ Unpaired 95% Wilson interval for each method’s trajectory success (n=100; random escalation is the five-seed mean, so its interval is approximate). Paired differences are given in the text. MethodTraj. succ.↑95% CI ‡ Action succ.↑ Cloud actions↓ Mixed success↑ Local-only48.0% [38.5, 57.7]70.5%0.0%– Random escalation60.0% [50.2, 69.1]78.2%30.0%34.0% Query-level router57.0% [47.2, 66.3]77.1%30.0%0.0% Self-consistency66.0% [56.3, 74.5]82.6%38.0%48.0% STEPGATE69.0% [59.4, 77.2]84.0%30.0%55.0% Strong-only82.0% [73.3, 88.3]91.0%100.0%– contrast, self-consistency attains 66.0% trajectory success but consumes 38.0% cloud actions. While a pure cloud strong model attains 82.0% trajectory success, STEPGATE closes over 60% of this headroom while preserving local autonomy for 70% of steps. Paired analysis. Over the 100 identical tasks, the trajectory-success difference of STEPGATE (paired bootstrap 95% CI, exact two-sided McNemarp) is+21.0[11.0, 31.0],p=0.0003 versus local-only. Compared to random escalation, the difference is+9.0[−0.6, 18.8],p=0.108 (paired against each of the five seeds, with the mean interval reported). It achieves+12.0[2.0, 22.0],p=0.029 versus query-level routing, and+3.0[−5.0, 11.0],p=0.629 versus self-consistency, which uses 8 percentage points more cloud actions. Consequently, the gains over local-only and query-level routing exclude zero, whereas those over random escalation and self-consistency do not, so we treat the latter two comparisons as suggestive. The unpaired intervals in Table 3 overlap across the four routed methods, which is consistent with this interpretation. Model scale and routing curves.Figure 3 summarizes the measured success-escalation trade-off across all three local-model sizes. Local-only success rises from 55.8% (0.5B) to 67.3% (1.5B) and 88.5% (3B), while strong-only is 92.3%. At the 50%-target sweep, learned routing provides substantial headroom for 1.5B (84.6% logistic; 88.5% gradient-boosted trees), but yields diminishing returns for 3B (90.4% vs. an 88.5% local baseline and a 90.8% random escalation baseline). Consequently, learned routing offers the greatest leverage when the local model leaves a substantial recoverable-error margin. As local SLM capacity approaches the strong model’s ceiling, the value of selective handoff naturally saturates. 020406080100 Strong-model step escalation rate (%) 40 50 60 70 80 90 100 End-to-end task success (%) Oracle (non-deployable) Random (budget-matched) Entropy threshold Self-consistency threshold Learned gate (LogReg) Learned gate (GBT) 020406080100 Strong-model step escalation rate (%) 60 70 80 90 100 End-to-end task success (%) Oracle (non-deployable) Random (budget-matched) Entropy threshold Self-consistency threshold Learned gate (LogReg) Learned gate (GBT) 020406080100 Strong-model step escalation rate (%) 80 85 90 95 100 End-to-end task success (%) Oracle (non-deployable) Random (budget-matched) Entropy threshold Self-consistency threshold Learned gate (LogReg) Learned gate (GBT) Figure 3: Measured end-to-end task success versus realized strong-model step escalation rate for Qwen2.5-0.5B, 1.5B, and 3B local models. Error bars are the task-level 95% confidence intervals from the executed routing sweep. The dashed oracle is non-deployable. The plot exposes an important scale effect: routing offers substantially more headroom for weaker local models, whereas the 3B local baseline is already close to the strong-model ceiling. Calibration and ablations.For the 1.5B full feature set, logistic regression obtains AUROC 0.829, AUPRC 0.832, Brier 0.144, and ECE 0.212. Gradient-boosted trees achieve an AUROC of 0.854 and an ECE of 0.121. The ablation at the 30% target is not monotonic: removing risk metadata improves logistic AUROC to 0.886 and downstream task success to 86.5% at 34.6% realized escalation, while 6 removing consistency or schema features leaves task success at 82.7%. Token uncertainty alone reaches only 76.9% success at 28.8% escalation. The risk ablation is not budget-matched: the no-risk gate escalates two more of 52 steps (34.6% vs. 30.8%) and completes two more tasks (86.5% vs. 82.7%), so the gain is consistent with the extra budget alone and neither supports nor refutes a benefit of risk metadata. At an equal 30.8% budget (threshold re-tuned on validation data) the no-risk gate reaches 82.7% task success and a 5.8% wrong-R2+ rate, versus 82.7% and 3.8% for the full gate (Appendix A.1 and Table 5): task success is identical, and the wrong-R2+ difference is one step of 52, too small to be conclusive. Because risk tiers are meant to shift thresholds for consequential actions rather than raise average accuracy, we do not claim that risk features improve task success. We interpret the remaining non-monotonicity as feature interactions and possible overfitting on a 52-task split (Appendix C) rather than a clean per-family ranking, arguing for calibration and feature selection rather than assuming that every uncertainty signal helps. Efficiency.Measured median local latency increases with SLM size: 0.301s (0.5B), 0.702s (1.5B), and 1.152s (3B), with corresponding local peak memory of 990, 3003, and 6058 MiB. The measured strong-model median latency is 1.528s. For the main 1.5B operating point, STEPGATE’s measured end-to-end latency is 0.777s versus 2.255s for strong-only, and remote tokens are 134.5 versus 475.0 (a disclosure-volume proxy, not a privacy guarantee). Local energy measurements are hardware-specific and are reported only in the appendix. They should not be generalized to other accelerators. Reproducibility and statistical scope. The routing sweep reports task-level 95% bootstrap con- fidence intervals. For instance, the 1.5B local-only estimate is 67.3% [53.8, 80.8] versus 92.3% [84.6, 98.1] for strong-only. Single-step differences between routers are evaluated on 52 tasks and interpreted descriptively (Appendix A.1). In contrast, multi-turn comparisons follow the paired proce- dure described above. Neither setting supports generalized claims beyond the evaluated benchmarks. The empirical findings reflect the evaluation of in-distribution benchmark tasks, scripted multi-turn observations, and the specified open-weight model pairings. 5 Conclusion We presented STEPGATE, an uncertainty-aware, step-level handoff mechanism for SLM agents. On the 52-task single-step benchmark, the 1.5B/7B pair improves task success from 67.3% (local- only) to 82.7% with only 30.8% escalation. In multi-turn evaluations, STEPGATE attains 69.0% trajectory success and 84.0% action success using a 30.0% cloud budget, outperforming local- only (48.0%/70.5%) and random escalation (60.0%/78.2%), with 55.0% of trajectories successfully completed via mixed local/cloud execution. In comparison, strong-only reaches 82.0% trajectory success at the expense of 100% remote execution. These results suggest that selective, step-level handoff offers an effective efficiency–reliability trade-off in this setting. However, they do not establish generality beyond one model family, one stronger backend, and scripted tasks. Furthermore, the risk tiers are empirical annotations rather than formal safety guarantees, remote-token counts do not ensure privacy, and the individual contribution of the risk features remains unestablished. Extending our empirical framework with finite-sample, distribution-free risk guarantees represents a compelling direction for future research. References P. Belcak, G. Heinrich, S. Diao, et al. Small Language Models are the Future of Agentic AI. arXiv:2506.02153, 2025. S. G. Patil, H. Mao, F. Yan, et al. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:48371–48392, 2025. I. Ong, A. Almahairi, V. Wu, et al. RouteLLM: Learning to Route LLMs from Preference Data. ICLR, 2025. D. Ding, A. Mallick, C. Wang, et al. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. ICLR, 2024. 7 P. Aggarwal, A. Madaan, A. Anand, et al. AutoMix: Automatically Mixing Language Models. NeurIPS, 37, 2024. L. Chen, M. Zaharia, and J. Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research, 2024. M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao. Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning. ICLR, 2024. P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini. Splitwise: Efficient Gen- erative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Sympo- sium on Computer Architecture (ISCA), p. 118–132, 2024. doi:10.1109/ISCA59077.2024.00019. A. Younesi, A. Shabrang Maryan, E. Oustad, Z. Najafabadi Samani, M. Ansari, and T. Fahringer. Splitwise: Collaborative Edge–Cloud Inference for LLMs via Lyapunov-Assisted DRL. In Pro- ceedings of the 18th IEEE/ACM International Conference on Utility and Cloud Computing (UCC 2025), Article 12, p. 12:1–12:11, 2025. doi:10.1145/3773274.3774267. Y. Zhong, S. Liu, J. Chen, et al. DistServe: Disaggregating Prefill and Decoding for Goodput- Optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), p. 193–210, 2024. W. Kwon, Z. Li, S. Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, p. 611–626, 2023. doi:10.1145/3600006.3613165. Y. Sheng, L. Zheng, B. Yuan, et al. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. ICML, PMLR 202:31094–31116, 2023. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On Calibration of Modern Neural Networks. ICML, PMLR 70:1321–1330, 2017. Y. Geifman and R. El-Yaniv. SelectiveNet: A Deep Neural Network with an Integrated Reject Option. ICML, PMLR 97:2151–2159, 2019. X. Wang, J. Wei, D. Schuurmans, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR, 2023. S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625–630, 2024. doi:10.1038/s41586-024-07421-0. P. Manakul, A. Liusie, and M. Gales. SelfCheckGPT: Zero-Resource Black-Box Halluci- nation Detection for Generative Large Language Models. EMNLP, p. 9004–9017, 2023. doi:10.18653/v1/2023.emnlp-main.557. Z. Hu, C. Liu, X. Feng, et al. Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in LLMs. NeurIPS, 37, 2024. S. Yao, J. Zhao, D. Yu, et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR, 2023. T. Schick, J. Dwivedi-Yu, R. Dessi, et al. Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS, 36, 2023. S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez. Gorilla: Large Language Model Connected with Massive APIs. NeurIPS, 37, 2024. L. E. Erdogan, N. Lee, S. Jha, et al. TinyAgent: Function Calling at the Edge. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 80–88, 2024. doi:10.18653/v1/2024.emnlp-demo.9. E. Debenedetti, J. Zhang, M. Balunovic, et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS Datasets and Benchmarks Track, 37, 2024. 8 Model Context Protocol Contributors. Model Context Protocol Specification, Revision 2025-11-25. 2025. https://modelcontextprotocol.io/specification/2025-11-25/. J. Han, W. Buntine, and E. Shareghi. Towards Uncertainty-Aware Language Agent. In Find- ings of the Association for Computational Linguistics: ACL 2024, p. 6662–6685, 2024. doi:10.18653/v1/2024.findings-acl.398. S. Hao, H. Zeng, H. Wei, and B. Jing. RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models. arXiv:2603.06616, 2026. T. Zhang, H. Shi, Y. Wang, et al. TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning. ICLR, 2026. V. Vovk, A. Gammerman, and G. Shafer. Algorithmic Learning in a Random World. Springer, New York, 2005. doi:10.1007/b106715. A Technical Appendix A.1 Budget-matched single-step comparison and equal-budget risk ablation Table 4 lists, for the 1.5B/7B pair on the 52-task single-step test split, the realized escalation rate next to each result, with unpaired 95% Wilson intervals computed from the reported counts (one task = 1.9points). Rows marked “matched” re-tune the router’s threshold (or, for random, its escalation probability) on validation data so that the realized test escalation equals STEPGATE’s 30.8% (16 of 52 steps). We retune the no-risk gate the same way. “As reported” rows are the measured values of Table 2 and the original ablation. Success–escalation curves for each router over the 10–50% range should be read together with this table. Wrong-R2+ is the executed wrong risky-action rate on actions annotated R2 or above (a benchmark annotation, not a real-world harm rate). Table 4: Single-step test (52 tasks), Qwen2.5-1.5B/7B. † Five-seed mean; interval approximate. “As reported” rows use the thresholds of the original experiment. “matched” rows are re-tuned on validation data to 30.8% realized escalation. MethodRealized esc. Task success 95% Wilson CI Wrong R2+ Local-only0.0%67.3%[53.8, 78.5]15.4% Random escalation, as reported † 33.8%75.4%[62.2, 85.1]10.8% Random escalation, matched † 30.8%74.7%[61.5, 84.5]11.2% STEPGATE (full)30.8%82.7%[70.3, 90.6]3.8% STEPGATE no-risk, as reported34.6%86.5%[74.7, 93.3]3.8% STEPGATE no-risk, matched30.8%82.7%[70.3, 90.6]5.8% Query-level, as reported42.3%73.1%[59.7, 83.2]15.4% Query-level, matched30.8%69.2%[55.7, 80.1]15.4% Self-consistency, as reported38.5%82.7%[70.3, 90.6]7.7% Self-consistency, matched30.8%78.8%[66.0, 87.8]9.6% Strong-only100.0%92.3%[81.8, 97.0]3.8% This appendix records implementation details and supporting measurements omitted from the main text. Single-step feature ablation.The detailed ablation table is moved here to prioritize the multi-turn trajectory result in the main paper. A.2 Formal decision problem and routing rule At agent stept, lets t = (x,h t ,T t )denote the agent state containing the user goalx, compact trajectory historyh t , and candidate tool schemasT t . The local SLM samples a proposed structured actiona t ∼ p S (·| s t ). LetY t ∈0, 1denote whether executinga t would fail under the environment evaluator. A lightweight gateg θ estimates failure probabilityq t = P(Y t = 1| φ t )from action-aware featuresφ t = φ(s t ,a t ). Based onq t , action risk tierr t , and remaining resource budgetb t , the routing policyπ(z t | q t ,r t ,b t )selects an actionz t ∈L,V,Ccorresponding to local execution (L), 9 bounded local verification (V), or cloud escalation (C). The optimal deployment policy solves the multi-objective loss minimization: min π E π [ℓ task + λ C C + λ D D + λ E E + λ P P],(2) whereCdenotes cloud invocations,Dlatency,Elocal energy, andPremote context exposure. For deployment, we instantiate an interpretable thresholding policy defined by risk- and budget- conditioned cutoffs τ L (r t ,b t ) < τ C (r t ,b t ): z t =    Lif q t ≤ τ L (r t ,b t ), Vif τ L (r t ,b t ) < q t < τ C (r t ,b t ), Cif q t ≥ τ C (r t ,b t ). (3) Thresholds are calibrated exclusively on validation trajectories. For state-changing or high-risk actions (r t ≥ R2), thresholds shift conservatively toward verification and escalation, while deterministic external authorization policies retain the ability to override routing decisions. Adaptive feature acquisition.Cheap features are computed from the first local decode. Expensive features such asK-sample consistency are acquired only when the cheap gate score lies in an ambiguity band[τ − A ,τ + A ]. This prevents uncertainty estimation from consuming more compute than it saves. All thresholds, K, and ambiguity bands are selected without test labels. SLM proposal a t Cheap features entropy, margin, schema Stage-1 gate q (1) t local execute Acquire K samples + state consistency Calibrated gate q (2) t local verify cloud handoff confident ambiguous Figure 4: Adaptive uncertainty acquisition. Expensive self-consistency is computed only for am- biguous actions. This appendix figure makes explicit which uncertainty signals incur additional local inference. Note that this diagram covers only sample-based consistency and a single bounded revision. Reference pseudocode.Algorithm 1 gives the complete per-step decision procedure summarized in Section 3 (“Algorithm”), spelling out the exact branching between local execution, the ambiguity-band retry/verification path, and cloud escalation, together with the safety rule that governs authorization. A.3 Feature definitions, calibration, and labels The frozen implementation should expose each feature separately so that ablations can be reproduced. Intrinsic features include mean token negative log-likelihood over the tool name and arguments, maximum token entropy, sequence length, and the probability margin between the top two tool names when tokenization permits; this family is conceptually related to token-level epistemic uncertainty Table 5: Ablation of the calibrated failure predictor (logistic regression), qwen2.5-1.5b, real BFCL subset (n=52 test tasks). Cloud budget fixed at 30% (threshold chosen on calibration split only). The risk gate escalates 2 more of 52 steps than the full gate. The last-but-one row re-tunes its threshold to the same 30.8% budget (AUROC/ECE are unchanged because the scoring model is the same). Gate featuresAUROC↑ ECE↓ Task succ. ↑ Cloud % All (STEPGATE)0.8290.21282.730.8 − consistency0.8320.23182.728.8 − schema/arguments0.8290.24182.730.8 − risk0.8860.11986.534.6 − risk, matched budget0.8860.11982.730.8 Token uncertainty only0.5650.07876.928.8 10 Algorithm 1 STEPGATE inference at one agent step Require: state s t , local model S, strong model M , gate g θ , thresholds τ L ,τ C , maximum retry R Ensure: executed action, updated state s t+1 1: Decode one structured proposal a t from S and compute cheap features φ c t 2: Compute q (1) t = g θ (φ c t ) 3: if risk-adjusted q (1) t ≤ τ L then 4:Execute a t 5: else if q (1) t lies in the ambiguity band then 6:Sample up to K− 1 additional local proposals 7:Compute tool/argument agreement and update q t 8:if q t < τ C and retry budget remains then 9:Perform one bounded local verification/revision 10:Re-score the revised action 11:else 12:Construct a minimized handoff context and query M 13:Validate the returned structured action against the schema and deployment policy 14:end if 15: else 16:Construct a minimized handoff context and query M 17:Validate the returned structured action against the schema and deployment policy 18: end if 19:Execute at most one authorized action, record the observation and routing telemetry, and advance to s t+1 Safety rule: schema validity or low uncertainty never grants authorization; application policy may force abstention or human confirmation. estimators such as TokUR (Zhang et al., 2026), though we use plain decode-time statistics rather than weight-perturbation ensembles. Structural features include JSON/schema validity, number and type of required arguments, enum constraints, nesting depth, and whether argument values are copied from or grounded in the observed state. Consistency features compareKstochastic proposals using exact tool agreement, normalized argument agreement, and execution-equivalence where a deterministic sandbox evaluator is available. State features include trajectory depth, number of prior failures, whether the current action depends on a previous observation, and candidate-tool count. Risk features are externally supplied and never inferred solely from the model. For each local proposal, the binary failure label is computed by the benchmark/environment before any cloud correction. Multi-turn tasks additionally retain trajectory-level success. The gate is trained only on local-model proposals from a calibration/training partition, preventing labels from strong-model outputs from leaking into the failure predictor. We recommend logistic regression and gradient-boosted trees as transparent baselines before any neural gate. Calibration should compare temperature scaling, isotonic regression, and uncalibrated scores using ECE and Brier score. We freeze the selected calibrator on validation data. We note that this calibration protocol targets good ranking and low average ECE, not a guaranteed per-deployment bound on misrouting risk. Readers wanting the latter should consult finite-sample conformal risk-control procedures such as RACER’s α-VOR calibration (Hao et al., 2026) or classical conformal prediction (Vovk et al., 2005), which we discuss as an open extension in Appendix C. A.4 Background: conformal prediction and risk control For readers unfamiliar with the guarantee referenced in Appendix C and Appendix B: conformal prediction constructs prediction sets that provably contain the true label with a user-specified proba- bility, using only exchangeability of calibration and test data and no distributional assumptions on the underlying model (Vovk et al., 2005). Conformal risk control generalizes this from set coverage to bounding the expected value of any monotone loss function by calibrating a single threshold on a held-out calibration set. RACER’sα-VOR calibration (Appendix B) is an instance of this idea specialized to model routing: the “loss” is whether the selected model set excludes every acceptable model, and the guaranteed bound holds for any finite calibration sample size, not just asymptotically. 11 Table 6: Feature taxonomy and reporting overview. “Extra decode” identifies signals that incur additional local generation and therefore must be accounted for in latency/energy. FamilyRepresentative signalsExtra decode? Intrinsictoken NLL, entropy, tool-choice margin, output lengthNo Structural schema validity, argument count/type, enum/nesting complexity No Consistencytool agreement, argument agreement, execution equiva- lence Yes State/history † trajectory depth, previous errors, observation depen- dence No Riskread-only/state-changing/destructive/external side effectNo † Evaluated exclusively within multi-turn sequential trajectories. In contrast, STEPGATE’s gate calibration (temperature/isotonic scaling; Appendix B) targets good average calibration (low ECE/Brier) on a validation split, which is a different and weaker property: it does not bound the probability of misrouting on any particular deployment sample, and the current implementation does not verify or rely on an exchangeability assumption across trajectory steps, which conformal-style guarantees would require. A.5 Risk taxonomy: concrete examples Section 3 and Appendix C describe the risk tiers R0–R3 as research annotations rather than a security standard. Table 7 gives representative tool calls for each tier, drawn from the calendar/email/file/search tool families used in our multi-turn environment (Appendix A.6), to make the annotation rule concrete for readers who wish to reproduce or extend it. Table 7: Worked examples of the risk taxonomy used to annotate tool calls. Tier assignment follows reversibility and blast radius, not tool name alone: two calls to the same underlying tool can receive different tiers depending on arguments (e.g., a narrowly scoped read vs. a bulk export). TierDescriptionExample calls R0Read-only, no external mu- tation search_events(range=...);get_email(id=...); list_files(folder=...); web_search(query=...) R1 Reversible,low-impact mutation create_draft_email(...)(notsent); add_calendar_event(private=True);rename_file(...) within the user’s own workspace R2Externallyvisibleor financially/privacy- relevant mutation send_email(to=external_recipient, ...); share_file(recipient=external, permission=view); reschedule_meeting(notify_attendees=True) R3 Destructive or difficult-to- reverse action delete_event(recurring=True) ; permanently_delete_file(...); bulk_delete_emails(filter=...); cancel_meeting(notify_all=True)for an externally or- ganized event Two practical notes follow from these examples. First, tier assignment is a function of the call (tool plus bound arguments), not just the tool name, since the same endpoint can be R0 in a narrowly scoped read and R2–R3 when arguments broaden its blast radius (e.g., a filtered vs. unfiltered bulk delete). Second, we fixed tier boundaries in our environment before test evaluation using the same deterministic rule set described in Section 4 of the main text and did not tune them against downstream task success to avoid the annotation itself becoming a source of test-set leakage. A.6 Multi-turn environment and baseline specification This appendix specifies the multi-turn evaluation referenced in Section 4 (Setup) to support replication. 12 Environment and task composition.The multi-turn split contains 100 held-out tasks executed in a sandboxed, scripted tool-use environment (not a live API), spanning four tool families: calendar (7 tools, e.g., search/create/update/delete events), email (6 tools, e.g., search/read/draft/send), file storage (5 tools, e.g., list/read/rename/share/delete), and web/knowledge search (2 tools). Each task specifies a natural-language user goal, an initial environment state (populated calendar/inbox/file tree), and a deterministic evaluator that checks final environment state and/or the sequence of executed actions against task-specific success criteria. Trajectories average 4.1 agent steps (min 1, max 8); a step ends in an executed tool call, and the trajectory ends when the evaluator’s goal condition is met, the agent abstains, or the step budget (8) is exhausted. Task difficulty varies along three axes we tag but do not stratify results by in the main text: whether disambiguation is required after an observation (e.g., multiple matching events), whether at least one R2/R3 action is required to complete the goal, and whether an early tool result invalidates the agent’s initial plan (forcing re-planning). Tasks were authored to cover all four tool families and all four risk tiers (Appendix A.5) at least twice each; no task or its paraphrase appears in the gate’s training or calibration data. Baseline configuration. All compared methods share the same underlying Qwen2.5-1.5B local and Qwen2.5-7B (4-bit) strong models, the same tool schemas, and the same evaluator; they differ only in the routing decision at each step. Local-only always executes the 1.5B model’s proposal. Strong-only always executes the 7B model’s proposal. Random escalation draws the routing decision i.i.d. per step from a Bernoulli distribution whose parameter is tuned so the realized cloud-action rate matches STEPGATE’s 30.0% (results averaged over five seeds). Query-level router computes the gate’s features once from the initial user goal and tool schemas (i.e., att = 0, before any observation) and commits to local or cloud execution for every step of the trajectory; it cannot react to later observations, which is the central contrast with STEPGATE. Self-consistency routing re-samples K = 5local proposals at every step and escalates when tool-choice agreement across samples falls below a fixed threshold tuned on the calibration split. STEPGATE is the full step-level, risk-adjusted gate described in Section 3, re-evaluated after every observation. All methods use the same minimized handoff-context construction (Section 3, “Handoff context minimization”) when escalating, so the latency/token differences in Section 4 are attributable to the escalation policy rather than to context size. Statistical scope.With 100 tasks, a 21-point gap corresponds to 21 additional successful trajectories out of 100. We report paired outcomes on identical task IDs: a paired bootstrap (10,000 resamples over tasks) for the difference in trajectory success, and an exact two-sided McNemar test on the discordant pairs. Random escalation is paired against each of its five seeds, and the mean interval and per-seedp-values are reported. Per-task outcomes and the analysis script are released with the artifact. The unpaired Wilson intervals in Table 3 are shown for reference only and are not a substitute for the paired test. A larger, pre-registered multi-turn benchmark and stratification by the three difficulty tags above remain future work (Appendix C). A.7 Dataset construction, splits, and leakage prevention The primary benchmark should be pinned to an immutable BFCL release or commit and evaluated using its official evaluator wherever possible. Report every included category separately and state all exclusions. If an execution-safe risk subset is created, annotation rules must be deterministic and released. Appendix A.5 gives the concrete tier examples used in this work. A suggested taxonomy is R0: read-only/no external mutation. R1 is a reversible low-impact mutation. R2 is an externally visible or financially/privacy-relevant mutation, and R3 is a destructive or difficult-to-reverse action. The taxonomy is a research annotation, not a security standard. Calibration, threshold selection, and final evaluation must be separated by task IDs and, where feasible, by tool families to test transfer to unseen tools. No final-test outcome may influence feature engineering, calibrator choice, thresholds, prompt selection, or retry count. For distribution- shift experiments, construct fixed transformations before evaluation: paraphrased tool descriptions, increased distractor-tool count, reordered schemas, longer trajectories, and held-out tool families. Preserve exact transformation seeds and generated artifacts. We flag distribution shift and cross- environment transfer (e.g., to ToolSandbox or WebArena-style tasks) as unevaluated in this work. 13 A.8 Prompts, handoff packet, and tool safety The artifact should contain verbatim system prompts and chat templates. The local actor prompt should request exactly one next action or explicit abstention, not hidden free-form chain-of-thought. The verifier prompt, if used, should receive the user subgoal, proposed action, relevant schema, and the minimal observation needed to judge correctness. Evaluation should score observable actions rather than private reasoning traces. The minimized cloud packet should be constructed from explicit fields: (1) current user subgoal; (2) relevant tool names and schemas; (3) compact state summary; (4) proposed local action; (5) uncertainty/risk metadata; and (6) only observations required to resolve the decision. Report both raw remote token count and a field-level disclosure inventory. Sensitive values should be redacted or tokenized whenever the strong model does not need their literal content. Full-context handoff is retained only as an upper-bound baseline for accuracy. All state-changing tools used in experiments should execute in mocks, sandboxes, or disposable accounts. The gate must not be treated as an access-control system. Before tool execution, an independent deterministic layer should enforce schema validation, allowlists, rate limits, and any required human confirmation. When used, MCP tool annotations are treated as untrusted hints rather than authorization. A.9 Baselines and budget matching Every routing method must be evaluated at multiple operating points. For a fair comparison, match the number of strong-model calls (or strong-model tokens, in a secondary analysis) by choosing each baseline threshold on validation data. We repeat random escalation across multiple seeds. Query-level routing makes a single decision before the first tool action and retains it throughout the trajectory (Appendix A.6 gives the exact procedure used in our multi-turn experiment). Step-level baselines may be re-evaluated after each observation. The oracle sees local failure labels and is reported only as an unattainable upper bound. Recommended operating points are approximately 0, 10, 25, 50, 75, and 100% cloud-step usage where attainable. Curves should report all attainable points rather than a single favorable threshold. If API rate limits or costs prevent dense sweeps, cache local proposals first and pre-register the selected operating points before strong-model evaluation. A.10 Metrics, uncertainty intervals, and statistical analysis Task success is the primary effectiveness metric. Cloud-step rate is the primary resource metric. Secondary metrics include per-action correctness, failure-to-abstain rate, risky wrong-action rate, strong/local token counts, end-to-end latency, peak memory, energy when measurable, and remote- context tokens. Measure router discrimination with AUROC and AUPRC, and calibration with Brier score and ECE. Selective behavior is measured with risk-coverage and success-cloud curves. For task-level proportions, report paired bootstrap 95% confidence intervals over task IDs with at least 10,000 resamples in the final artifact. When comparing two routers on the same binary task outcomes, use McNemar’s test or a paired randomization test and report the test definition. For learned gates, repeat training/calibration over at least three seeds and distinguish seed variability from task-resampling uncertainty. Do not interpret overlapping/non-overlapping confidence intervals as a substitute for a paired test. Report absolute effects alongside relative improvements. Readers seeking a router with a provable, rather than bootstrap-estimated, bound on misrouting probability should consider adapting finite-sample risk-control procedures such as RACER (Hao et al., 2026) or conformal risk control (Vovk et al., 2005) to the step-level, stateful setting. Pareto reporting. A router is practically useful only if it improves the frontier. For each method, plot task success against cloud-step rate and separately risky wrong-action rate against cloud-step rate. Report the area under the success-cloud curve only as a summary. The full curve remains the primary focus. A budget-matched claim is supported only when the compared methods use the same cloud budget within a declared tolerance. 14 A.11 Efficiency and hardware measurement protocol Hardware experiments should freeze the device, power mode, OS, drivers, inference runtime, model revision, quantization, context length, batch size, and decoding configuration. Perform a declared warm-up, then measure complete agent steps rather than model kernels alone. Report median and tail latency in addition to mean latency. If energy counters are available, state their source, sampling interval, integration procedure, idle-power treatment, and whether tool execution is included. API latency should be measured separately from local latency because network conditions are not reproducible across sites. To avoid misleading efficiency claims, account for the additional local samples used by self- consistency and any verifier calls. The principal resource table should therefore include locally generated tokens, strong-model generated/input tokens, total model invocations, and wall-clock time. We report monetary cost only with the exact price schedule and date used for calculation. A.12 Additional measured results Table 8 reports the measured 30%-budget results across the three local-model sizes. These measure- ments characterize the evaluated in-distribution setting; robustness to unseen tool families, schema paraphrases, distractor growth, and longer trajectories is outside this study’s empirical scope. Table 8: Per-model results at 30% fixed cloud budget, real BFCL subset. Three local/strong pairs were evaluated (a single strong model and three SLM sizes). Local / strong pairParamsTask succ.Cloud %Risky errorsLatency qwen2.5-0.5b / qwen2.5-7b0.5B73.126.93.80.351 qwen2.5-1.5b / qwen2.5-7b1.5B82.730.83.80.777 qwen2.5-3b / qwen2.5-7b3B90.426.93.81.153 020406080100 Local coverage (% steps executed without escalation) 0 10 20 30 40 Error rate among locally executed steps (%) qwen2.5-0.5b (entropy-ranked) qwen2.5-1.5b (entropy-ranked) qwen2.5-3b (entropy-ranked) Figure 5: Selective risk under entropy-ranked local coverage. The error rate among locally executed steps rises with coverage, most sharply for the 0.5B model and least for the 3B model. This plot evaluates ranking behavior; it is not the learned-gate routing curve. Calibration and efficiency details.Across local-model sizes, the full logistic gate reaches AUROC 0.729/0.829/0.899 for 0.5B/1.5B/3B, respectively. Full-feature ECE is 0.154/0.212/0.135. Measured efficiency statistics are shown below; energy is reported as measured for this hardware only. 15 0.00.20.40.60.81.0 Predicted failure probability 0.0 0.2 0.4 0.6 0.8 1.0 Observed failure frequency Perfect calibration Learned gate (LogReg) 0.00.20.40.60.81.0 Predicted failure probability 0.0 0.2 0.4 0.6 0.8 1.0 Observed failure frequency Perfect calibration Learned gate (LogReg) 0.00.20.40.60.81.0 Predicted failure probability 0.0 0.2 0.4 0.6 0.8 1.0 Observed failure frequency Perfect calibration Learned gate (LogReg) Figure 6: Reliability diagrams for the learned logistic failure gate. From left to right, the panels show Qwen2.5-0.5B, Qwen2.5-1.5B, and Qwen2.5-3B. The dashed diagonal denotes perfect calibration. The finite calibration/test sample sizes make individual bins noisy, so ECE and Brier scores are reported alongside these diagrams rather than inferred visually. 0100200300400 Mean strong-model tokens used per task (measured, avg input+output) 55 60 65 70 75 80 85 90 95 Task success (%) always local entropy threshold learned gate logistic regression self consistency threshold oracle nondeployable entropy threshold learned gate logistic regression random budget matched avg5seeds entropy threshold learned gate logistic regression always strongself consistency threshold always local oracle nondeployable learned gate logistic regression entropy thresholdlearned gate logistic regression random budget matched avg5seeds entropy threshold entropy threshold learned gate logistic regressionalways strongself consistency thresholdself consistency threshold always local oracle nondeployable learned gate logistic regressionentropy thresholdentropy thresholdlearned gate logistic regression random budget matched avg5seeds learned gate logistic regressionentropy thresholdalways strongself consistency thresholdself consistency threshold qwen2.5-0.5b qwen2.5-1.5b qwen2.5-3b Figure 7: Measured task success versus mean strong-model tokens per task (input+output). Each point is an executed routing configuration. The dashed connections are visual guides, not fitted performance models. The figure shows that token efficiency and task success must be considered jointly and that the preferred operating point depends on local model size. A.13 Diagnostic plots from the executed runs The diagnostic plots make four complementary aspects of the routing behavior visible. First, Figure 5 tests whether entropy is useful for ranking local actions by difficulty, independently of the learned gate. As the fraction of actions retained for local execution increases, the error rate among those actions also increases. The increase is steepest for the 0.5B model and shallowest for the 3B model, indicating that entropy-based selection separates easy from difficult actions more reliably for the weaker local model. This is a ranking result, however, and should not be interpreted as STEPGATE’s success-escalation curve. Second, Figure 6 assesses whether the logistic gate’s predicted failure probabilities are numerically meaningful, rather than merely correctly ordered. Curves close to the dashed diagonal indicate that actions assigned a probability nearpfail at roughly ratepwithin that bin; systematic deviations show over- or under-confidence. The sparse bins should be read cautiously because the calibration and test sets are small. The corresponding ECE values of 0.154, 0.212, and 0.135 for the 0.5B, 1.5B, and 3B models, respectively, and the reported Brier scores provide the quantitative comparison. 16 qwen2.5-0.5bqwen2.5-1.5b qwen2.5-3b 0 1 2 3 4 Per-call latency (s) qwen2.5-0.5bqwen2.5-1.5b qwen2.5-3b 0 1000 2000 3000 4000 5000 6000 Mean measured peak GPU memory (MiB) Local (SLM) Strong (7B, 4-bit) qwen2.5-0.5bqwen2.5-1.5b qwen2.5-3b 0 100 200 300 400 500 600 700 800 Per-call GPU energy (J, NVML power-draw integration) Figure 8: Measured per-call latency, peak GPU memory, and NVML power-draw integration for the local SLMs and the shared 7B 4-bit strong model. These measurements are specific to the hardware/runtime in use. They are not hardware-independent efficiency claims. 051015 Number of failing steps (test+calibration) parallel_function_checker_no_order:cannot_find_match value_error:list/tuple value_error:others simple_function_checker:missing_optional simple_function_checker:missing_required irrelevance_error:decoder_success_but_should_abstain malformed_output value_error:string parallel_function_checker_no_order:wrong_count 0.02.55.07.510.012.5 Number of failing steps (test+calibration) simple_function_checker:missing_optional value_error:string value_error:others parallel_function_checker_no_order:wrong_count irrelevance_error:decoder_success_but_should_abstain malformed_output 0246 Number of failing steps (test+calibration) multiple_function_checker:wrong_count simple_function_checker:missing_optional parallel_function_checker_no_order:cannot_find_match malformed_output irrelevance_error:decoder_success_but_should_abstain parallel_function_checker_no_order:wrong_count 5060708090100 Recovery rate if escalated to strong model (%) 406080100 Recovery rate if escalated to strong model (%) 20406080100 Recovery rate if escalated to strong model (%) Figure 9: Failure decomposition and escalation recoverability for the Qwen2.5-0.5B local model (left), Qwen2.5-1.5B local model (middle), and Qwen2.5-3B local model (right). The categories include irrelevance/abstention errors, malformed outputs, parallel-call count or matching errors, missing optional or required arguments, and value errors involving strings, lists/tuples, or other values. Red bars show observed local failure counts on the test and calibration sets, while the green line shows the recovery rate when the same failing step is escalated to the strong model. Recovery varies sharply by failure type, underscoring why local failure probability alone is not equivalent to the expected escalation benefit. Sparse categories should be interpreted cautiously. Third, Figure 7 shows the resource-quality trade-off in terms of mean strong-model input plus output tokens per task. Moving right means more remote inference, while moving upward means higher task success. The preferred point is therefore not necessarily the highest-success point: a configuration is attractive when it reaches a given success level with fewer strong-model tokens, and the best trade-off can differ by local-model size. The dashed lines only connect measured configurations. They are not interpolated or fitted performance curves. Finally, Figure 8 and Table 9 report the corresponding execution costs. They show how local-model size changes per-call latency and peak memory, and how those measurements compare with the shared 7B strong model. The NVML power integration is hardware-specific, not a portable energy estimate. Figure 9 adds a failure-type view: recovery after escalation varies by category, so a high local failure probability alone does not imply the same expected benefit from cloud handoff for every action. Together, these plots are descriptive diagnostics generated from the executed outputs rather than additional experiments. The selective-risk curve is entropy-ranked. The reliability diagrams use the learned logistic gate. The Pareto plot uses measured strong-model tokens per task, and the failure analysis combines test and calibration failures, as indicated on its axis. These distinctions keep the figures from being read as stronger evidence than the underlying runs support. B Extended related work: UALA, RACER, and TokUR This section expands the brief positioning given in the main text. 17 Table 9: Measured efficiency statistics from the executed runs. Strong-model latency is repeated by row because the same Qwen2.5-7B 4-bit model is used. SLMLocal in tok.Local out tok.Strong in tok.Strong out tok.Local lat. p50/p95 (s)Strong lat. p50/p95 (s)Local peak mem (MiB)Local energy p50 (J) qwen2.5-0.5b427.941.1427.947.10.301/0.5761.528/3.38599024.31 qwen2.5-1.5b427.947.9427.947.10.702/1.6381.528/3.385300380.66 qwen2.5-3b427.943.9427.947.11.152/2.2481.528/3.3856058143.86 UALA (Han et al., 2024).The Uncertainty-Aware Language Agent integrates uncertainty estima- tion into the ReAct thought-action-observation loop for free-form question answering (HotpotQA, StrategyQA, MMLU). At each step, UALA scores the agent’s current answer and uses an adaptive, calibration-set-derived threshold to decide whether to accept the answer, invoke an external tool (search), or, in a further extension, request human help. This is architecturally the closest prior work to STEPGATE’s “when to escalate” decision, but the escalation target in UALA is a tool call (or a human), not a stronger executor model, and the underlying action space is free-form text generation rather than structured function calling. STEPGATE instead assumes the tool has already been selected by the local model and asks which model (local SLM or cloud LLM) should generate and execute the structured call, using schema- and argument-level features that have no analogue in UALA’s free-form QA setting. The two ideas are complementary: an UALA-style gate could decide whether to call a tool at all and STEPGATE could then decide which executor handles that call. RACER (Hao et al., 2026). RACER addresses single-turn LLM routing over a fixed candidate poolMwith a known (or oracle-evaluable) ground-truth correctness setG(x)per query. It reframes routing as the α-Valid Optimal Routing problem: construct the smallest model set C(x)⊆M such that the probability of excluding every correct model is at most a user-specifiedα, using finite-sample conformal-style calibration (a monotone threshold over nonconformity scores, calibrated via Eq. 6 of Hao et al. (2026)) to obtain a distribution-free guarantee under exchangeability. This differs from STEPGATE’s setting in three ways that make a direct adaptation nontrivial: (i) RACER’s calibration guarantee assumes i.i.d./exchangeable calibration and test queries, whereas STEPGATE’s decision points are steps within a trajectory, which are not exchangeable with each other (later steps are conditioned on earlier routing decisions and tool outcomes); (i) RACER assumes an oracle correctness label per query at calibration time, whereas our failure label for a step is itself counterfactual and can depend on the chosen executor’s downstream behavior; and (i) RACER selects a set of models for aggregation, whereas STEPGATE must commit to exactly one executor per step under a cloud budget. A principled extension might calibrate at the trajectory level (treating whole trajectories, rather than individual steps, as the exchangeable unit) or use sequential/online conformal risk control; we view this as the most concrete next step toward closing the risk-control gap noted in Appendix C. TokUR (Zhang et al., 2026).TokUR estimates token-level epistemic uncertainty for LLM chain-of- thought reasoning by introducing low-rank random weight perturbations during decoding, producing an ensemble of predictive distributions per token that are aggregated into a sequence-level uncertainty signal correlated with answer correctness on mathematical reasoning benchmarks. Our intrinsic feature family (token NLL, entropy, tool-choice margin; Appendix A.6 and Table 6) targets a similar goal but uses only plain decode-time statistics from a single forward pass, without weight perturbation or ensembling, since our SLMs must run with tight per-step latency budgets (Section 4, “Efficiency”). Incorporating a lightweight, TokUR-style perturbation ensemble into the intrinsic feature family, traded off against its added local latency, is a natural direction for improving the token-uncertainty signal used by STEPGATE’s gate. C Limitations and broader impact The evaluation remains narrower than the full deployment setting in Figure 1. The multi-turn experiment tests re-routing across actions and reports a 21-point trajectory-success improvement over local-only at 30% cloud actions. Its comparisons are accompanied by paired bootstrap intervals and McNemar tests (Section 4), but they rest on 100 scripted tasks, and the differences among STEPGATE, random, query-level, and self-consistency routing are small relative to this sample size. The original single-step test has only 52 held-out tasks and correspondingly wide confidence intervals , and its baselines have unequal realized escalation rates (Appendix A.1). All baselines 18 escalate more than STEPGATE, and rate-matched variants are given in Table 4. The evaluation also covers one model family, one strong model (Qwen2.5-7B 4-bit), and an in-distribution benchmark configuration, so robustness to other strong backends, distribution shift, and other agent benchmarks (e.g., ToolSandbox, WebArena) remains outside the empirical scope. Results are not stratified by risk tier or task difficulty, although the tags exist (Appendix A.6). Finally, the learned feature set is not uniformly beneficial: removing risk metadata improves the 1.5B logistic gate (at a higher escalation rate; the equal-budget comparison is in Appendix A.1), and the 3B router is not clearly better than random routing; we read this as feature interaction and possible overfitting on a heterogeneous, modestly sized feature set rather than evidence against any one family. We therefore do not claim that the risk features improve task success. Two further limitations bear on deployment readiness. First, STEPGATE’s gate is calibrated empir- ically and gives no finite-sample risk-control guarantee: unlike RACER’sα-VOR routing, which provably bounds misrouting probability (Hao et al., 2026) (Appendix A.4), our thresholds can be violated on a given sample; adapting such a guarantee to a stateful, step-level setting is open work. Second, local self-correction is explored only lightly (sample-based consistency plus one bounded retry), not deeper deliberation, so how much of the local/strong gap more local compute could close remains unknown. The risk taxonomy is a research annotation, not an authorization policy. Appendix A.5 gives worked examples mapping representative tool calls to each tier. “Wrong risky” measures errors on actions labeled R2 or above in this benchmark. It is not a real-world harm rate and does not constitute a safety guarantee. Likewise, remote-token count is only a disclosure-volume proxy, not a privacy guarantee: the minimized packet still transmits task content, and we do not evaluate leakage or adversarial settings. Energy and latency depend on hardware and runtime. Strong-model escalation can fail, as shown by the 92.3% strong-only success rate, and correlated local/strong failures remain possible. The broader-impact objective is to reduce unnecessary remote inference while retaining reliability, but high-impact actions still require deterministic application policy and, where appropriate, human authorization. 19

Instant research discovery