Paper deep dive
ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, Dimitrios S. Nikolopoulos
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/14/2026, 1:41:14 AM
Summary
ConfigSpec is a profiling-based configuration selection framework for distributed speculative LLM serving that optimizes the trade-offs between goodput, verification cost efficiency, and energy efficiency across heterogeneous edge-cloud environments by modeling draft-target alignment and device-specific performance.
Entities (5)
Relation Signals (3)
ConfigSpec → optimizes → Speculative Decoding
confidence 95% · ConfigSpec, a configuration-selection framework for distributed speculative LLM serving.
ConfigSpec → profiles → NVIDIA Jetson AGX Orin
confidence 90% · ConfigSpec characterizes each draft model and edge platform using measurable quantities.
Llama-3.1-70B → verifies → Speculative Decoding
confidence 90% · a centralized verifier hosts a target model that enforces correctness under speculative decoding semantics
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model variants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configurationselection framework for distributed speculative LLM serving. ConfigSpec profiles edge devices and draft-target alignment, and models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge platforms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths (K*=2-10). Secondly, both cost and energy efficiency converge to K=2 due to a dominant bonus-token effect-with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw. These conflicts confirm that no single fixed configuration can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge-cloud LLM inference.
Tags
Links
- Source: https://arxiv.org/abs/2604.09722v1
- Canonical: https://arxiv.org/abs/2604.09722v1
Trouble viewing inline? Open PDF directly →
Full Text
38,524 characters extracted from source content.
Expand or collapse full text
ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving Xiangchen Li Virginia Tech Blacksburg, Virginia, USA lixiangchen@vt.edu Saeid Ghafouri Queen’s University Belfast Belfast, Northern Ireland, UK s.ghafouri@qub.ac.uk Jiakun Fan Virginia Tech Blacksburg, Virginia, USA jiakunfan@vt.edu Babar Ali Queen’s University Belfast Belfast, Northern Ireland, UK b.ali@qub.ac.uk Hans Vandierendonck Queen’s University Belfast Belfast, Northern Ireland, UK h.vandierendonck@qub.ac.uk Dimitrios S. Nikolopoulos Virginia Tech Blacksburg, Virginia, USA dsn@vt.edu Abstract Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model vari- ants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configuration- selection framework for distributed speculative LLM serving. Con- figSpec profiles edge devices and draft–target alignment, and mod- els drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge plat- forms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths (퐾 ∗ =2–10). Secondly, both cost and energy efficiency converge to퐾=2 due to a dominant bonus-token effect—with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw. These conflicts confirm that no single fixed configura- tion can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge–cloud LLM inference. CCS Concepts:• Computing methodologies→Distributed computing methodologies; Distributed artificial intelligence; Natural language processing;• Computer systems organization →Distributed architectures;• General and reference→Gen- eral conference proceedings. Keywords: Speculative Decoding, Large Language Models, Edge Computing, Distributed Inference, Token Verification ACM Reference Format: Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos. 2026. ConfigSpec: Profiling-Based Configu- ration Selection for Distributed Edge–Cloud Speculative LLM Serving. In 4th International Workshop on Testing Distributed Internet of Things Systems (TDIS ’26), April 27–30, 2026, Edinburgh, Scotland Uk. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3802513.3803483 This work is licensed under a Creative Commons Attribution 4.0 International License. TDIS ’26, Edinburgh, Scotland Uk © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2608-8/26/04 https://doi.org/10.1145/3802513.3803483 1 Introduction Deploying Large Language Models (LLMs) closer to end users has become increasingly attractive due to the need to reduce end-to- end latency, limit bandwidth usage [14], and preserve data local- ity and privacy [7]. Many interactive, streaming, and Internet of Things (IoT) applications in the fields of healthcare, industrial IoT, surveillance, etc., benefit from near-source inference execution, where responsiveness and data sensitivity are critical, enabling LLM inference on edge and mobile platforms [17]. For on-device inference, a wide range of model efficiency techniques have been explored, including quantization [15], pruning [5], and architec- tural simplification, which aim to reduce computation, memory footprint, and energy consumption on resource-constrained hard- ware. While these techniques can substantially lower inference cost and enable partial deployment of LLM models on edge devices, they face fundamental limits, as aggressive compression often leads to accuracy degradation and diminishing returns [2]. Consequently, fully executing high-capacity LLMs on edge devices alone remains impractical for many workloads, motivating hybrid cloud–edge exe- cution models in which IoT devices offload sensory insights to edge, where edge devices perform lightweight or partial inference while delegating expensive computation to centralized infrastructure [9]. Speculative decoding is a decoding paradigm for autoregressive language models in which a lightweight draft model proposes mul- tiple tokens ahead of time, and a higher-capacity target model sub- sequently verifies them against the target distribution. The target model accepts correct prefixes and discards the remaining, which eliminates divergence likelihood [8]. Crucially, speculative decod- ing preserves the output distribution of the target model despite using an auxiliary draft model [1,8]. This process decouples gener- ation into a drafting phase that prioritizes speed and a verification phase that enforces correctness [8,12]. By enabling multiple tokens to be proposed and validated in a single step, speculative decoding exposes opportunities to overlap computation, amortize verification cost, and restructure inference across heterogeneous resources. Speculative decoding can substantially improve throughput, sys- tem capacity, and cost efficiency in heterogeneous edge environ- ments [10,11,16]. By distributing generation across devices with different computational capabilities, these systems achieve better utilization of both edge and server resources while maintaining output quality. However, speculative decoding exposes a fundamen- tal challenge in practice that is orthogonal to mechanism design, namely configuration selection. Designers must decide which draft model variants to deploy on what edge devices, and how to set the arXiv:2604.09722v1 [cs.DC] 8 Apr 2026 TDIS ’26, April 27–30, 2026, Edinburgh, Scotland UkXiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos speculative decoding length, choices that directly impact perfor- mance, resource efficiency, monetary cost, and energy consumption across heterogeneous hardware. In realistic deployments, these con- figurations interact in non-trivial ways with hardware constraints, workload characteristics, and service-level objectives. Larger draft models typically improve token acceptance rates, but consume more compute and memory on the device, potentially limiting con- currency and degrading energy efficiency. No single configuration dominates across devices, workloads, and optimization objectives. Selecting configurations for distributed speculative LLM serving is challenging because system behavior depends on the interaction between draft model family, model size, quantization level, edge hardware, speculative decoding length, and target model align- ment. These factors jointly determine verified-token throughput, verification cost efficiency, and energy efficiency, and their effects are often non-intuitive. For example, increasing draft model size improves acceptance rate but may reduce goodput and increase energy per verified token; similarly, platform differences can dom- inate model-size differences. As a result, configuration decisions cannot be inferred from model accuracy or hardware capability alone, and require systematic profiling and cross-metric evaluation. In this paper, we present ConfigSpec, a framework that combines systematic device profiling, draft–target alignment measurement, and analytical performance modeling to evaluate configuration quality. ConfigSpec characterizes each draft model and edge plat- form using measurable quantities—drafting throughput, acceptance rate against a target model, and device power draw—and analyti- cally factors cloud verification effects into a set of parameters, map- ping these profiles to deployment-relevant metrics including good- put, verification cost efficiency, and energy efficiency. By grounding configuration evaluation in measurable primitives while preserv- ing the structural properties of speculative decoding, ConfigSpec enables fast, repeatable comparison of model families, quantization levels, and edge platforms prior to large-scale deployment. In summary, this paper makes the following contributions: •We identify configurations—specifically the joint choice of draft model, speculative decoding length, and edge platform— as a primary determinant of performance, cost, and energy in distributed edge–cloud speculative LLM serving. •We design ConfigSpec, a profiling-based configuration eval- uation framework that measures drafting throughput, draft– target acceptance rate, and device power, and systematically maps them to system goodput, cost, and device energy. •Exhaustive enumeration of the joint(푀,푄,퐾)configura- tion space reveals structurally conflicting optima: goodput favours smallest draft at device-dependent퐾 ∗ =2–10, while both cost and energy converge to퐾=2 via a bonus-token effect—yet cost selects the largest drafter and energy the smallest. These conflicts produce trade-offs of up to 2.9× in goodput, 2.2×in cost, and 7.8×in energy between objective- optimal configurations on same device, confirming that no single fixed setting can simultaneously optimise all metrics. 2 Related Work Related works are categorized into three groups: speculative de- coding algorithms, speculative decoding systems for edge and dis- tributed serving, and edge–cloud collaboration for LLM inference. Speculative Decoding Algorithms. Speculative decoding was originally introduced as a lossless acceleration technique for au- toregressive generation, enabling multiple tokens to be proposed in parallel by a lightweight draft model and verified by a larger target model. Leviathan et al. [8] formalize speculative decoding and prove that it preserves the output distribution of standard de- coding while achieving substantial speedups without retraining or architectural changes. Subsequent work extends speculative decod- ing to new regimes and workloads. MagicDec [13] demonstrates that speculative decoding remains effective for moderate to long sequences and high-throughput inference, identifying bottleneck shifts with batch size and proposing adaptive drafting strategies to improve throughput and latency. Speculative Decoding Systems for Edge and Distributed Serving. Recent systems integrate speculative decoding into prac- tical LLM serving architectures. EdgeLLM [16] applies speculative decoding to on-device inference, addressing memory constraints by combining draft models with efficient branch navigation, adap- tive fallback, and compute–IO pipelining. SLED [11] reinterprets speculative decoding as a mechanism for collaborative edge com- puting, allowing heterogeneous edge devices to draft tokens locally while a shared server verifies them in batches. WISP [10] further identifies wasted drafting time and verification interference as key scalability bottlenecks in distributed speculative serving and pro- poses dynamic drafting and SLO-aware batching to improve system capacity and goodput. Edge–Cloud Collaboration for LLM Inference. Beyond spec- ulative decoding, several works study collaborative inference across edge and cloud resources. CLONE [14] explores algorithm–hardware co-design for latency-aware LLM inference on edge devices, relying on profiling and deployment-driven evaluation to optimize perfor- mance and energy efficiency. EdgeShard [18] proposes partitioning LLMs into shards distributed across collaborative edge devices and cloud servers, optimizing device selection and partitioning deci- sions under heterogeneity and bandwidth constraints. While these approaches demonstrate the benefits of collaboration between edge and cloud, they do not model speculative decoding or address con- figuration selection across draft variants and speculative lengths. 3 System Framework ConfigSpec provides a configuration-profiling framework for dis- tributed speculative LLM serving across heterogeneous edge plat- forms, as illustrated in Figure 1. In the deployment model we con- sider, multiple edge devices execute inference requests using local draft models, while a centralized verifier hosts a target model that enforces correctness under speculative decoding semantics [8]. Con- figSpec evaluates configurations through profiling combined with an analytical model of speculative decoding rounds. 3.1 Framework Serving abstraction. An inference request consists of autore- gressive token generation. On each edge device, a draft model proposes tokens locally and submits them for verification by the target model. The verifier accepts the longest valid prefix and pro- duces a corrective token upon divergence. Only accepted tokens contribute to final output. ConfigSpec : Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM ServingTDIS ’26, April 27–30, 2026, Edinburgh, Scotland Uk Edge Devices Cloud Verifier Speculative Draft i i e """""""""""" > .. Target Model Draft Model A Draft Model B Draft Model C T V Accept Prefix: Token 1, Token 2 ~---| Speculative Token Block |--------- ] Verified Tokens 1 1 H . el il ecti L J L ) A i Configuration Evaluation Profiling Inputs Draft Throughput (tokens/s) v Accepted Tokens / Drafted Tokens $ Dicost, [Frmrrermemmmcmmossmeeess =» Estimate Goodput =p Estimate Cost Efficiency =P Estimate Energy per Verified Token 4 Maximize Goodput $ Minimize Cost Minimize Energy iifi Configuration Evaluation 4 Maximize Goodput ~ $ Minimize Cost Figure 1. System abstraction used by ConfigSpec. Heterogeneous edge devices generate speculative tokens using local draft models and interact with a centralized cloud verifier. Configuration profiling. For each draft model and edge plat- form, ConfigSpec profiles: (i) drafting throughput푣 푑 , (i) draft–target acceptance rate 훼(퐾), (i) device power draw 푃 . Configuration space and selection. A configuration consists of a target model at the verifier, a draft model variant, and an edge platform. Given the measured profile, ConfigSpec evaluates config- urations under objectives such as maximizing goodput, minimizing verification cost per token, or minimizing edge device energy per verified token. 3.2 Performance, Cost, and Energy Model We model distributed speculative decoding using the same round abstraction employed in our evaluation. In each speculative round, an edge device drafts퐾candidate tokens at rate푣 푑 (tokens/s), which are verified by a cloud-hosted target model with acceptance rate 훼(퐾) and verification latency푇 verify . Goodput. We define the goodput as the verified token through- put under the speculative decoding round model: 퐺(퐾)= 퐾 · 훼(퐾)+ 1 퐾/푣 푑 +푇 verify [tok/s],(1) where the numerator counts the expected accepted tokens per round (퐾훼(퐾)from the draft plus one bonus token at the first rejection), and the denominator is the round latency comprising local drafting time 퐾/푣 푑 and remote verification latency푇 verify . Verification Cost Efficiency. Under a token-priced billing model with unit price푝($/token), each speculative round processes ap- proximately퐾tokens at the verifier. The cost efficiency in accepted tokens per dollar is: 휂 cost = 퐾 · 훼(퐾)+ 1 퐾 · 푝 = 훼(퐾)+ 1/퐾 푝 [tok/$],(2) which depends only on the acceptance rate훼(퐾), speculative length퐾, and verifier price푝. In practice, continuous batching and parallel prefill/decode can influence this cost. Accounting for the existing conflicting objectives and diverse search space, we considered token-priced billing for decoding without batching. Table 1. Acceptance rate (훼) of draft models against each target model with speculative length of 5. Higher values indicate stronger alignment between draft and target distributions. Target: Llama-3.1-70BTarget: Qwen3-32B Draft Model휶Draft Model휶 Llama-3.2-1B0.462Qwen3-0.6B0.378 Llama-3.2-1B-Instruct0.546Qwen3-1.7B0.466 Llama-3.2-3B-Instruct0.572Qwen3-4B0.487 Llama-3.1-8B0.622Qwen3-8B0.522 Energy Efficiency. Let푃denote the average power draw of the edge device during drafting. Since verification occurs in the cloud, only local drafting time contributes to on-device energy consumption. The energy per verified token is: 퐸= 푃 · 퐾/푣 푑 퐾 · 훼(퐾)+ 1 [J/tok],(3) where the numerator is the drafting energy per round and the denominator is the expected number of accepted tokens. The purpose of ConfigSpec is configuration-level exploration. For each configuration (draft model, device, and target model), ConfigSpec measures푣 푑 ,훼(퐾), and푃, and evaluates퐺(퐾),휂 cost , and퐸using the analytical model above. Because휂 cost is independent of drafting speed and퐸depends only on푣 푑 and푃, configurations can be compared by profiling edge-side behavior and measuring draft–target alignment, while treating cloud verification latency as a parameter. 4 Evaluation We evaluate the effectiveness of distributed speculative decoding across heterogeneous edge platforms by profiling a comprehen- sive set of draft model configurations and speculative lengths. Our analysis proceeds along three axes—goodput, verification cost effi- ciency, and energy efficiency—and culminates in a unified selection framework that maps deployment constraints to recommended configurations. Experimental Setup. We deploy draft models on three repre- sentative edge platforms: Raspberry Pi 4B (RPi 4B, Cortex-A72, 8 GB RAM), Raspberry Pi 5 (RPi 5, Cortex-A76, 8 GB RAM), and NVIDIA Jetson AGX Orin (64 GB unified memory, Ampere GPU). All draft models are served viallama.cppin GGUF format with quantiza- tion variants ranging from Q4_K_M to Q8_0. Two target models are considered: Llama-3.1-70B and Qwen3-32B, both hosted on a remote cloud verifier. We use the instruction prompts from the Databricks Dolly 15K dataset [3] as input prompts. The dataset con- tains 15,011 instruction-following records created by Databricks contributors. 4.1 Goodput Analysis 4.1.1 Results. Fig. 2 reveals that the optimal speculative length 퐾 ∗ is not a universal constant but varies significantly with both device speed and draft model size. On the slowest device (RPi 4B), 퐾 ∗ =2 almost universally, as the high per-token drafting cost (1/푣 푑 ) penalises additional speculation. In contrast, the Jetson AGX Orin pushes퐾 ∗ to 5–8 for Llama and 4.5-7 for Qwen family, since its fast drafting makes the incremental cost of extra tokens negligible TDIS ’26, April 27–30, 2026, Edinburgh, Scotland UkXiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos 0.5 1.0 1.5 2.0 2.5 Goodput (tok/s) Llama-3.1-70B — RPi 4B 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 Llama-3.1-70B — RPi 5 4.5 5.0 5.5 6.0 6.5 7.0 7.5 Llama-3.1-70B — Jetson AGX Orin Llama-3.2-1B Llama-3.2-1B-Inst. Llama-3.2-3B-Inst. Llama-3.1-8B-Inst. 2345678910 Speculative Length K 0.5 1.0 1.5 2.0 2.5 Goodput (tok/s) Qwen3-32B — RPi 4B 2345678910 Speculative Length K 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Qwen3-32B — RPi 5 2345678910 Speculative Length K 4.0 4.5 5.0 5.5 6.0 Qwen3-32B — Jetson AGX Orin Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Goodput vs Speculative Length (T verify = 0.5s, ★ = optimal K) Figure 2. Goodput vs. speculative length across devices and models. 1b-inst-q4_k_m3b-inst-q4_k_m 1b-base-q4_k_m 8b-inst-q4_k_m 3b-inst-q8_0 0 2 4 6 Goodput (tok/s) Target: Llama-3.1-70B (K=5, T verify =0.5s) RPi 4BRPi 5Jetson AGX Orin 1.7b-base-q4_k_m 1.7b-base-q8_0 4b-base-q4_k_m 0.6b-base-q4_k_m 0.6b-base-q8_0 8b-base-q4_k_m 4b-base-q8_0 8b-base-q6_k 0 2 4 6 Goodput (tok/s) Target: Qwen3-32B (K=5, T verify =0.5s) Figure 3. Verified token generation speed. relative to the fixed verification latency푇 verify , which is effectively amortised over more candidates.푇 verify has been carefully selected based on historical experiments where the considered target models have been observed taking on average 0.5s to verify tokens, and it can vary for different target models and underlying hardware. Moreover, within each device, smaller draft models (e.g., Llama-3.2- 1B, Qwen3-0.6B) consistently favour higher퐾 ∗ than their larger counterparts, following the same cost–benefit logic. On fast devices the goodput peaks are broad, meaning the exact choice of퐾is forgiving; on slow devices the curves are nearly flat, making퐾 inconsequential. These observations suggest a practical guideline: set퐾=2 on resource-constrained platforms and퐾 ≥5 on GPU- class accelerators, with the precise value tuned per draft–target pair via a lightweight sweep. Goodput results. Fig. 3 reports the goodput for Llama- and Qwen-family draft models across three devices. In both families, the smallest draft models achieve the highest goodput: Llama-3.2-1B (Q4_K_M) reaches approximately 6.5 tok/s on the Jetson AGX Orin, while Qwen3-0.6B (Q4_K_M) reaches roughly 5.8 tok/s. Scaling up the draft model consistently degrades goodput despite improving훼; for instance, moving from Llama-3.2-1B to 3.1-8B raises훼from 0.46 to 0.62 yet cuts goodput by more than 4×on the Jetson. A notable observation is that the Jetson AGX Orin does not outperform the Raspberry Pi devices by as wide a margin as its raw drafting speed would suggest. While the Jetson drafts tokens 6.5–16.2×faster than the RPi 5, its goodput advantage is only 1.5–2×. This is because the verification latency푇 verify is shared across all devices—once the Llama-3.2-1B Llama-3.2-1B-Instruct Llama-3.2-3BLlama-3.1-8B 0K 250K 500K 750K 1000K 1250K Accepted Tokens per $ 736K 829K 858K 913K Target: Llama-3.1-70B Qwen3-0.6BQwen3-1.7B Qwen3-4BQwen3-8B 980K 1128K 1165K 1224K Target: Qwen3-32B Cost Efficiency of Draft Models (K=5) Figure 4. Cost efficiency of draft models. draft tokens are produced quickly, the system must still wait for the remote server to verify them. As푇 verify dominates the round- trip time, additional drafting speed yields diminishing returns on goodput. On the RPi 4B, all models above 1B fall below 1 tok/s, rendering them impractical for interactive use. These results reveal a consistent structural trend across both model families: on edge devices, goodput is governed by the interplay between drafting speed and verification latency. 4.2 Verification Cost Efficiency 4.2.1 Pricing Data. We adopt publicly listed inference pricing from major providers as representative verification costs. For the Llama-3.1-70B target, we use the Fireworks AI serverless tier for models exceeding 16B parameters, priced at $0.90 per 1M tokens [4]. For the Qwen3-32B target, we use Groq’s on-demand pricing at $0.59 per 1M tokens [6]. 4.2.2 Results. Fig. 4 shows that cost efficiency increases mono- tonically with draft model capacity in both families, consistent with Eq. (1). The largest drafts—Llama-3.1-8B (913K tokens/$, 훼= 0.62) and Qwen3-8B (1224K tokens/$,훼=0.52)—outperform their small- est counterparts by around 19.4% and 24.9%, respectively, driven en- tirely by higher acceptance rates. The Qwen configurations achieve uniformly higher cost efficiency due to a lower verification price ($0.59/M vs. $0.90/M for Llama). Combined with the goodput results in Section 4.1, this reveals a fundamental Pareto tradeoff: smaller draft models maximize goodput by maximizing푣 푑 , while larger draft models maximize cost efficiency by maximizing훼. The optimal choice therefore depends on whether the deployment prioritizes latency or monetary cost, a point we revisit in Section 4.4. 4.3 Energy Efficiency Energy comparison results. Fig. 5 shows that the edge device’s energy per verified token escalates steeply with draft model size on both platforms 1 . The smallest drafts—Llama-3.2-1B-Instruct Q4_K_M (0.63 J/tok on Jetson, 1.28 J/tok on RPi 5) and Qwen3-0.6B Q4_K_M (0.60 J/tok, 0.91 J/tok)—achieve the best energy efficiency across all configurations. Scaling to the largest drafts degrades effi- ciency by 4.17–5.14×in the Llama family (Llama-3.1-8B: 2.67 J/tok on Jetson, 6.58 J/tok on RPi 5) and up to 10.75×in the Qwen family (Qwen3-8B Q6_K: 4.25 J/tok on Jetson, 9.73 J/tok on RPi 5). The degradation is consistently less severe on the Jetson, whose higher compute throughput better amortizes the fixed power overhead. Speed–energy Pareto front. Fig. 6 visualizes the speed–energy tradeoff with iso-power curves at 15 W, 20 W, 40 W, and 60 W. In 1 Real-time power monitoring is much less accessible on the Raspberry Pi 4B than on the Raspberry Pi 5, which provides more practical platform-level support for power measurement. Therefore we report power results only for the Raspberry Pi 5. ConfigSpec : Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM ServingTDIS ’26, April 27–30, 2026, Edinburgh, Scotland Uk Llama-3.2-1B-Inst. (Q4_K_M) Llama-3.2-1B (Q4_K_M) Llama-3.2-3B (Q4_K_M) Llama-3.2-3B (Q8_0) Llama-3.1-8B (Q4_K_M) 0 2 4 6 J / Verified Token Energy Efficiency Comparison (Target: Llama-3.1-70B, K=5) RPi 5Jetson AGX Orin Qwen3-0.6B (Q4_K_M) Qwen3-0.6B (Q8_0) Qwen3-1.7B (Q4_K_M) Qwen3-1.7B (Q8_0) Qwen3-4B (Q4_K_M) Qwen3-4B (Q8_0) Qwen3-8B (Q4_K_M) Qwen3-8B (Q6_K) 0 5 10 J / Verified Token Energy Efficiency Comparison (Target: Qwen3-32B, K=5) Figure 5. Energy efficiency of draft models. 246810 Verified Goodput (tok/s) = (Kα(K)+1)/(K/tps+T verify ) 0.0 2.5 5.0 7.5 J / Verified Token 5 W 10 W 20 W 40 W 60 W Llama-3.2-1B-Inst. (Q4_K_M) 1.28 J/tok Llama-3.2-1B-Inst. (Q4_K_M) 0.64 J/tok Speed–Energy Tradeoff (Target: Llama-3.1-70B, K=5) RPi 5 Jetson AGX Orin 2468 Verified Goodput (tok/s) = (Kα(K)+1)/(K/tps+T verify ) 0 5 10 J / Verified Token 5 W 10 W 20 W 40 W 60 W Qwen3-0.6B (Q4_K_M) 0.91 J/tok Qwen3-0.6B (Q4_K_M) 0.60 J/tok Speed–Energy Tradeoff (Target: Qwen3-32B, K=5) Figure 6. Energy efficiency and speed (goodput) comparison. both model families, the smallest-draft model with lowest quantiza- tion bit-width on Jetson configurations occupy the Pareto-optimal corner (high goodput, low energy), while all RPi 5 configurations are Pareto-dominated by their Jetson counterparts. This architec- tural advantage stems from the Jetson’s GPU-accelerated inference, which exploits massive parallelism in matrix operations to deliver substantially higher throughput per watt, whereas the RPi 5’s CPU- only execution serializes these operations across a small number of cores, resulting in longer active inference time and proportionally greater energy expenditure per token. 4.4 Optimal Configuration Selection across the Three-Dimensional Space Sections 4.1–4.3 examined goodput, cost, and energy in isolation while fixing the speculative length at퐾=5. We now treat all three configuration knobs—draft-model variant푀, quantisation level 푄, and speculative length퐾—as a joint search space and ask: for a given LLM family and edge device, which triple(푀,푄,퐾)optimises each objective. 2 Methodology. For every (target, device) pair, we enumerate all feasible(푀,푄,퐾)triples with퐾 ∈ 2, . . .,10and evaluate three metrics: (i) goodput퐺(Eq. 1), (i) cost efficiency휂 cost (Eq. 2), and 2 Considering the power supply on the server is continuous, we only consider the power consumption and optimization on edge device side Table 2. Recommended configurations under different optimisation objectives.퐺: goodput (tok/s),휂 cost : cost efficiency (K tokens/$),퐸: energy per accepted token (J/tok). Best values per (target, device) group are bolded. Target Device ObjectiveConfiguration퐾 퐺 휂 cost 퐸 Llama-3.1-70B RPi 4B Max GoodputLlama-3.2-1B-Inst Q42 2.441334K— Min Cost/tokLlama-3.1-8B-Inst Q420.77 1401K— Min Energyno power data RPi 5 Max GoodputLlama-3.2-1B-Inst Q46 4.50763K0.84 Min Cost/tokLlama-3.1-8B-Inst Q421.55 1401K3.75 Min EnergyLlama-3.2-1B-Inst Q423.761334K 0.48 Jetson Max GoodputLlama-3.2-1B-Inst Q48 7.65623K0.85 Min Cost/tokLlama-3.1-8B-Inst Q424.35 1401K1.74 Min EnergyLlama-3.2-1B-Inst Q424.601334K 0.39 Qwen3-32B RPi 4B Max GoodputQwen3-0.6B Q42 2.811801K— Min Cost/tokQwen3-8B Q420.74 2048K— Min Energyno power data RPi 5 Max GoodputQwen3-0.6B Q47 3.86828K0.90 Min Cost/tokQwen3-8B Q421.49 2048K3.86 Min EnergyQwen3-0.6B Q423.481801K 0.41 Jetson Max GoodputQwen3-0.6B Q410 6.21633K0.93 Min Cost/tokQwen3-8B Q424.14 2048K1.88 Min EnergyQwen3-0.6B Q424.081801K 0.33 (i) energy per accepted token퐸(Eq. 3). Table 2 reports the config- uration that optimises each metric, together with all three metric values so that the trade-offs are directly visible. As the퐾and ac- ceptance rate (훼) have an intricate relation, for each푀,푄,퐾, edge device, and target model combination, we computed tailored훼(퐾). Observation 1: Goodput favours small, fast drafters with device-dependent퐾 ∗ . Across both families, the smallest Q4 quan- tised draft model—Llama-3.2-1B-Instruct for Llama-70B and Qwen3- 0.6B for Qwen3-32B—delivers the highest goodput on every device. The optimal speculative length퐾 ∗ , however, scales with device speed: on the RPi 4B the fixed verification overhead푇 verify already dominates at퐾=2; on the RPi 5 the sweet spot rises to퐾 ∗ =6– 7; and on the Jetson it climbs to퐾 ∗ =8–10. Comparing the Jet- son’s goodput-optimal퐺=7.65 tok/s (Llama) with the RPi 4B’s 퐺=2.44 tok/s reveals only a 3.1×advantage despite a∼22×raw speed gap, confirming that푇 verify compresses the goodput range. The cost of this goodput-maximising strategy is visible in the휂 cost column: on the Jetson it drops to 623K tok/$—less than half the cost-optimal 1401K tok/$. Observation 2: Cost efficiency is device-independent and always peaks at퐾=2 with the largest drafter. Since휂 cost = (훼(퐾)+1/퐾)/푝depends only on the acceptance rate and verifier price, the cost-optimal configuration is identical across all three devices: 8B-Instruct Q4 at퐾=2 for Llama-70B and Qwen3-8B Q4 at퐾=2 for Qwen3-32B. The key is the bonus-token effect: every verification round produces one auto-regressive output token regardless of how many drafted tokens are rejected. At퐾=2 this free token contributes 1/퐾=0.5 to the per-verified-token yield, whereas at퐾=10 it contributes only 0.1. Combined with the high acceptance rate of 8B drafters at short sequences (훼(2) ≈0.76),퐾= 2 yields훼(2)+1/2=1.26 accepted tokens per verified token—the maximum across the entire search space. The trade-off is stark: on the RPi 5 the cost-optimal configuration delivers only퐺=1.55 tok/s TDIS ’26, April 27–30, 2026, Edinburgh, Scotland UkXiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos (2.9×slower than the goodput-optimal 4.50 tok/s) and퐸=3.75 J/tok (7.8× worse energy than the energy-optimal 0.48 J/tok). Observation 3: Energy efficiency mirrors goodput’s model choice but universally locks to퐾=2. The energy-optimal draft model is the same small, fast model that maximises goodput—1B- Instruct Q4 for Llama, 0.6B Q4 for Qwen—because these draw the least power and minimise drafting time. Yet the optimal specula- tive length is universally퐾=2, diverging from goodput’s device- dependent퐾 ∗ . Since퐸only counts local drafting energy, setting 퐾=2 minimises the joules spent per round while the bonus- token effect keeps yield high. On the Jetson the energy-optimal 퐸=0.39 J/tok (Llama) is 17% lower than the RPi 5’s 0.48 J/tok despite higher idle power, because the Jetson completes each 2-token draft in a fraction of the time. Notably, the energy-optimal configuration still achieves reasonable goodput—퐺=4.60 tok/s on the Jetson (60% of the goodput-optimal 7.65)—making it an attractive default for battery-constrained deployments where moderate latency is acceptable. Key trade-offs. •Model size: Goodput and energy prefer the smallest drafter (maximum푣 푑 , minimum푃), while cost prefers the largest (maximum훼). On the RPi 5, switching from the cost-optimal 8B to the goodput-optimal 1B improves throughput by 2.9× and energy by 7.8×, but sacrifices 46% cost efficiency. •Speculative length퐾: Goodput benefits from longer spec- ulation on fast devices (more tokens amortise푇 verify ), while both cost and energy favour퐾=2 (the bonus-token effect dominates). Reducing퐾from the goodput-optimal퐾 ∗ to 2 costs up to 40% goodput but can halve energy per token. These results demonstrate that profiling-based configuration se- lection across the joint(푀,푄,퐾)space is essential: no single fixed setting can simultaneously optimise throughput, cost, and energy. 5 Conclusion ConfigSpec is a framework for configuration selection in distributed speculative LLM serving across cloud and edge. By modeling het- erogeneous edge devices and speculative decoding dynamics, Con- figSpec enables rapid exploration of trade-offs among edge model variants, quantisation levels, and speculative lengths without re- peated system deployment. Our analysis of the joint(푀,푄,퐾)con- figuration space reveals that the three optimisation objectives im- pose fundamentally conflicting pressures: goodput favours small, fast drafters at device-dependent퐾 ∗ that amortises verification latency, while both cost and energy efficiency converge to퐾=2 due to the dominant bonus-token effect—though cost selects the largest drafter to maximise acceptance rate whereas energy se- lects the smallest to minimise power draw. These structural con- flicts confirm that no single fixed configuration can simultane- ously optimise throughput, cost, and energy, underscoring the ne- cessity of profiling-based selection. The framework complements deployment-focused speculative decoding systems by supporting principled design-space exploration in disaggregated AI inference infrastructures. 6 Acknowledgements This work was supported by a research grant from the Depart- ment for the Economy, Northern Ireland (grant agreement USI-226), Virginia Tech College of Engineering (grant Major Grants Initia- tive Program), and by the National Science Foundation (grant No. 2315851). References [1] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Lau- rent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 (2023). [2]Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. 2025. Efficientqat: Efficient quantization-aware training for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10081–10100. [3]Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction- Tuned LLM.https://w.databricks.com/blog/2023/04/12/dolly-first-open- commercially-viable-instruction-tuned-llm [4]Fireworks AI. 2025. Pricing – Fireworks AI. https://fireworks.ai/pricing. Server- less tier: $0.90 / 1M tokens for models >16B parameters. Accessed: 2025-07-15. [5]Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning. PMLR, 10323–10337. [6]Groq. 2025. GroqCloud On-Demand Pricing. https://groq.com/pricing. Qwen3- 32B: $0.29 / 1M input tokens, $0.59 / 1M output tokens. Accessed: 2025-07-15. [7]Chengzhuo Han, Tingting Yang, Zhengqi Cui, and Xin Sun. 2025. A privacy- preserving and trustworthy inference framework for LLM-IoT integration via hierarchical federated collaborative computing. IEEE Internet of Things Journal (2025). [8] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning. PMLR, Honolulu, Hawaii, USA, 19274–19286. [9] Hui Li, Xiuhua Li, Qilin Fan, Qiang He, Xiaofei Wang, and Victor CM Leung. 2025. Adaptive model partitioning and pruning for collaborative DNN inference in mobile edge-cloud computing networks. IEEE Transactions on Mobile Computing (2025). [10] Xiangchen Li, Jiakun Fan, Qingyuan Wang, Dimitrios Spatharakis, Saeid Ghafouri, Hans Vandierendonck, Deepu John, Bo Ji, Ali R Butt, and Dimitrios S Nikolopou- los. 2026. WISP: Waste-and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching. arXiv preprint arXiv:2601.11652 (2026). [11] Xiangchen Li, Dimitrios Spatharakis, Saeid Ghafouri, Jiakun Fan, Hans Vandieren- donck, Deepu John, Bo Ji, and Dimitrios S Nikolopoulos. 2025. Sled: A speculative llm decoding framework for efficient edge serving. In Proceedings of the Tenth ACM/IEEE Symposium on Edge Computing. 1–8. [12]Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhi- hao Jia. 2023. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781 1, 2 (2023), 4. [13]Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. 2024. Mag- icDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding. In International Conference on Learning Representa- tions. [14] Chunlin Tian, Xinpeng Qin, Kahou Tam, Li Li, Zijian Wang, Yuanzhe Zhao, Minglei Zhang, and Chengzhong Xu. 2025. CLONE: customizing LLMs for efficient latency-aware inference at the edge. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 34, 23 pages. [15]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning. PMLR, 38087–38099. [16]Daliang Xu, Wangsong Yin, Hao Zhang, Xin Jin, Ying Zhang, Shiyun Wei, Meng- wei Xu, and Xuanzhe Liu. 2024. Edgellm: Fast on-device llm inference with speculative decoding. IEEE Transactions on Mobile Computing 24, 4 (2024), 3256– 3273. [17]Zhongzhi Yu, Zheng Wang, Yuhan Li, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reddy Bommu, Yang Zhao, and Yingyan Lin. 2024. Edge-llm: Enabling efficient large language model adaptation on edge devices via unified compression and adaptive layer voting. In Proceedings of the 61st ACM/IEEE Design Automation Conference. 1–6. [18] Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang. 2024. EdgeShard: Efficient LLM inference via collaborative edge computing. IEEE Internet of Things Journal (2024).