Paper deep dive
SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
Divya Jyoti Bajpai, Kishan Kumar Upadhyay, Manjesh Kumar Hanawal
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/14/2026, 6:06:02 AM
Summary
The paper introduces SPADE, a distributed inference framework that integrates speculative decoding across edge and cloud devices. It uses a compact draft model on the edge to generate candidate tokens rapidly and a large verifier model on the cloud to validate them in parallel. This approach shifts computation to the edge, significantly reducing cloud model calls by 76% while maintaining the accuracy of the full model without retraining.
Entities (8)
Relation Signals (8)
SPADE → reduces → Cloud Model Calls
confidence 95% · SPADE reduces the cloud model calls by 76% with zero loss in accuracy
SPADE → usestechnique → Speculative Decoding
confidence 95% · We present a distributed inference framework, SPADE, that integrates speculative decoding (SD) across edge and cloud.
SPADE → deployson → Edge Device
confidence 90% · A compact draft model deployed on the edge generates candidate tokens rapidly
SPADE → deployson → Cloud Server
confidence 90% · a large verifier model on the cloud validates these tokens in parallel
SPADE → evaluatedon → CNN/DailyMail
confidence 90% · Experimental results across multiple Natural Language Processing tasks using ... CNN/Dailymail datasets
SPADE → evaluatedon → SpecBench
confidence 90% · Experimental results across multiple Natural Language Processing tasks using SpecBench
Llama 3.2-1B → servesas → Draft Model
confidence 90% · we use LLaMA-3.2-1B as the draft model at the edge
LLaMA-3.1-8B → servesas → Verifier Model
confidence 90% · LLaMA-3.1-8B [6] as the target verification model on the cloud
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, \our{}, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that \our{} reduces the cloud model calls by $76\%$ with zero loss in accuracy as compared to the full model.
Tags
Links
- Source: https://arxiv.org/abs/2608.13076v1
- Canonical: https://arxiv.org/abs/2608.13076v1
Trouble viewing inline? Open PDF directly →
Full Text
26,552 characters extracted from source content.
Expand or collapse full text
SPADE: Speculative Decoding for Precise and Low-Cost Distributed Edge–Cloud Inference Divya Jyoti Bajpai, Kishan Kumar Upadhyay and Manjesh Kumar Hanawal Affiliation: MLiONS, Department of IEOR, IIT Bombay Mumbai, Maharashtra-400076, India divyajyoti.bajpai, 24n0453, mhanawal@iitb.ac.in Abstract Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, SPADE, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that SPADE reduces the cloud model calls by 76%76\% with zero loss in accuracy as compared to the full model. 11 1 Anonymized code: https://anonymous.4open.science/r/spade-anon-DC1F Index Terms: Speculative Decoding, Distributed inference. I Introduction Large Language Models (LLMs) have achieved remarkable breakthroughs in understanding and generating language. Yet, this surge in model scale comes with a hidden cost: larger models demand vast computational resources and memory, which are rarely available on mobile or edge devices. As a result, deploying state-of-the-art models outside of high-performance cloud servers becomes a major bottleneck. Existing literature has explored multiple strategies to alleviate this burden. Techniques such as model pruning [13], weight quantization [10], and knowledge distillation [8] reduce the size and computational load of models, making them feasible for smaller devices. However, they often come at the cost of reduced accuracy, as simplifying the network limits its ability to capture complex patterns. Even purpose-built, smaller variants of large models, while more efficient, consistently underperform compared to their full-scale counterparts. One alternative is to run the full model on the cloud. Hosting large models on powerful cloud servers can restore top-tier performance, enabling the use of models that are too large to run on mobile or edge devices. However, cloud-based inference introduces a key challenge: the computational cost grows with the duration for which the resources are utilized. As LLMs generate output in an autoregressive manner, they require sustained computation to process each input, which can result in longer inference times and increased costs. Fig. 1: A lightweight edge model generates tokens autoregressively, which are sent to a cloud model for parallel verification and consistency checking. Accepted tokens are kept, rejected ones are replaced, and generation continues from the updated context. These constraints highlight a critical challenge: how can we achieve high accuracy without incurring large inference times on the cloud? This motivates the need for distributed inference between edge and cloud devices, where a smaller version of the LLM is deployed on the edge while the full-scale model is hosted in the cloud. Inference fully on the edge can reduce the accuracy, and inference fully on the cloud will increase the cost to a greater extent. The central question that remains is how to further reduce the reliance on the cloud by utilizing the resources available at the edge device, while still maintaining the accuracy of the full model. Recently, Speculative Decoding (SD) has emerged as a powerful technique for accelerating LLM inference while preserving the accuracy of the full model. The approach relies on a two-model setup: a smaller, faster draft model that proposes speculative tokens and a larger, more accurate verifier model that evaluates them in parallel. Verified or accepted tokens are retained, while the first rejected token is replaced with the verifier’s predictions, after which the draft model continues generation conditioned on the corrected context. This iterative process cuts the autoregressive structure of LLMs to significantly reduce the expensive verifier calls—for instance, a 5050-token sequence that would normally require 5050 full model call can often be completed with only a few model calls—thus providing substantial efficiency gains without sacrificing output quality. We propose SPADE, a distributed inference framework across edge and cloud devices, addressing the dual challenges of efficiency and accuracy in large-scale LLM deployment. In this framework, the smaller model deployed on the edge acts as a draft generator, producing a sequence of speculative (draft) tokens locally. These draft sequences are then transmitted to the larger, full-scale model hosted on the cloud, which serves as a verifier to evaluate and correct the discrepancies in a single pass. By generating the draft tokens at the edge and only verifying them in the cloud, our approach significantly reduces the computational burden on the cloud, decreases inference latency, and minimizes the number of expensive model calls on the cloud, while maintaining the output fidelity of the full-scale model without any training requirements. Our method is detailed in Fig. 1 (details in Sec. I-C). The importance and usefulness of this method lie in its ability to enable scalable and resource-efficient deployment of large LLMs in real-world scenarios. It allows the edge device to handle most of the generative computation, reducing dependency on the cloud and improving the cloud computation cost. At the same time, the cloud ensures accuracy of the full model, so there is no compromise in output quality with minimal resource utilization of the cloud. This combination makes the deployment of large LLMs feasible in settings where purely cloud-based or purely edge-based solutions would be inefficient or impractical. In summary, our key contributions are as follows: • Distributed Speculative Decoding Framework: We introduce a novel edge-cloud distributed inference setup that leverages speculative decoding to efficiently split computation between a smaller edge model and a full-scale cloud model. • Reduce Cloud Computation: Our method significantly reduces the number of calls to the cloud model by using as the edge model to generate the fraft sequences. This lowers the inference time and resource usage without compromising accuracy. • Maintain Full-Scale Accuracy: The framework ensures that the final outputs are equivalent to those of the full cloud model, guaranteeing fidelity while benefiting from edge-side computation. This is achieved without any additional training requirements. • Empirical evaluations: We provide empirical results on various NLP tasks using SpecBench and CNN/DailyMail datasets, where the cloud computation time was reduced by 76%76\% with no loss in performance. I Related works We review prior works on distributed inference and speculative decoding related to our method. Distributed Inference: Recent works leverage heterogeneous resources across edge and cloud devices to distribute computation. 1) Layer-splitting: Neurosurgeon [9] partitions DNNs by executing initial layers on the edge and remaining layers on the cloud. 2) Encoder-side training: Methods improve edge encoders for efficient transmission, e.g., head-network distillation [12]. 3) Early-exit classifiers: Intermediate classifiers enable fast inference by exiting early for easy samples [2]. SplitEE [4] and I-SplitEE [1] optimize splitting and prediction. 4) Complexity-aware routing: DIMEE [3] routes samples based on estimated complexity, but relies on dataset-specific heuristics. Speculative Decoding: Speculative Decoding (SD) [15] accelerates autoregressive models using a lightweight draft model and a larger verifier for parallel validation. Self-Speculative Decoding: LayerSkip [7] reuse early layers of the large model as the draft. Distributed inference methods face key trade-offs: (i) lenient routing reduces performance, (i) strict routing increases cost, and (i) Complexity estimation lacks generalization. Our Approach: Our method differs as follows: • We apply speculative decoding to distributed inference, with a draft model on the edge and verifier on the cloud. • We achieve zero performance loss relative to the large model as proven in [11]. • Our framework generalizes across tasks and avoids dataset-specific heuristics. • SPADE is fully plug-and-play and requires no retraining. I Problem Setup We begin by introducing the general framework of autoregressive decoding, speculative decoding (SD), followed by our adaptation of SD for distributed inference. I-A Autoregressive Decoding in LLMs LLMs typically adopt the transformer architecture, consisting of an embedding layer, L stacked transformer blocks, and a final language modeling head. Let the input prompt be denoted by y1:t=(y1,y2,…,yt).y_1:t=(y_1,y_2,…,y_t). The embedding layer maps input tokens to embeddings x0x_0. At each layer l∈1,…,Ll∈\1,…,L\, the hidden representation evolves as xl+1t=xlt+fl(xlt),x_l+1^t=x_l^t+f_l(x_l^t), where fl(⋅)f_l(·) denotes the transformation at layer l (self-attention and feedforward). The final representation at the L-th layer, xLtx_L^t, is projected to logits via the language modeling head, oLt=g(xLt)o_L^t=g(x_L^t). Autoregressive decoding estimates the conditional distribution of the next token as P(yt+1∣y1:t)=softmax(oLt).P(y_t+1 y_1:t)=softmax(o_L^t). The next token yt+1y_t+1 is then selected either via greedy decoding (argmax) or sampling for diversity. A key limitation of this procedure is its computational cost: generating each new token (word) requires a full forward pass of the model over the entire context. Consequently, inference latency is dominated by model depth and memory requirements, leading to inefficiencies in large-scale models. I-B Speculative Decoding (SD) To mitigate the inefficiency of standard autoregressive decoding, SD employs a two-model pipeline that combines a smaller, faster model (draft model) with the full-scale model (verifier). The procedure consists of two stages: 1. Drafting Stage: The draft model autoregressively generates a block of d candidate tokens, yt+1:t+d=(yt+1,yt+2,…,yt+d),y_t+1:t+d=(y_t+1,y_t+2,…,y_t+d), conditioned on the prefix y1:ty_1:t. 2. Verification Stage: In a single forward pass, the verifier evaluates the block yt+1:t+dy_t+1:t+d against its probability distribution. Tokens consistent with the verifier are accepted directly. The first token yjy_j that is rejected is replaced by sampling from an adjusted distribution (explained next), and the draft model resumes generation from the updated prefix (y1,…,yj)(y_1,…,y_j) where j is the index of the token where the first rejection happens. This procedure ensures that the generated sequence is statistically identical [11] to what would have been produced by the verifier alone, while substantially reducing the number of expensive verifier calls. By verifying tokens in blocks rather than individually, SD achieves significant speedups in practice with minimal loss of performance. I-C Speculative Decoding for Distributed Inference We utilize speculative decoding in a distributed inference framework that leverages both edge and cloud resources. Draft Model ℳqM_q on Edge: A compact LLM ℳqM_q is deployed on the edge device to perform fast inference under constrained memory and bandwidth. Model selection at the edge is governed by available resources such as GPU memory, CPU throughput, and system bandwidth. The draft model must be sufficiently lightweight to generate speculative sequences without overwhelming the device. Verifier Model ℳpM_p on Cloud: The cloud hosts a larger, high-accuracy model ℳpM_p that validates the candidate tokens proposed by ℳqM_q. The computational demands of ℳpM_p make it unsuitable for edge deployment but ideal for cloud platforms, which can scale elastically with demand. Since cloud services often charge per model call, minimizing verifier invocations directly reduces inference cost. Verification Criterion: Let p(x)p(x) denote the target distribution from ℳpM_p, and q(x)q(x) the draft distribution from ℳqM_q. A candidate x∼q(x)x q(x) is accepted with probability α(x)=min(1,p(x)q(x)).α(x)= (1, p(x)q(x) ). If rejected, a replacement token is sampled from the adjusted distribution p′(x)=norm(max(0,p(x)−q(x))),p (x)=norm ( (0,\,p(x)-q(x)) ), which corrects the bias introduced by the draft distribution. This guarantees equivalence to verifier-only decoding [11]. The approach is depicted pictorially in Fig 1. Pipeline: A pseudo-code of our method is given in Algorithm 1. The procedure is summarized as follows: starting from a user prompt, the edge model ℳqM_q generates a block of d draft tokens. These tokens are sent to the cloud, where ℳpM_p verifies them in parallel using a single forward pass. All tokens accepted under α(x)α(x) are appended to the output. The first rejected token is replaced via sampling from p′(x)p (x), and the remaining draft tokens are discarded. The context is then updated with the accepted and corrected tokens, and the draft model resumes autoregressive generation for the next block until the end-of-sentence <eos><eos> is generated. This design guarantees that at least one token is appended to the sequence in each iteration, thereby ensuring progress. By batching verification in the cloud while exploiting low-latency drafting on the edge, the framework balances efficiency, accuracy, and cost in distributed inference. Algorithm 1 Speculative Decoding for Distributed Inference 1: Input: Prompt y1:my_1:m, Draft model ℳqM_q (edge), Verifier model ℳpM_p (cloud) 2: Initialize: Y←[]Y←[\,] ⊳ Generated sequence 3: while <eos><eos> token not predicted do 4: Edge: Generate d speculative tokens 5: for i∈[1,d]i∈[1,d] do 6: ym+i←ℳq(y1:m+i−1)∼q(⋅)y_m+i _q(y_1:m+i-1) q(·) 7: end for 8: Send ym+1:m+dy_m+1:m+d to cloud for verification 9: Cloud: In parallel, [ym+i∗←ℳp(y1:m+i−1)∼p(⋅)y_m+i^* _p(y_1:m+i-1) p(·) for i∈[1,d+1]i∈[1,d+1]] in a single forward pass. 10: Verify(ym+i,ym+i∗y_m+i,y_m+i^*) in parallel. 11: Accept the longest verified draft ym:m+ky_m:m+k 12: Sample y~m+k+1←norm(max(0,p−q)CLOSE y_m+k+1← norm(max(0,p-q) 13: Correct the rejected draft ym+k+1=y~m+k+1y_m+k+1= y_m+k+1. 14: Append tokens ym:m+k+1y_m:m+k+1 to Y 15: Update context: m←m+k+1m← m+k+1, y1:m+k+1y_1:m+k+1 16: end while 17: Return: Y TABLE I: Mean performance and efficiency metrics on the Spec-Bench dataset, comparing Target, Draft, and Speculative models across NLP tasks. Higher (↑ ) is better performance, lower (↓ ) indicates greater efficiency. Task / Metric Target Model Draft Model Our Model (SPADE) Task Scores (↑ ) Multi-turn Conversation 3.62 2.67 3.52 Translation 4.80 3.85 4.71 Summarization 4.61 4.41 4.55 Question Answering 4.25 3.08 4.15 Mathematical Reasoning 4.88 3.19 4.68 Retrieval-Augmented Generation 4.56 3.13 4.68 Overall Score (↑ ) 4.45 3.39 4.38 Efficiency Metrics Mean Target Model Calls (↓ ) 133.25 0.00 30.16 Average Throughput (tokens/s) (↑ ) 2.43 3.91 3.25 Cloud runtime (↓ ) 1.00× – 0.23× TABLE I: Performance comparison on the CNN / DailyMail summarization dataset. Higher values (↑ ) indicate better generation quality. Metric Target Model Draft Model Our Model (SPADE) BLEU-1 (↑ ) 23.76 22.33 23.39 BLEU-4 (↑ ) 07.57 06.49 06.98 ROUGE-1F1_F1 (↑ ) 38.38 36.05 37.99 ROUGE-LF1_F1 (↑ ) 24.32 22.49 23.92 CIDEr-D (↑ ) 02.50 01.15 03.19 Efficiency Metrics Mean Target Model Calls (↓ ) 127.30 0.00 30.79 Average Throughput (tokens/s) (↑ ) 1.21 2.82 1.95 Cloud runtime (↓ ) 1.00× – 0.24× Draft token length: The number of draft tokens, denoted as d, generated per edge call is a key control parameter in speculative decoding, governing the trade-off between computation and communication in the edge–cloud pipeline. A small d increases verification frequency and synchronization overhead, while a large d raises token rejection likelihood and draft computation cost, reducing overall speedup. From a systems perspective, d balances edge computation and cloud communication: smaller values make the system communication-bound, whereas larger values risk redundant local computation. Thus, d is treated as a hyperparameter, selected empirically by monitoring acceptance rates on an initial validation subset (typically ∼10 10 samples). The objective is to maximize throughput while maintaining a stable acceptance rate under realistic latency and compute constraints. IV Experiments Datasets: We use publicly available datasets, including CNN/DailyMail [14] for summarization, and Spec-Bench [15], a benchmark designed to evaluate speculative decoding methods. Spec-Bench consists of six subtasks—multi-turn conversation, summarization, translation, retrieval-augmented generation, question answering, and mathematical reasoning—allowing fair comparison of both speed and performance across diverse NLP settings. Setup: Our framework follows an edge–cloud design, where a small draft model operates at the edge and a larger verification model runs on the cloud. The edge setup uses an NVIDIA RTX 3080 GPU (12 GB RAM), providing a balance between computational capability and deployability for high-end edge devices. The cloud setup employs an NVIDIA RTX A6000 GPU (48 GB RAM), representing typical industrial-scale infrastructure for large model inference. Models: To instantiate this setup, we use LLaMA-3.2-1B as the draft model at the edge and LLaMA-3.1-8B [6] as the target verification model on the cloud. This configuration reflects a realistic distributed scenario in which a lightweight model performs draft generation, while the larger model validates outputs to ensure correctness during verification. Metrics: For the CNN/DailyMail dataset, we evaluate generated summaries using standard metrics. BLEU-1 and BLEU-4 assess n-gram precision, ROUGE-1 and ROUGE-L measure unigram recall and sequence-level overlap, and CIDEr-D evaluates semantic relevance using TF-IDF–weighted n-grams. All metric scores are normalized to a 0–100 scale for consistency. For the Spec-Bench dataset, performance is evaluated using Gemini-2.5-Flash-Lite [5] as an automatic judge, which assigns each generated response a score on a 1-5 Likert scale. Responses are assessed across six dimensions, correctness, instruction-following, completeness, clarity and coherence, conciseness, and factuality, with higher scores indicating better overall performance. Baseline: We compare against two baselines: (1) the Target Model, a large cloud model representing the upper bound in output quality, and (2) the Draft Model, a lightweight edge model providing low-latency outputs. (3) Our method SPADE bridges these extremes using speculative decoding, where the target model selectively verifies draft outputs. V Results and Analysis Fig. 2: Performance trade-off of our method, illustrating the variation in target model calls when the hyperparameter d (speculative token generation before one verifier call) increases. In Table I, we present results of our method on the SpecBench dataset, alongside two baselines: the Full Model (fully deployed on the cloud) and the Draft Model (operating solely at the edge). Evaluation uses an LLM-as-a-judge framework across multiple qualitative dimensions, correctness, instruction-following, completeness, clarity and coherence, conciseness, and factuality, with average scores reported as the overall performance metric. In addition to performance, we evaluate system efficiency using three indicators: average throughput (tokens per second), average number of cloud model calls, and average cloud runtime reduction. Our method achieves performance comparable to the full model while reducing cloud model calls by 77.4%, yielding significant efficiency gains with minimal quality degradation. Table I reports results on the CNN/DailyMail dataset using standard metrics (BLEU-1, BLEU-4, ROUGE-1, ROUGE-L, CIDEr-D) to measure linguistic quality and relevance, along with the same efficiency metrics. The results show that our method closely matches the accuracy and fluency of the full model, while reducing cloud calls by 76% and substantially lowering computational cost. Overall, these findings demonstrate that our approach maintains near cloud-level performance while significantly improving efficiency. Consistent results across SpecBench and CNN/DailyMail highlight its robustness and generalizability, supporting the use of selective, hybrid edge–cloud inference. Analysis on the value of d: In Figure 2, we analyze the effect of the number of draft tokens (d) on target model calls for CNN/DailyMail. Increasing d consistently reduces target model invocations, as more draft tokens decrease verification frequency. This effect is amplified by strong alignment between the draft and target models, leading to higher token acceptance rates. Poor alignment, however, would reduce acceptance and increase target calls. VI Conclusion We propose a distributed inference framework, SPADE, that leverages speculative decoding in a dual-model setup, with a lightweight draft model at the edge and a larger verification model on the cloud. By using the verifier only for parallel verification rather than autoregressive generation, SPADE reduces costly cloud invocations. Experiments across multiple NLP tasks show that SPADE significantly lowers cloud model calls while maintaining performance close to the full model. These results establish SPADE as a practical, scalable solution for latency-aware distributed inference, effectively balancing efficiency without any loss in performance. References [1] D. J. Bajpai, A. Jaiswal, and M. K. Hanawal (2024) I-splitee: image classification in split computing dnns with early exits. In ICC 2024 - IEEE International Conference on Communications, Vol. , p. 2658–2663. External Links: Document Cited by: §I. [2] D. J. Bajpai and M. K. Hanawal (2024) CeeBERT: cross-domain inference in early exit BERT. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 1736–1748. External Links: Link, Document Cited by: §I. [3] D. J. Bajpai and M. K. Hanawal (2025) Distributed inference on mobile edge and cloud: an early exit based clustering approach. In ICC 2025 - IEEE International Conference on Communications, Vol. , p. 5425–5430. External Links: Document Cited by: §I. [4] D. J. Bajpai, V. K. Trivedi, S. L. Yadav, and M. K. Hanawal (2024) SplitEE: early exit in deep neural networks with split computing. In Proceedings of the Third International Conference on AI-ML Systems, AIMLSystems ’23, New York, NY, USA. External Links: ISBN 9798400716492, Link, Document Cited by: §I. [5] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §IV. [6] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv e-prints, p. arXiv–2407. Cited by: §IV. [7] M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, et al. (2024) Layer skip: enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710. Cited by: §I. [8] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu (2019) Tinybert: distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351. Cited by: §I. [9] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, et al. (2017) Neurosurgeon: collaborative intelligence between the cloud and mobile edge. In ACM Computer Architecture News, Vol. 45, p. 615–629. Cited by: §I. [10] S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer (2021) I-bert: integer-only bert quantization. In International conference on machine learning, p. 5506–5518. Cited by: §I. [11] Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, p. 19274–19286. Cited by: 2nd item, §I-B, §I-C. [12] Y. Matsubara and M. Levorato (2021) Neural compression and filtering for edge-assisted real-time object detection in challenged networks. In 2020 25th International Conference on Pattern Recognition (ICPR), p. 2272–2279. Cited by: §I. [13] P. Michel, O. Levy, and G. Neubig (2019) Are sixteen heads really better than one?. Advances in neural information processing systems 32. Cited by: §I. [14] A. See, P. J. Liu, and C. D. Manning (2017) Get to the point: summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368. Cited by: §IV. [15] H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui (2024) Unlocking efficiency in large language model inference: a comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851. Cited by: §I, §IV.