Paper deep dive
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
Eunjeong Kim, Yeong Jun Jeon, Myeonggyun Han
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/13/2026, 5:16:33 AM
Summary
The paper introduces MemSpec, a memory-aware runtime for adaptive speculative decoding on edge devices. It addresses the mismatch between draft selection and availability by decoupling selection from execution. MemSpec uses a lightweight predictor to estimate draft effectiveness and a memory-aware scheduler to proactively manage resident working sets, reducing loading overhead. Experiments on Jetson Orin Nano show a 40.7% throughput improvement over state-of-the-art bandit-based methods.
Entities (9)
Relation Signals (8)
MemSpec → evaluatedon → Jetson Orin Nano
confidence 97% · Experiments on a Jetson Orin Nano show that MemSpec improves steady-state generation throughput
MemSpec → implements → Speculative Decoding
confidence 95% · MemSpec, a prediction-guided, memory-aware runtime for adaptive speculative decoding
MemSpec → runson → Edge devices
confidence 95% · MemSpec, a prediction-guided, memory-aware runtime for adaptive speculative decoding on edge devices.
MemSpec → usescomponent → Prediction Engine
confidence 95% · The runtime integrates three key components: a Prediction Engine
MemSpec → usescomponent → Draft Model Cache Manager
confidence 95% · a Draft Model Cache Manager that maintains a small resident working set
MemSpec → outperforms → Multi-armed bandit
confidence 92% · MemSpec improves steady-state generation throughput by 40.7% on average over state-of-the-art bandit-based adaptive methods
Evaluation → usesmodel → Llama-2-7b
confidence 90% · We evaluate a quantized LLaMA-2 7B target model
Prediction Engine → usesmodel → BERT
confidence 90% · We use a fine-tuned BERT encoder as the predictor
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to speculate multiple tokens, reducing expensive target model decoding steps. Its effectiveness depends heavily on draft selection, motivating adaptive methods that exploit variation across inputs and generation stages. On memory-constrained edge devices, however, these methods often fail to improve end-to-end throughput due to the overhead of switching between draft models. We identify a key limitation in this setting: the mismatch between draft selection and draft availability under tight memory budgets. To address this challenge, we present MemSpec, a prediction-guided, memory-aware runtime for adaptive speculative decoding on edge devices. MemSpec decouples draft selection from execution through proactive resident working-set management. A lightweight predictor estimates draft effectiveness from prompt and generation context, while a memory-aware scheduler reduces reactive model loading overhead. Experiments on a Jetson Orin Nano show that MemSpec improves steady-state generation throughput by 40.7% on average over state-of-the-art bandit-based adaptive methods while closely approaching the oracle upper bound.
Tags
Links
- Source: https://arxiv.org/abs/2608.10362v1
- Canonical: https://arxiv.org/abs/2608.10362v1
Trouble viewing inline? Open PDF directly →
Full Text
67,461 characters extracted from source content.
Expand or collapse full text
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices Eunjeong Kim Kyungpook National University Daegu, Republic of Korea kimeunjeong23@knu.ac.kr Yeong Jun Jeon Kyungpook National University Daegu, Republic of Korea jyj5219@knu.ac.kr Myeonggyun Han Kyungpook National University Daegu, Republic of Korea mhan@knu.ac.kr Abstract Speculative decoding accelerates autoregressive large lan- guage model (LLM) inference by using a lightweight draft model to speculate multiple tokens, reducing expensive tar- get model decoding steps. Its effectiveness depends heavily on draft selection, motivating adaptive methods that exploit variation across inputs and generation stages. On memory- constrained edge devices, however, these methods often fail to improve end-to-end throughput due to the overhead of switching between draft models. We identify a key limita- tion in this setting: the mismatch between draft selection and draft availability under tight memory budgets. To address this challenge, we present MemSpec, a prediction- guided, memory-aware runtime for adaptive speculative decoding on edge devices. MemSpec decouples draft selec- tion from execution through proactive resident working-set management. A lightweight predictor estimates draft ef- fectiveness from prompt and generation context, while a memory-aware scheduler reduces reactive model loading overhead. Experiments on a Jetson Orin Nano show that MemSpec improves steady-state generation throughput by 40.7% on average over state-of-the-art bandit-based adaptive methods while closely approaching the oracle upper bound. CCS Concepts:• Computer systems organization→Em- bedded systems;• Computing methodologies→Natural language processing. Keywords: Speculative Decoding, Large Language Models, On-Device AI, Memory Management, Adaptive Runtime ACM Reference Format: Eunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han. 2026. Mem- Spec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices. In Proceedings of the 27th ACM SIGPLAN/SIGBED International Conference on Languages, Com- pilers, and Tools for Embedded Systems (LCTES ’26), June 15–16, 2026, Boulder, CO, USA. ACM, New York, NY, USA, 13 pages. https: //doi.org/10.1145/3814943.3816174 This work is licensed under a Creative Commons Attribution 4.0 Interna- tional License. LCTES ’26, Boulder, CO, USA © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2721-4/2026/06 https://doi.org/10.1145/3814943.3816174 1 Introduction Large language models (LLMs) are increasingly moving be- yond cloud datacenters to memory-constrained edge plat- forms, including mobile devices, embedded systems, and edge servers [28,40]. As on-device AI adoption accelerates, achieving high-throughput LLM inference under tight mem- ory budgets has become a critical systems challenge. These constraints fundamentally reshape the design space of infer- ence optimization techniques. Speculative decoding [3,20,25] has emerged as an effec- tive approach for accelerating autoregressive inference by using a lightweight draft model to generate multiple can- didate tokens, which are then verified by a larger target model, thereby amortizing expensive target-model compu- tation. When the draft model accurately predicts the target model’s outputs, speculative decoding can significantly im- prove throughput while preserving output quality, making it particularly attractive for resource-constrained edge de- ployments. However, the effectiveness of speculative decoding is highly sensitive to the choice of draft model. Prior works [18,38,39] show that different draft models ex- hibit widely varying token acceptance rates across prompts and generation stages. In practice, such variation arises naturally from domain-specialized draft models (e.g., code, math, or legal), which perform well on their target domains but perform poorly on out-of-domain workloads. This moti- vates adaptive approaches that dynamically select or switch between multiple candidate drafts. On memory-constrained edge devices, however, this adap- tivity introduces a major systems challenge. Since only a small number of draft models can reside in fast memory, switching to a non-resident draft incurs substantial loading overhead. As a result, frequent draft transitions can negate potential throughput gains and even degrade overall perfor- mance. Existing approaches are not well suited to this setting. Traditional static methods [20,23–25] that use a single fixed draft model fail to exploit generation heterogeneity, result- ing in suboptimal performance. State-of-the-art adaptive methods typically rely on multi-armed bandit (MAB)-based approaches [14,18,22] to explore and select draft models at runtime, often improving token acceptance rates. However, these methods incur frequent draft switching and implicitly arXiv:2608.10362v1 [cs.OS] 11 Aug 2026 LCTES ’26, June 15–16, 2026, Boulder, CO, USAEunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han assume that selected drafts can be executed immediately. This assumption breaks down on edge devices, where mem- ory constraints limit draft residency. As a result, the cost of switching to non-resident drafts often outweighs the benefit of improved draft selection, and better draft selection does not necessarily translate into higher generation throughput. This limitation stems from a fundamental mismatch be- tween draft selection and draft availability: a draft that is predicted to be effective may not be immediately executable under memory constraints. On edge devices, this mismatch makes draft switching prohibitively expensive, rendering exploration-based adaptation inefficient. These observations lead to a key insight: adaptive speculative decoding on edge devices must be formulated not only as a selection problem, but also as a scheduling problem under memory constraints. To address this challenge, we propose MemSpec, a prediction-guided, memory-aware runtime for adaptive draft scheduling. Instead of relying on costly online explo- ration, MemSpec predicts promising draft models using both prompt features and recent generation context, and proactively aligns draft residency with future demand. By decoupling draft selection from execution, MemSpec always proceeds with the best currently resident draft while prepar- ing better candidates in the background, thereby avoiding blocking and excessive model loading overhead. Specifically, this paper makes the following contributions: •We identify a fundamental limitation of adaptive spec- ulative decoding on memory-constrained edge devices, namely the mismatch between draft selection and draft availability. We show that exploration-based methods incur excessive switching overhead, as selecting a bet- ter draft often requires loading non-resident models, limiting end-to-end throughput gains. •We propose MemSpec, a prediction-guided, memory- aware runtime that decouples draft selection from ex- ecution. MemSpec replaces online exploration with lightweight prediction and enables non-blocking de- coding by always selecting the best currently resident draft. •We design a memory-aware scheduling framework that proactively manages a small working set of draft models under tight memory budgets. Through coordi- nated prefetching and eviction, MemSpec aligns draft residency with predicted future demand and enables non-blocking adaptive decoding under memory con- straints. •We implement MemSpec on a Jetson Orin Nano and evaluate it across diverse workloads. MemSpec im- proves end-to-end throughput by 58.8% over static baselines and 40.7% over state-of-the-art adaptive methods, while achieving performance close to the dynamic oracle upper bound. 0.0 0.5 1.0 1.5 2.0 Alpaca LiveCode Bench Omni- MATH MMLU- Law MMLU- Medical GMEAN Normalized Acceptance General-StaticOracle-StaticOracle-Dynamic Figure 1. Impact of static and dynamic draft selection on token acceptance across datasets. 2 Background and Motivation This section presents three key observations that motivate MemSpec: (1) speculative decoding efficiency varies across workloads and generation stages, creating substantial head- room for adaptive draft selection; (2) on memory-constrained edge devices, switching to a non-resident draft incurs high latency, often exceeding several decoding iterations; and (3) exploration-based adaptive methods repeatedly incur this switching cost, limiting their effectiveness. 2.1 Speculative Decoding Efficiency and Acceptance In speculative decoding (SD), autoregressive generation pro- ceeds in iterative draft-and-verify steps. At each iteration, a draft model proposes candidate tokens, which are then verified by the target model. Let퐴denote the number of tokens accepted by the target model in one iteration. For a resident draft, throughput can be approximated as: Throughput≈ E[퐴] 퐿 iter ,(1) where퐿 iter is the average latency of one SD iteration. Im- provingE[퐴]is therefore the primary lever for increasing throughput. We evaluate a quantized LLaMA-2 7B target model (GPTQ INT4) with five 400M-parameter draft models: one general- purpose draft and four domain-specialized drafts trained for code, mathematical reasoning, legal, and medical workloads. Detailed setup is described in Section 4.1. Figure 1 compares three configurations: General-Static, Oracle-Static, and Oracle-Dynamic. General-Static uses a sin- gle general-purpose draft, Oracle-Static selects the best draft per prompt via offline evaluation, and Oracle-Dynamic dy- namically switches draft models within a single generation. Observation 1: Static selection is insufficient due to variability in draft effectiveness. Across datasets, Oracle- Static improves normalized acceptance by 40.3% on average over General-Static, demonstrating that selecting an appro- priate specialized draft model is critical for performance. Moreover, Oracle-Dynamic provides an additional 25.7% im- provement over Oracle-Static, indicating that the most ef- Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge DevicesLCTES ’26, June 15–16, 2026, Boulder, CO, USA 0.0 0.2 0.4 0.6 0.8 1.0 1.2 SD Iteration (Draft + Verify) Resident Draft Switch Non-Resident Draft Switch Latency (seconds) Figure 2. Latency comparison between SD iteration and draft switching overheads. fective draft can change within a single generation. These results show that even the best static selection is insufficient and that dynamic adaptation is necessary to fully exploit speculative decoding efficiency. 2.2 Memory Constraints and Draft Switching Overheads on Edge Platforms The headroom identified above does not directly translate into throughput on edge platforms. Unlike server-class sys- tems, edge devices cannot keep many draft models resi- dent in fast memory simultaneously. Switching to a better draft therefore requires loading a non-resident model from a slower memory tier, such as NVMe storage. We quantify this overhead on a Jetson Orin Nano using a 400M-parameter draft model. Figure 2 shows that loading a non-resident draft takes 2.7×longer than a single SD iter- ation. As a result, even infrequent switching can offset the throughput gains from improved acceptance. Observation 2: On memory-constrained edge devices, switching cost fundamentally limits adaptive draft se- lection. Because draft loading latency significantly exceeds per-iteration decoding latency, switching introduces substan- tial overhead. As the number of candidate drafts increases, cache misses become more frequent, further amplifying this cost. These results reveal a fundamental trade-off: while adap- tive draft selection can improve acceptance, switching to non- resident drafts incurs high loading latency that can negate throughput gains. Consequently, naive adaptive switching strategies are ineffective on memory-constrained edge de- vices. 2.3Limitations of Exploration-Based Adaptive Draft Selection State-of-the-art adaptive speculative decoding methods [14, 18,22] rely on multi-armed bandit (MAB)-based online ex- ploration to identify effective draft models during genera- tion. These approaches maintain multiple candidate drafts and adaptively select among them based on observed run- time feedback. In MAB-Sync, the runtime blocks to load and 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 General-Static Oracle-Static MAB-Sync MAB-Async Normalized Acceptance Figure 3. Normalized token acceptance of adaptive draft selection methods. 0.0 0.2 0.4 0.6 0.8 1.0 1.2 General-Static Oracle-Static MAB-Sync MAB-Async Norm. Execution Time Desired ExecutionFallback ExecutionDraft Load Wait Figure 4. Execution time breakdown of exploration-based methods. evaluate candidate drafts during exploration, incurring high switching overhead. In contrast, MAB-Async overlaps model loading with ongoing decoding to reduce blocking, but con- tinues execution with suboptimal drafts while waiting for preferred drafts to become available. Figure 3 shows that MAB-Sync improves normalized accep- tance by 34.5% on average over General-Static, approaching Oracle-Static. This confirms that exploration-based adapta- tion can effectively improve draft quality. However, these gains do not translate into throughput on memory-constrained edge devices. To understand why, we analyze execution time breakdown. Figure 4 shows that model loading dominates execution time in MAB-Sync. Al- though MAB-Async partially overlaps loading with execu- tion, it still spends a significant fraction of time—46.4% on average—executing suboptimal drafts while waiting for pre- ferred drafts to become resident. Observation 3: Exploration-based adaptation fails to improve throughput under memory constraints. While exploration improves draft selection, it incurs frequent model loading, which is prohibitively expensive on edge devices. As a result, improved acceptance does not translate into improved throughput. These results highlight a key limitation of existing adap- tive methods: they optimize which draft to use, but ignore whether the selected draft is immediately executable. Under tight memory constraints, this mismatch leads to excessive LCTES ’26, June 15–16, 2026, Boulder, CO, USAEunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han Target Model (Large Model) Draft Model (Small Model) Input T1T2T3 ... T1T2T3’ ... T3’ Fast Memory (GPU Mem.) In-Memory Draft Model Cache Prediction Engine Draft Model Cache Manager MemSpec Runtime Controller Evict Fetch & Async Prefetch Efficient Draft Model Runtime Data (Recently Generated Tokens) Slow Memory (CPU DRAM or SSD) Candidate Draft Models ... Figure 5. Overall architecture of MemSpec. fallback execution and diminished performance gains. Taken together, these observations show that improving acceptance alone is insufficient. An effective runtime must both identify promising drafts without repeated exploration and ensure their availability at runtime. This insight moti- vates MemSpec, a prediction-guided, memory-aware runtime for adaptive draft scheduling. 3 MemSpec Design Figure 5 presents the overall architecture of MemSpec, a prediction-guided, memory-aware runtime for adaptive draft scheduling in speculative decoding on edge devices. The run- time integrates three key components: a Prediction Engine that ranks candidate drafts based on decoding context, a Draft Model Cache Manager that maintains a small resident working set under memory constraints, and a Runtime Con- troller that orchestrates non-blocking adaptive decoding. The key goal of MemSpec is to realize the benefits of adaptive draft selection without incurring the high over- head of model switching under tight memory constraints. Rather than relying on online exploration, MemSpec predicts a small set of promising draft models for the current decod- ing context and maintains them as a resident working set. At runtime, decoding always proceeds with the best currently resident draft, while high-priority non-resident drafts are prefetched in the background. The central challenge is that, on memory-constrained edge devices, draft selection and execution are no longer tightly coupled: a draft predicted to be effective may not be im- mediately executable because it is not resident. MemSpec addresses this mismatch by formulating adaptive specula- tive decoding as a memory-aware runtime scheduling prob- lem, where prediction determines which drafts should be prepared, while execution is restricted to drafts that are cur- rently available. This design leads to two key principles: (1) non-blocking execution, which avoids stalling on non-resident drafts by always selecting from the resident set, and (2) proactive resi- dency management, which aligns the working set of drafts with predicted future demand. Together, these mechanisms allow MemSpec to achieve efficient adaptive decoding with- out repeated model switching or exploration overhead. 3.1 Design Overview Consider a speculative decoding run with candidate draft setDand resident cache capacity퐾, where at most퐾draft models can remain resident in fast memory. LetG 푖 ⊆ D denote the resident set at scheduling point푖, and let푑 푖 ∈ G 푖 denote the active draft used during interval 푖. MemSpec performs scheduling every푁speculative de- coding iterations. This interval-based design serves two pur- poses: (1) it amortizes the cost of prediction and runtime con- trol over multiple decoding steps, and (2) it enables overlap between decoding and asynchronous loading of non-resident drafts, which is critical for hiding model loading latency. At initialization, MemSpec selects the first draft using only the input prompt: 푑 0 = arg max 푑∈D 푃(푑 | 푥 prompt ). During generation, the context at scheduling point푖is defined as 푥 푖 =[푥 prompt ;푥 recent 푖 (푇)], where푥 recent 푖 (푇)denotes the most recent푇generated tokens. The Prediction Engine assigns each draft a score: 풑 푖 =푃(푑 | 푥 푖 ) | 푑 ∈ D. These scores are used in two ways. First, MemSpec derives a ranked list of candidate drafts: 푅 푖 = Sort(풑 푖 ), from which the cache manager derives the target resident set W 푖 = TopK(푅 푖 ). Second, it selects the active draft only from the currently resident set: 푑 ★ 푖 = arg max 푑∈G 푖 푃(푑 | 푥 푖 ). The key design abstraction of MemSpec is the separation between the target working setW 푖 and the active draft푑 ★ 푖 . Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge DevicesLCTES ’26, June 15–16, 2026, Boulder, CO, USA Prediction produces a ranked list of candidate drafts, the cache manager prepares a small working set from that list, and execution always proceeds with the best currently resi- dent draft. This design enables non-blocking decoding while grad- ually steering the cache toward more effective drafts. Con- ceptually, MemSpec balances two competing objectives: (1) following the most promising draft as the generation context evolves, and (2) minimizing costly model loading under a tight memory budget. The interval-based working-set design reconciles these objectives by transforming draft adaptation into a staged, overlapped process, rather than a sequence of stall-heavy immediate switches. 3.2 Prediction Engine The Prediction Engine ranks candidate draft models for the current decoding context. Unlike exploration-based ap- proaches, MemSpec uses an offline-trained model to directly predict context-to-draft matching, avoiding costly online probing of multiple drafts. We use a fine-tuned BERT encoder as the predictor: 풑 푖 = 푓 휃 (푥 푖 ), where풑 푖 = [푝 푖 (푑 1 ),푝 푖 (푑 2 ), . . .,푝 푖 (푑 |D| )]denotes the pre- dicted score for each draft. The input푥 푖 combines the prompt and recent output to- kens. The prompt captures global task semantics, while re- cent tokens reflect phase-dependent generation behavior (e.g., reasoning vs. code generation). Using only the prompt fails to capture such phase transitions, whereas relying only on recent tokens loses global task intent. Their combination enables effective context-aware draft ranking. Importantly, the Prediction Engine is designed to be light- weight. MemSpec does not require fine-grained utility es- timation or online evaluation of multiple drafts. Instead, it only needs a ranking that is sufficiently accurate to iden- tify a small set of promising candidates. This is sufficient because the downstream cache manager and runtime con- troller operate on relative priority rather than exact utility values. Output and training. The predictor outputs a probabil- ity for each draft: 푝 푖 (푑)= 푃(푑 | 푥 푖 ), ∀푑 ∈ D. It is trained offline using speculative decoding traces, where each context푥 푖 is labeled with the draft that achieves the highest decoding utility. This aligns prediction targets with system-level performance. Runtime usage. The Prediction Engine is invoked once every푁iterations. When invoked, it computes a ranked list of candidate drafts as summarized in Algorithm 1. Given the current context, it computes the per-draft score vector (lines 4–6), sorts drafts by score to obtain the ranked list푅 푖 (line 7), Algorithm 1 Context-Aware Draft Ranking 1:Inputs: prompt푥 prompt , recent tokens푥 recent 푖 (푇), candi- date setD 2: Output: ranked list 푅 푖 3: if 푖= 0 then 4: 풑 푖 ← 푓 휃 (푥 prompt ) 5: else 6: 풑 푖 ← 푓 휃 ([푥 prompt ;푥 recent 푖 (푇)]) 7: sort 푑 ∈ D by 푝 푖 (푑) to obtain 푅 푖 8: return 푅 푖 Algorithm 2 Draft Model Cache Update 1: Inputs: 푅 푖 ,G 푖 , capacity 퐾 , active draft 푑 푖 2: W 푖 ← top-퐾 drafts in 푅 푖 3: for each 푑 ∈W 푖 do 4: if 푑∉G 푖 and not being prefetched then 5: while|G 푖 |= 퐾 do 6:select 푑 evict ∈ G 푖 \(W 푖 ∪푑 푖 ) with lowest score 7: if no such draft exists then 8:break 9:evict 푑 evict 10: if |G 푖 |< 퐾 then 11: asynchronously prefetch 푑 and returns this list for downstream cache management (line 8). The target working setW 푖 is then derived by the cache manager from the top-퐾drafts in푅 푖 . Because prediction is performed only at interval boundaries, its overhead is amortized over multiple decoding steps. 3.3 Draft Model Cache Manager The Draft Model Cache Manager maintains draft residency under a tight memory budget. Because only퐾drafts can reside in fast memory, MemSpec explicitly manages a small working set instead of keeping all candidate drafts loaded. We consider two memory tiers: (1) a resident draft cache containing immediately executable drafts, and (2) backing storage containing drafts that require asynchronous loading. At each scheduling point, the cache manager derives the target setW 푖 from the ranked list푅 푖 by selecting the top-퐾 drafts (Algorithm 2, line 2). It then incrementally updates the resident setG 푖 without blocking execution. Policy. For each푑 ∈ W 푖 푖 , the system initiates asyn- chronous prefetch (lines 3–11). If space is required, it evicts drafts inG 푖 푖 in ascending order of predicted utility, while protecting the active draft푑 푖 whenever possible (lines 5–9). This simple top-퐾policy is effective in practice, as prediction already captures most of the utility variation across drafts. This design decouples long-horizon planning from im- mediate execution. Rather than switching to the globally best draft immediately, MemSpec incrementally reshapes the resident set so that promising drafts become available at LCTES ’26, June 15–16, 2026, Boulder, CO, USAEunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han Algorithm 3 MemSpec Runtime Loop 1: load target model 2: 푅 0 ← PredictDraftRanking(푥 prompt ) 3: load initial draft 푑 0 4: G 0 ←푑 0 5: UpdateCache(푅 0 ,G 0 ,퐾,푑 0 ) 6: 푖 ← 0 7: while not end-of-sequence do 8: for 푗= 1 to 푁 do 9:run draft 푑 푖 10: verify with target model 11:if end-of-sequence then 12:break 13: if not end-of-sequence then 14:collect 푥 recent 푖 (푇) 15:푥 푖+1 ← [푥 prompt ;푥 recent 푖 (푇)] 16: 푅 푖+1 ← PredictDraftRanking(푥 푖+1 ) 17:UpdateCache(푅 푖+1 ,G 푖 ,퐾,푑 푖 ) 18: 푑 푖+1 ← arg max 푑∈G 푖 푃(푑 | 푥 푖+1 ) 19: 푖 ← 푖+ 1 future scheduling points. In this sense, cache management is not merely a storage optimization, but a key component of adaptive draft scheduling. 3.4 Runtime Controller The Runtime Controller orchestrates decoding, prediction, cache updates, and draft switching. Its core policy is to al- ways execute the best currently resident draft and never stall for non-resident models. As summarized in Algorithm 3, MemSpec first predicts an initial ranked list from the prompt and loads the initial draft (lines 2–4). It then initializes the resident set and triggers cache preparation (line 5). During execution, the controller repeatedly performs three steps. First, it runs speculative decoding for the current inter- val using the active draft푑 푖 (lines 8–12). Second, it constructs the next context from the prompt and the most recent gen- erated tokens (lines 14–15). Third, it invokes prediction and cache update to prepare future drafts, and selects the best currently resident draft for the next interval (lines 16–18). If the top-ranked draft is not yet resident, execution con- tinues with the best available draft. Once loading completes, the new draft becomes eligible at the next scheduling point. This ensures that incomplete prefetch does not introduce blocking and converts draft adaptation into an overlapped process. This policy captures the key principle of MemSpec: the runtime should never sacrifice immediate progress to follow an unavailable draft. Instead, it continues decoding with the best runnable draft while opportunistically preparing better candidates in the background. This best-effort switching strategy enables adaptive decoding without incurring the Table 1. Hardware configuration. Component Specification PlatformNVIDIA Jetson Orin Nano GPUAmpere, 1024 CUDA cores, 32 Tensor cores CPU6-core Arm Cortex-A78AE Memory8GB LPDDR5 (102 GB/s) Storage1TB Samsung 990 PRO M.2 NVMe SSD severe overhead of naive dynamic switching. 3.5 System Implementation MemSpec is implemented in PyTorch and built on top of a high-performance speculative decoding engine presented in [31]. All models share a common tokenizer, and draft exe- cution follows the standard draft-and-verify process within a conventional speculative decoding pipeline. Each draft model maintains its residency state and prefetch status. Prefetching is performed asynchronously and over- lapped with decoding, allowing model loading to proceed without blocking execution. The active draft is protected from eviction whenever possible to ensure uninterrupted decoding. The Prediction Engine is invoked once every푁iterations, amortizing its runtime overhead. In practice, prediction ac- counts for only 3.9% of total execution time on average. Overall, MemSpec enables adaptive draft selection as a lightweight and non-blocking runtime mechanism, making it well-suited for memory-constrained edge deployments. 4 Evaluation We evaluate MemSpec along four key aspects: •Generation throughput: How much steady-state generation throughput improvement can MemSpec achieve over static and adaptive draft-selection strate- gies? • Performance breakdown: Why does MemSpec outperform exploration-based adaptive methods on memory-constrained edge devices? • Impact of design components: How much do the key design components of MemSpec contribute to per- formance? • Sensitivity analysis: How sensitive is MemSpec to runtime parameters such as scheduling interval, out- put length, and resident cache capacity? 4.1 Experimental Setup Platform. We evaluate MemSpec on a Jetson Orin Nano platform. Table 1 summarizes the hardware configuration. Software. We use PyTorch 2.9.1 with CUDA 12.6 on NVIDIA JetPack 6.2.1. All experiments are conducted under the same software environment. Models. We evaluate two target models: GPTQ INT4 LLaMA-2 7B [35] and GPTQ INT4 Qwen2.5 7B [34]. For Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge DevicesLCTES ’26, June 15–16, 2026, Boulder, CO, USA each model family, we construct five draft models: one general-purpose draft and four domain-specialized drafts (code, math, law, and medical). Each draft model has 400M parameters for LLaMA-2 and 0.5B parameters for Qwen2.5. Under the 8GB memory constraint of the Jetson Orin Nano, at most two draft models can remain resident simultaneously. Therefore, we use퐾=2 in the main evaluation unless oth- erwise stated. Draft models. Domain-specialized drafts are obtained via distillation following prior work [39]. Each draft is fine-tuned on domain-specific datasets: GSM8K [9] and MATH [13] (math reasoning), HumanEval [8] and MBPP [2] (code), Lex- GLUE [4] (law), and MedQA [16] and MedMCQA [27] (med- ical). Prediction model. The Prediction Engine uses a BERT- based model with a task-specific ranking head. The encoder is kept fixed, and only the ranking head is trained. Training data is collected by running speculative decoding on the same datasets used for draft fine-tuning and recording execution traces. The predictor is trained to select the most effective draft for each decoding context based on observed runtime utility. Training data is disjoint from the evaluation datasets to avoid data leakage and assess generalization. Workloads. We evaluate on five datasets covering diverse domains: Alpaca [32] (instruction following), LiveCodeBench- [15] (code), Omni-MATH [12] (math), MMLU-Law, and MMLU-Medical [13] (MMLU professional law and medicine). We randomly sample 100 prompts from each dataset. Runtime configuration. Unless otherwise stated, the scheduling interval is set to푁=4 and the output length is fixed to 128 generated tokens. We use greedy decoding with batch size 1. Throughput measurement. Throughput is measured as generated tokens per second during steady-state iterative decoding, including both draft execution and target veri- fication. We exclude prompt encoding time and focus on steady-state generation so that the reported performance reflects the efficiency of runtime draft scheduling itself. All results are averaged over three runs. For each prompt, we reset the runtime state, including draft cache contents and prediction context, to avoid cross-sample interference. When summarizing performance across workloads, we report the geometric mean (GMEAN). Baselines. We compare the following draft-selection strate- gies: • General-Static: uses a single general-purpose (i.e., not fine-tuned) draft model. •Oracle-Static: selects the best draft per input via of- fline evaluation but keeps it fixed during generation. •MAB-Async: a state-of-the-art exploration-based adaptive method that dynamically selects drafts with- out blocking execution. •Oracle-Dynamic: an empirical upper bound that se- lects the best draft at each scheduling point using the same scheduling interval as MemSpec, with or- acle knowledge of future draft utility under the same memory constraint. • MemSpec: our prediction-guided, memory-aware run- time. For MAB-Async, we implement an asynchronous bandit- based selection strategy that overlaps draft loading with ongoing decoding. This design gives the adaptive baseline the benefit of non-blocking loading and therefore represents a stronger comparison than a synchronous exploration strat- egy. We use a UCB-based policy similar to prior bandit-based adaptive speculative decoding approaches such as Bandit- Spec [14], and initially explore each draft model once before adaptive selection begins. During decoding, MAB-Async adaptively updates draft selection decisions based on ob- served runtime behavior. When the selected draft is non- resident, loading proceeds asynchronously while decoding continues with the currently resident draft. We evaluate ex- ploration coefficients(푐 ∈ 0.1,0.5,1.0,2.0,3.0)and use the best-performing configuration (푐= 2.0) in all experiments. These baselines cover static, adaptive, and oracle con- figurations. Comparing Oracle-Static and Oracle-Dynamic isolates the benefit of dynamic adaptation beyond optimal static selection, while comparing MAB-Async and MemSpec highlights the benefit of prediction-guided, memory-aware scheduling over exploration-based adaptation. 4.2 Overall Throughput Figure 6 shows normalized steady-state generation through- put (normalized to General-Static) across all workloads for both LLaMA-2 and Qwen2.5. MemSpec consistently out- performs both static baselines and the adaptive MAB-Async method across all datasets. Oracle-Static improves throughput by 22.5% on average over General-Static, demonstrating that selecting an appropri- ate draft model is critical for performance. However, Oracle- Dynamic achieves a further 24.9% improvement over Oracle- Static, revealing substantial additional headroom from dy- namic adaptation MemSpec closely approaches Oracle-Dynamic, achieving 95–97% of its throughput while remaining practical for de- ployment. Compared with MAB-Async, MemSpec improves throughput by approximately 40.7% on average across work- loads. These results reveal two important insights. First, there exists substantial dynamic headroom beyond static selection. While Oracle-Static already captures the best per-prompt draft, Oracle-Dynamic further improves performance, in- dicating that the most effective draft can change within a single generation. Second, realizing this headroom on edge devices requires more than better draft identification. Al- though MAB-Async improves selection quality, it fails to convert much of this benefit into throughput because effec- tive drafts are often not resident when needed. LCTES ’26, June 15–16, 2026, Boulder, CO, USAEunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han 0.0 0.5 1.0 1.5 2.0 Alpaca LiveCode Bench Omni- MATH MMLU- Law MMLU- Medical Alpaca LiveCode Bench Omni- MATH MMLU- Law MMLU- Medical GMEAN LLaMA-2Qwen2.5 Norm. Throughput General-StaticOracle-StaticMAB-AsyncOracle-DynamicMemSpec Figure 6. Normalized steady-state generation throughput across workloads. 0.0 0.2 0.4 0.6 0.8 1.0 1.2 MAB-Async MemSpec MAB-Async MemSpec LLaMA-2Qwen2.5 Execution Time Ratio of MemSpec to MAB-Async Desired Exec.Fallback Exec.Prediction Overhead Figure 7. Execution time breakdown of MAB-Async and MemSpec. MemSpec bridges this gap by aligning draft selection with runtime availability. By proactively maintaining a small set of high-utility drafts, it avoids excessive switching while preserving most of the gains of dynamic adaptation. The remaining gap to Oracle-Dynamic is relatively small, sug- gesting that prediction-guided scheduling is sufficient to approximate near-ideal behavior without exhaustive oracle knowledge. Overall, these results demonstrate that the key bottleneck on edge devices is not simply identifying better drafts, but ensuring that they are available at execution time. By jointly optimizing draft selection and draft residency, MemSpec captures most of the dynamic headroom of Oracle-Dynamic without requiring impractical offline enumeration. 4.3 Execution Breakdown To understand the performance gap, we decompose execu- tion time into three components: desired execution, fallback execution, and prediction overhead. Desired execution corre- sponds to decoding with the preferred draft, while fallback execution corresponds to decoding with a non-optimal resi- dent draft when the desired draft is not yet available. Figure 7 shows that MAB-Async spends a large fraction of execution time in fallback execution, accounting for 49.4% of total runtime on average across model families. This reflects a key limitation of exploration-based adaptation: even when a better draft is identified, it is often unavailable at execu- tion time, forcing the runtime to continue decoding with a suboptimal resident draft. In contrast, MemSpec reduces fallback execution to below 5.5% while substantially increasing the fraction of desired ex- ecution. Prediction overhead remains negligible, accounting for less than 3.9% of total runtime. A closer inspection reveals that this reduction is primar- ily due to improved temporal alignment between selection and residency. In MAB-Async, draft decisions are driven by online exploration and therefore react to past observations, but do not explicitly prepare high-utility drafts before they are needed. As a result, the runtime frequently spends time executing available but inferior drafts while preferred ones are still being loaded. By contrast, MemSpec uses prediction to anticipate near- future draft utility from both prompt semantics and recent generation context. This allows the runtime to initiate prefetch ahead of time and convert reactive switching into proactive preparation. The resulting increase in desired exe- cution shows that the performance advantage of MemSpec comes not only from selecting better drafts, but from ensur- ing that those drafts are actually runnable at the scheduling point. These results highlight a fundamental distinction between selection quality and execution quality. While MAB-Async can identify effective drafts, it does not ensure that they are available at execution time. MemSpec improves throughput by aligning draft selection with runtime availability through proactive residency management. 4.4 Impact of Design Components We analyze the contribution of MemSpec’s key design com- ponents by comparing the following configurations: •General-Static: a non-adaptive baseline with no pre- diction or runtime draft management. • Prediction-Only: uses prediction-guided draft selec- tion without proactive prefetching. • MemSpec: the full design with prediction-guided scheduling and memory-aware prefetching. Figure 8 shows that Prediction-Only improves throughput by 21.6% over General-Static, confirming that context-aware Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge DevicesLCTES ’26, June 15–16, 2026, Boulder, CO, USA 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 General-Static Prediction-OnlyMemSpec Normalized Throughput Figure 8. Throughput comparison for component analysis. 0.0 0.2 0.4 0.6 0.8 1.0 1.2 General-Static Prediction-OnlyMemSpec Norm. Execution Time Desired Exec.Fallback Exec.Prediction Overhead Figure 9. Execution breakdown for component analysis. draft selection is beneficial. However, it still falls short of MemSpec by 30.5%. Figure 9 explains this gap. Prediction-Only spends 21.3% of runtime in fallback execution, while MemSpec reduces this to below 5.3%. This shows that prediction alone is insufficient: performance gains are realized only when predicted drafts are made available at execution time. These results highlight an important system-level insight: prediction accuracy alone is not the limiting factor in adap- tive decoding under memory constraints. Even when the runtime can identify effective drafts, those drafts may still fail to improve throughput if they are unavailable when needed. Without proactive cache management, prediction quality does not directly translate into execution quality. MemSpec addresses this limitation by coupling prediction with residency management, ensuring that high-utility drafts are not only selected but also prepared in advance. This tight integration between prediction and scheduling is essential for realizing the full benefit of adaptive decoding. Overall, MemSpec derives its performance from combin- ing prediction with memory-aware scheduling, rather than prediction alone. More broadly, these results show that adap- tive speculative decoding on edge devices is fundamentally a joint optimization problem over draft selection and memory management. 4.5 Sensitivity Analysis We analyze the sensitivity of MemSpec to several key run- time parameters. 0.0 0.3 0.6 0.9 1.2 1.5 1.8 24816 Normalized Throughput Scheduling Interval Figure 10. Sensitivity to scheduling interval. 0.0 0.3 0.6 0.9 1.2 1.5 1.8 64128256 Normalized Throughput Output Sequence Length Figure 11. Sensitivity to output length. Scheduling interval. Figure 10 shows the effect of vary- ing the scheduling interval. Throughput improves from푁= 2 to푁=4, remains nearly unchanged at푁=8, and de- creases at푁=16. This trend reflects the trade-off between adaptation frequency and prefetch opportunity. When the interval is too small, frequent rescheduling leaves less time to overlap model loading with decoding. When the interval is too large, beneficial draft transitions are delayed. The relatively flat performance between푁=4 and푁=8 suggests that MemSpec does not require overly fine-grained parameter tuning to perform well. This is desirable for prac- tical deployment, where the best interval may vary slightly across platforms and workloads. Overall, these results in- dicate that a moderate interval provides the best balance, and that MemSpec remains robust over a reasonably wide operating range around the default setting. Output length. Figure 11 shows the effect of varying output length. MemSpec achieves larger gains as generation becomes longer. Compared with short outputs, longer gen- erations provide more opportunity to exploit intra-sequence variation and amortize the cost of draft adaptation. As a re- sult, runtime draft switching becomes increasingly beneficial as output length grows. This trend also indicates that the benefit of memory-aware scheduling becomes more pronounced as generation com- plexity increases. Longer outputs typically contain more phase transitions and greater intra-sequence heterogeneity, making static draft selection increasingly suboptimal. In such LCTES ’26, June 15–16, 2026, Boulder, CO, USAEunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 234 Normalized Throughput Resident Cache Capacity (K) MAB-AsyncMemSpec Figure 12. Sensitivity to resident cache capacity퐾on Jetson AGX Orin. settings, proactively adapting the resident working set pro- vides greater benefit than in short, relatively homogeneous generations. These results further support the central motivation of MemSpec: dynamic draft adaptation is especially important for longer and more heterogeneous generation workloads, and the advantage of MemSpec scales with the opportunity for adaptation. Resident cache capacity. We additionally evaluate sensi- tivity to resident cache capacity퐾, which controls how many draft models can remain resident simultaneously. Because the Jetson Orin Nano platform used in the main evaluation can hold at most two draft models under the default con- figuration, we conduct this additional analysis on a larger- memory Jetson AGX Orin platform. Figure 12 shows that MemSpec already achieves most of its performance benefit at relatively small cache capacities. While MAB-Async improves throughput from 1.13×to 1.29× as퐾increases from 2 to 4, MemSpec shows only modest improvement (from 1.47×to 1.51×). This behavior is consis- tent with the predictor analysis in Section 4.6, which showed high top-2 recall. Since high-utility drafts are already in- cluded in the resident working set in most cases, increasing cache capacity further provides limited additional benefit for MemSpec. In contrast, MAB-Async benefits more noticeably from larger resident cache capacity. As퐾increases, reactive draft loading becomes less frequent because the runtime expe- riences fewer non-resident draft selections. Consequently, the performance gap between MemSpec and MAB-Async de- creases slightly at larger퐾. Nevertheless, MemSpec consis- tently achieves higher throughput across all evaluated cache capacities, demonstrating that prediction-guided residency management remains beneficial even when more memory is available. 4.6 Discussion Predictor quality. MemSpec does not require perfectly accurate draft prediction to improve throughput. The predic- tor is primarily used to identify promising drafts for proac- Table 2. Predictor quality comparison. InputTop-1 Acc. (%)Top-2 Recall (%) Prompt-only31.955.8 Recent-only57.179.3 Prompt + Recent71.695.7 tive residency management rather than to select the exact optimal draft at every scheduling point. To evaluate predictor quality, we compare three predictor input configurations: prompt-only, recent-generated-tokens- only, and a combined configuration using both prompt and recent generated tokens. The combined predictor achieves the best ranking quality, improving top-1 accuracy to 71.6% and achieving a top-2 recall of 95.7%. This indicates that the optimal draft is al- most always included in the resident working set. These results support the key design principle of MemSpec: adap- tive scheduling mainly requires sufficiently accurate ranking to maintain high-utility drafts in the resident set, rather than perfect prediction. End-to-end latency considerations. Our main evalua- tion focuses on steady-state iterative decoding and therefore excludes prompt processing and initial model loading. To assess startup overhead, we additionally measure end-to-end latency including prompt processing, target-model prefill, initial draft loading, and draft-model preparation. Although including startup costs reduces relative gains, MemSpec still reduces end-to-end latency by 32.3% on aver- age over General-Static and by 24.9% over MAB-Async. These results remain consistent with our main findings: the pri- mary benefit of MemSpec comes from reducing repeated draft loading and fallback execution during iterative decod- ing. 5 Related Work 5.1 Speculative Decoding and Adaptive Draft Selection Speculative decoding accelerates autoregressive LLM infer- ence by using a lightweight draft model to propose candidate tokens that are then verified by a larger target model [5,20]. Prior work has improved this framework along multiple di- rections, including specialized or distilled draft models [39, 42] and system optimizations that reduce verification over- head or improve pipeline efficiency [23–25, 41]. More recent studies observe that draft effectiveness varies across inputs and even across generation stages within a sin- gle sequence. This has led to adaptive speculative decoding methods that select among multiple candidate drafts at run- time rather than relying on a single fixed drafter [14,18,22]. In particular, multi-drafter and bandit-based approaches im- prove acceptance through online exploration and adaptation Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge DevicesLCTES ’26, June 15–16, 2026, Boulder, CO, USA to observed runtime behavior. However, these methods primarily optimize which draft to select, implicitly assuming that the chosen draft can be executed immediately. That assumption is often invalid on memory-constrained edge devices, where only a small num- ber of drafts can remain resident and switching to a non- resident draft incurs substantial loading overhead. MemSpec differs from prior adaptive speculative decod- ing work by treating draft selection and draft availability as a coupled systems problem. Rather than focusing only on acceptance-rate improvement, it explicitly accounts for resi- dency and switching cost so that adaptive selection translates into throughput gains under tight memory budgets. 5.2 Adaptive Model Selection and Online Routing A broad line of research studies adaptive model selection in settings such as cascaded inference [10,26,36] and model routing [6,7,10,33]. These approaches dynamically select among candidate models based on input difficulty, confi- dence estimates, or runtime feedback to balance accuracy and efficiency. Some methods further employ online learning or bandit algorithms to manage the exploration–exploitation trade-off. MemSpec is related to this literature in that it also performs runtime model selection, but the underlying systems con- straints are fundamentally different. Conventional routing formulations typically assume that all candidate models are readily available for immediate execution, and thus optimize only the selection decision itself. In contrast, MemSpec operates in a setting where model availability is constrained by memory capacity and model loading latency. Unlike conventional adaptive routing set- tings, speculative decoding requires repeated draft decisions within a single generation trajectory, making switching over- head a first-order cost. MemSpec therefore extends adaptive model selection with residency-aware scheduling, separat- ing which drafts should be prepared from which resident draft should be executed immediately. 5.3 Memory-Constrained LLM Inference on Edge Devices Running LLMs on edge devices has motivated extensive work on reducing memory footprint and managing lim- ited memory resources. Existing approaches include model compression (e.g., quantization, distillation) to shrink model size [11,19,21,37] and parameter offloading across GPU, CPU, and storage tiers [1,17,29–31]. These techniques make large-model inference feasible on resource-limited platforms. Our work is complementary to these efforts but addresses a different challenge. Prior memory-optimization methods mainly focus on optimizing the execution of a single large model by reducing its footprint or by moving its parameters more efficiently across memory tiers. By contrast, MemSpec targets adaptive speculative decod- ing with multiple candidate draft models, where the main challenge is not only executing one model efficiently, but deciding which drafts should remain resident, which should be prefetched, and when switching is worthwhile. In this sense, prior edge inference techniques reduce the per-model memory cost, while MemSpec addresses the run- time scheduling problem of managing multiple drafts under a tight memory budget. These directions are orthogonal and potentially complementary. 6 Conclusion This paper presents MemSpec, a memory-aware runtime system for adaptive draft scheduling in speculative decoding on memory-constrained edge platforms. Through detailed characterization on a Jetson Orin Nano device, we show that although generation heterogeneity creates substantial opportunities for improving token acceptance, adaptive draft switching incurs significant model loading overhead—often exceeding multiple decoding iterations—which limits end- to-end throughput gains. To address this challenge, MemSpec formulates adaptive speculative decoding as a residency-constrained schedul- ing problem. By combining prediction-guided draft rank- ing with memory-aware residency management, MemSpec proactively aligns draft selection with runtime availability, allowing decoding to proceed with the best currently resi- dent draft while asynchronously preparing better candidates in the background. Experimental results show that Mem- Spec improves steady-state generation throughput by 40.7% on average over state-of-the-art adaptive baselines, while achieving 95–97% of an oracle dynamic upper bound under the same memory constraints. These results demonstrate that the primary bottleneck in adaptive speculative decoding on edge devices is not draft selection quality, but ensuring timely availability of effective drafts. Acknowledgements This research was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korean gov- ernment (MSIT) (No. RS-2025-24535034), and in part by the Advanced GPU Utilization Support Program funded by the Korean government (MSIT). Myeonggyun Han is the corre- sponding author. References [1] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatam- ifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a flash: Efficient Large Language Model Inference with Limited Memory. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Lun- Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Bangkok, Thailand, 12562–12584. doi:10.18653/v1/2024.acl-long.678 [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, LCTES ’26, June 15–16, 2026, Boulder, CO, USAEunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [3] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. MEDUSA: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. In Proceedings of the International Conference on Machine Learning (ICML). Vienna, Austria. [4]Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Smaranda Muresan, Preslav Nakov, and Aline Villav- icencio (Eds.). Dublin, Ireland, 4310–4330. doi:10.18653/v1/2022.acl- long.297 [5] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Lan- guage Model Decoding with Speculative Sampling. arXiv:2302.01318 [6]Lingjiao Chen, Matei Zaharia, and James Zou. 2022. Efficient Online ML API Selection for Multi-Label Classification Tasks. In Proceedings of the 39th International Conference on Machine Learning, Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.), Vol. 162. Baltimore, Maryland, USA, 3716–3746. https://proceedings.mlr.press/v162/chen22ad.html [7]Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research 2024 (2024). https://openreview.net/forum?id=cSimKw5p6R [8]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, An- drew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Lan- guage Models Trained on Code. arXiv:2107.03374 [9]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hee- woo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Train- ing Verifiers to Solve Math Word Problems. arXiv:2110.14168 [10]Jasper Dekoninck, Maximilian Baader, and Martin Vechev. 2025. A Unified Approach to Routing and Cascading for LLMs. In Proceedings of the International Conference on Machine Learning (ICML), Vol. 267. Vancouver, Canada. https://openreview.net/forum?id=AAl89VNNy1 [11]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transform- ers. In Proceedings of the International Conference on Learning Repre- sentations (ICLR). Kigali, Rwanda. https://openreview.net/forum?id= tcbBPnfwxS [12]Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2024. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models. arXiv:2410.07985 [13]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. New Orleans, LA, USA. https://openreview.net/forum? id=7Bywt2mQsCe [14] Yunlong Hou, Fengzhuo Zhang, Cunxiao Du, Xuan Zhang, Jiachun Pan, Tianyu Pang, Chao Du, Vincent Y. F. Tan, and Zhuoran Yang. 2025. BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms. In Proceedings of the International Conference on Machine Learning (ICML), Vol. 267. Vancouver, Canada. https://openreview.net/forum? id=ghkWIlliZ8 [15]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Sto- ica. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [16]Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences 11, 14 (2021). doi:10.3390/app11146421 [17]Sowoong Kim, Eunyeong Sim, Youngsam Shin, YeonGon Cho, and Woongki Baek. 2024. Activation Sequence Caching: High-Throughput and Memory-Efficient Generative Inference with a Single GPU. In Proceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques (PACT ’24’). New York, NY, USA, 78–90. doi:10.1145/3656019.3676945 [18]Taehyeon Kim, Hojung Jung, and Se-Young Yun. 2025. A Unified Framework for Speculative Decoding with Multiple Drafters as a Ban- dit. https://openreview.net/forum?id=5haYLrlyGj [19]Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. 2025. DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. In Proceedings of the Inter- national Conference on Machine Learning (ICML), Vol. 267. Vancouver, Canada. https://openreview.net/forum?id=rc65N9xIrY [20] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Infer- ence from Transformers via Speculative Decoding. In Proceedings of the International Conference on Machine Learning (ICML). Honolulu, Hawaii, USA. [21]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-Aware Weight Quantiza- tion for On-Device LLM Compression and Acceleration. In Pro- ceedings of Machine Learning and Systems (MLSys), P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. Santa Clara, CA, USA, 87–100. https://proceedings.mlsys.org/paper_files/paper/2024/file/ 42a452cbafa9d64e9ba4a95c1ef21-Paper-Conference.pdf [22]Hongyi Liu, Jiaji Huang, Zhen Jia, Youngsuk Park, and Yu-Xiang Wang. 2026. Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs. In Proceedings of the International Conference on Learning Representations (ICLR). Rio de Janeiro, Brazil. https://openreview.net/forum?id=JMmljf895g [23]Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. 2025. PEARL: Parallel Speculative Decoding with Adaptive Draft Length. In Proceedings of the International Conference on Learning Representations (ICLR). Singapore. https://openreview.net/forum?id= QOXrVMiHGK [24] Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Na- dav Timor, Moshe Wasserblat, and Roy Schwartz. 2024. Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models. In Proceedings of The 4th NeurIPS Efficient Natu- ral Language and Speech Processing Workshop, Mehdi Rezagholizadeh, Peyman Passban, Soheila Samiee, Vahid Partovi Nia, Yu Cheng, Yue Deng, Qun Liu, and Boxing Chen (Eds.), Vol. 262. Vancouver, Canada, 456–467. https://proceedings.mlr.press/v262/mamou24a.html [25] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Lan- Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge DevicesLCTES ’26, June 15–16, 2026, Boulder, CO, USA guage Model Serving with Tree-Based Speculative Inference and Verification. In Proceedings of the 29th ACM International Confer- ence on Architectural Support for Programming Languages and Op- erating Systems, Volume 3 (ASPLOS ’24’). New York, NY, USA, 932–949. doi:10.1145/3620666.3651335 [26]Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Se- ungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. 2025. Faster Cascades via Speculative Decoding. In Proceedings of the International Conference on Learning Representations (ICLR) (Singa- pore). https://openreview.net/forum?id=vo9t20wsmd [27] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A Large-Scale Multi-Subject Multi-Choice Dataset for Medical Domain Question Answering. In Proceedings of the Con- ference on Health, Inference, and Learning, Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (Eds.), Vol. 174. 415 Main Street, Cambridge, MA USA, 248–260. https://proceedings. mlr.press/v174/pal22a.html [28]Ruiyang Qin, Jun Xia, Zhenge Jia, Meng Jiang, Ahmed Abbasi, Peipei Zhou, Jingtong Hu, and Yiyu Shi. 2024. Enabling On-Device Large Lan- guage Model Personalization with Self-Supervised Data Selection and Synthesis. In Proceedings of the 61st ACM/IEEE Design Automation Con- ference (DAC ’24’). New York, NY, USA. doi:10.1145/3649329.3655665 [29]Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Lan- guage Models with a Single GPU. In Proceedings of the International Conference on Machine Learning (ICML). [30]Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). New York, NY, USA, 590–606. doi:10.1145/3694715.3695964 [31]Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhi- hao Jia, and Max Ryabinin. 2024. SpecExec: Massively Parallel Spec- ulative Decoding for Interactive LLM Inference on Consumer De- vices. In Advances in Neural Information Processing Systems (NeurIPS), A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom- czak, and C. Zhang (Eds.), Vol. 37. Vancouver, Canada, 16342– 16368.https://proceedings.neurips.c/paper_files/paper/2024/file/ 1d91d5689e251d27993a3c2182dddcf7-Paper-Conference.pdf [32] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-Following LLaMA Model. https:// github.com/tatsu-lab/stanford_alpaca. [33]Ben Taylor, Vicent Sanz Marco, Willy Wolff, Yehia Elkhatib, and Zheng Wang. 2018. Adaptive Deep Learning Model Selection on Embedded Systems. In Proceedings of the 19th ACM SIGPLAN/SIGBED Interna- tional Conference on Languages, Compilers, and Tools for Embedded Sys- tems (LCTES ’18) (Philadelphia, PA, USA). 31–43. doi:10.1145/3211332. 3211336 [34]Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [35]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xi- ang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [36]Neeraj Varshney and Chitta Baral. 2022.Model Cascading: To- wards Jointly Improving Efficiency and Accuracy of NLP Systems. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Abu Dhabi, United Arab Emirates, 11007–11021. doi:10.18653/v1/2022.emnlp-main.756 [37]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the In- ternational Conference on Machine Learning (ICML), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Vol. 202. Honolulu, Hawaii, USA, 38087– 38099. https://proceedings.mlr.press/v202/xiao23c.html [38]Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. 2025. Decoding Speculative Decoding. In Proceedings of the Annual Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Albuquerque, New Mexico, 6460–6473. doi:10.18653/ v1/2025.naacl-long.328 [39]Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, and Se- Young Yun. 2024. Towards Fast Multilingual LLM Inference: Specula- tive Decoding and Specialized Drafters. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Miami, Florida, USA, 10789–10802. doi:10.18653/v1/2024.emnlp-main.602 [40] Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. 2025. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices. IEEE Transactions on Mobile Computing 24, 8 (2025), 7059–7073. doi:10.1109/TMC.2025.3546466 [41]Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, and Zhaopeng Tu. 2025. Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Suzhou, China, 16696–16708. doi:10.18653/v1/2025.emnlp-main.844 [42]Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. 2024. DistillSpec: Improving Speculative Decoding via Knowledge Distillation. In Proceedings of the International Con- ference on Learning Representations (ICLR). Vienna, Austria. https: //openreview.net/forum?id=rsY6J3ZaTF