Paper deep dive
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, Jiliang Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/9/2026, 3:38:37 AM
Summary
This paper presents a comprehensive cross-layer measurement study of mobile Large Language Model (LLM) inference, analyzing performance and energy efficiency across CPU, GPU, and NPU backends alongside five mainstream inference frameworks. The authors introduce PowerBench, a fine-grained profiling tool for backend-specific energy attribution. Key findings indicate that framework-induced performance gaps are significantly amplified on NPUs, NPUs excel in compute-bound prefilling while CPUs dominate memory-bound decoding, and suboptimal resource scheduling causes up to 40% energy waste. An optimized configuration is proposed to reduce energy consumption by up to 54.8%.
Entities (8)
Relation Signals (5)
NPU → hasphasepreference → Prefill Phase
confidence 98% · NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding.
PowerBench → enablescapability → Fine-grained Energy Attribution
confidence 95% · PowerBench enables fine-grained, backend-specific energy attribution by capturing both full-SoC and per-compute-unit consumption.
CPU → hasphasepreference → Decode Phase
confidence 95% · CPUs outperform all other backends in memory-bound decoding.
Scheduling Inefficiencies → causes → Energy Waste
confidence 92% · Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste.
Inference Frameworks → exhibitsperformancegapon → NPU
confidence 90% · Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., this http URL, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.
Tags
Links
- Source: https://arxiv.org/abs/2607.05475v1
- Canonical: https://arxiv.org/abs/2607.05475v1
Trouble viewing inline? Open PDF directly →
Full Text
77,784 characters extracted from source content.
Expand or collapse full text
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference Guanyu Cai caigy25@mails.tsinghua.edu.cn Tsinghua University Beijing, China Ruiming Tian 22301131@bjtu.edu.cn Beijing Jiaotong University Beijing, China Lang Yang yanglang21@mails.tsinghua.edu.cn Tsinghua University Beijing, China Zhouhong Ren 23126462@bjtu.edu.cn Beijing Jiaotong University Beijing, China Jinliang Yuan yuanjinliang@tsinghua.edu.cn Tsinghua University Beijing, China Lingkun Li lkli@bjtu.edu.cn Beijing Jiaotong University Beijing, China Jiliang Wang jiliangwang@tsinghua.edu.cn Tsinghua University Beijing, China Abstract Deploying Large Language Models (LLMs) on mobile de- vices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frame- works (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop Power- Bench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond tradi- tional device-level measurements. Our study yields three crit- ical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10×using custom operators due to divergent offloading and quantiza- tion strategies. (2) We identify a distinct “phase split” where NPUs excel at compute-bound prefilling, while CPUs outper- form all other backends in memory-bound decoding. This is driven by the NPU’s preference for large, fixed-shape work- loads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substan- tial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy con- sumption by up to 54.8% on the NPU backend across three datasets. Keywords Mobile LLM, Performance Measurement, Energy Efficiency GENIE Intra-backend llama.cpp Cross backend Framework Backend Scheduling NPUCPU GPU Bottlenecks PowerBench CPU/GPU/NPU Energy Split Prefill/Decode Throughput 2 3 1 Measure Identify Guide MLLM MNN MLC-LLM Figure 1: Overview of our cross-layer measurement. PowerBenchmeasures backend-specific energy and throughput to identify bottlenecks and guide optimiza- tion for mobile LLM inference. 1 Introduction Large Language Models (LLMs) have demonstrated remark- able capabilities in language understanding, reasoning, and multimodal interaction [1,2]. Deploying LLMs on mobile devices is becoming increasingly attractive, motivated by the need for lower latency, stronger privacy, and offline accessi- bility [3]. This is particularly important for LLM-powered mobile agents, which sustain interactive inference while invoking tools and manipulating user interfaces [4,5]. Sus- taining agent workflows demands both high throughput for interactive responsiveness and low energy consumption to preserve battery life, making efficiency the central challenge. This challenge has motivated a growing body of optimization arXiv:2607.05475v1 [cs.AR] 6 Jul 2026 Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang efforts across the LLM inference stack, including inference frameworks [6–8], execution backends [6,9], and resource scheduling [10, 11]. However, the current research landscape remains frag- mented, as existing works typically focus on isolated compo- nents of the LLM inference stack. This fragmentation limits the community’s ability to develop a holistic understanding of mobile LLM efficiency. While recent measurement stud- ies [11–14] have demonstrated the feasibility of on-device LLMs—characterizing the impact of model scale, quantiza- tion, and CPU/GPU backends—they leave two critical blind spots. First, despite NPUs emerging as a primary target for deployment, their performance characteristics across diverse frameworks remain poorly understood. Second, the synergis- tic impact of resource scheduling across heterogeneous back- ends on latency and energy consumption remains largely underexplored. We summarize the research coverage of our work with prior studies in Table 1. To address these blind spots, we conduct a comprehensive cross-layer measurement study of mobile LLM inference, as shown in Figure 1. It is the first to simultaneously inves- tigate NPU execution and resource scheduling. Our study encompasses five mainstream frameworks (e.g., llama.cpp and GENIE), three heterogeneous backends (CPU, GPU, and NPU), and diverse resource scheduling policies (e.g., DVFS). Measurement methodology: Our analysis is under- pinned by a unified measurement framework. To quantify energy efficiency, we developPowerBench, a lightweight, framework-agnostic instrumentation library. Unlike prior coarse-grained, device-level profiling,PowerBenchenables fine-grained, backend-specific energy attribution by captur- ing both full-SoC and per-compute-unit consumption. For throughput measurement, we instrument each framework to standardize workload execution, including precise con- trol over prompt token injection and end-of-sequence token replacement. This ensures that prefill and decode through- put are evaluated under consistent prompt and generation lengths. Collectively, these methods allow us to construct what is, to our knowledge, the first comprehensively con- trolled benchmark for mobile LLM inference efficiency, fea- turing over 400 configurations across various models, de- vices, frameworks, and inference stages. Our results reveal three key findings. (1) Framework-induced performance gaps are substan- tially amplified on NPUs. While performance diversity across frameworks is expected, we observe that NPU execu- tion significantly exacerbates these disparities. Even frame- works sharing the same vendor-provided backend (e.g., Qual- comm QNN) exhibit clear performance gaps: GENIE achieves 1219.2 tokens/s, outperforming MNN’s 700.9 tokens/s. This difference primarily stems from suboptimal offloading strate- gies, where inter-layer communication and layout conver- sion introduce significant overhead. The performance gap is further widened by operator implementation: in our evalua- tion, GENIE reaches 1463.7 tokens/s during prefilling, sur- passing llama.cpp (115.1 tokens/s) by over 15×, despite both maintaining nearly identical decoding throughput (∼23 to- kens/s). This divergence is driven by the differential scala- bility of operators; specifically, QNN’s optimized MatMul reduces compute cycles by over 20×compared to llama.cpp’s implementation for large input sizes. Furthermore, while activation quantization enhances throughput by over 2× and cuts energy consumption by over 50%, it may incur non-negligible accuracy degradation, particularly in smaller models. (2) Backend efficiency exhibits a distinct “phase split” between prefilling and decoding. The optimal execution backend depends fundamentally on the inference phase. NPU backends dominate the compute-bound prefilling stage, achieving peak throughput exceeding 1,400 tokens/s. How- ever, this ranking reverses during the decoding stage: CPUs typically emerge as the fastest backend (achieving 70 to- kens/s), with GPUs following closely, while NPUs consis- tently lag behind. We attribute this to a fundamental archi- tectural mismatch. Decoding operates over a continuously growing context, while static-graph NPU execution requires a large context window to be preallocated in advance. This poorly matches the mobile NPU’s preference for large, de- terministic, fixed-shape graphs and introduces avoidable per-token overhead, leaving 20.0–23.8% decode throughput headroom as context grows from 16 to 4096 tokens. (3) Inefficient resource scheduling leaves substantial headroom for optimization. Backend-specific energy attri- bution uncovers critical inefficiencies that remain obscured by coarse, device-level measurements. We find that dur- ing NPU-accelerated inference, the host CPU can account for up to 30% of the total system energy due to aggressive polling and synchronization, despite contributing minimally to model computation. By simply fine-tuning RPC polling intervals and NPU sleep latency, energy consumption per token can be reduced by 30.9–37.8% and 44.6–50.9%, respec- tively, typically with negligible throughput degradation. Fur- thermore, suboptimal thread-core affinity causes up to 35% performance variance, while fine-grained DVFS (Dynamic Voltage and Frequency Scaling) tuning can recover approx- imately 50% of wasted energy on certain backends. These findings underscore that mobile LLM efficiency is not just a kernel-level challenge but a complex scheduling problem requiring both intra-backend calibration and cross-backend coordination. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference Table 1: Comparison of research coverage, scheduling factors, and energy measurement granularity. Work Framework CoverageBackend CoverageScheduleEnergy Measurement llama.cppMLC- LLM MLLMMNNGENIECPUGPUNPUDVFS Impact Dispatch Impact Compute Unit Whole Device Lu et al. [12]✓–✓– Guo et al. [14]✓–✓–✓– Li et al. [13]✓–✓– MELTing Point [15]✓–✓–✓ Ours✓ Collectively, We estimate that coordinating CPU frequency with NPU sleep and polling parameters can reduce energy consumption by up to 54.8% on NPU. •First NPU-centric measurement: We present a com- prehensive measurement study across five frameworks (llama.cpp, MNN, GENIE, MLLM, MLC-LLM) and three hardware backends (CPU, GPU, NPU). By systematically exploring NPU-based deployment, we fill a critical blind spot in prior research and provide a holistic view of the mobile LLM ecosystem. •Fine-grained profiling toolkit: We develop a lightweight, framework-agnostic instrumentation suite that enables fine-grained power and throughput measurements. Unlike prior coarse device-level profiling, our toolkit uncovers hidden energy bottlenecks and execution inefficiencies at the compute-unit level (CPU, GPU, and NPU), providing the precision necessary for deep power analysis. • Hidden bottlenecks and best practices: Through ex- tensive experimentation, we derive actionable insights spanning kernel scalability, activation quantization, and backend coordination. We identify a critical "phase split" in efficiency and propose a phase-aware co-design of sched- uling and backend selection to guide future mobile LLM optimizations. 2 Background 2.1 LLM Inference Phases LLMs employ transformer architectures [16] with two dis- tinct inference phases that exhibit fundamentally different computational characteristics: Prefill Phase: Processes the entire input prompt in paral- lel to generate the KV cache [17,18]. This phase performs matrix-matrix multiplications (GEMM) with high arithmetic intensity, making it largely compute-bound; performance is therefore determined mainly by the backend’s peak FLOPS (floating-point operations per second). Decode Phase: Generates output tokens autoregressively, one token at a time. This phase performs matrix-vector mul- tiplications (GEMV) as it fetches the entire KV cache for each token generation. Its arithmetic intensity is much lower Table 2: Heterogeneous backend compute peaks for flagship mobile SoCs [19]. Vendor SoCCPU FP32 † GPU FP32 NPU INT8 ‡ (TFLOPS) (TFLOPS) (TOPS) Qualcomm 8 Elite Gen 50.493.68> 34 MediaTekDimensity 9500 0.405.27> 50 AppleA19 Pro0.302.48> 35 † CPU peaks are estimated from clock frequencies and per-core vector fused multiply-add throughput. ‡ NPU peaks are taken from the most recent SoCs with publicly available data. than that of the prefill phase, making it memory-bound; performance is therefore determined mainly by memory bandwidth. This phase difference fundamentally shapes backend per- formance and bottlenecks, as shown in Section 6. 2.2 Heterogeneous Compute Backends Modern mobile SoCs integrate three heterogeneous compute units that share the same main memory but differ substan- tially in peak compute capability, as summarized in Table 2. CPUs consist of ARM big.LITTLE clusters with vector ac- celeration for parallel arithmetic. Recent CPUs provide less than 0.5 TFLOPS under FP32. They offer the greatest flex- ibility and the lowest control overhead, making them suit- able for irregular workloads and latency-sensitive execution. GPUs consist of parallel cores exposed through compute APIs such as OpenCL. Recent mobile GPUs provide around 5 TFLOPS under FP32, about one order of magnitude higher than CPUs. NPUs are dedicated AI accelerators designed for dense tensor computation. Recent NPUs expose around 50 TOPS under INT8 arithmetic, suggesting another order-of- magnitude increase in nominal compute capability. However, this performance usually depends on vendor-specific SDKs and restricted operator support [20,21], resulting in less flexible execution models. Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang Table 3: Overview of representative mobile LLM in- ference frameworks. Symbols denote backend status: ✗ unsupported; ❍ supported; ✓ targeted. Framework Maintainer CPU GPU NPUVersion llama.cpp[22]Community ✓ ✓ ✓ Custom kernel (open source) eadc418 MNN[23]Alibaba✓ ✓ ✓ QNN kernel (closed source) 510ac8f MLC-LLM[24] Community ❍ ✓ ✗ 8f49ea6 MLLM[25]Community ❍ ❍ ✓ QNN kernel (closed source) 10d3d6a GENIE[26]Qualcomm ❍ ✗ ✓ QNN kernel (closed source) 2.39.0 2.3 Mobile Inference Frameworks Table 3 summarizes representative mobile LLM inference frameworks used in our study and reveals a highly heteroge- neous support landscape across different backends. Among them, llama.cpp has the most active open-source commu- nity, MNN benefits from commercial support, MLLM rep- resents an early NPU-oriented LLM inference framework, GENIE provides a Qualcomm vendor-optimized framework, and MLC-LLM offers a compiler-based cross-platform de- ployment stack. While CPU and GPU execution are broadly available, NPU support is limited and implemented through two different approaches. Frameworks such as GENIE, MNN, and MLLM adopt a QNN-based path, delegating execution to Qualcomm’s AI Engine Direct and Hexagon software stack and therefore inheriting the same vendor-provided kernels [20,21,26]. In contrast, llama.cpp follows a custom- kernel path, implementing operators at a lower level in open source. This approach provides greater flexibility and con- trol but also demands substantially higher engineering effort. The table further shows that backend support is not merely a binary capability distinction: frameworks vary in which backends they prioritize and in the maturity of each backend implementation. This diversity is important for our study because it indicates that performance differences may arise not only from the hardware backends themselves, but also from frameworks’ offloading strategies and operator imple- mentation choices, as shown in Section 5. 3 Infrastructure and Methodology Figure 2 illustrates our end-to-end evaluation infrastruc- ture. A host machine orchestrates all experiments, including model preparation, deployment, run configuration, and result collection. It communicates with target devices through An- droid Debug Bridge (ADB). Before each run, the host follows a protocol that includes thermal conditioning, background isolation, and repeated trials to ensure reproducibility (Sec- tion 3.4). The testbed comprises four smartphones. On each device, we evaluate five frameworks, namely llama.cpp, MLC- LLM, MLLM, MNN, and GENIE, across the CPU, GPU, and NPU backends. To collect energy measurements consistently CPU/GPU/NPU Energy CPU Prefill/Decode Throughput GPUNPU Host Server Quantized Models Devices Launch Llama.cppGENIE MLC-LLM MLLMMLC-LLM ...... PowerBench Plugin Report Qwen2.5-1.5B Qwen2.5-7B Llama3.2-1B Llama3.2-3B Phi-3.5-3.8B ... Inference Frameworks Figure 2: Overview of our evaluation infrastructure. across frameworks, we develop a lightweight instrumen- tation libraryPowerBenchand plug it into each framework binary (Section 3.3). For throughput measurements, the same harness standardizes benchmark settings across frameworks, including token size, runtime, and repeated runs (Section 3.2). The evaluated models include Llama 3.2-1B, Llama 3.2-3B, Qwen 2.5-1.5B, Qwen 2.5-7B, and Phi 3.5-3.8B in quantized deployment formats. Unless otherwise specified, we use 4-bit weight-only quantization (w4). All MLLM deployments use 4-bit weight and 16-bit activation quantization (w4a16), and within GENIE only Qwen 2.5-7B uses w4a16; the remaining GENIE models use w4. Overall, this controlled benchmark space spans more than 400 configurations across models, devices, frameworks, backends, and inference stages. The measurement results are streamed back to the host for offline analysis. 3.1 Hardware Platform We conduct experiments on four smartphones spanning three generations of Qualcomm Snapdragon platforms, as summarized in Table 4. The testbed covers SoCs from 2023 to 2025, including SM8650, SM8750, and SM8850, captur- ing both generational evolution and device-level variation. Across these devices, CPU and GPU frequencies increase steadily across generations, and the NPU evolves from V75 to V81. All phones provide 16 GB DRAM, offering a consis- tent memory-capacity baseline. We use Qualcomm platforms because they are common in prior mobile LLM studies, es- pecially those involving NPU execution [1,6,27], and be- cause their NPU stack currently has the broadest support across mainstream frameworks, enabling controlled evalua- tion across CPU, GPU, and NPU backends. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference Table 4: Devices used in our experiments. DeviceYear SoCCPUGPU NPU DRAM Xiaomi 172025 SM88502×4.6 GHz + 6×3.62 GHz1.20 GHzV8116 GB OnePlus 15 2025 SM88502×4.6 GHz + 6×3.62 GHz1.20 GHzV8116 GB Xiaomi 152024 SM8750 2×4.32 GHz + 6×3.53 GHz 1.10 GHzV7916 GB Xiaomi 142023 SM8650 1×3.3 GHz + 3×3.15 GHz 2×2.96 GHz + 2×2.27 GHz 1.00 GHzV7516 GB 3.2 Throughput Profiling We measure prefill and decode throughput. We also mea- sure layer and operator execution latency. Because differ- ent frameworks expose different benchmark and runtime interfaces, we build a benchmark tool for each framework and standardize the measurement protocol by controlling token size, benchmark runtime, and the number of repeated runs. This design keeps the workload scale and measure- ment duration comparable. Prefill throughput is computed as the prompt length divided by the elapsed time from in- ference start to the emission of the first output token. De- code throughput is computed as the number of generated tokens divided by the elapsed time between the first and the last emitted output tokens. To analyze the sources of performance differences across frameworks, we benchmark representative layers and operators covering the dominant LLM workloads, including attention and FFN layers, prefill GEMM operators, and decode GEMV operators. For each op- erator, we construct isolated micro-benchmarks using tensor shapes that match those encountered in real inference. 3.3 Energy Profiling We use four per-token energy metrics (μJ/token): SoC, CPU, GPU, and NPU. Here, SoC energy denotes total chip energy, including CPU, GPU, NPU, and other on-chip components, enabling efficiency analysis at multiple granularities. To at- tribute energy consumption to individual compute units, we developPowerBench, a lightweight C++ instrumentation li- brary built on top of the Qualcomm Power Telemetry (QPT) driver stack [28]. QPT exposes cumulative energy counters for power management integrated circuit (PMIC)-regulated power zones through the Linuxpowercapsysfs interface, covering the whole SoC as well as rails for the CPU clus- ters, GPU, and NPU. Unlike coarse device-level approaches such as Androiddumpsys batterystatsor external power monitors [10,15],PowerBenchautomatically discovers avail- able power zones, maps them to logical compute units, and records energy deltas throughstart()/stop()calls. Total energy is derived from counter differences, while a back- ground sampling thread records temporal power traces at configurable intervals for auxiliary analysis. The library is header-only and can be integrated into existing C++ bench- marks with minimal code changes. We use it uniformly across frameworks to ensure measurement consistency. This backend-level granularity allows us to identify and optimize energy waste that coarse device-level measurements cannot isolate, as shown in Section 7. 3.4 Experimental Protocol To ensure measurement reliability and reproducibility, we fol- low a strict experimental protocol. Before each experiment, devices are cooled to below 28 ◦ C to avoid thermal throttling. We further disable background services, notifications, and network connectivity. Devices run in airplane mode with the screen turned off, and only the inference process is ac- tive during measurement. Each configuration includes one warm-up run followed by at least three recorded trials, and we report the mean across repeated runs unless otherwise stated. 4 Benchmark and Research Questions We first conduct a benchmark across devices, models, frame- works, and backends to establish a holistic view of the current performance landscape. All frameworks are evaluated under their default high-performance settings, with 4-bit weight quantization and no activation quantization unless otherwise noted. This benchmark provides a unified view of throughput and energy for 256-token prefill and decode workloads. The major results are shown in Table 5. Full results are reported in Table 9 in the appendix. Summary of the table. First, prefill throughput is usu- ally highest on NPU when a mature NPU path is available; many GENIE and MLLM entries exceed 1,000 tokens/s, while CPU and GPU entries are lower. Second, decode throughput shifts toward CPU and GPU: for small and medium models, CPU and GPU often remain in the 20–70 tokens/s range, while NPU decode frequently falls behind. Third, model size and device generation behave largely as expected, as larger models reduce throughput and increase per-token energy. Expected trends from the benchmark. Some conclu- sions are intuitive and largely confirm prior understanding. Prefill is a large, token-parallel workload, so accelerator- oriented backends are expected to do well. The table shows clear NPU advantages for prefilling on supported frame- works. Throughput and energy broadly scale with model size: moving from 1B/1.5B models to 3B/7B models generally lowers throughput and raises energy consumption. Intriguing phenomena. Beyond these expected trends, the table also reveals several observations that cannot be ex- plained by model size or backend class alone. (1) Framework gaps are strongly amplified on NPUs. On the same OnePlus 15 device and Qwen2.5-1.5B model, GENIE reaches 1463.7 tokens/s in NPU prefill, while MNN reaches 700.9 tokens/s and llama.cpp reaches only 115.1 tokens/s; the same model on CPU or GPU shows much smaller framework gaps. (2) Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang Table 5: Averaged Throughput and Energy across Devices, Models, Frameworks, and Backends with 256 tokens. ModelBackend Framework Quantization PrefillDecode Throughput (tokens/s)Energy (휇J/token)Throughput (tokens/s)Energy (휇J/token) Xiaomi 17 OnePlus 15 Xiaomi 15 Xiaomi 14 Xiaomi 17 OnePlus 15 Xiaomi 17 OnePlus 15 Xiaomi 15 Xiaomi 14 Xiaomi 17 OnePlus 15 Qwen2.5-1.5B CPUllama.cppw4417.3299.9329.2157.23.3e41.5e451.655.753.934.51.2e59.7e4 CPUMNNw4349.5228.3259.4259.5 2.3e41.5e419.047.549.545.71.3e59.2e4 GPUllama.cppw4569.7754.8680.9365.71.0e41.1e438.850.348.631.81.1e51.4e5 GPUMNNw4392.2434.1406.1272.58.6e31.7e439.112.345.826.29.4e49.9e4 GPUMLC-LLMw424.342.445.9141.85.7e45.0e424.619.629.916.81.3e51.3e5 NPUGENIEw41213.0 1463.71219.2–6.8e35.6e323.123.020.7–4.3e53.2e5 NPUllama.cppw4107.6115.182.351.48.2e47.6e431.933.318.313.82.0e51.6e5 NPUMNNw4814.3700.9694.4593.88.3e38.1e39.910.316.914.82.1e52.5e5 NPUMLLMw4a16904.5966.8814.6–3.2e34.1e334.034.332.7–7.5e48.7e4 Qwen2.5-7B CPUllama.cppw485.458.655.134.11.1e57.9e412.614.313.28.75.0e54.4e5 CPUMNNw480.330.058.450.89.8e41.0e59.612.212.27.24.6e53.9e5 GPUllama.cppw4137.5171.6141.884.64.0e46.5e410.212.211.75.43.2e55.7e5 GPUMNNw478.4102.186.262.44.1e49.1e48.910.612.711.73.1e55.1e5 NPUGENIEw4a16859.11411.11020.9–9.4e36.9e314.616.813.6–4.0e54.0e5 NPUllama.cppw434.835.127.013.01.6e56.0e59.19.27.76.46.9e56.0e5 Llama3.2-1B CPUllama.cppw4326.8287.6386.3166.31.1e41.6e476.1 72.270.141.91.4e57.4e4 CPUMNNw429.8184.8324.8321.98.7e31.5e43.261.666.655.47.9e46.6e4 GPUllama.cppw4744.0986.8862.1450.16.3e35.4e346.164.457.122.51.4e51.1e5 GPUMNNw4692.1669.8534.4371.36.9e31.3e416.821.660.744.45.5e4 7.9e4 GPUMLC-LLMw4–189.3–31.619.940.120.11.0e59.3e4 NPUGENIEw41887.12316.11747.9–4.2e34.4e325.926.323.6–2.9e53.0e5 NPUllama.cppw4132.0141.393.542.37.6e44.6e441.341.621.013.82.4e51.3e5 NPUMNNw41215.6228.91043.9850.75.7e36.1e313.813.821.118.61.8e51.9e5 NPUMLLMw4a162163.2 2322.21716.0 1624.42.8e33.2e358.462.550.951.36.3e48.6e4 Llama3.2-3B CPUllama.cppw4119.7109.8114.858.65.4e44.1e429.9 30.527.117.12.9e52.1e5 CPUMNNw4123.466.190.8113.16.8e44.4e412.424.924.023.7 2.1e51.8e5 GPUllama.cppw4224.8357.6306.6161.12.2e42.1e420.025.025.211.11.8e52.5e5 GPUMNNw4201.2248.7205.3145.32.2e43.5e424.412.125.417.12.0e52.0e5 NPUGENIEw4615.7815.3656.3 –1.3e41.1e48.912.210.8–7.0e57.1e5 NPUllama.cppw448.143.034.115.31.6e51.2e519.119.310.13.94.4e52.9e5 NPUMNNw4533.597.2454.5440.11.3e41.4e46.010.58.27.54.3e57.3e5 NPUMLLMw4a16929.3 886.2–674.87.4e38.3e3– The best backend flips sharply between prefilling and decod- ing. For example, on Qwen2.5-1.5B decode on OnePlus 15, llama.cpp on CPU reaches 55.7 tokens/s, compared with 23.0 tokens/s for GENIE on NPU, indicating a phase-dependent backend split rather than a single universally best accelera- tor. (3) NPUs are widely marketed as highly energy-efficient AI accelerators, yet their measured decode energy is often surprisingly high. On OnePlus 15 with Qwen2.5-1.5B decode, GENIE on NPU consumes 3.2×10 5 휇J/token, compared with 9.7×10 4 휇 J/token for llama.cpp on CPU and 1.4×10 5 휇 J/token for llama.cpp on GPU. The same pattern appears on Xiaomi 17, where GENIE’s NPU decode consumes 4.3×10 5 휇J/token, far above 1.2× 10 5 휇J/token on CPU and 1.1× 10 5 휇J/token on GPU. Thus, despite the NPU’s reputation for energy effi- ciency, its end-to-end decode energy can be roughly 2–4× higher than CPU/GPU alternatives. These anomalies suggest that the efficiency problem spans framework design, backend architecture, and resource scheduling. These intriguing observations motivate three research questions: •RQ1: Framework gaps. What causes the large perfor- mance gaps across NPU frameworks? • RQ2: Backend optimization. Why does backend perfor- mance ranking reverse between prefilling and decoding? •RQ3: Resource Scheduling. How effective is resource scheduling in reducing unnecessary energy overhead? 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 Length N 10 6 10 7 10 8 Matmul cycles llama-shape 26.9× gap QNN llama.cpp 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 Length N qwen-shape 30.7× gap Figure 3: Comparison of matmul cycles between the QNN implementation used by GENIE and llama.cpp. 5 RQ1: Framework Gaps Mobile inference frameworks share a common pipeline, in- cluding graph conversion, quantization, backend selection, operator offloading, and runtime optimization. The bench- mark overview in Section 4 shows that framework gaps are particularly large on NPUs, especially in prefill. We find three major causes behind this amplification: offloading strategy, operator scaling, and activation quantization. 5.1 Finding 1: Substantial Offloading Overhead Both GENIE and MNN use Qualcomm QNN for NPU acceler- ation, yet their performance differs sharply. GENIE achieves nearly 2×the throughput of MNN. We find that this gap is mainly caused by different offloading granularity. GENIE executes the full graph on the NPU, whereas MNN keeps Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference Table 6: Workload breakdown for Qwen2.5-1.5B: GE- NIE (compute cycles) vs. MNN (end-to-end latency). Framework Component Latency Share GENIE NPU Attention44.79% NPU others55.21% MNN CPU Attention1.06% NPU Plugins98.94% self-attention on the CPU and offloads the feed-forward and other modules to the NPU. Table 6 shows a clear divergence between compute share and runtime share. In GENIE, non-attention modules account for 55.21% of true NPU compute cycles, but in MNN, NPU plugins occupy 98.94% of end-to-end latency. This indicates that MNN’s plugin time includes substantial overhead be- yond computation, such as CPU–NPU remote process calls and memory layout conversion, which significantly reduce NPU efficiency. 5.2 Finding 2: Operator Scaling Amplifies NPU Gaps On Qwen2.5-1.5B with the OnePlus 15 NPU backend, GE- NIE reaches 1463.7 tokens/s in NPU prefill, while llama.cpp reaches only 115.1 tokens/s, a 15×gap. In decode, however, the throughputs are nearly identical (23.0 vs. 22.8 tokens/s). This asymmetry indicates the importance of operator scala- bility across workload shapes. We isolate NPU MatMul efficiency by sweeping the output width푁, which corresponds to token-parallel width in prefill and is fixed to one in decode. Figure 3 shows a clear turning point: at decode-like sizes (푁 ≤ 4), llama.cpp is comparable to QNN. At푁=256, however, QNN reduces MatMul cycles by 26.9×–30.7×. At푁=1024, the gap remains 28.9×–48.0×. Prefill exposes large-푁GEMM; QNN scales efficiently in this regime, and GENIE inherits that advantage. Decode ex- poses small-푁kernels, where the operator advantage largely disappears, so llama.cpp remains competitive. Thus, large-푁 operator optimization is the bottleneck in open-source NPU paths. 5.3 Finding 3: Activation Quantization is Powerful but Risky Activation quantization is the third major source of frame- work divergence. On GENIE, comparing W4 (4-bit weight quantization) with W4A16 (4-bit weight and 16-bit activation quantization) for Llama-3.2-1B and Llama-3.2-3B shows con- sistent end-to-end gains. W4A16 improves throughput by 2.16×and 2.58×in prefill and by 2.63×and 2.34×in decode, while reducing energy per token by 53.5%–66.9%. 0.0 2.5 5.0 Speed (token/s) 1e3 Prefill 0 5 1e1 Decode 128256512102420484096 Prompt Tokens 0.0 0.5 1.0 Energy (uJ/token) 1e4 128256512102420484096 Generated Tokens 0 5 1e5 CPU GPU NPU Others Llama3.2-1B W4 Llama3.2-1B W4A16 Llama3.2-3B W4 Llama3.2-3B W4A16 Figure 4: The impact of activation quantization on throughput and energy in the GENIE framework. Conv2d Eltwise_Binary Softmax MatMul StridedSlice Output Rms Concat Operation Type 0.0 2.5 5.0 7.5 10.0 W4A16 Speedup 7.68× (37.7%) 2.96× (25.1%) 2.55× (23.8%) 0.92× (5.5%) 0.96× (5.0%) 0.69× (1.3%) 0.63× (0.9%) 1.95× (0.5%) Prefill Conv2d Eltwise_Binary MatMul StridedSlice Softmax Concat Output Rms Operation Type 0.0 2.5 5.0 7.5 10.0 12.5 6.85× (58.2%) 5.43× (24.0%) 0.84× (9.4%) 0.48× (5.2%) 1.34× (1.8%) 1.47× (1.1%) 1.53× (0.2%) 0.35× (0.1%) Decode Figure 5: Operator speedup breakdown for W4 vs W4A16 of Llama-3.2-3B. Percentages denote each op- erator’s contribution to the total compute cycles. The speedup is concentrated rather than uniform. Figure 5 shows that in Llama-3.2-3B, the gains are dominated by a few hot operators: in prefill, Conv2d, Eltwise_Binary, and Softmax account for 86.6% of W4 cycles and all accelerate substantially; in decode, Conv2d and Eltwise_Binary alone account for 82.2% of cycles and dominate the gain. Activation quantization may substantially reduce accu- racy. Perplexity, where lower is better, reflects the model’s language modeling quality. Table 7 shows strong model de- pendence: W4A16 increases perplexity by only 1.04% on Llama-3.2-3B, but by 30.36% on Llama-3.2-1B. Thus, acti- vation quantization can remove a substantial performance bottleneck on NPUs, but only when model accuracy is robust enough to tolerate it. 5.4 Summary RQ1 shows three key causes to framework gaps are par- tial offloading that amplifies communication and memory overhead, poor scaling for large operators, and missing or accuracy-limited activation quantization. Together, these mechanisms explain why frameworks on the same hardware can still differ by about 2×, and why custom-kernel paths can fall an order of magnitude behind vendor-optimized stacks in prefill. Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang Table 7: Perplexity (lower is better) comparison be- tween W4 and W4A16 quantization on WikiText-2. ModelPPL (W4) PPL (W4A16) RelativeΔ PPL (%) Llama-3.2-1B16.7321.81+30.36% Llama-3.2-3B11.5011.62+1.04% 6 RQ2: Backend Optimization CPU, GPU, and NPU backends expose different execution models, leading to distinct tradeoffs across inference stages. We compare them to identify backend-specific strengths and bottlenecks. 6.1 Finding 4: NPU’s Decode Weakness Although NPUs dominate compute-bound prefill [6,29], this advantage disappears in decode. Figure 7 shows that NPU decode is governed mainly by context size rather than out- put length. Throughput drops from 61.6 to 47.0 tokens/s for Llama-3.2-1B and from 43.9 to 35.1 tokens/s for Qwen2.5- 1.5B as context grows from 16 to 4096 tokens, i.e., by 23.8% and 20.0%, respectively. By contrast, at fixed context size, the curves for 15, 63, and 255 generated tokens nearly over- lap, showing that the per-token cost is set by context length rather than the remaining decode length. Decode emits one token per step, exposing little paral- lelism while repeatedly reading the growing KV cache. the workload is therefore memory-bound rather than compute- bound. In this regime, NPU peak compute is underutilized. NPU further amplifies the problem by executing decode with shape-specialized static graphs. To support longer decode, NPU typically set a large context size (e.g. 4096) and Each step becomes more expensive. This explains why context size, not the number of generated tokens, dominates NPU decode throughput. 6.2 Finding 5: CPU Backends Dominate Decode Throughput Across the four devices, CPU backends generally deliver the highest decode throughput. Figure 6 shows that CPU usually outperforms, or at least matches, GPU across model families and configurations; GPU wins only in a few isolated cases, which do not overturn the overall trend. More importantly, GPU is often a near-equivalent substi- tute for decode throughput. As decode length grows, the CPU advantage typically narrows, while sustained decoding on CPU increases occupancy and tightens CPU-side scheduling constraints. Offloading decode to GPU can therefore preserve near-CPU throughput while reducing CPU pressure, making GPU a practical auxiliary backend when CPU resources are scarce or reserved for concurrent tasks. With appropriate scheduling, GPU can also be more energy-efficient in some regimes; we analyze these cases in Section 7.2 and Section 7.3. 6.3 Summary RQ2 reveals strong phase and length dependence. NPU is best for compute-bound prefill, but decode shifts to an invocation- and bandwidth-dominated regime in which static-graph over- head and poor amortization expose the NPU’s main bottle- neck. CPU is usually the best decode backend because of its low dispatch overhead and effective cache use, while GPU becomes attractive for long-context decode when it can de- liver near-CPU throughput with lower CPU pressure and sometimes better energy efficiency. 7 RQ3: Scheduling Strategy Figure 8 shows that even when computation is offloaded to the GPU or NPU, the host CPU still accounts for a substan- tial share of total system energy due to dispatch, polling, synchronization, and runtime coordination. We study this scheduling problem from two perspectives: inter-backend co- ordination between the CPU and accelerator, including RPC polling, NPU sleep control, and CPU frequency selection for GPU/NPU backends; and intra-backend tuning within each processor, including thread–core affinity and backend-local DVFS. Prior work [10,11] has focused mainly on CPU and GPU scheduling, leaving NPU-specific tuning and cross-unit interactions underexplored. 7.1 Finding 6: NPU Invocation Optimization We first examine inter-backend coordination for the NPU backend. In modern mobile SoCs, the CPU and NPU operate independently and communicate through remote procedure calls (RPC). During LLM inference, the CPU sends parame- ters via RPC messages and the NPU executes operations; re- turn values are sent back asynchronously. This design makes end-to-end efficiency sensitive to two host–accelerator con- trol parameters: RPC polling interval and NPU sleep latency. RPC polling interval: Figure 9 sweeps the RPC polling interval from 5 to 65535휇s, with 9999휇s as the default setting in GENIE and MLLM. Across both models and both phases, throughput is largely insensitive to this setting, and mov- ing from the default to the energy-optimal setting changes throughput by only−7.2% to+0.1%. The energy effect is much stronger. Switching from the default to short polling intervals (5–20휇s) reduces total energy per token by 30.9%– 37.8%. The dominant source of this gain is the NPU rather than the host CPU. The NPU energy component drops by 49.8%–70.8%, whereas the CPU component changes only modestly. This indicates that faster completion detection Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference 128256512102420484096 Decode Length 0 50 100 token/s 81 52 76 54 45 37 46 44 42 38 30 26 24 21 21 19 17 14 20 20 20 19 17 15 26 24 19 16 14 11 19 20 19 17 14 11 47 47 46 41 35 28 39 37 37 35 32 28 14 13 8 11 10 8 9 10 10 10 9 8 XiaoMi 17 128256512102420484096 Decode Length 69 73 71 65 56 32 64 41 62 57 48 38 29 18 29 26 19 15 25 23 24 24 21 17 25 25 24 21 18 10 26 24 25 22 17 13 15 14 14 13 12 11 12 12 12 12 11 9 56 50 42 51 44 33 48 36 40 43 37 29 OnePlus 15 128256512102420484096 Decode Length 0 50 token/s 36 47 66 60 51 43 57 57 54 49 42 30 28 27 26 24 22 18 25 25 24 23 20 17 24 22 22 20 17 13 23 23 21 19 16 12 57 55 53 50 45 38 48 48 46 43 39 32 14 13 13 12 11 10 12 12 12 11 10 9 XiaoMi 15 128256512102420484096 Decode Length 41 42 41 37 32 24 20 23 22 19 15 12 18 17 16 14 13 10 11 11 11 10 9 7 14 15 13 12 10 7 10 10 10 9 7 6 36 27 33 23 22 19 32 33 32 31 27 23 9 7 7 7 7 6 9 8 8 8 7 6 XiaoMi 14 llama 1B (CPU) llama 1B (OpenCL) llama 3B (CPU) llama 3B (OpenCL) phi3 3B (CPU) phi3 3B (OpenCL) phi3_5 (CPU) phi3_5 (OpenCL) qwen 1.5B (CPU) qwen 1.5B (OpenCL) qwen 7B (CPU) qwen 7B (OpenCL) qwen2 1.5B (CPU) qwen2 1.5B (OpenCL) qwen2 7B (CPU) qwen2 7B (OpenCL) qwen7B (CPU) qwen7B (OpenCL) Figure 6: CPU and GPU Decode Throughput Across Devices for Each Model 1664256102420484096 Context Size 50 60 70 Throughput (tokens/s) Llama3.2-1B 1664256102420484096 Context Size 30 40 Qwen2.5-1.5B n_gen=15n_gen=63n_gen=255n_gen=511 Figure 7: Decode throughput of the QNN NPU backend across context sizes. Throughput mainly depends on context size. 12825651210242048409660008192 Token Size 0 2.5 5 7.5 Energy (uJ/token) CPU CPU CPU CPU CPU CPU CPU CPU GPU GPU GPU GPU GPU GPU GPU GPU NPU NPU NPU NPU NPU NPU NPU NPU ×10 5 CPU energy GPU energy NPU energy Other energy CPU backend GPU backend NPU backend Figure 8: Backend decode energy on Xiaomi 17 with Qwen2.5-1.5B. shortens the time the NPU remains unnecessarily active, while the extra host polling overhead is very small. NPU sleep latency: Figure 9 also sweeps NPU sleep la- tency from 5 to 65535휇s, with GENIE and MLLM default- ing to 40휇s. Here, the default is too aggressive for energy efficiency. The energy-optimal points shift to much larger values, between 1280 and 65535휇s depending on model and phase, and can reduce total energy per token by 44.6%–50.9%. Moreover, this saving does not sacrifice throughput, which ranges from a negligible 0.9% drop to a 16.2% increase; for Llama decode, the same setting is both throughput-optimal and energy-optimal. The reduction is driven primarily by the accelerator itself, whose NPU energy component falls by 66.7%–100.0%. The overall pattern suggests that, for mo- bile LLM inference, allowing the NPU to remain idle longer before wake/sleep transitions yields a substantially better efficiency point. 7.2 Finding 7: Thread-Core Affinity Matters We find that CPU thread–core affinity is not only an im- portant configuration for the CPU backend itself, but also a major factor shaping efficiency for GPU and NPU execution. CPU and NPU backends favor matched thread and core counts. For both the CPU and NPU backends, blindly increasing the thread count can significantly degrade decode throughput. As shown in Figures 10a and 10c, performance is typically highest when the thread count roughly matches the number of active cores; once the thread count exceeds core availability, throughput drops sharply and can even become impractically low. In other words, what matters is not core selection alone, but thread–core co-design. NPU execution prefers minimal host participation. Figures 10c and 10f further show that the NPU backend achieves its best throughput and energy efficiency with the smallest CPU thread count. Additional CPU threads mainly add orchestration overhead and contention, rather than use- ful parallelism. GPU execution needs enough, but not excessive, host concurrency. The GPU backend exhibits a more nuanced pattern. As shown in Figures 10b and 10e, setting the CPU thread count above the number of active cores has limited impact on throughput and can even reduce energy in some cases, but using too few threads can significantly hurt GPU throughput. This suggests that GPU execution benefits from sufficient host-side concurrency for dispatch and synchro- nization, yet does not follow the same “more threads is better” logic as CPU-only execution. Taken together, Figure 10 shows that thread affinity is a system-level scheduling knob rather than a CPU-only opti- mization. Even when the GPU or NPU performs most arith- metic work, the host-side thread configuration still shapes end-to-end throughput and energy. Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang 0.0 2.5 Speed (token/s) 1e3 Prefill 0.0 2.5 1e1 Decode 0.0 2.5 1e3 Prefill 0.0 2.5 1e1 Decode 5 2080 320 128051209999 2048065535 RPC Polling Time (us) 0 5 Energy (uJ/token) 1e3 34.3% lower 5 2080 320 128051209999 2048065535 RPC Polling Time (us) 0.0 2.5 1e5 37.8% lower 5 204080 320 12805120 2048065535 NPU Sleep Latency (us) 0 5 1e3 46.5% lower 5 204080 320 12805120 2048065535 NPU Sleep Latency (us) 0.0 2.5 1e5 50.9% lower CPUGPUNPUOthersQwen2.5-1.5BLlama3.2-1BDefault setting Figure 9: Impact of CPU–NPU invocation parameters on throughput and energy consumption. 1M0L2M0L3M0L4M0L5M0L6M0L0M1L1M1L2M1L3M1L4M1L5M1L6M1L0M2L1M2L2M2L3M2L4M2L5M2L6M2L Mask (M/L) 1 2 3 4 5 6 7 8 9 Thread <5<5 <5<5<5<5<5 <5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5 0 50 tokens/s (a) CPU: Speed. 1M0L2M0L3M0L4M0L5M0L6M0L0M1L1M1L2M1L3M1L4M1L5M1L6M1L0M2L1M2L2M2L3M2L4M2L5M2L6M2L Mask (M/L) 1 2 3 4 5 6 7 8 9 Thread 0 50 tokens/s (b) OpenCL: Speed. 1M0L2M0L3M0L4M0L5M0L6M0L0M1L1M1L2M1L3M1L4M1L5M1L6M1L0M2L1M2L2M2L3M2L4M2L5M2L6M2L Mask (M/L) 1 2 3 4 5 6 7 8 9 Thread <5 <5<5<5 <5<5<5<5 <5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5 <5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5<5 0 20 tokens/s (c) NPU: Speed. 1M0L2M0L3M0L4M0L5M0L6M0L0M1L1M1L2M1L3M1L4M1L5M1L6M1L0M2L1M2L2M2L3M2L4M2L5M2L6M2L Mask (M/L) 1 2 3 4 5 6 7 8 9 Thread 0 20 SoC Energy (J) (d) CPU: Energy. 1M0L2M0L3M0L4M0L5M0L6M0L0M1L1M1L2M1L3M1L4M1L5M1L6M1L0M2L1M2L2M2L3M2L4M2L5M2L6M2L Mask (M/L) 1 2 3 4 5 6 7 8 9 Thread 0 25 SoC Energy (J) (e) OpenCL: Energy. 1M0L2M0L3M0L4M0L5M0L6M0L0M1L1M1L2M1L3M1L4M1L5M1L6M1L0M2L1M2L2M2L3M2L4M2L5M2L6M2L Mask (M/L) 1 2 3 4 5 6 7 8 9 Thread 0 500 SoC Energy (J) (f) NPU: Energy. Figure 10: Impact of CPU thread masking on throughput (top row) and system energy (bottom row) across backends for Qwen2.5-1.5B decode on Xiaomi 17. 7.3 Finding 8: Energy-Unfriendly DVFS We next examine DVFS from both backend-local and cross- backend perspectives. We sweep CPU and GPU frequency levels, and we sweep NPU operating points over the joint NPU core/bus voltage-corner space. Across these settings, the defaults are often not energy-optimal. They can over- provision host compute for accelerator-heavy workloads, miss better mid-range operating points, and ignore phase- specific behavior. Available frequency scales. CPU cluster policy0 (cpu0– cpu5) and cluster policy6 (cpu6–cpu7) expose different fre- quency ranges. Because of vendor-enforced DVFS behavior, some frequency points cannot be fixed or maintained. Ac- cordingly, we use 21 valid CPU frequency-level combinations and 15 GPU frequency levels in our evaluation. The NPU frequency states are configured at the governor level and include disable, min, svs2, svs, svs_plus, nom, nom_plus, turbo, turbo_plus, turbo_l2, turbo_l3, and max. Backend-local DVFS. Default backend-local DVFS poli- cies leave substantial efficiency on the table, but the best response depends strongly on which processor is on the critical path. For the CPU backend, Figure 11b shows that the default DVFS setting is far from energy-optimal. Downclocking to frequency levels 8–9 reduces energy consumption by roughly 50% relative to the default setting, with only about a 13% drop in throughput. However, not all lower frequencies are bene- ficial: reducing the CPU frequency level from 20 to 14 incurs a 20% throughput loss while increasing energy consumption by 14%. In contrast, lowering the frequency level from 20 to 8 achieves a much better trade-off, reducing throughput by 20% while cutting energy consumption by 42%. For the GPU backend, the trade-off is much less favorable overall. Figure 11a shows that downclocking significantly reduces throughput, while the energy benefit is limited. At level 16, throughput is already 27% lower than the default set- ting, whereas the energy reduction remains modest; further downclocking from level 16 to 13 causes an additional 30% throughput drop with only limited further energy savings. NPU frequency scaling differs fundamentally from the CPU and GPU backends. While DVFS on CPU and GPU can be analyzed primarily along a single frequency axis, NPU scaling depends on the joint configuration of both NPU core and bus voltage corners. We therefore sweep the full core vcorner×bus vcorner space and visualize the results as a heatmap in Figure 12. The results show that NPU DVFS Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference 01234567891011121314D GPU Frequency Level 0.0 0.5 1.0 1.5 Energy (uJ/token) 1e5 0.9 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.9 0.9 0.9 1.0 1.4 10 20 30 40 Throughput (token/s) CPU_L CPU_M GPU Other Soc Total Default Energy (D) Throughput (token/s) Default Throughput (D) (a) GPU frequency scaling. 0134567891011121314151617181920D CPU Frequency Level (lowest frequency = level 0) 0 1 2 3 4 Energy (uJ/token) 1e5 0.8 0.9 0.9 1.4 0.7 1.2 1.3 0.8 0.8 4.1 1.0 1.2 1.2 1.5 1.4 1.3 1.3 1.2 1.4 1.3 1.4 20 40 60 Throughput (token/s) CPU_L CPU_M GPU Other Soc Total Throughput (token/s) Default Energy (D) Default Throughput (D) (b) CPU frequency scaling. 01234567891011121314151617181920D CPU Frequency Level (lowest frequency = level 0) 0.0 0.5 1.0 1.5 Energy (uJ/token) 1e5 0.7 0.7 0.6 0.6 0.6 0.7 0.7 0.7 0.9 0.9 0.9 0.9 0.8 0.9 0.9 0.9 0.9 1.0 1.0 1.0 1.1 1.4 40 45 50 Throughput (token/s) CPU_L CPU_M GPU Other (Soc - L - M - GPU) Soc Total Throughput (token/s) Default Energy (D) Default Throughput (D) (c) CPU frequency scaling for GPU. Figure 11: Throughput and energy under frequency scaling in llama.cpp for Llama 3.2 1B decode with 256 tokens. is inherently a two-dimensional operating-point problem rather than a single monotonic knob, and they reveal three consistent patterns. First, throughput-optimal points concentrate at very high voltage corners. Second, energy-optimal points are usually different from throughput-optimal points, but the size of the trade-off depends strongly on model and phase. In Qwen prefill, the two optima are close: moving from (max, max) to the energy-optimal (turbo_l2, turbo_l3) lowers energy by only 2.0% while reducing throughput by 6.5%. In the other three cases, however, the separation is substantial. Qwen de- code saves 22.1% energy at the energy-optimal (turbo_plus, nom) point, but throughput drops by 25.5%. Llama prefill saves 23.7% energy at (nom, turbo_l2) with a 31.3% through- put reduction, while Llama decode saves 19.1% energy at (turbo_plus, turbo_l3) with an 11.7% throughput drop. Third, the most efficient operating points stay in the mid- to-high vcorner region rather than at the lowest settings. None of the energy-optimal points falls into min/svs/svs2; instead, they cluster around nom to turbo_plus for the core and nom to turbo_l3 for the bus. This means aggressive under-clocking is not the right policy for these NPU work- loads. The heatmap suggests that the useful search region lies in a relatively narrow band of medium-to-high voltage corners. This band also shifts across models and between prefill and decode. Cross-backend DVFS. Backend-local policies overlook strong CPU–accelerator interactions and can therefore miss globally efficient operating points. CPU downclocking can benefit the GPU backend: Fig- ure 11c shows that when running the GPU backend, a higher CPU frequency is not always better. At CPU frequency level 9, throughput increases by 13% relative to the default setting, while energy consumption decreases by 51%. More gener- ally, mid-range CPU frequency levels (5–10) can outperform higher levels (20–26) in both throughput and energy, indicat- ing that excessive host frequency can add overhead rather than useful work. CPU frequency scaling strongly affects NPU efficiency: Figure 13 fixes the CPU at several frequency levels during NPU inference and compares them with the default DVFS setting. Across both prefill and decode, the results show that the default DVFS policy is not energy-optimal for NPU exe- cution. Lower CPU frequency levels generally reduce total energy per token while preserving comparable throughput, indicating that host-side CPU provisioning under DVFS is often excessive for accelerator-dominated inference. This effect is especially clear in prefill, where throughput remains largely unchanged while energy is consistently reduced, with the largest reduction reaching 23.7%. A similar pattern also appears in decode, where the best case achieves a 39.7% en- ergy reduction, although throughput varies more across fre- quency levels, suggesting stronger device- and configuration- dependent trade-offs. Overall, these results show the impact of DVFS on NPU and GPU inference. Unlike pure CPU execution, where low- ering frequency often trades speed for energy, NPU- and GPU-bound workloads frequently tolerate substantial CPU downclocking because the critical path remains on the ac- celerator. 7.4 Summary RQ3 shows that scheduling is critical to mobile LLM effi- ciency. The main gains come from reducing host-side over- head through better CPU–NPU invocation settings, proper thread–core affinity, and backend-aware DVFS. Overall, the best policy is phase- and backend-aware rather than relying on default settings. 8 Optimizations and Future Directions Guided by these findings, we identify a practical energy- oriented best-practice configuration for mobile LLM infer- ence on NPU: use QNN with full-graph offloading to the NPU, build a computation graph with a context length that is suitable for the target workload, and reduce both CPU- and NPU-side overhead by tuning the CPU polling interval, CPU frequency, and NPU sleep latency. Specifically, we set the RPC polling interval to 20μs, the NPU sleep latency to 65535μs, and fix the CPU frequency at the lowest level to avoid unnecessary CPU boosting during NPU execution. This cuts prefill energy by 53.6% with a slight throughput gain of 0.8%, and cuts decode energy by 54.8% Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang off min svs2 svs svs+ nom nom+ turbo t-l2 t-l3 t+ max Core vcorner Qwen2.5-1.5B | Prefill 428429429426430426431428430430431431 583599600602567609619612620620620622 598565600618567616616612614621619622 727673731831682816847833837845848857 773736810940748920995968999101610101038 88989780410719921048114811011174125711711268 92489183085411251095120811751241132812361324 84297986389011951164126512461326144413501386 10229021021127011901246134513561451155814481617 10431054917131112701273142814221493155214261659 85587910009049041332133412821364148213761437 103910561027132312881445139415001487159415171667 Qwen2.5-1.5B | Decode 6.896.876.896.886.906.916.736.856.896.936.916.94 9.119.429.419.789.769.789.789.749.739.829.779.81 9.409.429.419.779.749.419.679.779.729.859.779.83 10.910.29.6312.312.912.413.312.812.813.412.813.0 11.510.110.014.410.114.115.414.615.116.215.215.5 11.810.410.410.610.516.714.816.217.318.317.319.8 12.212.210.610.610.617.015.216.718.019.318.021.0 12.412.410.716.516.418.415.619.419.520.518.822.4 12.512.510.910.917.016.018.217.920.222.619.623.9 12.712.712.317.217.216.219.120.520.421.920.823.2 10.712.510.816.716.717.818.819.719.720.919.221.9 12.312.612.717.217.116.219.520.520.523.020.623.1 Llama3.2-3B | Prefill 218220220220220220220220220221220220 308309309317317315317318318318318319 308296309317296314317315317318317319 386365381411368423432432434441438441 436405431476409486516507523526522534 491452493578458565609598623646622656 514505470477609596649636664695665693 544546495500602641684718723768727722 563521581676693702773754802847802814 591593594698668725756733826847744885 500559503514681729701713749796752742 587594594743712782803806803862803906 Llama3.2-3B | Decode 3.603.653.713.713.703.703.653.713.723.743.733.75 4.954.914.925.035.265.265.225.285.265.325.265.32 4.984.984.895.065.265.275.275.265.265.235.285.32 5.545.935.276.456.986.767.097.007.197.257.187.31 5.916.315.587.705.747.528.068.078.288.668.338.57 6.195.875.868.278.439.359.249.019.5210.39.3810.4 6.426.605.976.166.259.248.509.389.8710.69.9411.3 6.636.756.158.209.099.588.869.7310.511.410.312.2 5.967.266.268.698.549.3010.111.111.212.010.912.5 6.767.027.268.658.809.4510.810.411.212.611.013.5 5.876.936.208.728.349.0110.010.810.811.910.712.2 7.187.136.959.348.889.3710.210.711.112.211.213.1 off min svs2 svs svs+ nom nom+ turbo t-l2t-l3 t+ max Bus vcorner off min svs2 svs svs+ nom nom+ turbo t-l2 t-l3 t+ max Core vcorner 6.76.97.06.46.96.26.76.06.86.86.97.4 5.05.05.25.04.74.85.14.85.05.05.25.8 5.34.85.35.24.94.44.94.95.15.25.05.8 4.24.44.64.14.23.64.03.83.84.14.04.4 4.14.24.43.54.13.63.43.53.53.53.63.9 4.34.04.23.43.63.33.27.43.13.13.13.2 4.24.14.24.03.43.33.33.13.33.13.13.8 4.44.34.24.33.53.33.33.23.13.13.23.7 5.04.54.63.84.23.53.53.13.12.83.03.3 5.25.15.04.14.13.63.73.83.23.23.63.1 4.54.64.44.34.33.33.23.33.03.03.13.8 4.55.04.83.53.73.43.23.23.43.03.12.9 off min svs2 svs svs+ nom nom+ turbo t-l2t-l3 t+ max Bus vcorner 3.93.94.13.74.14.23.73.94.14.34.24.5 3.13.23.13.12.83.03.13.02.93.13.13.4 3.03.13.13.12.72.82.93.03.03.23.13.5 2.72.52.72.42.12.12.32.22.32.42.32.5 2.62.72.72.22.71.92.02.02.02.11.92.1 2.62.62.62.72.71.81.81.81.71.81.71.9 2.62.72.72.72.71.71.81.71.61.71.71.9 2.62.72.71.91.91.71.71.71.61.61.61.7 2.62.82.92.71.81.81.71.71.51.61.61.8 2.83.02.91.92.01.81.71.71.81.61.61.6 2.72.72.71.91.91.41.71.61.61.61.61.7 2.72.62.92.01.91.81.51.61.71.61.61.6 off min svs2 svs svs+ nom nom+ turbo t-l2t-l3 t+ max Bus vcorner 1.31.41.41.41.31.31.41.31.31.41.31.5 1.11.11.11.01.00.91.11.01.01.11.01.2 1.11.01.11.11.01.01.11.01.01.11.01.2 0.90.90.90.80.80.80.80.80.80.80.80.9 0.90.90.90.80.80.70.80.80.80.80.80.8 0.90.90.90.80.90.70.80.70.70.80.70.8 0.90.90.90.90.80.70.70.70.80.80.70.9 0.90.90.90.90.80.80.80.80.70.80.71.0 1.11.01.11.01.00.90.80.80.90.80.80.9 1.21.21.11.01.11.01.00.90.90.90.90.9 1.01.01.00.90.80.80.80.80.80.80.80.9 1.11.21.21.01.01.00.90.91.00.91.00.9 off min svs2 svs svs+ nom nom+ turbo t-l2t-l3 t+ max Bus vcorner 7.57.57.77.97.47.57.37.77.98.07.98.6 6.16.16.15.86.05.95.76.05.86.45.96.5 6.36.16.05.96.05.86.15.95.86.05.86.5 5.45.45.64.94.94.64.54.64.64.94.65.2 5.45.55.64.65.34.34.23.94.24.54.34.7 5.55.55.64.53.94.14.14.14.04.04.14.2 5.55.55.65.65.54.24.34.24.14.23.94.3 5.75.86.04.94.94.24.54.24.34.04.14.3 6.76.16.45.25.14.84.64.44.44.24.34.7 7.06.96.05.85.65.04.84.75.04.64.74.7 6.45.76.14.84.94.54.44.34.23.84.24.3 6.96.96.95.55.75.25.05.04.84.34.84.7 0.8 1.2 1.6 1e3 3.0 4.5 6.0 1e3 1.0 1.5 2.0 1e1 1.6 2.4 3.2 4.0 1e5 4 6 8 1e2 0.8 1.0 1.2 1.4 1e4 0.50 0.75 1.00 1.25 1e1 4.5 6.0 7.5 1e5 Throughput (token/s) Energy (uJ/token) Throughput-optimalEnergy-optimal Figure 12: Impact of NPU voltage-corner scaling on throughput and energy during inference. Table 8: Estimated end-to-end impact of the energy- optimized GENIE configuration on the MathQA, RoleBench, and LongBench datasets. DatasetLatencyEnergy MathQA174187.8→ 201244.4 s (+15.5%)127774.4→ 57754.2 J (-54.8%) RoleBench407485.2→ 469985.6 s (+15.3%)335279.9→ 152037.8 J (-54.7%) LongBench89531.2→ 88848.8 s (-0.8%)733648.2→ 340622.2 J (-53.6%) with a 13.4% throughput loss. Table 8 reports the estimated end-to-end impact on the MathQA [30], RoleBench [31], and LongBench [32] datasets. Across these datasets, total energy consistently decreases by 53.6%–54.8%. End-to-end latency rises by 15.1%–15.5% on MathQA and RoleBench, but slightly decreases by 0.8% on LongBench. This is because LongBench is more prefill-dominant, so it benefits from the slight prefill speedup and is less affected by decode slowdown. We further optimize llama.cpp’s CPU decode by tuning thread-core affinity, removing one mid core from the thread placement, and lowering the CPU frequency to Level 17, which improves throughput by 1.5% while reducing energy consumption by 20.0%. For GPU decode, we find that CPU- side tuning remains critical: adjusting the CPU frequency from its default setting to Level 10, together with thread- core affinity optimization, improves throughput by 10.8% and reduces energy by 36.4%. These findings indicate that, even when inference is offloaded to GPU or NPU backends, CPU- side scheduling significantly affects end-to-end efficiency and must be incorporated into holistic optimization. Our findings reveal several optimization opportunities: Multi-graph switching on NPU. NPU decode suffers from static graph execution: frameworks often compile a large graph for long contexts even when the current se- quence is short, creating unnecessary overhead. A promis- ing direction is to pre-compile multiple graphs for different context windows and switch among them during decoding. The key challenge is efficient switching, including KV-cache reuse and state migration. Heterogeneous phase-aware 0 2 Speed (token/s) 1e3 Prefill 0 2 1e1 Decode 1235913DVFS CPU Freq Level 0 5 Energy (uJ/token) 1e3 23.7% lower 1235913DVFS CPU Freq Level 0.0 2.5 1e5 39.7% lower CPUGPUNPUOthersXiaomi 17Oneplus 15DVFS Figure 13: Impact of CPU frequency scaling on through- put and energy during NPU inference for Qwen 1.5B, across prefill and decode on Xiaomi 17 and OnePlus 15. pipelines. Current frameworks usually bind inference to a single backend, while our results show that prefill and de- code favor different processors. A promising direction is a runtime that dispatches prefill to the NPU and decode to the CPU. The key challenge is minimizing tensor copy and manipulation overhead. Phase- and model-aware NPU frequency scaling. Our DVFS results suggest that NPU operating points should not be fixed globally. The best oper- ating point depends on both phase and model. Future work may jointly tune NPU frequency with workload features to balance throughput and energy. Holistic resource schedul- ing. CPU thread-core affinity, polling behavior, sleep settings, and DVFS materially affect NPU and GPU inference even when the CPU is not the main compute engine. This sug- gests that future systems should jointly coordinate resources rather than tuning only the active processor. Overall, these findings suggest that mobile LLM inference still has substantial cross-layer optimization headroom span- ning frameworks, backends, and scheduling strategies. 9 Related Work On-device LLM Measurement Studies: Prior work estab- lishes key baselines for on-device LLM inference. MELTing Point [15] and MobileAIBench [33] benchmark mobile LLM Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference performance and resource usage, while later studies ana- lyze runtime bottlenecks, model scaling, SLM behavior, and multi-instance execution [12–14,34]. These works show that mobile LLM inference is increasingly practical with suitable models and system settings. However, they usually cover only limited framework-backend combinations, focus mainly on CPU/GPU execution, and rely on coarse whole-device en- ergy measurements, leaving NPU behavior, cross-framework differences, and backend-specific coordination overheads insufficiently understood. On-device LLM Optimizations: A large body of re- search improves LLM efficiency through model compres- sion and runtime optimization, including quantization and compression [35–37], efficient attention and KV-cache man- agement [17,38–42], and speculative decoding [27,43]. On mobile platforms, recent systems further explore sparsity- aware execution, NPU offloading, heterogeneous GPU–NPU collaboration, memory-aware deployment, architecture co- design, and scheduling optimization [6–11,29,44–49]. These studies show that substantial gains can come from optimizing specific mechanisms. However, they usually target one frame- work, one backend path, or one optimization dimension at a time. Our work instead provides a benchmark and a uni- fied cross-layer measurement view across five frameworks, three backends, and systematic scheduling policies, enabling us to jointly analyze framework diversity, phase-dependent backend behavior, and fine-grained energy inefficiency. 10 Conclusion In this paper, we presented the first comprehensive cross- layer measurement study of mobile LLM inference across five representative frameworks and three heterogeneous back- ends. We also developedPowerBench, a lightweight profiling plugin that enables backend-specific throughput and energy attribution beyond coarse device-level measurements. Our findings exposed how framework design, backend optimiza- tion, and resource scheduling jointly determine end-to-end mobile LLM efficiency. More importantly, we revealed sub- stantial room for efficiency optimization, especially for NPU execution. References [1]Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tai, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Sid- dhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro- Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gau- rav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- finetuned language models. J. Mach. Learn. Res., 25(1):70:3381–70:3433, January 2024. [2] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In Proceedings of NeurIPS, 2023. [3]Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. On-Device Language Models: A Comprehensive Review. arXiv preprint arXiv:2409.00088, September 2024. [4] Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, and Yuanchun Li. AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation. arXiv preprint arXiv:2412.18116, May 2025. [5]Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. AutoDroid: LLM-powered Task Automation in Android. In Proceedings of ACM MobiCom, 2024. [6] Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Meng- wei Xu, and Xuanzhe Liu. Fast On-device LLM Inference with NPUs. arXiv preprint arXiv:2407.05858, December 2024. [7]Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. PowerInfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282, 2024. [8]Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices . IEEE Transactions on Mobile Computing, 24(08): 7059–7073, August 2025. [9]Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yubin Xia, Pinjie Xu, and Haibo Chen. Characterizing Mobile SoC for Accelerat- ing Heterogeneous LLM Inference. In Proceedings of the ACM SOSP, 2025. [10] Zhengxiang Huang, Chaoyue Niu, Zhaode Wang, Jiarui Xue, Hanming Zhang, Yugang Wang, Zewei Xin, Xiaotang Jiang, Chengfei Lv, Fan Wu, and Guihai Chen. MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selection. arXiv preprint arXiv:2506.19884, June 2025. [11]Zongpu Zhang, Pranab Dash, Y. Charlie Hu, Qiang Xu, Jian Li, and Haibing Guan. Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency. arXiv preprint arXiv:2507.02135, July 2025. [12]Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Wei Liu, Jian Luan, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. Demys- tifying Small Language Models for Edge Deployment. In Proceedings of ACL, 2025. [13]Xiang Li, Zhenyan Lu, Dongqi Cai, Xiao Ma, and Mengwei Xu. Large Language Models on Mobile Devices: Measurements, Analysis, and Insights. In Proceedings of the Workshop on Edge and Mobile Foundation Models, 2024. [14] Qingzhe Guo, Tu Ouyang, and An Wang. Large Language Models on Mobile Devices: A Measurement Study of Single- and Multi-Instance Execution. In Proceedings of the 2nd International Workshop on Edge and Mobile Foundation Models, 2025. [15] Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. MELTing Point: Mobile Evaluation of Language Transform- ers. In Proceedings of ACM MobiCom, 2024. [16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, 2017. [17]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of EMNLP, 2023. [18]Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient gen- erative llm inference using phase splitting. In Proceedings of ISCA, 2025. [19]Smartphone Processors Ranking List [2025] - NanoReview. URL https: //nanoreview.net/en/soc-list/rating. Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang [20]Qualcomm Technologies, Inc. Qualcomm AI Engine Direct SDK | Qualcomm Developer, . URL https://w.qualcomm.com/developer/ software/qualcomm-ai-engine-direct-sdk. [21] Qualcomm Technologies, Inc. Hexagon NPU SDK | Qualcomm Devel- oper, . URL https://w.qualcomm.com/developer/software/hexagon- npu-sdk. [22]Georgi Gerganov. llama.cpp, 2023. URL https://github.com/ggml- org/llama.cpp. [23] Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yu Cai, Tianhang Yu, Chengfei Lv, and Zhihua Wu. Mnn: A universal and efficient inference engine. arXiv preprint arXiv:2002.12418, 2020. [24] MLC Team. Mlc-llm, 2023. URL https://github.com/mlc-ai/mlc-llm. [25]Ruyi Yi, Xiangyu Li, et al. mllm: Fast multimodal llm on mobile devices, 2024. URL https://github.com/UbiquitousLearning/mllm. [26]Qualcomm Technologies, Inc.Gen ai inference extensions. https://w.qualcomm.com/developer/software/gen-ai-inference- extensions, 2026. Accessed: 2026-03-14. [27]Daliang Xu, Wangsong Yin, Hao Zhang, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, and Xuanzhe Liu. EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding. IEEE Transactions on Mobile Computing, 24(4):3256–3273, April 2025. [28]OnePlusOSS.android_kernel_oneplus_sm8850: Qualcomm power telemetry and powercap driver sources.https: //github.com/OnePlusOSS/android_kernel_oneplus_sm8850/tree/ 6504e3d0385a951de4848bc81ce19ce8f8145dbe/drivers/powercap/ qcom, 2026. Commit 6504e3d0385a951de4848bc81ce19ce8f8145dbe, accessed 2026-03-13. [29]Zixu Hao, Jianyu Wei, Tuowei Wang, Minxing Huang, Huiqiang Jiang, Shiqi Jiang, Ting Cao, and Ju Ren. Scaling LLM Test-Time Compute with Mobile NPU on Smartphones. arXiv preprint arXiv:2509.23324, September 2025. [30]Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, June 2019. [31]Noah Wang, Zy Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computa- tional Linguistics: ACL 2024, 2024. [32]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhid- ian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context under- standing. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024. [33] Rithesh Murthy, Liangwei Yang, Juntao Tan, Tulika Manoj Awal- gaonkar, Yilun Zhou, Shelby Heinecke, Sachin Desai, Jason Wu, Ran Xu, Sarah Tan, Jianguo Zhang, Zhiwei Liu, Shirley Kokane, Zuxin Liu, Ming Zhu, Huan Wang, Caiming Xiong, and Silvio Savarese. Mo- bileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases. arXiv preprint arXiv:2406.10290, June 2024. [34]Haoxin Wang, Xiaolong Tu, Hongyu Ke, Huirong Chai, Dawei Chen, and Kyungtae Han. lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models. In Proceedings of ACM/IEEE SEC, 2025. [35]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, October 2022. [36]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei- Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, June 2023. [37]Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774, January 2023. [38]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135, May 2022. [39]Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, July 2023. [40]Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Systems, 7, 2025. [41]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. arXiv preprint arXiv:2309.06180, September 2023. [42]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Joseph E. Gonzalez, Ion Stoica, and Matei Zaharia. H 2 O: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306.14048, June 2023. [43] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.17192, November 2022. [44]Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. PowerInfer: Fast large language model serving with a consumer-grade GPU. In Proceedings of the ACM SOSP, 2024. [45]Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Mengwei Xu, and Xuanzhe Liu. WIP: Efficient LLM prefilling with mobile NPU. In Proceedings of the Workshop on Edge and Mobile Foundation Models, 2024. [46] Kaiyuan Liu, Xiaobo Zhou, and Li Li. m 2 2LLM: A Multi-Dimensional Optimization Framework for LLM Inference on Mobile Devices . IEEE Transactions on Parallel & Distributed Systems, 36(10):2014–2029, Oc- tober 2025. [47]Jinliang Yuan, Chen Yang, Dongqi Cai, Shihe Wang, Xin Yuan, Zeling Zhang, Xiang Li, Dingge Zhang, Hanzi Mei, Xianqing Jia, Shangguang Wang, and Mengwei Xu. Mobile foundation model as firmware. In Proceedings of ACM MobiCom, 2024. [48]Siyuan He, Zhantong Zhu, Yandong He, and Tianyu Jia. LP-Spec: Lever- aging LPDDR PIM for efficient LLM mobile speculative inference with architecture-dataflow co-optimization. arXiv preprint arXiv:2508.07227, August 2025. [49]Wangsong Yin, Daliang Xu, Mengwei Xu, Gang Huang, and Xu- anzhe Liu. Dynamic sparse attention on mobile SoCs. arXiv preprint arXiv:2508.16703, 2025. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference Table 9: Averaged Throughput and Energy across Devices, Models, Frameworks, and Backends with 256 tokens. ModelBackend Framework Quan. PrefillDecode Throughput (tokens/s)Energy (휇J/token)Throughput (tokens/s)Energy (휇J/token) Xiaomi 17 OnePlus 15 Xiaomi 15 Xiaomi 14 Xiaomi 17 OnePlus 15 Xiaomi 17 OnePlus 15 Xiaomi 15 Xiaomi 14 Xiaomi 17 OnePlus 15 Qwen2.5-1.5B CPUGENIEw4135.6123.6129.298.93.6e43.3e431.327.431.120.98.0e49.5e4 CPUllama.cppw4417.3299.9329.2157.23.3e41.5e451.6 55.753.934.51.2e59.7e4 CPUMNNw4349.5228.3259.4259.52.3e41.5e419.047.549.545.71.3e59.2e4 GPUllama.cppw4569.7754.8680.9365.71.0e41.1e438.850.348.631.81.1e51.4e5 GPUMNNw4392.2434.1406.1272.58.6e31.7e439.112.345.826.29.4e49.9e4 GPUMLC-LLMw424.342.445.9141.85.7e45.0e424.619.629.916.81.3e51.3e5 NPUGENIEw4 1213.0 1463.71219.2–6.8e35.6e323.123.020.7–4.3e53.2e5 NPUllama.cppw4107.6115.182.351.48.2e47.6e431.933.318.313.82.0e51.6e5 NPUMNNw4814.3700.9694.4593.88.3e38.1e39.910.316.914.82.1e52.5e5 NPUMLLMw4a16904.5966.8814.6–3.2e3 4.1e334.034.332.7–7.5e48.7e4 Qwen2.5-7B CPUGENIEw427.826.829.018.91.7e51.5e57.56.77.15.43.5e54.2e5 CPUllama.cppw485.458.655.134.11.1e57.9e412.614.313.28.75.0e54.4e5 CPUMNNw480.330.058.450.89.8e41.0e59.612.212.27.24.6e53.9e5 GPUllama.cppw4137.5171.6141.884.64.0e46.5e410.212.211.75.43.2e55.7e5 GPUMNNw478.4102.186.262.44.1e49.1e48.910.612.711.73.1e55.1e5 GPUMLC-LLMw4–29.7–10.013.411.28.05.9e58.8e5 NPUGENIEw4a16 859.1 1411.11020.9–9.4e36.9e314.616.813.6–4.0e54.0e5 NPUllama.cppw434.835.127.013.01.6e56.0e59.19.27.76.46.9e56.0e5 Llama3.2-1B CPUGENIEw4182.2167.4177.3134.42.8e42.3e439.935.639.328.16.3e47.5e4 CPUllama.cppw4326.8287.6386.3166.31.1e41.6e476.1 72.270.141.91.4e57.4e4 CPUMNNw429.8184.8324.8321.98.7e31.5e43.261.666.655.4 7.9e46.6e4 GPUllama.cppw4744.0986.8862.1450.16.3e35.4e346.164.457.122.51.4e51.1e5 GPUMNNw4692.1669.8534.4371.36.9e31.3e416.821.660.744.45.5e4 7.9e4 GPUMLC-LLMw4–189.3–31.619.940.120.11.0e59.3e4 NPUGENIEw41887.12316.11747.9–4.2e34.4e325.926.323.6–2.9e53.0e5 NPUllama.cppw4132.0141.393.542.37.6e44.6e441.341.621.013.82.4e51.3e5 NPUMNNw41215.6228.91043.9850.75.7e36.1e313.813.821.118.61.8e51.9e5 NPUMLLMw4a16 2163.2 2322.21716.0 1624.42.8e33.2e358.462.550.951.36.3e48.6e4 Llama3.2-3B CPUGENIEw439.759.262.145.76.2e46.8e415.914.015.712.01.7e52.0e5 CPUllama.cppw4119.7109.8114.858.65.4e44.1e429.930.527.117.12.9e52.1e5 CPUMNNw4123.466.190.8113.16.8e44.4e412.424.924.023.72.1e51.8e5 GPUllama.cppw4224.8357.6306.6161.12.2e42.1e420.025.025.211.11.8e52.5e5 GPUMNNw4201.2248.7205.3145.32.2e43.5e424.412.125.417.12.0e52.0e5 GPUMLC-LLMw4–67.8–19.820.921.213.32.8e53.9e5 NPUGENIEw4615.7815.3656.3–1.3e41.1e48.912.210.8–7.0e57.1e5 NPUllama.cppw448.143.034.115.31.6e51.2e519.119.310.13.94.4e52.9e5 NPUMNNw4533.597.2454.5440.11.3e41.4e46.010.58.27.54.3e57.3e5 NPUMLLMw4a16 929.3 886.2–674.87.4e38.3e3– Phi-3.5-mini CPUGENIEw4–52.037.5–13.69.1– CPUllama.cppw497.383.385.646.88.2e44.5e424.124.922.415.43.3e52.6e5 GPUllama.cppw4 216.0281.3228.8118.12.0e43.5e419.624.122.710.21.9e52.8e5 GPUMLC-LLMw4–53.1–19.320.920.114.73.3e54.0e5 NPUGENIEw4a16–1039.4 746.8–12.69.1– NPUllama.cppw440.344.126.318.51.7e51.3e52.73.04.61.41.6e61.3e6