Paper deep dive
Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence
Eziyo Ehsani, Luca Giamattei, Ivano Malavolta, Roberto Pietrantuono
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/31/2026, 1:57:29 AM
Summary
This paper investigates the trade-offs between performance, energy consumption, and generation quality for Large Language Models (LLMs) running on edge devices. By profiling eight models (0.5B to 9B parameters) on a Samsung Galaxy S25 Ultra, the authors identify a 'quantization-energy paradox' where importance-aware quantization provides negligible energy savings compared to standard methods. They also highlight that Mixture-of-Experts (MoE) architectures offer superior energy efficiency relative to their storage capacity, and propose a mid-sized model sweet spot (e.g., Qwen2.5-3B) for sustainable on-device deployment.
Entities (5)
Relation Signals (3)
Samsung Galaxy S25 Ultra â executes â LLMs
confidence 95% ¡ empirical case study on a flagship Android device, the Samsung Galaxy S25 Ultra
Mixture-of-Experts â providesbetterefficiencythan â Dense Architectures
confidence 90% ¡ Mixture-of-Experts (MoE) architectures defy the standard size-energy trend
Importance-aware quantization â yieldsnegligiblesavingscomparedto â Mixed-precision methods
confidence 90% ¡ we found it yields negligible energy savings compared to standard mixed-precision methods
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The migration of Large Language Models (LLMs) from cloud clusters to edge devices promises enhanced privacy and offline accessibility, but this transition encounters a harsh reality: the physical constraints of mobile batteries, thermal limits, and, most importantly, memory constraints. To navigate this landscape, we constructed a reproducible experimental pipeline to profile the complex interplay between energy consumption, latency, and quality. Unlike theoretical studies, we captured granular power metrics across eight models ranging from 0.5B to 9B parameters without requiring root access, ensuring our findings reflect realistic user conditions. We harness this pipeline to conduct an empirical case study on a flagship Android device, the Samsung Galaxy S25 Ultra, establishing foundational hypotheses regarding the trade-offs between generation quality, performance, and resource consumption. Our investigation uncovered a counter-intuitive quantization-energy paradox. While modern importance-aware quantization successfully reduces memory footprints to fit larger models into RAM, we found it yields negligible energy savings compared to standard mixed-precision methods. This proves that for battery life, the architecture of the model, not its quantization scheme, is the decisive factor. We further identified that Mixture-of-Experts (MoE) architectures defy the standard size-energy trend, offering the storage capacity of a 7B model while maintaining the lower energy profile of a 1B to 2B model. Finally, an analysis of these multi-objective trade-offs reveals a pragmatic sweet spot of mid-sized models, such as Qwen2.5-3B, that effectively balance response quality with sustainable energy consumption.
Tags
Links
- Source: https://arxiv.org/abs/2603.26603v1
- Canonical: https://arxiv.org/abs/2603.26603v1
Trouble viewing inline? Open PDF directly â
Full Text
118,872 characters extracted from source content.
Expand or collapse full text
Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence Eziyo Ehsani â , Luca Giamattei â , Ivano Malavolta â , Roberto Pietrantuono â â University of Naples Federico I, Naples, Italy â Vrije Universiteit Amsterdam, Amsterdam, The Netherlands AbstractâThe migration of Large Language Models (LLMs) from cloud clusters to edge devices promises enhanced privacy and offline accessibility, but this transition encounters a harsh reality: the physical constraints of mobile batteries, thermal limits, and, most importantly, memory constraints. To navigate this landscape, we constructed a reproducible experimental pipeline to profile the complex interplay between energy consumption, latency, and quality. Unlike theoretical studies, we captured granular power metrics across eight models ranging from 0.5B to 9B parameters without requiring root access, ensuring our findings reflect realistic user conditions. We harness this pipeline to conduct an empirical case study on a flagship Android device, the Samsung Galaxy S25 Ultra, establishing foundational hypotheses regarding the trade-offs between generation quality, performance, and resource consump- tion. Our investigation uncovered a counter-intuitive quantization- energy paradox. While modern importance-aware quantization successfully reduces memory footprints to fit larger models into RAM, we found it yields negligible energy savings compared to standard mixed-precision methods. This proves that for battery life, the architecture of the model, not its quantization scheme, is the decisive factor. We further identified that Mixture-of-Experts (MoE) architectures defy the standard size-energy trend, offering the storage capacity of a 7B model while maintaining the lower energy profile of a 1B to 2B model. Finally, an analysis of these multi-objective trade-offs reveals a pragmatic sweet spot of mid-sized models, such as Qwen2.5-3B, that effectively balance response quality with sustainable energy consumption. Index TermsâLarge Language Models, Edge AI, Quantization, Edge Computing, llama.cpp I. INTRODUCTION Large Language Models (LLMs) are increasingly being integrated into interactive systems that must operate under strict latency, privacy, and availability constraints. This paradigm shift has accelerated interest in edge AI, which involves executing inference directly on end-user devices such as smartphones, wearables, and embedded systems. For many real-world ap- plications, local execution is not merely desirable but strictly necessary. For example, edge inference becomes critical when network connectivity is unreliable or entirely denied, when user data is highly privacy-sensitive and cannot be transmitted to external cloud servers, or when interactive responses must be delivered with consistently low latency to maintain a seamless user experience. Furthermore, offloading computation to the edge significantly reduces cloud infrastructure costs and server-side energy consumption by distributing the heavy computational burden across billions of decentralized consumer devices [1]. This decentralization also alleviates the massive carbon footprint associated with maintaining centralized data centers dedicated to continuous generative inference [2]. However, the transition from massive cloud infrastructure to mobile hardware fundamentally changes the feasibility constraints of deploying these models. Smartphones operate under extremely tight limits in compute throughput, memory bandwidth, and thermal headroom, and they are inherently bottlenecked by being powered by a finite battery. LLM inference is particularly challenging in this resource-constrained setting because it combines massive memory footprints, which are required to store both the multi-billion parameter model weights and the continuously growing KV cache, with sustained compute demand during autoregressive decoding. This token generation phase is heavily memory-bound, meaning that the speed of data transfer between RAM and the processor often limits performance more than raw computational capacity [3]. The architectural gap between processor speed and memory bandwidth has widened significantly, making the transfer of multi-billion parameter matrices for every single generated token a severe hardware bottleneck [4]. As a result, a model that is highly accurate but rapidly drains the battery or triggers hardware thermal throttling can quickly become impractical for real-world deployment, regardless of its raw generative quality or theoretical capabilities. Understanding these complex trade-offs requires reliable and granular energy profiling, yet accurate power measurement on consumer devices remains notoriously difficult. Traditional approaches to power profiling rely on external power monitors, such as the Monsoon hardware power monitor [5], which provide high-fidelity electrical measurements directly from the power source. However, these techniques typically require invasive hardware modifications, such as bypassing the internal battery entirely, and are therefore poorly aligned with the goal of evaluating intact consumer phones under realistic usage conditions. Conversely, software-only estimators based on CPU frequency states are much easier to deploy but can be highly unreliable for long-running LLM workloads. This unreliability stems from the fact that processor frequency is dynamically influenced by thermal throttling, operating system governors, and background system activity, meaning that static power models quickly lose accuracy as the device heats arXiv:2603.26603v1 [cs.SE] 27 Mar 2026 up [6]. Because modern mobile operating systems continuously adjust voltage and frequency in a non-linear fashion to prevent hardware damage, relying solely on software heuristics often leads to significant estimation errors during sustained inference sessions [7]. Consequently, many existing studies focus on evaluating latency or compression techniques in isolation, leaving the joint relationship between energy consumption, memory feasibility, performance, and generation quality severely under-explored on actual mobile hardware. In this paper, we address this critical research gap through a comprehensive empirical evaluation of on-device LLM inference conducted entirely on an unrooted flagship Android device, specifically the Samsung Galaxy S25 Ultra. We propose a reproducible, non-intrusive measurement pipeline that accurately estimates workload-attributable energy directly from native system APIs, thereby preserving the ecological validity of the device. Using llama.cpp [8] as the execution runtime, we systematically benchmark eight open-source LLMs spanning a spectrum of 0.5B to 9B parameters. This selection includes both standard dense architectures and advanced sparse Mixture-of-Experts (MoE) architectures. Furthermore, we evaluate these models under two distinct 4-bit quantization schemes: the widely used mixed-precisionQ4_K_Mformat and the importance-awareIQ4_XSformat. We rigorously evaluate each hardware-software configuration across multiple dimensions, including performance metrics such as prefill speed, generation speed, overall latency, and Time to First Token (TTFT). We concurrently measure resource usage through peak memory footprints, evaluate energy efficiency via total Joules and Joules per token, and assess output quality using both reference-based evaluation with BERTScore [9] and reference- free evaluation with G-Eval [10]. To achieve these goals, we address the following research questions: ĂRQ 1 . Is it feasible to deploy Large Language Models on mobile devices in an efficient manner? Investigating whether it is feasible to deploy Large Language Models on mobile devices in an efficient manner addresses the baseline viability of executing these workloads locally. It directly examines the harsh physical realities of smartphones, specifically restricted RAM capacity, thermal ceilings, and finite battery limitations. The primary objective is to determine if local execution of models ranging from 0.5 billion to 9 billion parameters can transcend theoretical possibility and achieve practical usability without inducing hardware failure, severe thermal throttling, or unacceptable user friction due to high latency. ĂRQ 2 . To what extent does quantization matter for on- device Large Language Models? Exploring to what extent quantization matters for on-device Large Language Models probes the actual efficacy of different compression algorithms on mobile hardware. The research investigates whether advanced, importance-aware formats trans- late into proportional energy savings and latency reductions. By explicitly contrasting standard mixed-precision methods with complex codebook-based quantization, the analysis aims to uncover if runtime weight reconstruction introduces process- ing overhead that completely negates theoretical efficiency benefits. ĂRQ 3 . How do different model sizes and foundational architectures affect performance on mobile hardware? Analyzing how different model sizes and foundational architectures affect performance on mobile hardware isolates these variables to examine their direct impact on the memory wall during the autoregressive decoding phase. The inquiry evaluates how scaling dense parameter counts exponentially increases memory traffic. Furthermore, it rigorously tests whether sparse architectures, specifically Mixture-of-Experts, can successfully decouple the active compute burden from the total parameter capacity, thereby altering traditional performance scaling laws and energetic costs on resource- constrained devices. By synthesizing these diverse metrics, this paper makes three primary contributions to the field of sustainable edge intelligence: Contributions to Sustainable Edge Intelligence â˘We provide a methodology for reproducible energy profiling on unrooted Android systems, delivering an automated pipeline for measuring workload-attributable energy from time-series battery telemetry without requiring rooting or hardware modification. â˘We present a detailed quantization and architecture trade-off characterization through a controlled bench- mark across various model sizes and architectures, specif- ically contrasting dense models with Mixture-of-Experts while directly comparing theQ4_K_MandIQ4_XS quantization paradigms in terms of latency, energy, and peak memory demands. â˘We conduct a joint quality and efficiency analysis via an empirical case study on a flagship Android device. This analysis thoroughly investigates the multi-objective trade-offs among energy, speed, resource consumption, and generation quality. Crucially, this includes a critical examination of how reference-based metrics can misrepre- sent extractive generation behavior and demonstrates how reference-free evaluation protocols effectively complement them to provide a more accurate reflection of cognitive capabilities. Ultimately, these comprehensive empirical evaluations trans- late into highly practical implications for sustainable edge deployment. By exposing the complex interplay between compression, architecture, and physical hardware constraints, we provide software engineers and researchers with actionable guidelines on how to optimally balance model scale and quantization strategies against the unforgiving battery and thermal realities of consumer mobile devices. I. RELATED WORK This section surveys the research literature most relevant to sustainable on-device LLM inference. We organize the discussion around four central themes: energy efficiency in natural language processing and deployment strategies, model quantization techniques for large language models, on-device inference systems paired with mobile energy measurement methodologies, and automated experimentation frameworks designed for reproducible systems measurement. A. Energy Efficiency in NLP and Deployment Strategies Empirical work on sustainable natural language processing has historically examined both model-side efficiency techniques and system-side deployment choices. The broader Green AI movement, introduced by Schwartz et al. [11], catalyzed this shift by explicitly advocating for the inclusion of efficiency metrics alongside traditional accuracy measurements during both training and inference phases. Furthermore, Strubell et al. [12] highlighted the massive carbon footprint associated with training and deploying dense Transformer models, providing a strong environmental imperative to push inference toward optimized edge devices. The sheer computational complexity of Transformer architectures, driven by the quadratic scaling of self-attention mechanisms and massive parameter counts, has made energy efficiency a critical prerequisite for deployment outside of data centers. Yuan et al. [13] studied Knowledge Distillation as a practical approach to reducing inference cost at the architectural level. By comparing heavily parameterized Transformer models, such as BERT and GPT-2 style architec- tures, against their distilled variants, they demonstrated how transferring representations from a large teacher to a compact student model fundamentally alters the execution profile. They reported that distillation can significantly reduce overall energy consumption while simultaneously improving inference time, reinforcing Knowledge Distillation as an effective method for permanently lowering compute demand without requiring specialized hardware accelerators. Other studies shift the focus from how the model is constructed to the energy implications of where the inference actually executes. The dichotomy between cloud-based and edge-based execution involves complex trade-offs between net- work transmission costs and local computation costs. Nguyen et al. [14] compared client-side energy usage when fetching LLM-generated content from a remote server via a network interface versus generating the equivalent text entirely on- device. Across multiple language models and varying content lengths, they found remote fetching to be substantially more energy-efficient for the client device in most scenarios. They attributed this energy gap primarily to the exceptionally high compute cost of local autoregressive generation and the strong linear correlation between execution time and total energy consumption on mobile systems. This highlights a critical tension in edge AI: while local execution guarantees privacy and offline availability, it heavily taxes the local power budget. B. Model Quantization for LLMs Running massive language models locally is highly attractive for ensuring enhanced privacy and data sovereignty, but practical deployment at the edge is strictly limited by available memory, computational power, and energy efficiency con- straints. Abstreiter et al. [15] evaluated the feasibility of local generative inference on single-board computers using a short story summarization task. They found that while quantization helps mitigate memory overhead, resource bottlenecks remain a severe constraint for the on-device user experience. The authors noted that sustained local generation frequently leads to noticeable latency, hardware overheating (which necessitated an external heatsink during their evaluation), and high energy consumption that raises concerns for battery-powered environ- ments. This thereby highlights the urgent need for efficiency techniques that improve holistic system usability rather than merely proving theoretical feasibility. Post-Training Quantization has emerged as the primary en- abler of on-device LLM deployment. By reducing the precision of model weights from 16-bit floating-point representations to 8-bit or 4-bit integer formats, quantization directly mitigates the memory bandwidth bottleneck, which is the primary limiting factor during the token-by-token generation phase. Approaches such as SmoothQuant [16], alongside foundational techniques like GPTQ proposed by Frantar et al. [17], which utilizes approximate second-order information to aggressively compress weights, allow multi-billion parameter models to fit entirely within the limited memory budgets of mobile SoCs. Additionally, Dettmers et al. [18] demonstrated with their int8 quantization framework that managing extreme outlier features in the activation space is absolutely critical for preventing catastrophic performance degradation at low precisions, further complicating the deployment landscape. More recently, importance-aware quantization methods such as Activation-aware Weight Quantization [19] have advanced the state of the art. These techniques propose protecting a small subset of highly sensitive, salient weights using activation statistics derived from calibration datasets to significantly reduce quantization error at extremely low bit widths. How- ever, the vast majority of quantization evaluations in the literature emphasize theoretical memory footprint reductions and zero-shot accuracy metrics. The end-to-end physical consequences of these different quantization layouts on mobile CPUs, particularly regarding sustained energy consumption and decoding latency, remain much less well characterized. This is especially true for advanced codebook and indirection- based formats likeIQ4_XS, which require additional memory lookups and runtime reconstruction logic that may introduce unexpected computational overhead during inference. C. On-Device and Edge LLM Inference Systems The migration of LLM inference workloads from specialized cloud infrastructure to ubiquitous edge devices has rapidly accelerated due to mounting privacy concerns and the necessity for low-latency interactive systems. Early edge inference efforts relied almost exclusively on much smaller models or overly aggressive compression techniques that severely degraded cognitive capabilities. However, recent advancements in inference runtimes, most notablyllama.cpp[8] and cross-platform compilation frameworks like Apache TVM introduced by Chen et al. [20], have successfully demonstrated that quantized, multi-billion parameter models can indeed run on standard consumer devices. A growing number of benchmarking studies evaluate this new reality. For example, the comprehensive on-device evaluation conducted by Laskaridis et al. [21] measured throughput and latency across various mobile CPUs and NPUs. Their work emphasizes critical hardware bottlenecks, specifically the limited memory bandwidth characteristic of mobile SoCs and the inherently sequential nature of autoregressive decoding, which forces the system into a memory-bound state with low arithmetic intensity. Furthermore, fundamental algorithmic improvements such as FlashAttention by Dao et al. [22] have shown that making inference runtimes heavily input- output aware can drastically reduce memory reads and writes, though seamlessly integrating these advanced optimizations into quantized mobile CPU kernels remains an ongoing system engineering challenge. While these foundational works firmly establish technical feasibility and carefully characterize generation speed, physical energy consumption is often treated as a secondary metric or approximated via higher-level software estimators. Moreover, the integration of sparse architectures, such as Mixture-of-Experts, introduces an entirely different compute and memory trade-off. By routing tokens only to specific subnetworks, MoE models offer low active compute requirements but demand exceptionally high resident memory capacity to store all experts simultaneously. This specific ar- chitectural paradigm remains comparatively under-explored within the tight constraints of mobile execution environments. D. Mobile Energy Measurement Accurate and reproducible energy measurement on commer- cial smartphones is a notoriously difficult systems challenge due to restricted developer access to physical telemetry, complex thermal dynamics, and unpredictable background operating system activity. Foundational studies in mobile computing, such as the comprehensive smartphone power analysis conducted by Carroll and Heiser [23], demonstrated early on that component- level power draw is highly non-linear and context-dependent. External power monitors, such as the Monsoon hardware power monitor [5], are frequently treated as the accepted gold standard in the literature because they capture electrical power directly at the source with high temporal resolution. However, deploying these physical monitors typically requires physically disassembling the device, bypassing the internal battery, and utilizing invasive hardware setups. This is highly impractical for large-scale consumer-device studies and fundamentally alters the thermal mass and heat dissipation characteristics of the phone, making it less representative of real-world user conditions. Conversely, software-only approaches often attempt to esti- mate energy consumption indirectly by tracking CPU frequency states and utilizing pre-computed device power models. While highly scalable, these estimators can be highly inaccurate for long-running and computationally variable workloads like LLM inference. Pathak et al. [24] highlighted the extreme difficulty of software-based energy accounting, noting that hidden system- call overheads and lingering hardware tail-states frequently confound software power models. During extended generation tasks, aggressive thermal throttling continuously alters the frequency and power relationship, and leakage current increases non-linearly with processor temperature, causing static software estimators to diverge sharply from physical reality. Recent systems research has therefore explored leveraging native on- device APIs, such as the AndroidBatteryManager[25], to periodically log voltage and current in a completely non- intrusive manner. While providing a highly realistic operating environment, this approach requires exceptionally careful experimental controls, including strict background process iso- lation, consistent screen state management, empirical baseline subtraction, and extensive repeated runs to produce reliable and statistically significant comparative measurements. E. Automated Experimentation and Measurement Ensuring reproducibility is a long-standing methodological challenge in empirical mobile systems and software engineering research. The historical context of this challenge is deeply rooted in the field of green software engineering, initially pioneered by Hindle et al. [26], which established the absolute necessity of rigorous, automated profiling to uncover the true energy signatures of software execution. The state of a mobile operating system is highly dynamic, and minor environmental variations can significantly skew performance profiling. Orchestration frameworks such as Android Run- ner [27] and Experiment Runner [28] address this by fully automating experiment execution and standardizing the data collection pipeline. Android Runner specifically targets Android applications and supports declarative experiment definitions, enabling researchers to conduct systematic runs with perfectly consistent setup and teardown behaviors. Experiment Runner generalizes this operational philosophy into a flexible, platform- agnostic Python framework that heavily emphasizes automation, artifact persistence, and graceful crash recovery. Both orchestration systems adopt highly extensible plugin architectures to seamlessly integrate a wide variety of profilers and measurement tools. For example, Android Runner natively supports mobile-oriented profilers such as Monsoon [5] and Trepn [29], while Experiment Runner has been successfully integrated with software-based carbon and energy estimators such as CodeCarbon [30] and EnergiBridge [31]. The design principles behind these orchestration frameworks directly motivate the rigorous run-level controls implemented in our experimental pipeline, including automated initialization, abso- lute repeatability, strict workload isolation, managed thermal cool-down periods, and structured artifact collection. Positioning of this work. Building directly upon these intersecting lines of systems research, our study provides a comprehensive, joint empirical evaluation of multiple critical dimensions simultaneously. We rigorously assess energy consumption, performance capabilities encompassing prefill and generation speeds, time to first token, end-to- end latency, physical resource usage tracking peak memory footprint, and linguistic output quality. We conduct this analysis across multiple LLM scales and architectural paradigms on an unrooted, unmodified commercial Android smartphone. Crucially, we isolate the effects of compression by directly com- paring a standard mixed-precision 4-bit quantization scheme against a complex importance-aware 4-bit scheme, revealing the practical limits of algorithmic efficiency optimizations when deployed on physical edge hardware. I. METHODOLOGY This section describes the comprehensive experimental design utilized to profile on-device large language model inference within an Android environment. We detail the selected models and their respective quantization schemes, the specific hardware and software configurations, the execution pipeline and environmental controls established to ensure strict reproducibility, the energy measurement and post-processing procedures, and the final evaluation metrics used to quantify system performance and generation quality. A. Models and Quantization To capture realistic mobile trade-offs across the current open- source ecosystem, we evaluated a diverse set of instruction- tuned large language models spanning a parameter scale from 0.5B to 9B. This specific range was deliberately selected because models smaller than half a billion parameters often lack the cognitive reasoning capacity required for meaningful human assistance, whereas models exceeding 9B parameters typically surpass the unified memory limits of contemporary mobile devices, leading to out-of-memory errors or relying on prohibitively slow storage swap space. Our selection includes both standard dense Transformer models and a sparse Mixture- of-Experts model. Candidate models were selected based on their objective rankings in the Hugging Face Open LLM Leaderboard [32]. To obtain comprehensive coverage across varying computational scales, we selected the highest-ranked model within each approximate one billion parameter band up to the 9B limit. We strictly prioritized instruction-tuned base variants rather than task-specific fine-tunes. Instruction-tuned models are explicitly trained to follow human directions and engage in dialogue [33], [34], making them the most representative choice for evaluating general-purpose on-device digital assistants. Models falling within the 4B to 6B parameter range were intentionally excluded from this study because the publicly available candidates in this specific band were substantially lower-ranked in terms of benchmark quality than several smaller models, rendering them less informative for a systematic, size- based empirical comparison. Table I lists all evaluated models along with their architectural specifications. TABLE I EVALUATED MODELS AND ARCHITECTURES. ModelParams (B)ArchFormatRef Qwen2-05B0.494DenseGGUF[35] Qwen2.5-1.5B1.5DenseGGUF[36] Phi-22.78DenseGGUF[37] Qwen2.5-3B3.086DenseGGUF[36] OLMoE-1B-7B-01256.919MoEGGUF[38] Qwen2.5-7B7.616DenseGGUF[36] Meta-Llama-3.1-8B8.03DenseGGUF[39] Gemma-2-9B9.0DenseGGUF[40] Allselectedmodelswereexecutedusingthe llama.cpp[8] inference engine, a highly optimized framework designed specifically for resource-constrained edge environments. A standard 7B parameter model stored in 16-bit floating-point precision requires over 14 gigabytes of memory, which far exceeds the available RAM allocated to user-space applications on most flagship smartphones [41]. Therefore, we quantized each model using thellama-quantize[42] utility into two widely used 4-bit formats supported by the GGUF ecosystem. The first format evaluated isQ4_K_M, a block-wise mixed- precision post-training quantization approach. Prior research demonstrates that uniform quantization below 4 bits in block- wise methods inevitably leads to catastrophic degradation in model quality and increased perplexity, whereas exceeding 6 bits increases memory usage beyond feasible smartphone RAM capacities [21], [43]. TheQ4_K_Mscheme provides a highly practical architectural balance by preserving the most sensitive parts of the neural network, especially attention layers and feed-forward network components, at a higher 6- bit precision. Simultaneously, it aggressively compresses less sensitive and redundant weights to 4-bit precision. This hybrid strategy maintains overall model quality and semantic reasoning capabilities while successfully keeping the memory footprint low enough for reliable on-device deployment. The second format evaluated isIQ4_XS, an advanced importance-aware quantization method heavily inspired by activation-aware weight quantization techniques. Foundational research in this area [19] shows that protecting a very small fraction of highly salient weights, sometimes as little as one percent of the total parameter count, can significantly reduce quantization error while still achieving massive compression ratios. This quantization process begins with calibration data that is statistically similar to the original training distribution of the model. This data is passed through the network to calculate activation statistics, estimate importance metrics, and form a detailed sensitivity map of the model weights. Instead of simply rounding weights to nearest fixed values,IQ4_XS employs a highly optimized codebook-based approach. The most important weights are assigned discrete values closely matching their original continuous representations to minimize accuracy loss, while less important weights undergo far more aggressive compression. The model ultimately stores pointer indices directing to entries in a shared numerical codebook, which the device runtime environment references and reconstructs dynamically during inference. This methodology focuses mathematical precision exactly where it is most needed and significantly reduces the overall resident memory footprint. Figures 1 and 2 illustrate the structural differences between these two quantization strategies. Fig. 1.Q4_K_M: block-wise mixed precision with higher precision for more sensitive components. Fig. 2. IQ4_XS: importance-aware 4-bit quantization using calibration data and codebook-based reconstruction. B. Experimental Setup and Orchestration We designed the evaluation framework to accurately reflect a realistic, unrooted consumer smartphone setting while main- taining strict, automated run-to-run reproducibility suitable for empirical software engineering research. The complete testing system comprises three primary compo- nents. The host controller is a dedicated MacBook Air M4 that exclusively executes the Experiment-Runner [28] orchestration framework. This controller is responsible for autonomously defining experiments, deploying necessary artifacts to the device, initiating and terminating hardware measurements, and collecting all resultant telemetry logs. The target device and active inference engine is a Samsung Galaxy S25 Ultra running the Android 16 operating system. This specific device executes thellama.cppbinary for CPU-only inference, allocating up to 8 processing threads to fully saturate the heterogeneous multi-core architecture of the mobile processor. Finally, the communication interface handles all automated device control strictly via Wireless Android Debug Bridge (ADB) [44]. This wireless protocol is an absolute necessity for power profiling, as establishing a physical USB connection would introduce charging currents from the host machine that would inherently corrupt all discharge-based energy measurements recorded by the device. C. Execution Pipeline and Reproducibility Controls We structured the methodology as a strictly controlled pipeline consisting of a one-time global setup phase, a repeated experimental run loop, and a final environmental restoration phase (Fig. 3). This highly deliberate design ensures pristine measurement isolation across consecutive runs and systematically mitigates major environmental confounds, such as unpredictable background OS activity and accumulated thermal carry-over from previous computations. Pre-experiment configuration: Before collecting any empir- ical data, we comprehensively standardize the device state during the one-time setup phase. We first define a global configuration table specifying the model name, the targeted task type, and a requirement of 30 distinct repetitions to ensure statistically stable and robust results. The smartphone screen is programmatically forced on and locked at minimum brightness. This is a critical requirement because allowing the display to turn off triggers deep Android OS idling behaviors that drastically reduce CPU clock frequencies and limit processing throughput to save battery [45], thereby artificially skewing latency and energy metrics. Furthermore, background activities, such as periodic Google Play services, automated bug reporting, and application maintenance tasks, are forcibly isolated. This prevents undocumented CPU usage spikes and secondary battery drains from interfering with our baseline measurements. Finally, we completely clear the system logcat to establish a pristine logging environment. Model deployment and warm-up: We automatically push all required LLM artifact files and thellama.cppexecution engine to the local storage of the device. An initial warm-up phase is then executed. This preliminary inference run prepares the device memory, populates essential system caches, stabilizes CPU frequency scaling governors, and completely mitigates the severe impact of cold-start penalties on power behavior. Experimental loop: Following the one-time global setup, the orchestration framework enters a continuous execution loop for each defined iteration. Run-level log isolation is strictly enforced by clearing thelogcatbuffer again at the start of every single run. If system logs were only cleared at the beginning of the entire experiment, previous run data would accumulate endlessly, making it extremely difficult to parse, separate, and analyze individual trials without data contamina- tion. The on-device battery logger is subsequently initialized with a high-resolution sampling rate of 100 milliseconds. This is immediately followed by a designated spin-up period of one to two seconds, which grants the underlying monitoring system sufficient time to stabilize its telemetry reporting before the massive compute load of inference begins. During the actual interaction stage, the language model is invoked utilizing model-specific instruction templates. This is necessary because every model has been fine-tuned on different datasets and implicitly expects a specific prompt structure to function optimally. The generated textual outputs are continuously written to the local storage of the device. To guarantee that each run remains fully isolated from the next, the host machine pulls these output files over the wireless network and immediately deletes them from the smartphone. Without this vital cleanup step, leftover text files and KV cache artifacts could gradually consume device storage, potentially interfering with filesystem performance and subsequent telemetry analysis. Post-run collection and restoration: Once generation con- cludes, the battery logger and all active system profilers are gracefully terminated. All recorded time-series logs containing granular device temperature, current, voltage, and synchronized timestamps are pulled back to the host controller. In the final data population stage, raw performance metrics and the textual responses are consolidated into a structured CSV file for downstream evaluation. Following the successful completion of all scheduled runs, an after-experiment phase actively restores the smartphone to its normal operating state by terminating any lingering monitoring services, clearing residual experiment logs, and resetting the screen timeout and system configurations to their original default parameters. Phase 1: One-Time Setup Configure experiment Screen on Background isolation Deploy & Warm-up 1) Start Run Clear logcat 2) Start Measurement Start logger (100ms) + 2s spin-up 3) Run Inference llama.cpp(local output) 4) Stop Measurement Stop logger 5) Collect & Cleanup Pull logs/outputs Delete device artifacts 6) Cool-Down 200s pause next run Fig. 3. Controlled execution pipeline. Setup is executed once; measurement and inference are repeated per run with run-level isolation and thermal management. Reproducibility parameters. We strictly fixed decoding and runtime parameters to ensure perfect mathematical comparabil- ity across all trials. The maximum output generation length is strictly bounded to exactly 100 tokens, as variable generation lengths would inherently skew total energy measurements and make cross-model comparisons invalid. The context window size is set to 512 tokens, generation temperature is fixed at 0 to force deterministic greedy decoding, and the inference process utilizes exactly 8 CPU threads. To actively manage thermal dynamics and prevent heat saturation, a 200-second cool-down period is enforced between every run, allowing the passive cooling of the smartphone chassis to return the processor to a baseline temperature. Furthermore, the internal battery range is strictly maintained between 80 percent and 100 percent charge, evaluating only one specific model per testing round. This prevents the natural voltage drops that consistently occur at lower charge levels, such as the electrical shift from 4.4 V at full charge down to lower thresholds, ensuring all models are evaluated under identical electrical conditions. The input prompt text is completely fixed, though the exact token count mathematically varies between 93 and 119 tokens across models strictly due to inherent vocabulary and tokenizer differences. D. Energy Measurement and Post-Processing Accurate power profiling on consumer smartphones presents significant methodological challenges due to the highly re- stricted nature of modern mobile operating systems and the tightly integrated physical design of contemporary mobile hardware. Systems researchers generally face a forced choice among three potential measurement paradigms, each carrying distinct and profound trade-offs between measurement fidelity and ecological validity. (1) The first and traditionally most precise method involves utilizing an external hardware power monitor, such as a Monsoon [5] power meter. This approach captures high- fidelity electrical measurements directly at the hardware source. However, this specific technique requires physically opening the device and completely bypassing the internal smartphone battery to connect the external meter directly to the delicate power terminals of the motherboard. This highly invasive hardware modification fundamentally compromises the physical integrity of the device and completely nullifies the realistic operating conditions of a consumer smartphone. Furthermore, it permanently alters the thermal mass and heat dissipation characteristics of the device, which is a critical compounding factor when evaluating sustained, heat-generating workloads like autoregressive text generation. (2) A second alternative is to extract current and voltage met- rics directly through the Android Debug Bridge [44] utilizing the built-in hardware sensors of the system. While this software- based approach is considerably less invasive than hardware modification, it encounters severe operating system security and access barriers. Modern Android devices, particularly those updated with strict kernel-level security policies implemented after 2024, actively block external unprivileged clients from querying low-level power metrics to successfully prevent side- channel attacks and user data leakage [46]. Consequently, obtaining these precise hardware measurements via external diagnostic interfaces is practically impossible without gaining administrative root access to the operating system. We explicitly and intentionally avoided rooting the device to ensure that our measurements accurately reflect the unmodified, stock software environment that a typical consumer would experience, as elevated privileges can inadvertently alter the foundational power-management behavior of the operating system. (3) To successfully overcome these critical limitations, we adopted a third approach that relies on an on-device background monitoring application leveraging the native Android Battery- Manager API [25], [47]. We developed a dedicated logging service that is installed directly on the target smartphone and operates entirely within user space. Because this monitoring tool runs natively as a standard internal application rather than an external diagnostic probe, the Android security framework explicitly permits its requests for system telemetry. During the execution pipeline, when the orchestration framework initiates the inference workload, it seamlessly triggers this internal monitoring application to concurrently measure energy consumption. This method successfully bypasses external security restrictions while effectively preserving the realistic deployment conditions of the device. This dedicated application is configured to periodically log battery voltage, battery current, internal device temperature, and high-resolution unix timestamps. We established a rapid sampling interval of 100 milliseconds to successfully capture the highly dynamic power fluctuations that are characteristic of processor-intensive generative AI tasks. To ensure that the recorded energy data accurately reflects the inference process rather than OS initialization overhead, we incorporate a brief spin-up period of one to two seconds, allowing the monitoring system to perfectly stabilize before the actual model execution commences. The generated logs are stored locally on the device during the run and are subsequently retrieved by the host controller strictly after the inference generation completes, thereby preventing the wireless data transfer process from interfering with the active measurement window (Fig. 4). Fig. 4. Measurement workflow. A user-space monitoring app logs voltage, current, and temperature at fixed intervals whilellama.cppruns on-device. Logs are pulled to the host after each run. Following rigorous data collection, we apply a mathematical post-processing procedure to completely isolate the energy consumed specifically by the language model. Instantaneous gross power is computed by multiplying the sampled voltage and current values at every discrete time step. Because an idle smartphone continuously consumes non- trivial background power to maintain basic system networking functions and persistent display illumination, we must perform a mandatory baseline subtraction. We conducted a completely separate baseline experiment by recording the device in a completely idle state for two full hours under perfectly matched environmental conditions, specifically maintaining the screen on at minimum brightness with no foreground workload active. We calculate the mean baseline powerP baseline directly from this extended idle trace and subtract it from the instantaneous gross power to estimate the true net power attributable exclusively to the inference workload. Finally, the total energy consumption for a given experi- mental run is computed by integrating this net power over the bounded measurement time using trapezoidal integration, which is highly effective for high-frequency discrete sampling. WithNdistinct samples recorded at timestampst n , we approximate the total Joules consumed during inference as follows: E = Nâ1 X n=0 P net (t n ) + P net (t n+1 ) 2 ¡ (t n+1 â t n ). E. Evaluation Metrics We rigorously report results across multiple dimensions, including hardware performance, physical resource usage, energy efficiency, and semantic output quality. Performance metrics include prefill speed measured in tokens per second, which represents the prompt processing throughput. Because prefill processes the input sequence in parallel, it primarily reflects the computational capacity of the device. We also measure generation speed in tokens per second for autoregressive decoding throughput. Because decoding is inherently sequential, it is heavily bound by the memory bandwidth of the device. Additionally, we track Time to First Token to measure the startup latency until the initial token appears, and end-to-end latency for the combined duration of both the prefill and generation phases. Efficiency metrics strictly comprise total integrated energy measured in Joules, the specific energy consumed per generated token, and the absolute peak memory measured in gigabytes, encompassing the model weights, the dynamic KV cache, and runtime buffers. Output quality is assessed through two deeply complemen- tary tracks utilizing both reference-based and reference-free evaluation methodologies. For reference-based evaluation, we utilize BERTScore [9]. Instead of relying on rigid exact word overlap metrics, BERTScore intelligently uses deep contextual embeddings to measure the underlying semantic similarity between the generated response of a model and a human- authored gold-standard reference summary. This provides a highly robust evaluation of semantic quality, particularly for summarization tasks where vocabulary and sentence structure may vary wildly but the fundamental underlying meaning must remain absolutely consistent. However, during comprehensive evaluation, we observed an important and recurring limitation regarding reference-based metrics when applied across a highly diverse set of model sizes. Smaller models frequently achieved higher semantic similarity scores than their larger counterparts, even when human inspection revealed their summaries were visibly lower in actual quality and readability. This paradox occurred because smaller models heavily tended to copy large, contiguous portions of the source text with minimal restructuring or abstraction. Because BERTScore fundamentally measures semantic similarity, outputs that closely resemble the original source text inherently receive artificially high scores. In stark contrast, larger models were far more likely to aggressively paraphrase, restructure sentences logically, and produce more natural, human-like summaries. Although these sophisticated outputs were often vastly superior in readability and cognitive abstraction, they occasionally received lower BERTScore values simply because they diverged substantially from the exact wording of the reference material. This revealed that the metric heavily favored extractive behavior over complex abstractive summarization, artificially biasing the evaluation toward less capable models [48]. Consequently, to correct for this metric bias, we relied heavily on reference-free qualitative evaluation to accurately capture the true coherence, faithfulness, and overall summary quality beyond mere surface-level similarity. We implemented a sophisticated LLM-based judging framework heavily inspired by the G-Eval method [10]. In this advanced setup, a highly capable language model independently assesses the quality of each generated response according to strict cognitive criteria without ever relying on a rigid gold reference. The evaluation pipeline scores generated responses based on faithfulness to ensure the summary accurately reflects the factual source content without hallucination, relevance to confirm the response focuses purely on critical information rather than trivial details, and coherence to verify the text is logically structured and flows naturally. These individual cognitive dimensions are mathematically combined into an overall quality score reported as the final evaluation metric. We statistically analyze per-run measurements based on the 30 repetitions recorded per condition. Normality across the dataset is initially assessed using the Shapiro-Wilk test. Because power profiling and latency distributions on mobile operating systems are consistently non-normal due to discrete processor frequency scaling states, we employ non-parametric tests. We utilize the Friedman test for omnibus comparisons across all models, and paired Wilcoxon signed-rank tests combined with the Holm-Bonferroni correction for rigorous post-hoc multi-model and quantization rankings. IV. RESULTS This section reports the comprehensive empirical findings derived from our extensive on-device profiling experiments. We present detailed quantitative results spanning performance, which includes both prefill and generation throughput, end- to-end latency, physical energy consumption, and rigorous statistical significance testing. Unless explicitly stated otherwise, all plots, distributions, and summary statistics are computed from the full distribution of per-run measurements, specifically capturing 30 discrete repetitions for every unique model and quantization condition to ensure high statistical confidence and accurately capture the natural variance of the mobile operating system. A. Performance Analysis We measure performance across two fundamentally distinct operational phases of large language model inference: prompt processing throughput, referred to as the prefill phase, and au- toregressive decoding throughput, referred to as the generation phase. Figure 5 demonstrates a massive and highly consistent performance separation between these two computational phases across all evaluated neural architectures and both quantization schemes. Prefill versus generation dynamics: Across all hardware and software configurations, prefill throughput drastically exceeds generation throughput. This massive discrepancy is deeply rooted in the underlying mathematics of the Transformer architecture and the physical limitations of von Neumann computing architectures. The prefill phase processes the entire input sequence simultaneously, allowing the inference engine to utilize highly optimized, highly parallel matrix-matrix multiplications to efficiently populate the initial Key-Value cache [49]. In stark contrast, the generation phase is inherently sequential. Each newly generated token fundamentally depends on all previously generated tokens, forcing the system to rely on memory-bound matrix-vector multiplications. This strict sequential dependency prevents parallel execution across the sequence length, slamming the processor into the memory wall, a phenomenon where the speed of data transfer from RAM completely dictates performance rather than the arithmetic speed of the CPU [50]. This drastically increases the end-to-end runtime and establishes the generation phase as the dominant bottleneck and the primary contributor to total inference time for any moderately long output sequence. Scaling with model size: As strictly anticipated by theoretical computational complexity bounds and established scaling laws [51], smaller models consistently achieve much higher throughput, while larger multi-billion parameter models execute significantly slower under both 4-bit quantization formats. This clear downward trend directly reflects the exponentially increased compute requirements and the massive memory traffic associated with larger parameter counts and bloated intermediate attention states. Every single generated token requires loading the entirety of the model weights from the system RAM into the limited L1 and L2 caches of the mobile CPU, creating a severe memory bandwidth bottleneck that mobile systems, constrained by low-power LPDDR memory interfaces, are fundamentally poorly equipped to handle [52]. Dense versus Mixture-of-Experts architectures: We observe a highly distinct and extremely favorable operational behavior for the sparse Mixture-of-Experts model, OLMoE-1B-7B. Although this specific architecture possesses a massive total Qwen2-0.5B Qwen2.5-1.5B Phi-2 Qwen2.5-3B OLMoE-1B-7B-0125 Qwen2.5-7B Llama3.1-8B Gemma2-9B 0 50 100 150 200 Prefill Speed (tokens/s) Q4KM IQ4XS (a) Prefill Speed Qwen2-0.5B Qwen2.5-1.5B Phi-2 Qwen2.5-3B OLMoE-1B-7B-0125 Qwen2.5-7B Llama3.1-8B Gemma2-9B 0 10 20 30 40 50 60 Generation Speed (tokens/s) Q4KM IQ4XS (b) Generation Speed Fig. 5. Throughput across models and quantization schemes. Prefill benefits from parallel processing of the prompt, whereas generation is slower due to autoregressive decoding and higher per-token overhead. parameter count nearing seven billion, the internal routing mechanism ensures that only a carefully selected, highly specialized subset of neural experts is actively engaged per token [53]. This dynamic, input-dependent routing drastically reduces the active computational burden per token relative to standard dense models of a similar total size. Consequently, despite occupying the storage footprint of a massive model, OLMoE-1B-7B produces a processing throughput that closely mirrors the performance of much smaller dense models in the one to two billion parameter class. This highly beneficial algorithmic sparsity effect appears consistently and prominently in both the prefill and generation throughput results. B. Latency Figure 6 visualizes the total inference latency, explicitly decomposed into the initial prefill time and the subsequent generation time. Each individual bar represents the median end-to-end latency required to produce a sequence of up to 100 output tokens. Latency breakdown and user experience: Absolute latency increases substantially and non-linearly with model size. The stacked graphical breakdown clearly illustrates that the autoregressive generation phase completely dominates the total latency budget, especially for the larger models in our cohort, which perfectly aligns with the sequential, heavily memory- bound nature of the decoding process. This reality has profound implications for practical human deployment. Established foundational guidelines in human-computer interaction dictate that interaction latencies exceeding 200 milliseconds rapidly become perceptible to human users, and systemic delays extending beyond two to three seconds severely disrupt natural conversational flow, degrade user trust, and cause severe cognitive friction [54], [55]. As our empirical results starkly demonstrate, several of the larger models, such as the 8B and 9B variants, exhibit staggering end-to-end latencies extending well beyond 10 seconds per query. Consequently, while these massive models are technically feasible to execute on an unrooted mobile device, they are practically unusable for real-time, interactive digital assistant applications where immediate responsiveness is the paramount design requirement. C. Energy Consumption and Trade-offs We comprehensively evaluate both the initial startup energy cost, which is closely correlated with Time to First Token, and the sustained energy efficiency, rigorously quantified as physical Joules consumed per generated token. Figure 7 maps the complex relationship between initial latency and sustained energy cost across the entire spectrum of models and both quantization schemes. Larger neural architectures inevitably exhibit both signifi- cantly higher startup latencies and substantially elevated energy costs per token. For example, the Meta-Llama-3.1-8B configu- ration demonstrates severe startup latency and an exceptionally high sustained energy cost per token when compared to the smaller, more agile models. Similarly, the Gemma-2-9B model exhibits massive energy costs that are entirely consistent with its extreme computational complexity, massive parameter count, and enormous memory bandwidth demands. In sharp contrast, the smaller models, notably Qwen2-0.5B, manage to produce the first token almost instantaneously and consume a mere fraction of the energy per token, positioning them as the only genuinely viable candidates for heavily latency-sensitive and strictly battery-constrained mobile applications. Total energy and the role of architecture: Figure 8b illustrates that total physical energy consumption increases almost exponentially with model scale for standard dense architectures. A single inference run utilizing Gemma-2-9B routinely exceeds 300 Joules, whereas the highly efficient Qwen2-0.5B Qwen2.5-1.5B Phi-2 Qwen2.5-3B OLMoE-1B-7B-0125 Qwen2.5-7B Llama3.1-8B Gemma2-9B 0 5 10 15 20 25 30 Latency (s) Prefill Q4KM Generation Q4KM Prefill IQ4XS Generation IQ4XS Fig. 6. Inference latency breakdown. Qwen2-0.5B Qwen2.5-1.5B Phi-2 Qwen2.5-3B OLMoE-1B-7B-0125 Qwen2.5-7B Llama3.1-8B Gemma2-9B 0 2 4 6 8 Time to First Token (s) Q4KM (Time) IQ4XS (Time) Q4KM (Energy) IQ4XS (Energy) 0 1 2 3 4 Energy per Token (Joules) Fig. 7. Comparison of Time to First Token (Bars, Left Axis) and Energy per Token (Lines, Right Axis). Qwen2-0.5B operates near a mere 50 Joules for the exact same summarization task. A highly notable deviation from this strict size-based scaling law is the MoE model. OLMoE-1B- 7B consumes an amount of energy that is entirely comparable to much smaller dense models, despite harboring a massive total parameter count. This rigorously confirms that the reduced active compute per token achieved through sparse expert routing directly and directly translates into physical battery savings on the hardware level. The limited impact of quantization format: Across the vast majority of evaluated models, the measured energy and speed differences between the standardQ4_K_Mformat and the complexIQ4_XSformat are surprisingly modest, especially when compared to the massive disparities between different model families and overall parameter scales. This suggests a critical architectural insight. Under CPU-only inference on contemporary mobile hardware, the baseline model architecture and the overall volume of active arithmetic compute dominate physical energy consumption far more than the specific algorithmic choice between these two distinct 4-bit compression formats. The computational overhead required to unpack and reconstruct the codebook indices inIQ4_XSintroduces signifi- cant branching and cache lookup penalties [56]. These processor inefficiencies appear to completely offset the theoretical energy savings gained by moving fewer total bytes across the system bus from RAM. Speed versus quality trade-off: Figure 8c reveals a strict Pareto frontier governing the general trend. Higher generation speed almost universally corresponds to lower cognitive quality as measured by the G-Eval framework. Faster models naturally tend to be much smaller in capacity, which drastically improves physical efficiency metrics but severely limits their internal rep- resentations, hindering their ability to produce deep abstraction, logical restructuring, and high semantic coherence in complex summarization outputs. Conversely, massive models achieve superior semantic quality at the entirely unacceptable cost of crippled throughput and severe, rapid battery drain. Energy versus quality trade-off: Figure 8a visually demon- strates that the highest-quality configurations are inextricably linked to the most intensely energy-consuming execution profiles. Gemma-2-9B achieves the absolute best generative quality but demands a substantial and ultimately unsustainable energy budget for a portable device. However, we critically observe that raw parameter count alone does not strictly dictate the quality outcome. For instance, Qwen2.5-7B manages to achieve incredibly better quality while consuming measurably less total energy than Meta-Llama-3.1-8B. This clearly indicates that base architectural design choices, tokenizer efficiency, and pre-training dataset quality deeply influence the ultimate shape of the quality and efficiency frontier. Energy distributions and thermal stability: Figure 9 visual- izes the complete per-run energy distributions using violin plots. These distributions heavily reinforce the strong, mathematically predictable scaling trends for dense models and highlight the fascinating MoE deviation, showing OLMoE-1B-7B clustering tightly alongside the 1.5B and 3B dense models in absolute energy draw, completely defying its large resident weight capacity. Across all evaluated models, the two quantization variants produce broadly similar distribution shapes and internal variances. This specific variance spread indicates that while thermal throttling and background operating system scheduling inevitably introduce some expected run-to-run noise over 30 repetitions, the aggregate physical energy behavior of the device remains remarkably stable and bounded. 050100150200250300350 0 0.2 0.4 0.6 0.8 1 Total Energy Consumption (Joules) G-Eval Score (a) Energy vs. Quality 050100150200250300350 0 20 40 60 Total Energy Consumption (Joules) Generation Speed (tokens/s) Quantization Model IQ4XSGemma2-9B Q4KMLlama3.1-8B Qwen2.5-7B OLMoE-1B-7B Qwen2.5-3B Phi-2 Qwen2.5-1.5B Qwen2-0.5B (b) Energy vs. Speed 0204060 0 0.2 0.4 0.6 0.8 1 Generation Speed (tokens/s) G-Eval Score (c) Speed vs. Quality 012345 0 0.2 0.4 0.6 0.8 1 Model Size (GB) G-Eval Score (d) Size vs. Quality Fig. 8. Multi-objective trade-offs across models and quantization schemes. D. Statistical Analysis We rigorously apply non-parametric statistical testing methodologies, as mobile telemetry measurements rarely conform to ideal normal distributions. This non-normality is primarily due to the discrete, step-based nature of mobile processor dynamic voltage and frequency scaling, commonly referred to as DVFS, alongside abrupt thermal throttling events dictated by the kernel [57]. Normality testing: We first comprehensively assess the assumption of normality for all five key performance and efficiency metrics using the rigorous Shapiro-Wilk test, detailed in Table I. For every single evaluated metric, the calculated p-value is significantly less than the standard 0.05 threshold, formally rejecting the null hypothesis that the empirical samples are normally distributed. Consequently, we must rely exclusively on robust non-parametric tests for all subsequent inferential analyses to prevent mathematical artifacts and false positive conclusions. Model effects and multi-comparison rankings: To formally test whether the fundamental choice of the baseline language model significantly impacts the observed performance and physical efficiency metrics, we utilize the Friedman omnibus test, presented in Table I. Across all evaluated metrics, we observe extremely large chi-squared statistics coupled TABLE I SHAPIROâWILK TEST RESULTS. MetricW-Statisticp-value Generation Speed0.73753.1716e-19 Energy Consumption0.88007.7721e-13 Memory Usage0.88852.6160e-12 Latency0.85402.6560e-14 Time to First Token0.85744.0479e-14 with vanishingly small p-values. This provides overwhelming statistical evidence of massive and highly significant differences across the model cohort. To conduct the required post-hoc phase and precisely isolate these specific differences, we perform multiple pairwise Wilcoxon signed-rank tests combined with the strict Holm- Bonferroni correction to rigorously control the family-wise error rate across dozens of simultaneous statistical comparisons. Table I also reports the calculated Wilcoxon signed-rank test mean ranks, where a lower numerical rank indicates vastly supe- rior performance. The ultra-compact Qwen2-0.5B consistently and absolutely dominates the board, ranking best across all latency and efficiency-related metrics. Qwen2.5-1.5B and the Fig. 9. Total energy per run across models and quantization schemes. Distributions summarize 30 repetitions per configuration. sparse OLMoE-1B-7B form a highly competitive and tightly clustered second tier. The massive dense models, specifically Meta-Llama-3.1-8B and Gemma-2-9B, consistently rank at the absolute bottom, statistically reflecting their overwhelmingly high latency, massive memory footprints, and extreme energy costs. However, from a practical systems deployment perspec- tive, it is critical to heavily emphasize that while Qwen2-0.5B completely dominates the energy, performance, and memory trade-offs, it incurs a severe and easily measurable penalty in terms of actual cognitive generation quality, as definitively reflected in its substantially lower G-Eval scores. Quantization effects and statistical reality: To mathemati- cally quantify the true physical impact of the selected quantiza- tion formats beyond their theoretical footprint reductions, we directly compare the pairedQ4_K_MandIQ4_XSruns using the Wilcoxon signed-rank test. Table IV summarizes these highly revealing and somewhat counter-intuitive results. While the codebook-basedIQ4_XSachieves an absolute statistical victory in minimizing peak memory usage, the structurally sim- plerQ4_K_Machieves statistically significant victories across all critical dynamic operational metrics, explicitly including generation speed, overall latency, and most importantly, total physical energy consumption. Summary of key findings and practical deployment implica- tions: Synthesizing this massive corpus of comprehensive data, the empirical results dictate several absolute, foundational rules for mobile language model inference. First, the memory-bound autoregressive generation phase overwhelmingly dominates end-to-end latency and physical energy draw, entirely dwarfing the prompt prefill phase. Second, baseline neural architecture and aggregate parameter scale shape the energy and speed profiles far more aggressively than advanced post-training com- pression techniques. Third, while highly advanced codebook quantization likeIQ4_XSoperates as a highly effective tool for shoehorning massive parameter arrays into rigidly constrained RAM footprints, it introduces severe processing overhead that strictly prevents it from translating memory savings into actual battery life savings relative to mathematically simpler block- wise schemes likeQ4_K_M. Finally, sparse Mixture-of-Experts architectures successfully shatter traditional dense scaling laws, offering a highly distinct and extremely advantageous physical efficiency profile. This architecture effectively provides the cognitive reasoning capacity of a massive model with the active computational energy draw of a much smaller one. Based directly on these rigorously established empirical trade- offs, we can clearly extract actionable deployment paradigms tailored to highly specific engineering constraints. If an application demands absolute top-tier cognitive reasoning and the target device possesses abundant RAM alongside massive thermal dissipation headroom, large dense models like Gemma- 2-9B are mandatory. However, developers must willingly accept rapid battery degradation and severe, highly noticeable latency penalties. Conversely, if strict battery preservation and immediate interactivity are the unyielding primary constraints, deploying highly distilled models in the 0.5B to 1.5B class provides the only mathematically viable engineering path, accepting a known and significant penalty in complex summa- rization and reasoning quality. For heavily balanced, general- purpose consumer applications that require acceptable, human- like quality without instantly rendering the smartphone unusable due to intense heat or crippling latency, mid-sized dense models in the 3B parameter range, or highly efficient sparse architectures like OLMoE, definitively represent the optimal engineering sweet spot. These carefully balanced configurations deliver manageable thermal and energy footprints, reasonable user responsiveness, and highly robust reasoning capabilities. TABLE I FRIEDMAN TEST (OMNIBUS) & PAIRWISE WILCOXON SIGNED-RANK TEST (HOLMâBONFERRONI CORRECTED) FOR MODEL COMPARISONS. ModelGenerationEnergyMemoryLatencyTTFT FriedmanĎ 2 413.07415.83420.00417.78415.46 Global p-value3.7573e-859.5526e-861.2227e-863.6554e-861.1513e-85 Qwen2-0.5B1.001.001.001.001.00 Qwen2.5-1.5B2.932.952.002.932.95 Phi-24.074.023.004.014.08 Qwen2.5-3B4.934.984.004.984.92 OLMoE-1B-7B2.072.055.002.072.05 Qwen2.5-7B6.336.086.006.036.00 Meta-Llama-3.1-8B6.676.987.006.977.88 Gemma-2-9B8.007.938.008.007.12 V. DISCUSSION This section critically interprets the empirical findings derived from our measurements and highlights the broader practical implications for deploying large language models on resource-constrained mobile devices. We specifically focus on understanding why memory-oriented quantization does not necessarily translate into proportional energy savings, analyzing the unique opportunities and inherent physical constraints of sparse Mixture-of-Experts architectures, addressing the methodological realities and compromises of power profiling on commercial consumer smartphones, and emphasizing the critical role of thermal dynamics in sustained on-device token generation. A. The QuantizationâEnergy Paradox: Memory Savings Do Not Equal Battery Savings A central and somewhat counter-intuitive finding of our empirical evaluation is that the choice between the two highly distinct 4-bit quantization formats, specificallyQ4_K_Mversus IQ4_XS, has a surprisingly limited effect on total energy consumption. This minimal variance stands in stark contrast to the much larger energy differences induced by baseline model size and foundational neural architecture. While the importance- awareIQ4_XSformat consistently and successfully reduces peak memory usage, it does not provide consistent or propor- tional energy savings and frequently exhibits slightly higher end-to-end latency during the generation phase. This behavioral divergence suggests a critical micro- architectural reality: under CPU-only inference on contempo- rary mobile hardware, the physical energy consumption during autoregressive decoding is not determined solely by the gross weight memory footprint or the theoretical reduction in memory bandwidth utilization. Instead, the dominant energetic cost is the end-to-end per-token execution overhead, which arises from a highly complex combination of arithmetic compute, memory traffic including continuous Key-Value cache updates, and underlying runtime kernel behavior [58]. In particular, importance-aware and sub-byte formats such asIQ4_XS inherently rely on additional mathematical reconstruction logic. As thoroughly documented by Nagel et al. [59], extreme low-bit quantization schemes typically require dynamic dequantization at runtime because mobile arithmetic logic units are historically optimized for 32-bit or 16-bit precision. The utilization of codebook-based indirections specifically requires the CPU to perform supplementary lookup operations and complex bit-shifting algorithms to unpack the compressed weights into a usable floating-point or integer format before any matrix multiplication can actually occur. This heavy indirection directly disrupts the highly optimized, sequen- tial memory access patterns typically exploited by modern processor prefetchers, frequently introducing severe CPU pipeline stalls and costly cache misses [60]. These architectural inefficiencies drastically increase the active computational work required per token, thereby consuming any electrical energy saved by transferring fewer total bytes across the system bus. We therefore interpretIQ4_XSand similar advanced compression schemes primarily as capacity enablers. They make larger, more capable models technically feasible within the strict confines of limited mobile RAM, rather than acting as guaranteed efficiency optimizers capable of extending battery life. TABLE IV WILCOXON SIGNED-RANK TEST RESULTS FOR QUANTIZATION COMPARISON. MetricWp-valueWinner Generation Speed111790.00423 Q4_K_M Energy Consumption89673.36e-07Q4_K_M Memory Usage02.82e-41 IQ4_XS Latency29661.32e-26Q4_K_M Time to First Token8992.24e-36 Q4_K_M Consequently, for software engineering applications where battery preservation and immediate system responsiveness are the primary design objectives, selecting a fundamentally smaller language model or a natively more efficient architecture typi- cally yields vastly larger energetic gains than simply switching between two competing 4-bit quantization formats. Conversely, if the absolute main constraint for a deployment scenario is fitting a highly capable model into a severely restricted memory budget without triggering operating system out-of-memory terminations,IQ4_XSremains a remarkably strong choice. It successfully lowers the peak RAM footprint while retaining competitive semantic quality, provided the developer is willing to accept the hidden energetic and computational overhead of runtime weight reconstruction. B. The Promise and Constraint of Mixture-of-Experts Ar- chitectures The evaluated sparse Mixture-of-Experts model, OLMoE- 1B-7B, demonstrates a highly distinctive and promising mobile operational profile. It possesses a high resident weight capacity that is entirely comparable to a standard 7B dense model, yet its measured latency and physical energy consumption track much closer to significantly smaller dense models in the 1B to 2B parameter class. This behavioral pattern perfectly aligns with the fundamental MoE inference mechanism, where a dynamic routing network ensures that only a carefully selected subset of neural experts is activated per generated token [61]. This fundamentally changes the computational profile of the device, corroborating the architectural theories proposed by Zhou et al. [62], who demonstrated that intelligent expert routing can drastically decouple parameter count from pure inference latency. By entirely decoupling the total parameter count from the active parameter count, this architecture drastically reduces the active arithmetic compute required per generated token. This explicit decoupling is deeply attractive for mobile deployment because it effectively shifts the traditional quality and efficiency frontier. Higher effective cognitive capacity and superior abstractive summarization can be achieved without paying the massive, unsustainable computational cost typically associated with dense network scaling. However, the MoE paradigm introduces a severe and critical constraint specifically for smartphones: all individual experts must remain fully loaded in the system memory at all times, even if only a minuscule fraction of them is actively utilized for any given token. Because mobile operating systems utilize unified memory architectures shared across the CPU, GPU, and background processes, a massive resident footprint places immense pressure on the entire device. Gale et al. [63] similarly emphasize that the memory bandwidth required to fetch scattered expert weights can severely bottleneck MoE inference on unoptimized edge hardware, shifting the limitation from computational bound to memory-capacity bound. If the operating system is forced to swap dormant experts to slower internal flash storage, the latency penalty during expert retrieval would render the model entirely unusable for real-time interaction. As a result, MoE architectures can offer excellent speed and energy characteristics only if the target device possesses sufficient RAM headroom to securely store the complete expert set alongside the rapidly expanding Key- Value cache and all necessary runtime buffers. The Mixture- of-Experts paradigm is therefore a highly promising research direction for achieving top-tier generative quality at a bounded energy cost, but it is currently most suitable for premium flagship devices equipped with extraordinarily large RAM budgets. For applications targeting broader device ecosystems or heavily memory-constrained mid-range phones, smaller dense models may currently remain the most robust and predictable engineering option. C. Power Profiling on Consumer Smartphones: Absolute Accuracy versus Ecological Practicality Reliable and granular energy measurement on commercial smartphones is a notoriously formidable challenge due to artificially restricted telemetry access, highly unpredictable background operating system activity, and complex hardware thermal dynamics. External power monitors, such as the Monsoon hardware power monitor [5], provide high-fidelity electrical measurements directly at the circuit level and are widely considered the gold standard within hardware engineer- ing disciplines. However, deploying these physical systems often requires physically disassembling the device, bypassing the native battery management system, and establishing invasive hardware setups. This methodology is highly impractical for large-scale consumer-device studies and is fundamentally incompatible with evaluating an intact phone in a realistic, unmodified configuration, as removing the chassis completely alters the natural heat dissipation pathways of the phone. On the other end of the spectrum, software-based estimation tools, such as frequency-state power models, are infinitely easier to deploy but can be wildly unreliable for sustained, computa- tionally heavy LLM workloads. During long inference sessions lasting several minutes, CPU clock frequencies constantly and unpredictably fluctuate due to aggressive operating system governors and mandatory thermal throttling mechanisms. At- tempting to map these volatile frequency states to static power consumption models can therefore introduce substantial margin of error. As established by foundational micro-architecture studies from Skadron et al. [64], static power dissipation, primarily driven by leakage current within silicon architectures, increases exponentially with internal hardware temperature. Throughout our preliminary investigations, we observed that even microscopic changes in ambient room temperature or routine background network activity could severely perturb the inferred power models. Consequently, static software estimators consistently fail to capture this thermal-electrical feedback loop, causing them to diverge sharply from physical reality during sustained edge workloads [65]. For these precise reasons, we adopted a highly controlled on-device measurement strategy based directly on the native AndroidBatteryManagertelemetry APIs. By combining this internal telemetry with extremely strict experimental controls, including locked screen states, background process isolation, rigorous baseline power subtraction, and dozens of repeated runs over Wireless ADB to avoid USB charging noise, we established a highly reliable profiling pipeline. This specific approach consciously trades a minor degree of absolute electri- cal accuracy for immense practical reproducibility and total realistic device configuration. Preserving the ecological validity of the physical device is often the absolute central requirement for conducting meaningful, externally valid comparative studies of mobile machine learning inference. D. Thermal Dynamics and Throttling as a First-Class Hardware Constraint Thermal behavior is not merely a secondary artifact of computation; it is a first-class, dominant constraint for sustained on-device generative artificial intelligence. Consumer smartphones rely entirely on passive cooling mechanisms, dissipating heat outward through tightly packed glass and metal chassis. As the device rapidly heats up under the immense computational load of continuous matrix multiplications, mobile systems-on-chip immediately engage self-preserving thermal management mechanisms. As explored by Esmaeilzadeh et al. [66] in their foundational analysis of dark silicon, modern mobile processors are entirely physically incapable of power- ing all transistors simultaneously without rapidly exceeding catastrophic thermal design limits. These hardware-level governors drastically reduce CPU clock frequencies and throttle processing performance to remain safely within established thermal design power limits. This unavoidable physical reality can massively increase token generation latency and fundamentally change the overall energy dynamics of the device. Because total energy is mathematically the integral of power over time, artificially extending the generation phase through severe thermal throttling ultimately increases the total energy drain of the device, especially for longer summarization or dialogue sessions where thermal energy inevitably accumulates within the chassis [67]. To mathematically reduce this severe environmental con- found during our empirical study, we rigorously enforced a 200-second resting cool-down period between every single experimental run and continuously monitored the internal silicon temperature to keep all experiments within a highly stable, ambient operating range of 22 to 28 degrees Cel- sius. Nevertheless, real-world deployments by end-users will inevitably encounter far broader and more hostile thermal variations, such as executing models during outdoor use in direct sunlight, running inference while the device is actively charging, or generating text while heavy background applications consume residual CPU cycles. This inescapable hardware reality strongly suggests that practical, production-ready on-device digital assistants must evolve to incorporate sophisticated energy-aware and thermal- aware control policies. For example, the software architecture could be designed to dynamically shift computation to a much smaller, less capable fallback model when the internal battery temperature exceeds a critical threshold. Alternatively, the application could autonomously reduce the maximum allowable generation length, increase the deployment of speculative decoding techniques, or adaptively lower decoding parameters to gracefully degrade the user experience without completely draining the battery or causing the operating system to forcefully terminate the application due to critical hardware overheating. VI. LIMITATIONS While the experimental methodology of this study was rigorously designed to be highly reproducible and strictly representative of an unrooted, real-world consumer device setting, several inherent architectural and methodological limitations naturally affect both the universal generalizability of the findings and the absolute fidelity of the physical measurements. Acknowledging these boundaries is essential for properly contextualizing the empirical results. A. Restricted Access to Low-Level Power Telemetry A theoretically straightforward approach to high-fidelity power profiling would be to read instantaneous battery voltage and current directly via the exposed system interfaces of the Android Debug Bridge. In practice, however, modern Android security architectures and strict kernel-level policies restrict access to many of these low-level electrical signals for non-privileged users to proactively prevent sophisticated side- channel attacks and unauthorized hardware telemetry extraction. As detailed by Yan et al. [68], malicious actors can potentially exploit high-resolution power traces to infer sensitive user activities or cryptographically secure data, prompting operating system vendors to lock down these diagnostic interfaces. On contemporary flagship devices, detailed micro-ampere current and precise voltage readings are entirely unavailable to external debugging clients unless the device operating system is explicitly rooted. Rooting the smartphone would technically provide the elevated administrative access necessary for deeper telemetry, but this approach was intentionally and explicitly avoided for two primary methodological reasons. First, the rooting process inevitably modifies the core operating system, often requiring custom kernels that fundamentally alter the native power- management behavior, the CPU frequency scaling governors, and the proprietary thermal throttling thresholds originally programmed by the manufacturer. Modifying these deep system parameters would completely undermine the central research goal of evaluating inference under realistic, stock user conditions. Second, rooting a flagship device typically trips irreversible hardware security fuses, permanently voids device warranties, and disables critical system functionalities, rendering this methodology impractical and largely unfeasible for many researchers or industry practitioners seeking to replicate the pipeline. Therefore, our energy measurements rely entirely on user-space telemetry acquired through the native Android BatteryManager APIs [47]. While this software- based approach guarantees high practical reproducibility and perfectly preserves the ecological validity of the physical device, the sampling frequency and absolute precision may be marginally less accurate than invasive external hardware instrumentation. B. Single-Device Hardware Scope We intentionally designed this research framework as an empirical case study centered specifically on a single flagship smartphone, the Samsung Galaxy S25 Ultra. In empirical software engineering terminology, carefully framing this work as an exploratory case study effectively mitigates the imme- diate threat to external validity, as our primary objective is to formulate strongly grounded research hypotheses for future broad validation rather than to hastily claim universal architectural generalizability [69]. Although this specific mobile platform represents the abso- lute bleeding edge of contemporary high-end Android hardware, featuring a state-of-the-art neural processor and exceptionally high-bandwidth memory, the specific quantitative results and exact energy values will inevitably vary on phones possessing different architectural characteristics. Devices equipped with different systems-on-chip, narrower memory bus widths, less efficient cache hierarchies, constrained RAM capacities, older battery chemistry implementations, or inferior passive thermal cooling designs will exhibit radically different performance profiles. In particular, mid-range or lower-end edge devices will experience exponentially stronger memory pressure during inference. This severe memory limitation forces the operating system to aggressively page memory to flash storage, prompting much earlier thermal throttling interventions by the operating system and resulting in drastically lower sustained token throughput [70]. These severe hardware bottlenecks would fundamentally shift the observed speed, energy, and cognitive quality frontier downward, making the deployment of larger models entirely impossible on budget hardware. C. Workload and Task Specificity We evaluate the language models using a single, strictly bounded task type tailored for document summarization. This evaluation utilizes a completely fixed decoding configuration executing greedy decoding with a temperature strictly set to zero, and enforcing a maximum generation length of exactly 100 output tokens. While this rigid standardization is absolutely necessary to ensure mathematical comparability across dozens of varying models and configurations, other generative workloads will inevitably yield vastly different execution profiles. Alternative applications, such as dynamic multi-turn dialogue, tool-augmented agentic prompting, unconstrained long-form creative generation, complex code synthesis, or advanced chain-of-thought reasoning, place entirely different structural demands on the mobile hardware. For example, unconstrained long-form outputs force the internal Key-Value cache of the model to grow linearly with every single generated token. As rigorously analyzed by Pope et al. [3], this rapid cache expansion consumes enormous amounts of precious mobile RAM and severely saturates the memory bandwidth, extending the total time spent in the highly inefficient autoregressive decoding phase. Consequently, these sustained workloads may amplify the physical energy drain and trigger severe thermal throttling effects far beyond the specific boundaries observed and documented within this isolated summarization study. D. Quantization and Optimization Scope Our current investigation focuses on comparing two highly prevalent 4-bit quantization formats, specifically the block- wiseQ4_K_Mand the importance-awareIQ4_XS, evalu- ated entirely within the establishedllama.cppand GGUF software ecosystem. We explicitly do not evaluate more aggressive experimental compression techniques, such as sub- 4-bit or ternary quantization, structured network pruning, deep knowledge distillation, or speculative decoding algorithms. While these advanced systems optimizations could theoretically alter the latency and energy trade-offs, aggressive sub-byte compression frequently destroys the semantic coherence of models smaller than ten billion parameters, rendering them useless for practical human assistance. Furthermore, speculative decoding, which utilizes a smaller draft model to predict tokens for a larger target model, might successfully mask apparent latency for the user but could paradoxically increase total physical energy consumption due to the necessity of keeping two distinct neural networks resident in memory and executing simultaneously [71]. Additionally, we purposefully constrain our focus exclusively to CPU-only inference. While modern smartphone systems-on- chip include highly capable proprietary Neural Processing Units and Graphics Processing Units, the software ecosystem for het- erogeneous mobile acceleration remains severely fragmented across different silicon vendors. Shifting the execution pipeline to hardware accelerators would completely change the fundamental system bottlenecks. As demonstrated by Cass [72], transitioning from CPU to highly parallel accelerators shifts the primary constraint from memory bandwidth limitations directly to severe accelerator dispatch latency and costly memory transfer overheads between the CPU and the NPU, thereby producing entirely different energetic conclusions. E. Environmental and Runtime Variability Despite implementing extraordinarily rigorous experimental controls, including strict background process isolation, a highly restricted operational battery range, and mandatory enforced thermal cool-down intervals, all commercial consumer devices inherently exhibit unavoidable operational variability. This systemic variance is driven by invisible operating system thread scheduling, residual background maintenance services, cellular network polling, the physical degradation of battery health over time, and subtle fluctuations in ambient room temperature. These uncontrolled environmental factors can introduce minor run-to-run electrical noise and may slightly influence the absolute energy values recorded by the telemetry sys- tem. For instance, as a lithium-ion battery ages, its internal electrical resistance naturally increases, causing voltage to drop more aggressively under the heavy sustained load of matrix multiplications [73]. Our carefully structured repeated- run design, combined with robust non-parametric statistical analysis, successfully mitigates this background noise for drawing accurate comparative conclusions between the tested models. Nevertheless, a certain degree of baseline variance remains an inherent and inescapable reality of conducting empirical performance measurement on highly complex, un- rooted physical edge devices. VII. CONCLUSIONS AND FUTURE WORK This paper presented a highly rigorous, reproducible method- ology designed to thoroughly profile on-device large language model inference operating within the strict constraints of an unrooted commercial Android smartphone. By moving beyond isolated performance metrics, we established a comprehensive framework for jointly evaluating hardware performance, physi- cal energy consumption, peak resident memory utilization, and semantic output quality. Utilizing thellama.cppexecution engine in conjunction with a custom native telemetry applica- tion based on the Android BatteryManager API, we successfully bypassed external hardware requirements and operating system security restrictions to gather pristine, high-resolution power data. Through careful empirical baseline subtraction and precise numerical integration over 100-millisecond sampling intervals, we systematically benchmarked eight diverse open-source models ranging from a compact 0.5 billion to a massive 9 billion parameters. As highlighted by Reddi et al. [74] in their foundational work on mobile machine learning benchmarking, establishing standardized and entirely reproducible measure- ment pipelines is essential for tracking genuine progress in edge artificial intelligence, effectively separating theoretical software improvements from actual physical hardware efficiency gains. We evaluated these architectures under two distinct 4-bit quantization schemes, specifically the block-wiseQ4_K_M and the importance-awareIQ4_XS, and strictly repeated each unique hardware and software configuration 30 times to ensure the highest degree of statistical confidence and to mitigate unavoidable environmental and thermal variance. The synthesis of our empirical results reveals three distinct and highly practical operating regions that software engineers must navigate depending on their specific deployment goals and hardware constraints. For operational scenarios requiring high cognitive intelligence and prioritizing absolute generative quality, massive dense models such as Gemma-2-9B and Meta- Llama-3.1-8B achieve the strongest semantic reasoning and deepest logical abstraction. However, they incur the highest physical energy cost and suffer from crippling end-to-end latency that destroys interactive user experiences. Consequently, these massive architectures are best suited for infrequent, exceptionally high-value background queries where rapid battery drain and delayed responses are explicitly deemed acceptable by the user. Conversely, for high-efficiency mo- bile applications where battery preservation and immediate interactivity are the primary engineering concerns, ultra-small models like Qwen2-0.5B deliver the fastest possible responses and the absolute lowest energy consumption per inference run. While this efficiency comes with a mathematically measurable reduction in complex summary quality and reasoning capability, these nimble models represent an excellent fit for generating short, tightly bounded outputs, performing rapid background text parsing, and facilitating frequent, low-latency user interface interactions. A highly practical middle ground is firmly established by mid-sized models such as Qwen2.5-3B and Phi- 2. These intermediate architectures provide a highly favorable engineering compromise, achieving moderate and sustainable energy consumption alongside acceptable latency bounds while maintaining highly competitive generative quality, ultimately making them the most attractive default option for continuous, daily-use mobile digital assistants. Beyond identifying these specific operational regions, three broader and more fundamental conclusions regarding mobile machine learning stand out from our analysis. First, foun- dational architecture and the volume of active compute absolutely dominate physical mobile efficiency. The empirical performance of the MoE model profoundly demonstrates that sparse neural activation can successfully approach the extremely low energy and latency profiles of much smaller dense models while simultaneously maintaining the vast reasoning capacity of a massive network. However, engineers must carefully balance this massive computational advantage against the high resident memory requirements necessary to keep all experts continuously loaded in RAM. Second, theoretical memory savings do not necessarily imply physical energy savings. The sophisticatedIQ4_XSformat successfully and reliably reduces peak memory utilization, but its end-to-end energy and latency benefits are extremely limited when directly compared to the mathematically simplerQ4_K_Mformat. This strongly indicates that the per-token runtime computational overhead required to constantly reconstruct codebook indices can completely offset the anticipated energy benefits of a reduced weight footprint. Third, semantic quality evaluation in resource-constrained environments absolutely requires a multi- view approach. Traditional reference-based similarity metrics heavily reward lazy extractive behavior, whereas reference- free judging via G-Eval far better captures true cognitive coherence, abstractive reasoning, and ultimate usefulness for complex summarization tasks. This directly corroborates recent findings by Zheng et al. [75], who conclusively demonstrated that strong LLM-as-a-judge frameworks exhibit a significantly higher correlation with actual human preferences than classical embedding-based similarity metrics. Overall, achieving genuinely sustainable edge intelligence requires the research and engineering communities to ele- vate physical energy consumption and thermal dissipation constraints to the status of first-class optimization objectives, equal in importance to benchmark accuracy and theoretical parameter compression. Practical deployment decisions must meticulously prioritize inherently efficient model families and sparse architectures, and engineers must rigorously validate their optimization choices under realistic, highly constrained physical device conditions rather than relying on theoretical parameter counts or idealized laboratory metrics alone. This paradigm shift firmly aligns with the broader environmental imperatives outlined by Patterson et al. [76], emphasizing that mitigating the expanding carbon footprint of massive gener- ative models requires aggressively pushing highly optimized inference tasks toward decentralized, low-power edge hardware. Looking forward, we identify several critical directions to significantly extend the foundational work presented in this study. One vital avenue is the transition to fully native on-device benchmarking, which involves integrating the entire orchestration framework, the inference engine, and the background monitoring service directly into a single, cohesive Android application. This architectural shift would entirely eliminate the need for host-side ADB control and enable continuous profiling under far more realistic, chaotic user interaction patterns, specifically evaluating the energetic impact of foreground and background application switching, incoming operating system notifications, and highly extended multi-turn dialogue sessions where context windows grow dynamically. Another highly promising research direction is the develop- ment of adaptive, energy-aware inference orchestration frameworks. This focuses on the algorithmic design of dynamic operating system policies that can intelligently adjust model selection and decoding parameters in real-time. For example, a system could autonomously manipulate maximum token generation limits, restrict KV-cache expansion, engage speculative decoding, or dynamically route queries to a much smaller fallback model based entirely on live device temperature readings, current battery degradation states, and strict latency targets. Such adaptive cascading architectures, conceptually similar to the cloud-based API routing strategies proposed by Chen et al. [77], must be rigorously adapted and engineered specifically for localized thermal and energetic management on mobile processors. Furthermore, aggressively expanding the scope of evaluated workloads and hardware ecosystems is necessary to validate these hypotheses universally. By including deeply complex additional tasks such as logical reasoning, local code generation, tool-use execution, and unconstrained long-form generation across multiple devices spanning drastically different low- end and high-end systems-on-chip and thermal envelopes, re- searchers can significantly improve the generalizability of these findings. As demonstrated by Xiao et al. [78], managing the virtually unbounded growth of the Key-Value cache in extended long-context scenarios represents a critical memory bottleneck that will profoundly impact mobile energy limits and requires entirely new streaming inference paradigms. Investigating these limits will fundamentally help establish desperately needed, standardized mobile LLM energy benchmarks for the broader scientific community. Finally, investigating the energetic realities of heterogeneous hardware acceleration through specialized mobile GPUs, NPUs, and hybrid execution paths will be absolutely critical. Future systems research must precisely quantify the inflection points where these dedicated hardware accelerators genuinely reduce overall physical energy consumption versus the sce- narios where initial dispatch overhead, kernel initialization, and massive memory transfer latencies between the CPU and the heterogeneous accelerator ultimately dominate and actively degrade the energetic efficiency of the system. REFERENCES [1] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, âEdge computing: Vision and challenges,â IEEE Internet of Things Journal, vol. 3, no. 5, p. 637â646, 2016. [2]C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, C. Bai et al., âSustainable AI: Environmental implications, challenges and opportunities,â Proceedings of Machine Learning and Systems, vol. 4, p. 795â813, 2022. [3]R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, âEfficiently scaling transformer inference,â Proceedings of Machine Learning and Systems, vol. 5, 2023. [4]R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Fang et al., âDeepSpeed-inference: enabling efficient inference of transformer models at unprecedented scale,â in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2022, p. 1â15. [5]Monsoon Solutions, Inc., High Voltage Power Monitor (P/N: A10F) User Manual, Monsoon Solutions, Inc., Bellevue, WA, USA, 2024. [Online]. Available: https://w.msoon.com/high-voltage-power-monitor [6]J. C. McCullough, Y. Agarwal, J. Chandrashekar, S. Kuppuswamy, A. C. Snoeren, and R. K. Gupta, âEvaluating the effectiveness of model-based power characterization,â in Proceedings of the 2011 USENIX Annual Technical Conference (USENIX ATC â11), 2011. [7]C. Yoon, D. Kim, W. Jung, C. Kang, and H. Cha, âAppScope: Application energy metering framework for Android smartphones using kernel activity monitoring,â in Proceedings of the 2012 USENIX Annual Technical Conference (USENIX ATC â12), 2012. [8]G. Gerganov and llama.cpp contributors, âllama.cpp: LLM inference in C/C++,â https://github.com/ggml-org/llama.cpp, 2023. [9]T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, âBERTScore: Evaluating text generation with BERT,â in Proceedings of the 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, Apr 2020. [Online]. Available: https://openreview.net/forum?id=SkeHuCVFDr [10]Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, âG-Eval: NLG evaluation using GPT-4 with better human alignment,â in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).Singapore: Association for Computational Linguistics, Dec 2023, p. 2511â2522. [Online]. Available: https://aclanthology.org/2023.emnlp-main.153 [11] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, âGreen AI,â Communications of the ACM, vol. 63, no. 12, p. 54â63, 2020. [12] E. Strubell, A. Ganesh, and A. McCallum, âEnergy and policy consid- erations for deep learning in NLP,â in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, p. 3645â3650. [13] Y. Yuan, J. Zhang, Z. Zhang, K. Chen, J. Shi, V. Stoico, and I. Malavolta, âThe impact of knowledge distillation on the energy consumption and runtime efficiency of nlp models,â in Proceedings of the 2024 IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI (CAIN â24). Lisbon, Portugal: ACM, 2024. [14]V. Nguyen, V. Dhopate, H. Huynh, H. Bouhlal, A. Annengala, G. L. Scoccia, M. Martinez, V. Stoico, and I. Malavolta, âOn-device or remote? on the energy efficiency of fetching llm-generated content,â in Proceedings of the 2025 IEEE/ACM 4th International Conference on AI Engineering - Software Engineering for AI (CAIN â25). IEEE, 2025, p. 72â82. [15] M. Abstreiter, âSometimes painful but certainly promising: Feasibility and trade-offs of language model inference at the edge,â in Proceedings of the 4th Workshop on Machine Learning and Systems (EuroMLSys â24).Athens, Greece: ACM, 2024, p. 1â8. [Online]. Available: https://doi.org/10.1145/3642970.3655835 [16]G. Xiao, J. Lin, F. Seide, S. Han et al., âSmoothquant: Accurate and efficient post-training quantization for large language models,â in Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. [17] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, âGPTQ: Accurate post-training quantization for generative pre-trained transformers,â in Proceedings of the 11th International Conference on Learning Represen- tations (ICLR), 2023. [18]T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, âLLM.int8(): 8-bit matrix multiplication for transformers at scale,â in Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS â22), vol. 35, 2022, p. 30 318â30 332. [19]J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, âAWQ: Activation-aware weight quantization for LLM compression and acceleration,â in Proceedings of the 7th MLSys Conference (MLSys 2024), 2024, santa Clara, CA. [Online]. Available: https://arxiv.org/abs/2306.00978 [20]T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze et al., âTVM: An automated end-to-end optimizing compiler for deep learning,â in Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI â18), 2018, p. 578â594. [21]S. Laskaridis, K. Katevas, L. Minto, and H. Haddadi, âMELTing Point: Mobile evaluation of language transformers,â in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking (MobiCom â24). Washington D.C., USA: ACM, Nov 2024, p. 890â907. [Online]. Available: https://doi.org/10.1145/3636534.3690668 [22]T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. R Ě e, âFlashAttention: Fast and memory-efficient exact attention with IO-awareness,â in Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS â22), vol. 35, 2022, p. 16 344â16 359. [23]A. Carroll and G. Heiser, âAn analysis of power consumption in a smartphone,â in Proceedings of the 2010 USENIX Annual Technical Conference (USENIX ATC â10), vol. 14, Boston, MA, 2010, p. 21â21. [24]A. Pathak, Y. C. Hu, and M. Zhang, âWhere is the energy spent inside my app? Fine grained energy accounting on smartphones with Eprof,â in Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys â11), 2011, p. 29â42. [25]S2-group, âBatterymanager-companion: Companion app for the bat- terymanager plugin for android-runner,â https://github.com/S2-group/ batterymanager-companion/, 2024. [26]A. Hindle, A. Wilson, K. Rasmussen, E. J. Jedwab, R. Godfrey, and P. Sweeney, âGreen mining: investigating power consumption across versions,â in Proceedings of the 34th International Conference on Software Engineering (ICSE â12). IEEE, 2012, p. 1305â1308. [27]I. Malavolta, E. M. Grua, C.-Y. Lam, R. de Vries, F. Tan, E. Zielinski, M. Peters, and L. Kaandorp, âA framework for the automatic execution of measurement-based experiments on android devices,â in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE â20). ACM/IEEE, 2020. [28]M. Karsten, A. C. Dragomir, R. Apsan, V. Stoico, and I. Malavolta, âExperiment Runner: A tool for the automatic orchestration of experiments targeting software systems,â Science of Computer Programming, vol. 239, p. 103415, Jan 2025. [29] Qualcomm Technologies, Inc., âTrepn power profiler,â Qualcomm Developer Network, 2024, accessed: 2026-02-12. [Online]. Available: https://developer.qualcomm.com/forums/software/trepn-power-profiler [30]V. Schmidt, K. Goyal, A. Joshi, B. Feld, L. Conell, N. Laskaris, D. Blank, J. Wilson, S. Friedler, and S. Luccioni, âCodecarbon: Estimate and track carbon emissions from machine learning computing,â 2021. [31]J. Sallou, L. Cruz, and T. Durieux, âEnergibridge: Empowering soft- ware sustainability through cross-platform energy measurement,â arXiv preprint arXiv:2312.13897, 2023. [32]E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero, L. Tunstall, and T. Wolf, âOpen LLM Leaderboard,â Hugging Face Space, 2023. [Online]. Available: https://huggingface.co/ spaces/open-llm-leaderboard/openllmleaderboard [33]L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., âTraining language models to follow instructions with human feedback,â in Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS â22), vol. 35, 2022, p. 27 730â27 744. [34]J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, âFinetuned language models are zero-shot learners,â in Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021. [35]A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang et al., âQwen2 technical report,â arXiv preprint arXiv:2407.10671, 2024. [Online]. Available: https: //arxiv.org/abs/2407.10671 [36]QwenTeam,âQwen2.5technicalreport,â arXiv preprint arXiv:2412.15115, 2024. [Online]. Available: https://arxiv.org/abs/ 2412.15115 [37] M. Javaheripi and S. Bubeck, âPhi-2: The surprising power of small language models,â Microsoft Research Blog, Dec 2023. [Online]. Available: https://w.microsoft.com/en-us/research/blog/ phi-2-the-surprising-power-of-small-language-models/ [38]N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, P. Walsh, O. Tafjord, N. Lambert et al., âOLMoE: Open mixture-of-experts language models,â arXiv preprint arXiv:2409.02060, 2024. [Online]. Available: https://arxiv.org/abs/2409.02060 [39]A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letak, A. Mathur, A. Schelten, A. Yang, A. Fan et al., âThe Llama 3 herd of models,â arXiv preprint arXiv:2407.21783, 2024. [Online]. Available: https://arxiv.org/abs/2407.21783 [40] G. DeepMind, âGemma 2: Improving open language models at a practical size,â arXiv preprint arXiv:2408.00118, 2024. [Online]. Available: https://arxiv.org/abs/2408.00118 [41] A. Ignatov, R. Timofte, W. Chou, K. Wang, M. Wu, T. Hartley, and L. Van Gool, âAI benchmark: Running deep neural networks on Android smartphones,â in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, p. 0â0. [42] G. Gerganov and llama.cpp contributors, âllama.cpp quantize tool,â 2023, accessed: 2026-02-08. [Online]. Available: https://github.com/ ggml-org/llama.cpp/tree/master/examples/quantize [43] T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, â8-bit optimizers via block-wise quantization,â in International Conference on Learning Representations, 2022. [Online]. Available: https: //arxiv.org/abs/2110.02861 [44]Google, âAndroid Debug Bridge (ADB),â 2024, accessed: 2026-02-12. [Online]. Available: https://developer.android.com/tools/adb [45]M. A. Hoque, M. Siekkinen, and J. K. Nurminen, âUnderstanding the energy consumption of Android app idle states,â Pervasive and Mobile Computing, vol. 24, p. 68â86, 2015. [46]M. Li, Y. Gao, S. F. Al-Sarawi, and D. Abbott, âPower side-channel attacks on mobile devices: A survey,â IEEE Access, vol. 10, p. 6718â 6736, 2022. [47]Google, âAndroid BatteryManager API reference,â 2024, accessed: 2026-02-08. [Online]. Available: https://developer.android.com/reference/ android/os/BatteryManager [48] A. R. Fabbri, W. Kry Ě sci Ě nski, B. McCann, C. Xiong, R. Socher, and D. Radev, âSummEval: Re-evaluating summarization evaluation,â Transactions of the Association for Computational Linguistics, vol. 9, p. 391â409, 2021. [49] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and C. Re, âEfficient memory management for large language model serving with PagedAttention,â in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP â23), 2023, p. 611â626. [50]W. A. Wulf and S. A. McKee, âHitting the memory wall: implications of the obvious,â ACM SIGARCH Computer Architecture News, vol. 23, no. 1, p. 20â24, 1995. [51]J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, âScaling laws for neural language models,â arXiv preprint arXiv:2001.08361, 2020. [52] R. David, J. Duke, A. Jain, V. J. Reddi, N. Jeffries, J. Li, N. Krentz, T. Cruesoe, and P. Warden, âTensorFlow Lite Micro: Embedded machine learning for TinyML systems,â Proceedings of Machine Learning and Systems, vol. 3, p. 800â811, 2021. [53]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, âOutrageously large neural networks: The sparsely-gated mixture-of-experts layer,â arXiv preprint arXiv:1701.06538, 2017. [54]R. B. Miller, âResponse time in man-computer conversational transac- tions,â in Proceedings of the fall joint computer conference, part I, 1968, p. 267â277. [55] J. Nielsen, Usability engineering. Morgan Kaufmann, 1993. [56]S. Han, H. Mao, and W. J. Dally, âDeep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,â in Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015. [57] J. Kong, S. W. Chung, and K. Choi, âEnergy-efficient thermal manage- ment for multiprocessor systems-on-chip,â in Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE â13). IEEE, 2013, p. 1119â1124. [58]A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, âA survey of quantization methods for efficient neural network inference,â arXiv preprint arXiv:2103.13630, 2021. [59]M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. Van Baalen, and T. Blankevoort, âA white paper on neural network quantization,â arXiv preprint arXiv:2106.08295, 2021. [60]S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, âI- BERT: Integer-only BERT quantization,â in Proceedings of the 38th International Conference on Machine Learning (ICML).PMLR, 2021, p. 5506â5518. [61]W. Fedus, B. Zoph, and N. Shazeer, âSwitch transformers: Scaling to trillion parameter models with simple and efficient sparsity,â Journal of Machine Learning Research, vol. 23, no. 120, p. 1â39, 2022. [62]Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon et al., âMixture-of-experts with expert choice routing,â in Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS â22), vol. 35, 2022, p. 7103â7114. [63] T. Gale, D. Narayanan, C. Young, and M. Zaharia, âMegaBlocks: Efficient sparse training with mixture-of-experts,â Proceedings of Machine Learning and Systems, vol. 5, p. 288â304, 2023. [64] K. Skadron, M. R. Stan, W. Huang, S. Velusamy, K. Sankaranarayanan, and D. Tarjan, âTemperature-aware microarchitecture,â ACM SIGARCH Computer Architecture News, vol. 32, no. 2, p. 2â13, 2004. [65] V. Keller, R. Lachaize, V. Gramoli et al., âPower and energy characteriza- tion of ARM processors,â in Proceedings of the 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2014, p. 116â126. [66]H. Esmaeilzadeh, E. Blem, R. St. Amant, K. Sankaralingam, and D. Burger, âDark silicon and the end of multicore scaling,â IEEE Micro, vol. 32, no. 3, p. 122â134, 2011. [67]Y. Wang, Y. Li, X. Zheng, and H. Liu, âTowards sustainable AI: a comprehensive study of carbon footprints in large language models,â arXiv preprint arXiv:2310.03093, 2023. [68]C. Yan, X. Ji, K. Wang, Q. Jiang, Z. Jin, and W. Xu, âUnderstanding and mitigating the security risks of voice-driven interfaces,â in Proceedings of the 29th USENIX Security Symposium (USENIX Security 20), 2020, p. 2625â2642. [69]P. Runeson and M. H Ě ost, âGuidelines for conducting and reporting case study research in software engineering,â Empirical Software Engineering, vol. 14, no. 2, p. 131â164, 2009. [70]V. Gogte, W. Wang, A. Kolli, and T. F. Wenisch, âSoftware wear management for persistent memories,â in Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST â19), 2019, p. 45â 58. [71]Y. Leviathan, M. Kalman, and Y. Matias, âFast inference from transform- ers via speculative decoding,â in Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, 2023, p. 19 274â 19 286. [72]S. Cass, âTaking AI to the edge: Armâs new neural processing units,â IEEE Spectrum, vol. 56, no. 5, p. 16â17, 2019. [73]S. Pelletier, O. Jabali, G. Laporte, and M. Veneroni, âLithium-ion battery degradation: what you need to know,â Physical Chemistry Chemical Physics, vol. 19, no. 32, p. 21 231â21 245, 2017. [74]V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Maximov, T. Choudhury, D. Gregg et al., âMLPerf inference benchmark,â ACM SIGARCH Computer Architecture News, vol. 48, no. 1, p. 50â65, 2020. [75]L. Zheng, W.-L. Chiang, Y. Sheng, S. Hao, Z. Wu, J. Ba, Z. L. Jiang, Z. Wu, A. Mirza, Z. Li et al., âJudging LLM-as-a-judge with MT-Bench and chatbot arena,â in Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS â23), vol. 36, 2023. [76]D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, âCarbon emissions and large neural network training,â arXiv preprint arXiv:2104.10350, 2021. [77]L. Chen, M. Zaharia, and J. Zou, âFrugalGPT: How to use large language models while reducing cost and improving performance,â arXiv preprint arXiv:2305.05176, 2023. [78]G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, âEfficient streaming language models with attention sinks,â in Proceedings of the 12th International Conference on Learning Representations (ICLR), 2023.