Paper deep dive
This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs
Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas, Vishal Banwari, Paul Lukowicz, Jakob Karolus
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 5:16:43 AM
Summary
This paper investigates using inference time as a proxy for estimating the energy consumption of API-based Large Language Models (LLMs). By benchmarking Mistral models locally on various NVIDIA GPU architectures (Ampere and Hopper series) and comparing these results with API-based inference times, the authors demonstrate that time measurements can effectively ground energy cost estimations, despite the inherent opacity of API-based infrastructure.
Entities (5)
Relation Signals (3)
CarbonTracker → measures → Energy Consumption
confidence 95% · Energy consumption is recorded with CarbonTracker
Inference Time → servesasproxyfor → Energy Consumption
confidence 95% · we investigate inference time measurements as a proxy to approximate the associated energy costs of API-based LLMs
Mistral-7B-Instruct-v0.3 → isequivalentto → Open-Mistral-7B
confidence 90% · Mistral-7B-Instruct-v0.3 (Local), equivalent to Open-Mistral-7B (API)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The energy consumption of Large Language Models (LLMs) is raising growing concerns due to their adverse effects on environmental stability and resource use. Yet, these energy costs remain largely opaque to users, especially when models are accessed through an API -- a black box in which all information depends on what providers choose to disclose. In this work, we investigate inference time measurements as a proxy to approximate the associated energy costs of API-based LLMs. We ground our approach by comparing our estimations with actual energy measurements from locally hosted equivalents. Our results show that time measurements allow us to infer GPU models for API-based LLMs, grounding our energy cost estimations. Our work aims to create means for understanding the associated energy costs of API-based LLMs, especially for end users.
Tags
Links
- Source: https://arxiv.org/abs/2603.15699v1
- Canonical: https://arxiv.org/abs/2603.15699v1
Trouble viewing inline? Open PDF directly →
Full Text
33,568 characters extracted from source content.
Expand or collapse full text
This Is Taking Too Long - Investigating Time as a Proxy for Energy Consumption of LLMs Lars Krupp Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany lars.krupp@dfki.de Daniel Geißler Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany daniel.geissler@dfki.de Francisco M. Calatrava-Nicolas Centre for Applied Autonomous Sensor Systems (AASS) ̈ Orebro University ̈ Orebro, Sweden francisco.calatrava-nicolas@oru.se Vishal Banwari Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany vishal.banwari@dfki.de Paul Lukowicz Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany paul.lukowicz@dfki.de Jakob Karolus Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany jakob.karolus@dfki.de Abstract—The energy consumption of Large Language Models (LLMs) is raising growing concerns due to their adverse effects on environmental stability and resource use. Yet, these energy costs remain largely opaque to users, especially when models are accessed through an API — a black box in which all information depends on what providers choose to disclose. In this work, we investigate inference time measurements as a proxy to approximate the associated energy costs of API-based LLMs. We ground our approach by comparing our estimations with actual energy measurements from locally hosted equivalents. Our results show that time measurements allow us to infer GPU models for API-based LLMs, grounding our energy cost estimations. Our work aims to create means for understanding the associated energy costs of API-based LLMs, especially for end users. Index Terms—Large Language models, energy consumption, energy estimation, sustainability I. INTRODUCTION Recent progress in the field of deep learning and especially Large Language Models (LLMs) has transformed what arti- ficial intelligence is capable of in a wide range of domains, ranging from natural language understanding to other fields such as healthcare and robotics [1], [2]. Society experiences significant changes that move LLMs well beyond research settings into everyday use [3]. Large, nationally representative surveys show sharp growth in consumer adoption of generative AI tools for routine tasks, information seeking, and work/study support [4]–[6]. In recent years, the number of proposed models has increased significantly [7], further accelerating the adoption of generative AI across society. This technological development has been fueled by in- creasingly large-scale architecture, big corpora of data, and high-performance computing (HPC) infrastructure. As a result, many companies and research institutions have entered a competitive race to develop more formidable models, aiming to push the boundaries of the current state-of-the-art. However, this growth comes at significant computational costs, arising from both the training and inference stages of these models. The resulting energy demands, both from GPU-level power draw and system-level electricity consumption [8], [9], have reached new levels. Some companies, such as Google, are even constructing dedicated nuclear reactors to sustain their AI infrastructure [10]. Beyond technical and economic challenges, the environmental implications of this escalating energy con- sumption are becoming a great concern. Regrettably, company policies regarding the disclosure of energy consumption are often opaque, hiding true costs behind inconspicuous web user interfaces, cf. ChatGPT [11]. Closed- source models and sparse details on data center infrastructure make robust energy estimation impossible [7], [8]. No direct access to weights and the model’s inference configuration further prevent researchers from conducting transparent mea- surements of the LLM’s energy footprint and, ultimately, assessing the efficiency of API-based LLM interactions. Yet, there remains one metric that can be directly measured: task completion time. I.e., the time it takes for a model to process a single token. Consequently, computation time can prove valuable when estimating gross energy consumption of API-based LLMs. However, factors such as employed GPUs, model size and architecture, load factor, and many others still play a pivotal role and can drastically change computation times. Grounding results via locally deployed models with predefined options is crucial to consolidating time-based estimations. In this work, we investigate the potential of inference completion time as a robust proxy for estimating energy consumption in large language models (LLMs). To this end, we present a benchmark designed to analyze the relationship between inference time and energy usage under controlled experimental conditions for both locally hosted and API-based models. We focus on Mistral models [12], as they are available in both open-weight and API-accessible versions. The goal of this work is to evaluate whether inference time can serve as an effective proxy for energy estimation in API-based Mistral arXiv:2603.15699v1 [cs.PF] 16 Mar 2026 models and consolidate our time-based estimates via locally measured computation times under representative conditions. Our code and evaluation results are available on GitHub 1 . While our work is not the first to propose API-based energy estimation, cf. [13], our approach grounds API-based energy estimation through local measurements using the same model run on representative computing infrastructure. Our results show that inference time measurements on API-based LLMs, grounded by equivalent measurements on local models, allow us to infer the GPU configurations used on external servers. Our work provides evidence of a direct correspondence be- tween API-based large language models and their local coun- terparts in terms of computation time and, ultimately, energy consumption. I. RELATED WORK Starting with the inception of the attention mechanism [14], large language models, such as ChatGPT, have been gain- ing traction, leading to fierce competition between different model families like OpenAI [11], Llama [15], deepseek [16], qwen [17], and mistral [18]. However, outside of competing on different benchmarks, there is ongoing discussion about the topic of open-source and proprietary models. While pro- prietary models often show superior performance on bench- marks [19], their inner workings frequently remain opaque, in contrast to open-source LLMs. Especially regarding the energy consumption of proprietary models, information is sparse and often unreliable, if available at all. The energy footprint of LLMs spans both training and inference phases and constitutes one of the most pressing sustainability challenges in AI. Recent large-scale systems such as LLaMA-65B are estimated to consume around 5× 10 5 kWh during training [9], [20]. Despite significant gains from modern datacenter efficiency and AI-optimized hardware, such runs can still emit tens to hundreds of tons of CO 2 when powered by fossil-based energy [20], [21]. Advances in model sparsity (e.g., mixture-of-experts), efficient accelerator architectures, and low-carbon datacenters can reduce energy intensity by one to three orders of magnitude [20], [22]. For inference, even though a single LLM query may con- sume only 0.3 to 1 Wh, the cumulative demand of millions of daily requests may scale to annual petawatt-hour levels by 2026 [23], [24]. Inefficiencies from idle GPU power draw, throughput limits, and non-proportional energy scaling further amplify emissions, whereas techniques such as quan- tization, batching, and dynamic scheduling can mitigate these effects [23], [25]. Accurately assessing the energy footprint of large language models remains challenging. Recent tools such as Carbon- Tracker [26] and CodeCarbon [27] aim to provide real-time monitoring of power consumption and CO 2 emissions by in- tegrating hardware-level energy data with regional grid carbon intensity. 1 https://github.com/DFKIEI/LLM-Energy Recent studies have begun to quantify the energy and environmental costs associated with both the training and evaluation stages of deep learning systems. On the training side, Geißler et al. [28] analyzed how hyperparameters can influence the energy demand of neural networks, showing that suboptimal hyperparameter choice can increase the en- ergy consumption even when achieving similar accuracy. As model scales have grown, these concerns have extended from conventional deep networks to large language models (LLMs). Fernandez et al. [29] presented a comprehensive study of the energy implications of LLM inference, examining how model architecture, decoding strategies, and software frameworks affect power use. Jegham et al. [13] further introduced time- based estimation methods to assess the energy consumption of API-served models, yet lacking grounding through actual measurements. Here, Krupp et al. [30] depict an approach for benchmarking the energy consumption of locally run web agents and expose the complexity of estimating the energy cost for web agents that use API-based LLMs. I. METHODOLOGY Building upon these works, we propose a methodology that leverages locally executed models to inform the energy estimation of API-based systems, using inference computation time as a common factor for energy consumption across both setups. Our core assumption is that when computational hardware operates at or near optimal utilization, its power draw remains approximately constant, allowing total energy consumption to be estimated as a function of execution time. However, since the exact hardware specifications for API- based models are undisclosed, we additionally benchmark models locally across different GPU architectures to establish a plausible mapping of energy estimates to GPU fleets. We then compare these results with the inference completion times measured from API-based models. Our benchmark consists of synthetic prompts (generated with llama3-70b) from four input–output configurations (cov- ering technical, creative, educational, and business-oriented tasks with injected topics) that differ in sequence length (short = 2,048 tokens, long = 8,192 tokens). An example educational prompt may look like this: ”Provide a thorough explanation of advanced mathematics”. We selected the LLMs based on their availability in both open-weight and API-based variants, as well as their model size. These criteria enabled us to assess the accuracy of our time-based proxy energy estimation approach more reliably by testing both models locally and through their API, while ensuring that the models remain lightweight enough to fit on a single GPU when tested locally. Specifically, we used Mistral-7B-Instruct-v0.3 (Local), equivalent to Open-Mistral- 7B (API), and Mistral-NeMo-Instruct-2407 (Local), equivalent to Open-Mistral-NeMo (API). A. Experiment Protocol To ensure reproducibility and consistency, all tests are conducted following a structured experimental protocol. This applies to both locally executed runs as well as runs initiated on API-based models. Each run processes a fixed sequence of 100 synthetic prompts from our benchmark. This determin- istic ordering eliminates sampling variance, ensuring that all models and hardware platforms receive identical workloads. Execution proceeds in structured passes, where each con- figuration is executed repeatedly to ensure statistical relia- bility. For every configuration, the full benchmark is run independently ten times, allowing mean and standard deviation values to be computed for all runtime and energy metrics. In particular, for API-based models, we schedule runs throughout the day to capture variances due to times of high load. For each locally executed run, all prompts are processed in parallel batches of eight to ensure deployment with sufficient GPU utilization. Idle cycles between queries are minimized to ensure consistent energy-per-token comparability. B. Local Hardware Architecture Selection To assess the architectural impact on runtime and en- ergy efficiency, local experiments are performed across six NVIDIA GPUs representing two generations (Ampere, Hop- per) and form factors (SXM, PCIe): A100-40GB, A100- 80GB, A100-PCI, H100, H100-PCI, and H200. These devices differ markedly in thermal design power (TDP) and optimal sustained load range as shown in Table I. The comparison focuses on the divergence between server- grade SXM modules and PCIe variants (A100-PCI, H100- PCI). Server versions benefit from higher TDP ceilings, ex- ternal cooling, and high-bandwidth NVLink interconnects, whereas PCIe versions have less power and much larger auxiliary losses, such as power conversions and cooling, which are addressed on-chip. Public sources do not disclose the exact GPU fleet serving Mistral APIs. However, Mistral reports training on H100 GPUs [31], and MLPerf v4.1 shows H100/H200-class acceler- ators dominate datacenter LLM inference [32]. Since Mistral models are offered via major clouds [33], H100/H200-class GPUs are most likely used for API-hosted deployments. C. Local LLM Initialization The evaluated models, Mistral-7B-v0.3 and Mistral-NeMo- Instruct-2407 (12B), are executed in FP16 precision with TF32 acceleration on Ampere and Hopper GPUs, using device- map=”auto” for balanced memory allocation. KV-caching and left-padding are applied to optimize batch consistency and reduce redundant computation. To ensure comparability, temperature is fixed at 0.7 across all runs, for both local and API-based models, mimicking real use cases. Random seeds (42) are synchronized across Python, NumPy, PyTorch, and CUDA. Prompts are passed directly to the model without system instructions, ensuring that runtime and energy variations result solely from model architecture, precision, and decoding configuration. D. Local Energy Tracking Energy consumption is recorded with CarbonTracker [26], which samples GPU power via NVIDIA System Management Interface (SMI) and integrates the full board-level energy. In contrast, TDP reflects only the chip’s power envelope, serving as a theoretical lower bound. The difference between the two indicates the additional power drawn by auxiliary components. This gap is most evident for PCIe GPUs, where less efficient power delivery and higher conversion losses cause CarbonTracker values to exceed TDP estimates, while server- grade GPUs show closer alignment. Comparing both metrics thus reveals how chip-level specifications underestimate real- world energy use, especially in PCIe configurations. E. API-Based Energy Estimation As outlined in our experiment protocol, we performed N = 10 independent executions on both a local open-weights deployment and an API-based deployment. For each local run i ∈ 1, 2, . . . , N, we tracked energy consumption during code execution. The resulting energy consumption is denoted as E loc i , and the wall-clock duration of each run as T loc i . Consequently, average power draw for a local run can be calculated as follows: P loc i = E loc i T loc i , averaging over all runs yields P loc . Based on this local power calculation, we can predict the energy consumption of the equivalent API-based model, if run on a comparable computing setup: b E api = P loc · ̄ T api ,(1) where ̄ T api denotes the mean API execution time over all API runs. To allow comparability between runs and models 2 , we normalized our results by token count. Consequently, for each run, we normalized both energy E i and time T i by the token count of this run N (i) tokens . This yields mean energy per token ̄ E token and mean inference time per token ̄ T token . IV. RESULTS This section presents the evaluation of energy consumption for large language models (LLMs) from the Mistral family, ex- ecuted both locally and through API-based services. We begin by analyzing the computational time required to complete the benchmark under both scenarios (local vs. API), as it forms the basis for subsequent energy estimation. Next, we present the energy consumption analysis, reporting both the total energy and energy per token (see Section I-E). Finally, we validate the energy predictions for the local setup by comparing them with estimates derived from the Thermal Design Power (TDP) parameter of each GPU. A. Benchmarking Computation Time Table I shows the runtime on our proposed benchmark over ten runs for both scenarios (local and API). The com- parison between local and API-based executions highlights a substantial difference in computational efficiency. Both free and paid API run the benchmark faster than any local GPU configuration. Compared to the best-performing local setup, 2 Output token count was not consistent across models. TABLE I TDP AND OPTIMAL WORK LOAD RANGE GPUVRAM Thermal Design Power (TDP) [W] Optimal TDP Load Range [W] A100-40GB40 GB400340–380 A100-80GB80 GB400340–380 A100-PCI40 GB250213–238 H10080 GB700595–665 H100-PCI94 GB400340–380 H200140 GB700595–665 TABLE I TABLE DEPICTING MEAN AND STANDARD DEVIATION OF TIME (S) TO COMPUTE THE BENCHMARK. GPUS ARE ORDERED CHRONOLOGICALLY BY RELEASE DATE. ModelTypeGPUmean± sd Mistral-7B Local A100-40GB2447.± 8.85 A100-80GB2308.± 8.36 A100-PCI2525.± 14.7 H1001662.± 22.5 H100-PCI1820.± 9.24 H2001559.± 4.72 Free-API-718.± 43.8 Paid-API-727.± 28.8 Mistral-NeMo Local A100-40GB3865.± 8.13 A100-80GB3612.± 14.0 A100-PCI4003.± 16.4 H1002140.± 23.1 H100-PCI2353.± 11.9 H2002018.± 8.17 Free-API-1260.± 161. Paid-API-1159.± 31.2 the API execution achieves a reduction in computation time of approximately 54% for Mistral-7B and approximately 43% for Mistral-NeMo, indicating differences in the setup, such as system prompts or parallelization. The marginal difference between the free and paid APIs (on the order of a few seconds, within one standard deviation) suggests that both rely on similar backend resources, with variations likely due to server load or queueing effects. In contrast, local executions display longer but more predictable runtimes (smaller standard deviation), where performance depends on the underlying hardware. Within the local setup, a clear hierarchy emerges: newer GPUs (H100 and H200) consistently complete the benchmark faster than the older Ampere series models. In addition, we offer a fine-grained comparison using the average time per token. As seen in Figure 1, the ̄ T token of the different configurations are clustered by GPU generation for local runs. For both LLMs, we observe two local clusters (see Figure 1), the A-Cluster and the H-Cluster. The API-based ̄ T token for Mistral-NeMo are close to the H-cluster, however, this is not the case for Mistral-7B, where runtimes are slightly higher than for the H-Cluster GPUs. Section IV-C puts these results into perspective regarding energy consumption. Cluster H Cluster H Cluster A Cluster A 0.006 0.008 0.010 0.012 Mistral-7B Mistral-NeMo LLM Time/Token (s) GPU A100-40GB A100-80GB A100-PCI H100 H100-PCI H200 Free API Paid API Fig. 1. Boxplot showing the computation time per token ̄ T token in seconds for running the same benchmark on both models across different local GPUs and API executions. Cluster A includes A100 GPUs, while Cluster H groups H100 and H200 GPUs. B. Translating Time to Energy In this section, we present the results related to the use of time in energy estimation for the API models (c.f. Sec- tion I-E). In addition, we compare this value with the energy consumption based on the Thermal Design Power (TDP) for each GPU. We use an average sustained power equal to 0.9 × TDP as a conservative estimate where cooling cost and energy losses through heat radiation are not included. Empirically grounded estimates of actual energy use during high-throughput inference are reported between 85 % and 95 %, cf. [8], [31], [34]. Based on these findings, Table I reports the optimal TDP load values adopted in our analysis. Applying this information to the calculated runtimes, we can calculate a lower bound for the energy consumption. As shown in Table I, the differences in energy consumption between TDP and local measurements can vary widely, depending on the GPU architecture (SXM or PCIe), as outlined in Section I-B. C. Estimating the Energy Consumption for API-Based LLMs By contrasting the measured time per Token ̄ T token of our GPU setups with the results from the API-based models, we can pinpoint likely GPU setups for the given LLMs. We do note that, while ̄ T token might be comparable, we cannot make any evidence-based statements on the actual GPU setups behind the API-based models. As such, this comparison is based on reports about compute clusters and our confirming findings through inference time measurements regarding those TABLE I AVERAGE SUSTAINED POWER AND ENERGY USE ESTIMATED VIA TDP VS. LOCALLY MEASURED (CARBON TRACKER). EACH SUPER-LEVEL GROUP PER TOKEN AND TOTAL ENERGY. ModelGPUTotal tokens TDP (calculated)Carbon Tracker (measured)Difference (% of local) mean [%] Per token (mWh)Total (Wh)Per token (mWh)Total (Wh) Mistral-7B A100-40GB2862190.86± 0.003245.± 0.8851.06± 0.023303.± 6.5419 A100-80GB2862190.81± 0.003231.± 0.8361.10± 0.024314.± 6.7526 A100-PCI2862190.55± 0.003158.± 0.9210.949± 0.007272.± 1.9542 H1003276740.88± 0.012291.± 3.941.07± 0.016350.± 5.3716 H100-PCI3276740.56± 0.003182.± 0.9240.904± 0.009296.± 2.8939 H2003276740.83± 0.003273.± 0.8271.04± 0.011340.± 3.5420 Mistral-NeMo A100-40GB3293451.15± 0.002387.± 0.8131.47± 0.027485.± 8.7520 A100-80GB3293451.10± 0.004361.± 1.401.54± 0.015507.± 4.8329 A100-PCI3293450.75± 0.003250.± 1.021.32± 0.007435.± 2.2343 H1003328601.13± 0.012375.± 4.041.37± 0.019456.± 6.2818 H100-PCI3328600.71± 0.004235.± 1.191.17± 0.013388.± 4.2539 H2003328601.06± 0.004353.± 1.431.33± 0.021444.± 7.0020 TABLE IV COMPARISON BETWEEN MEASURED AND CALCULATED TOTAL ENERGY COSTS FOR THE API VERSION (FREE AND PAID) OF MISTRAL-7B AND MISTRAL-NEMO. ModelAPI cost Number of Tokens H100-PCI Total energy (Wh) Mistral-7BfreeM111272.± 6351.100.60± 6.00 Mistral-7BfreeC111272.± 6351.62.31± 3.56 Mistral-7BpaidM110882.± 5922.100.23± 5.60 Mistral-7BpaidC110882.± 5922.62.09± 3.32 Mistral-NeMofreeM173743± 4967.203.28± 5.81 Mistral-NeMofreeC173743± 4967.123.36± 3.53 Mistral-NeMopaidM174496.± 5815.204.16± 6.80 Mistral-NeMopaidC174496.± 5815.123.89± 4.13 M = measured; C = calculated. GPU setups. Likewise, Figure 1 reveals that the H100-PCI is a good representative setup for both Mistral models. Finally, we calculated the estimate of mean energy con- sumption for the API-based models b E api as seen in Equa- tion (1) using the average number of tokens computed by the paid and free API and energy per token results for both TDP calculation and local measurements. For Mistral-7B and for Mistral-NeMo, the results are shown in Table IV showcasing differing energy consumptions. Note that TDP calculations are likely underestimated, cf. Section I-D. We will discuss additional factors for these results in the next section. V. DISCUSSION Our results confirmed the necessity for a strict experiment protocol. Deviations in deployment-specific factors, such as variations in system prompts and minor model version differ- ences between API and open-source releases, may easily lead to skewed time measurements as depicted in the comparison between Table I and Table IV. Here, the API and local executions produce different numbers of tokens, making raw total energy values misleading across settings. Consequently, normalizing energy values given the amount of processed tokens is essential for accurate results. Figure 1 shows these normalized values for API-based and local models. In particular, it highlights a close alignment between the API-based deployment and Cluster H (local deployment equipped with NVIDIA Hopper series GPUs) for Mistral-NeMo. However, we note that this does not imply that both deployments necessarily consume the same energy per token. Server-side optimization techniques, such as tensor or pipeline parallelism, may be applied to reduce latency at the expense of higher power consumption. For instance, hosting Mistral-7B on multiple GPUs shortens the runtime but proportionally increases total energy consumption per second. Nonetheless, we know that most data centers are using a mixture of Nvidia Ampere and Nvidia Hopper series GPUs. Further, our local time per token results are within one stan- dard deviation of the API-based results. As we select models that fit into a single GPU, it is reasonable to conclude that the two measurements directly correspond, ultimately confirming our hypothesis that inference time per token is a reasonable proxy for energy estimation in API-based LLMs. A. Limitations This work relies on the assumption that the time we measure for the API-based models is dominated by GPU computation time, with other contributions being negligible. This assump- tion allowed us to reduce the complexity of our evaluation, such as only one GPU and a single batch size. In practice, however, real-world deployments may introduce additional sources of latency and energy consumption that are not directly observable in our proposed experimental setup. These include multi-GPU execution, as well as tensor, pipeline, and expert parallelism strategies, which can change the runtime–energy trade-off (e.g., reducing latency at the cost of increased power draw). While our approach therefore provides a reasonable gross estimate of the energy consumption of LLMs accessed through an API, a more fine-grained characterization of these effects can improve accuracy. As future work, systematic ablation studies, including parallelism strategies, hardware configurations, and batch sizes, are necessary to quantify the impact of these factors on both runtime and energy use. VI. CONCLUSION In this work, we investigated the potential of inference time as a proxy for estimating the energy consumption of API- based language models. Our results show that time per token can be used to imply the type of GPU configurations used behind API-based models. Likewise, gross energy estimations for these particular models are possible, providing researchers with a means to judge the energy efficiency of API-based models. VII. ACKNOWLEDGMENTS This work is funded and supported by the EU’s Hori- zon Europe research and innovation program through the ”SustainML” project (101070408), the Carl Zeiss Foundation through the project ”Sustainable Embedded AI”, the German BMFTR through the ”Cross-Act” project (01IW25001) and the Wallenberg AI, Autonomous Systems and Software Program (WASP) under the Knut and Alice Wallenberg Foundation. REFERENCES [1] J. Wang, E. Shi, H. Hu, C. Ma, Y. Liu, X. Wang, Y. Yao, X. Liu, B. Ge, and S. Zhang, “Large language models for robotics: Opportunities, challenges, and perspectives,” Journal of Automation and Intelligence, vol. 4, no. 1, p. 52–64, 2025. [2] J. Haltaufderheide and R. Ranisch, “The ethics of chatgpt in medicine and healthcare: a systematic review on large language models (llms),” NPJ digital medicine, vol. 7, no. 1, p. 183, 2024. [3] J. Howarth, “Most visited websites in the world (august 2025),” Webpage, 2025. [Online]. Available: https://explodingtopics.com/blog/ most-visited-websites [4] F. Simon, R. K. Nielsen, and R. Fletcher, “Generative ai and news report 2025: how people think about ai’s role in journalism and society,” 2025. [5] O. McClain, “34% of us adults have used chatgpt, about double the share in 2023,” Pew Research Center, available at: Link to the cited article, 2025. [6] Ofcom,“Onlinenation2024report,”Ofcom,London, UK,Tech.Rep.,Nov.2024,accessed:October23,2025. [Online].Available:https://w.ofcom.org.uk/siteassets/resources/ documents/research-and-data/online-research/online-nation/2024/ online-nation-2024-report.pdf?v=386238 [7] N. Maslej, L. Fattorini, R. Perrault, Y. Gil, V. Parli, N. Kariuki, E. Capstick, A. Reuel, E. Brynjolfsson, J. Etchemendy et al., “Artificial intelligence index report 2025,” arXiv preprint arXiv:2504.07139, 2025. [8] S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V. Gadepally, “From words to watts: Benchmarking the energy costs of large language model infer- ence,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2023, p. 1–9. [9] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy con- siderations for modern deep learning research,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 09, 2020, p. 13 693–13 696. [10] A. Hunt, “Google to fund development of three nuclear power sites,” 2025. [Online]. Available: https://w.world-nuclear-news.org/articles/ google-to-fund-elementl-to-prepare-three-nuclear-power-sites [11] O. et al., “Gpt-4 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2303.08774 [12] M. AI, “Google to fund development of three nuclear power sites,” Webpage, 2025. [Online]. Available: https://mistral.ai/ [13] N. Jegham, M. Abdelatti, L. Elmoubarki, and A. Hendawi, “How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,” arXiv preprint arXiv:2505.09598, 2025. [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [15] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv e-prints, p. arXiv–2407, 2024. [16] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024. [17] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [18] Mistral, “Mistral nemo,” 2024. [Online]. Available: https://mistral.ai/ news/mistral-nemo [19] C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Dey et al., “Livebench: A challenging, contamination-limited llm benchmark,” arXiv preprint arXiv:2406.19314, 2024. [20] D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,” arXiv preprint arXiv:2104.10350, 2021. [21] P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, and J. Pineau, “Towards the systematic reporting of the energy and carbon footprints of machine learning,” Journal of Machine Learning Research, vol. 21, no. 248, p. 1–43, 2020. [22] D. Zhao, N. C. Frey, J. McDonald, M. Hubbell, D. Bestor, M. Jones, A. Prout, V. Gadepally, and S. Samsi, “A green (er) world for ai,” in 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 2022, p. 742–750. [23] M. ̈ Ozcan, P. Wiesner, P. Weiß, and O. Kao, “Quantifying the energy consumption and carbon emissions of llm inference via simulations,” arXiv preprint arXiv:2507.11417, 2025. [24] R. Desislavov, F. Mart ́ ınez-Plumed, and J. Hern ́ andez-Orallo, “Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,” Sustainable Computing: Informatics and Sys- tems, vol. 38, p. 100857, 2023. [25] C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, C. Bai et al., “Sustainable ai: Environmental implications, challenges and opportunities,” Proceedings of machine learning and systems, vol. 4, p. 795–813, 2022. [26] L. F. W. Anthony, B. Kanding, and R. Selvan, “Carbontracker: Tracking and predicting the carbon footprint of training deep learning models,” arXiv preprint arXiv:2007.03051, 2020. [27] B. Courty, V. Schmidt, S. Luccioni, Goyal-Kamal, MarionCoutarel, B. Feld, J. Lecourt, LiamConnell, A. Saboni, Inimaz, supatomic, M. L ́ eval, L. Blanche, A. Cruveiller, ouminasara, F. Zhao, A. Joshi, A. Bogroff, H. de Lavoreille, N. Laskaris, E. Abati, D. Blank, Z. Wang, A. Catovic, M. Alencon, M. Stechly, C. Bauer, L. O. N. de Ara ́ ujo, JPW, and MinervaBooks, “mlco2/codecarbon: v2.4.1,” May 2024. [Online]. Available: https://doi.org/10.5281/zenodo.11171501 [28] D. Geißler, B. Zhou, M. Liu, S. Suh, and P. Lukowicz, “The power of training: How different neural network setups influence the energy demand,” in International Conference on Architecture of Computing Systems. Springer, 2024, p. 33–47. [29] J. Fernandez, C. Na, V. Tiwari, Y. Bisk, S. Luccioni, and E. Strubell, “Energy considerations of large language model inference and efficiency optimizations,” arXiv preprint arXiv:2504.17674, 2025. [30] L. Krupp, D. Geißler, V. Banwari, P. Lukowicz, and J. Karolus, “Pro- moting sustainable web agents: Benchmarking and estimating energy consumption through empirical and theoretical analysis,” arXiv preprint arXiv:2511.04481, 2025. [31] NVIDIA, “Mistral ai and nvidia unveil mistral nemo 12b,” https://blogs. nvidia.com/blog/mistral-nvidia-ai-model/, 2024, accessed: 2025-10-29. [32] —,“Nvidiablackwellsetsnewstandardforgenai inmlperfinferencev4.1,”https://blogs.nvidia.com/blog/ mlperf-inference-benchmark-blackwell/,2024,accessed:2025-10- 29. [33] AmazonWebServices,“Mistralaimodelsnowavailable onamazonbedrock,”https://aws.amazon.com/blogs/aws/ mistral-ai-models-now-available-on-amazon-bedrock/, 2024, accessed: 2025-10-29. [34] NVIDIA,“Nvidiaa100tensorcoregpuarchitecture,” NVIDIACorporation,Tech.Rep.,2020.[Online].Avail- able:https://images.nvidia.com/aem-dam/en-z/Solutions/data-center/ nvidia-ampere-architecture-whitepaper.pdf