Paper deep dive
Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware
Zeyu Yang, Wes Armour
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/14/2026, 9:52:59 AM
Summary
This paper measures the inference energy efficiency of 86 open-weight LLM checkpoints across seven benchmarks and six hardware platforms. It finds that most model configurations are not on the score-energy Pareto frontier, often using 2.6-4.8x more energy than necessary for their capability. Enabling reasoning modes increases energy by ~3.7x on average. FP8 quantization reduces energy by ~25% with minimal score impact, while hardware changes rescale energy but preserve relative rankings. The study argues for reporting benchmark scores alongside measured energy consumption.
Entities (10)
Relation Signals (6)
86 open-weight model checkpoints → evaluatedon → seven benchmarks
confidence 95% · We directly measure this trade-off for 86 open-weight model checkpoints, evaluated on seven benchmarks
86 open-weight model checkpoints → evaluatedon → six hardware platforms
confidence 95% · evaluated on seven benchmarks and six hardware platforms
Reasoning Mode → increasesenergyby → 3.7x
confidence 90% · enabling thinking increases energy by about 3.7× on average
FP8 → reducesenergyby → 25%
confidence 90% · FP8 reduces energy by roughly a quarter with little change in score
Most configurations → usesmoreenergythan → Pareto frontier
confidence 90% · Most configurations lie away from the score–energy Pareto frontier: they use a median of 2.6–4.8× more energy
Hardware Change → rescalesenergy → energy
confidence 85% · changing hardware largely rescales energy without changing the relative ranking of models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability. We directly measure this trade-off for 86 open-weight model checkpoints, evaluated on seven benchmarks and six hardware platforms ranging from a consumer GPU to an eight-GPU server. We find large differences in energy efficiency even among models with similar capability. Most configurations lie away from the score--energy Pareto frontier: they use a median of 2.6--4.8$\times$ more energy than another configuration that scores at least as well, with differences reaching 27$\times$. Reasoning can improve capability, but enabling thinking increases energy by about 3.7$\times$ on average, and the benefit depends strongly on the model family and output budget. Generated-token count is also an imperfect proxy for energy, misordering 18--24\% of model pairs. In contrast, FP8 reduces energy by roughly a quarter with little change in score, while changing hardware largely rescales energy without changing the relative ranking of models. These results show that capability alone gives an incomplete picture of efficiency. We argue that benchmark scores should be reported together with measured energy consumption under a specified serving regime and output budget.
Tags
Links
- Source: https://arxiv.org/abs/2610.14389v1
- Canonical: https://arxiv.org/abs/2610.14389v1
Trouble viewing inline? Open PDF directly →
Full Text
85,608 characters extracted from source content.
Expand or collapse full text
Preprint MEASURING THE INFERENCE ENERGY EFFICIENCY OF OPEN-WEIGHT LLMS ACROSS BENCHMARKS, REASONING MODES, AND HARDWARE Zeyu Yang Department of Engineering Science University of Oxford zeyu.yang@eng.ox.ac.uk Wes Armour Department of Engineering Science University of Oxford wes.armour@oerc.ox.ac.uk ABSTRACT Benchmark scores tell us what an LLM can do, but not how much energy is re- quired to deliver that capability. We directly measure this trade-off for 86 open- weight model checkpoints, evaluated on seven benchmarks and six hardware plat- forms ranging from a consumer GPU to an eight-GPU server. We find large differ- ences in energy efficiency even among models with similar capability. Most con- figurations lie away from the score–energy Pareto frontier: they use a median of 2.6–4.8× more energy than another configuration that scores at least as well, with differences reaching 27×. Reasoning can improve capability, but enabling think- ing increases energy by about 3.7× on average, and the benefit depends strongly on the model family and output budget. Generated-token count is also an im- perfect proxy for energy, misordering 18–24% of model pairs. In contrast, FP8 reduces energy by roughly a quarter with little change in score, while changing hardware largely rescales energy without changing the relative ranking of models. These results show that capability alone gives an incomplete picture of efficiency. We argue that benchmark scores should be reported together with measured en- ergy consumption under a specified serving regime and output budget. 1INTRODUCTION Two LLMs with similar benchmark scores can require very different amounts of energy to serve. Among the open-weight models we evaluate that score above 80% on AIME, the GPU energy re- quired per correct answer spans roughly 33×, from 1.5 to 50 Wh depending on the model and the hardware serving it (Fig. 12; Appendix B.2). Conventional leaderboards capture the difference in capability, but not the energy required to deliver it. This distinction is becoming increasingly important as inference accounts for a growing share of AI computation. Around three fifths of the energy Google reports for machine learning is attributed to inference rather than training (Patterson et al., 2022), while the output lengths of reasoning models have grown rapidly (Emberson et al., 2025). More broadly, global data-centre electricity consump- tion was approximately 415 TWh in 2024 and is projected to more than double by 2030, with AI as a major driver (IEA, 2025). Evaluating model capability without considering the energy required to deliver it therefore gives an incomplete picture of inference efficiency. Measuring this trade-off is not as simple as counting parameters, FLOPs, or generated tokens. These quantities are often used as proxies for inference cost, but do not uniquely determine physical en- ergy consumption (Henderson et al., 2020; Dehghani et al., 2022; Fernandez et al., 2025). Direct measurements are also sensitive to serving conditions: batching, sequence length, hardware, preci- sion, and the serving stack can all substantially change the energy required for the same workload (Saad-Falcon et al., 2025; Chung et al., 2026; Pizzini Cavagna et al., 2026). Meanwhile, capability leaderboards typically report benchmark scores without the corresponding measured energy con- sumption. The capability–energy trade-off across a broad set of current LLMs under a common high-utilisation serving regime therefore remains poorly characterised. In this work, we measure this trade-off directly. We evaluate 86 open-weight checkpoints, corre- sponding to 134 serving configurations, on seven scored benchmarks spanning instruction following, 1 arXiv:2610.14389v1 [cs.AI] 9 Oct 2026 Preprint knowledge, science, mathematics, coding, and tool use. Models are served with saturated contin- uous batching on six hardware platforms ranging from a consumer GPU to an eight-GPU server. Rather than assigning each model a fixed energy-per-token value, we meter GPU energy consump- tion using each benchmark’s own traffic, including the additional tokens generated by reasoning modes. We then compare benchmark score against measured energy consumption and analyse the resulting score–energy Pareto fronts. Contributions Our main contributions and findings are: • A large-scale measurement of capability against directly measured inference energy. In the main cross-model comparison, only 21 of 82 configurations lie on a Pareto front; off-front configurations use a median of 2.6–4.8× the energy of an alternative scoring at least as well, and up to 27× (Fig. 1). • A controlled characterisation of the energy cost of reasoning and longer outputs. Across 44 paired hybrid configurations, enabling thinking increases energy by about 3.7×, while the capability gain depends strongly on the model lineage (Fig. 3). Increasing the output budget can recover substantial additional capability, but at up to 6.1× the energy; late tokens are also 2.0–4.9× more energy-intensive (Fig. 4). • Evidence that commonly used inference-cost proxies can give a different picture from physical energy. Generated-token count misorders energy for 18–24% of model pairs, and score–token Pareto fronts retain only 10 of the 25 models selected by the corresponding score–energy fronts (Fig. 2). Parameter count is an even weaker predictor of energy. • A cross-platform analysis of deployment choices. Vendor-provided FP8 weights reduce GPU energy by roughly a quarter with no measurable change in aggregate score (Fig. 5), while the same configurations consume 0.5–2.2× as much energy across hardware plat- forms but preserve almost the same energy ranking (ρ = 0.99; Fig. 6). 2PRELIMINARIES: WHAT SERVING ENERGY DEPENDS ON Serving a language model has two main phases. Prefill processes the input sequence in parallel and is typically compute-intensive, while decode generates tokens autoregressively and is often memory- bandwidth-bound at small batch sizes (Pope et al., 2023; Patel et al., 2024b). Each decode step accesses the model parameters and accumulated KV state, making memory movement a major cost. Continuous batching improves utilisation by serving many active requests together and replacing completed requests as execution proceeds (Yu et al., 2022; Kwon et al., 2023). This allows model execution to be shared across more requests and increases throughput. Since GPU power typically grows much less than throughput as concurrency increases (Patel et al., 2024a), higher utilisation generally reduces energy per token until KV-cache capacity limits further batching (Kwon et al., 2023; Chung et al., 2026). Energy per token is therefore not a property of the model alone. It depends on the model, hardware, serving system, and workload. Batch size can change per-token energy by several-fold (Chung et al., 2026), while input and output lengths can produce much larger variation on the same model and GPU (Pizzini Cavagna et al., 2026). Long outputs are especially costly because they generate more tokens while increasing KV-cache usage and reducing achievable concurrency (Chung et al., 2026). In finite workloads, concurrency can also fall as shorter requests finish, so late tokens may be generated in smaller batches even when the GPU remains highly powered. A single energy- per-token value therefore cannot reliably represent the cost of a benchmark, and simple models based only on prompt and output token counts can incur substantial error under batched serving (Pizzini Cavagna et al., 2026; Dutt et al., 2025). Meaningful comparisons therefore require energy to be measured under a defined workload, serving regime, and hardware platform. This motivates our use of direct GPU level energy measurements under high-utilisation continuous batching, rather than assigning each model a fixed energy-per- token value. Our measurement boundary excludes host and data-centre overhead. For context, Google’s production measurements report total serving energy at roughly 1.7× active accelerator energy after accounting for host systems, idle capacity, and infrastructure overhead (Elsworth et al., 2025). Section 4 describes the workload construction and measurement procedure. 2 Preprint 3RELATED WORK Energy benchmarking Several benchmarks directly measure LLM inference energy. ML.ENERGY evaluates more than 40 models and workloads (Chung et al., 2025; 2026), while MLPerf Power reports system efficiency subject to workload-specific accuracy requirements (Tschand et al., 2024). The AI Energy Score measures energy per query at batch size 1 (Luccioni et al., 2025; 2024). These efforts establish the value of direct measurement, but do not jointly characterise capability and energy across a broad model set under a common high-utilisation serving regime, where batching, sequence shape, software, and hardware can substantially affect energy (Fernandez et al., 2025; Chung et al., 2026; Saad-Falcon et al., 2025). Serving efficiency and reasoning length Systems work reduces inference energy through fre- quency, parallelism, batching, and scheduling (You et al., 2023; Patel et al., 2024a; Stojkovic et al., 2025), while other work controls or shortens reasoning traces (Snell et al., 2024; Muennighoff et al., 2025; Aggarwal & Welleck, 2025; Chen et al., 2024). Response-length studies further show substan- tial variation across models and increasingly long reasoning traces (Poddar et al., 2025; Emberson et al., 2025). Our focus is complementary: we measure how these choices affect the energy required to deliver benchmark capability. Energy proxies and metrics Several works connect capability to estimated inference cost. Token Arena estimates joules per correct answer from hardware power assumptions (Gao et al., 2026); Oviedo et al. (2026) estimate per-query energy from published throughput and power data; and Ca- pability Frontier and OckBench use monetary cost or token counts (Fowler et al., 2026; Du et al., 2025). Analytical models similarly estimate energy from FLOPs or token counts (Faiz et al., 2024; Wilkins et al., 2024; Pizzini Cavagna et al., 2026), although accuracy can degrade as serving con- ditions vary (Pizzini Cavagna et al., 2026; Dutt et al., 2025; Fu et al., 2025). Prior work therefore cautions against universal efficiency proxies (Schwartz et al., 2020; Henderson et al., 2020; De- hghani et al., 2022). We instead retain capability and measured energy as separate objectives and analyse their Pareto frontier. Joint energy and capability measurement A smaller body of work directly measures energy while evaluating model quality. Intelligence per Watt evaluates more than 20 models across eight accelerators, primarily at batch size 1 (Saad-Falcon et al., 2025); as Appendix C.1 shows, this regime can have substantially higher energy per token than saturated serving. Jin et al. (2025) study one model family at fixed batch size, while Stuhlmann et al. (2025); Masthan Ali et al. (2026) consider smaller model sets under light-load conditions. Caravaca et al. (2025) combine published capability scores with measured energy for a synthetic workload. In vision, Yang & Armour (2025) jointly measured accuracy and energy for 1,200 architectures at saturating batch sizes. Together, these leave open the capability–energy frontier across current LLMs under high-utilisation serving. We address this with scored benchmark traffic, saturated continuous batching, and con- trolled comparisons of reasoning mode, output budget, precision, and hardware. 4METHODOLOGY Models We evaluate 86 open-weight checkpoints from eleven model families: Qwen, DeepSeek, Llama, Gemma, GLM, IBM Granite, Mistral, Phi, Nemotron, MiniMax, and gpt-oss. The check- points were released between July 2024 and August 2026, span 0.8B–400B parameters, and in- clude 43 dense and 43 mixture-of-experts models. Within each family, we include instruction-tuned checkpoints across the available size range, subject to compatibility with our pinned serving stack. Vendor-published FP8 and NVFP4 weight variants are included as separate checkpoint entries. Our unit of study is a serving configuration, rather than the model weights alone. For hybrid models, reasoning mode is controlled through the chat template and changes both capability and generation length, so each mode forms a separate configuration. Single-mode checkpoints contribute one con- figuration, giving 134 in total. Sampling follows the generation configuration shipped with each checkpoint or the settings recom- mended in its model card; where neither is available, we use temperature 0.6 and top-p 0.95. This preserves the intended serving behaviour of each checkpoint rather than standardising decoding across models. The full model list and sampling provenance are given in Appendix A.1. 3 Preprint Hardware Models are grouped by weight footprint into three memory tiers: 32, 96, and 512 GB: the weight budget available to the model while retaining headroom for the KV cache. Each tier is assigned only to platforms with sufficient memory (Table 1). We examine CPU weight offloading separately in Appendix C.2. The GB10 is capped at 96 of its 128 GB unified memory, leaving the remainder for the operating system and serving stack. For the main cross-model comparison, the 32, 96, and 512 GB tiers use the RTX 5090, RTX PRO 6000, and 8×H100 as their reference platforms, respectively. The 96 GiB model set is ad- ditionally served on three other platforms, spanning three GPU generations and three memory tech- nologies, enabling like-for-like comparison of the same serving configurations across hardware. Full system and software details are given in Appendix A.2. Table 1: Serving platforms. Multi-GPU platforms are NVLink-connected and use tensor parallelism, plus expert parallelism for MoE models on the 8-GPU node. PlatformClassArch.MemoryModel Tier GeForce RTX 5090desktopBlackwell32 GB GDDR732 2× RTX A6000workstationAmpere2×48 GB GDDR696 GH200serverHopper96 GB HBM396 GB10 (DGX Spark)desktopBlackwell128 GB unified LPDDR5X96 RTX PRO 6000 BlackwellworkstationBlackwell96 GB GDDR796 8× H100 SXMserverHopper8×80 GB HBM3512 Benchmarks The seven-benchmark suite is summarised in Table 2. All headline metrics use deterministic scoring without an LLM judge. MMLU-Pro and BFCL use stratified subsets sampled once with a fixed seed and then frozen as ID lists, so every configuration receives the same questions. We use multiple seeded attempts for the higher-variance GPQA and AIME benchmarks. AIME 2026 and LiveCodeBench provide recent evaluation items with lower exposure to older training corpora. Benchmarks define the problems and scoring rules, but not a common generation budget. We there- fore use benchmark-specific output caps held fixed across models, ranging from 4K to 32K tokens. A generation that reaches the cap is scored normally; if truncation removes the extractable answer, it is counted as incorrect. We do not resample truncated answers or assign partial credit. Instead, we report truncation rates, treating completion within the common budget as part of model performance. We separately examine the sensitivity of score and energy to larger output budgets in Fig. 4. Table 2: The benchmark suite. k = seeded attempts. cap = maximum new tokens. † per turn. BenchmarkAbilityItemskCapHeadline metric IFEval (Zhou et al., 2023)instr. following54118Kprompt-level strict acc. MMLU-Pro (1k) (Wang et al., 2024)knowledge1,000116Kexact match GPQA Diamond (Rein et al., 2023)science198232Kexact match MATH-500 (Lightman et al., 2024)math500116Kexact match AIME 25/26 (Zhang & Team, 2026)competition math60432Kexact match LiveCodeBench (Jain et al., 2025)coding175132Kpass@1 (executed) BFCL v3 (Patil et al., 2025)tool use2,07314K † macro avg. 17 categories Evaluation harness All benchmark harnesses communicate with the serving backend through a common OpenAI-compatible logging proxy. The proxy applies sampling parameters, output caps, and per-attempt seeds, and records prompt, cached, and generated token counts. Before evaluation, a smoke test checks the chat template, reasoning parser, answer extraction, and truncation behaviour. Deployment We serve all models with a pinned version of vLLM (Kwon et al., 2023) under sat- urated continuous batching. Continuous batching, chunked prefill, and prefix caching are enabled. We standardise the serving regime rather than a numerical concurrency: each platform retains the engine’s default admission limit, while client concurrency is set to 1.5× that limit to maintain a continuous backlog and leave batching decisions to the server scheduler. Prefix-cache state is reset between benchmark runs, while reuse within a run is retained as part of the workload. Energy measurement We use accelerator-level energy as our primary measurement boundary. On discrete GPUs, board power is sampled from NVIDIA’s onboard sensors and integrated over 4 Preprint time; multi-GPU energy is summed across devices (Yang et al., 2024). On the GH200 we use the GPU+HBM reading, while on the GB10 we use the GPU-rail energy counter. Package-level measurements for both superchips are recorded separately and reported in Appendix A.3. CPU and host memory on the discrete-GPU systems, power-supply losses, cooling, and other infrastructure overheads are outside the primary measurement boundary. Energy is measured from each benchmark’s own request traffic rather than from a synthetic work- load. We identify the active serving interval from server-side token counters and exclude harness startup and CPU-side scoring. The first 10% and final 15% of the active interval are removed to exclude ramp-up and drain-out, leaving the saturated steady-state region. Scoring and aggregation Each benchmark is scored using its standard metric (Table 2) after any reasoning trace is removed by the model-specific parser. Scores are averaged over the k seeded attempts. Generated-token counts include reasoning tokens, so the additional generation produced by reasoning modes remains part of their measured cost. We treat capability and energy as separate objectives rather than combining them into a score-per- energy ratio, which can favour inexpensive configurations with poor absolute capability. A configu- ration is Pareto efficient if no other configuration in the comparison achieves at least its score while using no more energy, with a strict improvement in at least one objective. For the cross-benchmark summary, capability is the arithmetic mean of the seven headline scores, while energy is the geometric mean of energy per attempt across the seven benchmark components. The geometric mean gives each benchmark equal weight on a multiplicative scale rather than allow- ing the highest-energy workloads to dominate. Appendix A.4 analyses the effect of this aggregation. 5THE CAPABILITY–ENERGY FRONTIER We first compare capability with energy across the three reference tiers. Fig. 1 places each serving configuration in the score–energy plane, where higher scores and lower energy are preferable. 10 1 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 30 40 50 60 70 Mean Score over the Test Suite (%) 32G Tier: RTX 5090, BF16 (n=28) 1B4B12B marker area = total parameters Gemma-4-E2B Gemma-4-E4B Nemotron-3-Nano-4B Phi-4-mini Gemma-4-E4B Qwen3.5-0.8B Qwen3.5-4B Nemotron-3-Nano-4B Qwen3.5-9B Ministral-3-3B DeepSeek-R1-Distill-Qwen-7B Gemma-4-12B Llama-3.1-8B Qwen3.5-9B Gemma-4-12B Granite-4.2-8b 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 50 55 60 65 70 75 80 85 Mean Score over the Test Suite (%) 96G Tier: RTX PRO 6000, BF16 (n=27) 8B16B32B marker area = total parameters Nemotron-3-Nano-30B-A3B Ministral-3-8B-Reasoning Gemma-4-26B-A4B GLM-4.7-Flash Ministral-3-14B Gemma-4-31B Gemma-4-26B-A4B Gemma-4-31B DeepSeek-R1-Distill-Qwen-32B Qwen3.6-27B Granite-4.2-30b 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 65 70 75 80 Mean Score over the Test Suite (%) 512G Tier: 8x H100, FP8 (n=27) 100B200B400B marker area = total parameters Llama-4-Maverick Mistral-Small-4 GLM-4.5-Air Mistral-Medium-3.5 Qwen3.8-Flash-Next Qwen3.5-122B-A10B Qwen3.5-397B-A17B GLM-4.5 Nemotron-3-Super Qwen3.5-397B-A17B Mistral-Medium-3.5 GLM-4.7 Qwen DeepSeek Llama Gemma GLM IBM Granite Mistral Phi Nemotron MiniMax non-thinking / instruct thinking dense: solid, all parameters active MoE: total translucent, active solid Pareto front Figure 1: Capability–energy frontiers across the three memory tiers. Each panel shows one tier on its reference platform (BF16 for 32/96 GiB; FP8 for 512 GiB). x: geometric-mean GPU energy per attempt; y: mean suite score. Marker encoding follows the legend; steps show the Pareto fronts. Fig. 10 gives the fully labelled version. Most configurations are dominated Across the three reference-tier comparisons, only 21 of 82 configurations lie on a Pareto frontier. For the remaining configurations, the cheapest alternative that scores at least as high uses a median of 2.6–4.8× less energy, depending on the tier; the largest difference is 27×. On the RTX PRO 6000, for example, Granite-4.2-30b with thinking scores 77.6 at 5 Preprint 1,307 Wh, while Gemma-4-26B-A4B without thinking scores 79.5 at just 49 Wh. Large differences in energy therefore remain even after capability is held approximately fixed. The highest scores come at a steep energy premium Thinking configurations achieve the highest score on all three frontiers, but much of the final gain in capability is expensive. In every tier, a non- thinking configuration comes within five points of the highest score while using only a quarter to a sixth of its energy (Qwen3.5-9B, Gemma-4-31B, and Qwen3.5-122B-A10B). Overall, non-thinking configurations account for 15 of the 21 frontier members, compared with 6 of the 42 thinking con- figurations. We examine the effect of reasoning mode and the output budget in Section 6. Energy correlates more strongly with generated-token count (Spearman ρ = 0.70–0.79) than with active parameter count (ρ = 0.28–0.52), but token count alone still does not explain which configu- rations are energy efficient. Fig. 2 separates these two components. 0.1M0.2M0.5M1M2M5M Generated Tokens (Suite Geo Mean) 0.05 0.1 0.2 0.5 1 2 Energy per Generated Token (J) 3 Wh10 Wh30 Wh 100 Wh 300 Wh 1000 Wh 3000 Wh RTX 5090 gemma-3-12b-it gemma-4-E2B-it [NT] granite-4.2-8b [T] Qwen3.5-2B [T] 0.2M0.5M1M2M5M Generated Tokens (Suite Geo Mean) 0.2 0.5 1 2 10 Wh30 Wh100 Wh 300 Wh 1000 Wh RTX PRO 6000 Nemotron-3-Nano-30B-A3B-FP8 [NT] gemma-4-31B-it [NT] granite-4.2-30b [T] Qwen3.6-35B-A3B-FP8 [T] 0.2M0.5M1M2M Generated Tokens (Suite Geo Mean) 1 2 30 Wh100 Wh300 Wh 1000 Wh 3000 Wh 8x H100 Llama-3.1-70B-Instruct Mistral-Small-4-119B-2603 [NT] Qwen3-235B-A22B-Thinking-2507 DeepSeek-V4-Flash [T] Qwen DeepSeek Llama Gemma GLM IBM Granite Mistral Phi Nemotron MiniMax gpt-oss non-thinking / instruct thinking dense: solid, all parameters active MoE: total translucent, active solid equal suite energy the machine's medians marker area = total parameters (key in Figure 1) Figure 2: Energy varies through both generation length and energy per token. Each panel shows all complete configurations on a reference platform, across weight formats. x: geometric-mean generated tokens per attempt; y: GPU energy per generated token, including prefill. Their product gives the suite energy in Fig. 1. Dashed diagonals show equal energy; dotted lines show platform medians. Marker encoding follows Fig. 1. Generated tokens are not a reliable proxy for energy Across the three reference platforms, generated-token counts vary by 6–14× between the 90th and 10th percentiles, while energy per generated token varies independently by 3–10×. The two quantities are nearly uncorrelated (Pearson r =−0.23 to +0.02), so generating more tokens does not imply that each token is more expensive. Token count nevertheless captures part of the variation in total energy, explaining 60–85% of its log- scale variance. Much of this association reflects the separation between thinking and non-thinking configurations: within either group, variation in token count and energy per token are of comparable importance. As a ranking proxy, however, generated tokens misorder energy for 18–24% of config- uration pairs. Two non-thinking configurations with nearly identical token counts—Gemma-3-12B and Gemma-4-E2B on the RTX 5090—differ by 23× in energy. This difference is large enough to change which models appear efficient. On the three tiers, the score–token frontier shares only 4 of 11, 3 of 6, and 3 of 8 members with the corresponding score– energy frontier. It excludes configurations that generate many tokens cheaply, such as Gemma-4- 26B-A4B and Qwen3.5-122B-A10B, while admitting Qwen3-235B-A22B-Instruct, which gener- ates relatively few tokens at roughly 3 J each. Token-based cost measures (Du et al., 2025) can therefore identify a substantially different set of efficient models from direct energy measurement. 6REASONING AND OUTPUT BUDGET The frontier analysis suggests that reasoning can reach higher capability, but often at substantially higher energy. We isolate this effect using hybrid checkpoints that can be served with and without reasoning while keeping the weights and hardware fixed. 6 Preprint RTX 5090 RTX PRO 6000 8x H100 all three −20 −10 0 10 20 30 40 n=14n=16n=14n=44 Score Change (% Points) RTX 5090 RTX PRO 6000 8x H100 all three 0.6 0.8 1.0 1.2 1.4 1.6 1.8 n=14n=16n=14n=44 Score Change (Relative) RTX 5090 RTX PRO 6000 8x H100 all three −2 0 2 4 6 8 10 12 14 n=14n=16n=14n=44 Energy Change (kWh) RTX 5090 RTX PRO 6000 8x H100 all three 1 2 5 10 20 n=14n=16n=14n=44 Energy Change (Relative) QwenNemotronGLMGemmaIBM GraniteMistralDeepSeek Figure 3: Effect of enabling reasoning with the same weights on the same hardware. Each point compares one hybrid checkpoint with thinking disabled and enabled on its tier’s reference platform (44 pairs), coloured by lineage; boxes show quartiles by platform and pooled. Left: change in suite score, shown in points and as a ratio. Right: change in GPU energy for one pass over the suite, shown in kWh and as a ratio (log scale). Reasoning is consistently expensive, but its capability gain is not Across the 44 paired config- urations, enabling thinking increases the suite score by a median of 4.5 points (mean 6.3; 95% CI 3.6–9.1), while multiplying the energy of a suite pass by 3.7× on average. Eight of the 44 pairs lose score despite consuming more energy. The score gain varies much more by model lineage than by hardware. We find no detectable plat- form effect on either the score or energy change (Kruskal–Wallis p = 0.60 and 0.88, respectively), whereas the difference between lineages is significant (p < 0.001). Nemotron hybrids gain a median of 20.6 points and none loses score; GLM hybrids gain 12.0 points in the median. Qwen behaves very differently: its median change is−0.7 points, with 8 of 14 pairs scoring lower when thinking is enabled. More generally, the gain from reasoning decreases as the corresponding non-thinking score rises (Spearman ρ =−0.69). These losses, however, are entangled with the output budget. Enabling thinking raises the average share of requests truncated at the benchmark cap from 3% to 17%. For the four Qwen configura- tions with the largest score losses, 25–62% of requests are truncated. Under the optimistic clean- completion bound of Fig. 11, these configurations would instead reach suite scores of 78–88. The measured benefit of reasoning is therefore conditional not only on the model lineage, but also on how much generation budget it is given. 10 1 10 2 10 3 Energy per Attempt (Wh, Equal Requests) 0 20 40 60 80 100 AIME 2026 - Score (%) 10 2 10 3 10 4 Energy per Attempt (Wh, Equal Requests) 0 20 40 60 80 100 LiveCodeBench - Score (%) marker area = output-token cap 8K32K128K 0-88-1616-3232-6464-128 Output Position (K Tokens) 500 1000 2000 Throughput (Tokens per Second) 0-88-1616-3232-6464-128 Output Position (K Tokens) 0.2 0.5 1 2 5 Energy per Generated Token (J) Qwen3.5-9B (RTX 5090)Nemotron-3.5-Lightning-30B-A3B (RTX 5090)DeepSeek-V4-Flash (8x H100)GLM-5.3-Flash (max) (8x H100) Figure 4: Increasing the output budget can recover capability at higher energy cost. Four thinking configurations are rerun from 8K to 128K output tokens; the main evaluation uses 32K. Left: score against GPU energy per attempt, with marker area proportional to the output cap; energy is normalised to the request count. Right: throughput and energy per generated token by output position. Dashed lines: RTX 5090; solid lines: 8×H100. The output budget can change both scores and model rankings Fig. 4 reruns four thinking configurations beyond the 32K cap used in the main evaluation. Increasing the cap beyond 32K improves AIME 2026 by 7–25 points and LiveCodeBench by 6–21 points, but requires 1.1–6.1× as much energy. The return on this additional energy varies sharply between models. For the final 7 Preprint budget doubling, Nemotron-3.5-Lightning requires 1 Wh for each additional AIME point, compared with 239 Wh for GLM-5.3-Flash; on LiveCodeBench the values are 5 and 1,869 Wh per point. More importantly, the budget can change the relative ordering of models. DeepSeek-V4-Flash, for example, ranks last among the four configurations on AIME 2026 at 32K but first at the largest output cap. The scores in the main frontier should therefore be interpreted as capability under a common output budget, rather than as estimates of each model’s unconstrained maximum capability. Late tokens become more expensive as the KV state grows Within the same runs, energy per generated token rises 2.0–4.9× from the first 8K window of output to the last, while throughput falls to 0.19–0.46×. Longer sequences can increase cost in two ways. Their growing inference state increases memory pressure and reduces achievable concurrency; in attention layers whose per-token work grows with context length, later tokens can also require more state access. In our serving logs, KV-cache usage remains close to capacity while the number of running sequences falls to 0.19–0.37×; GPU power, however, remains at 0.78–0.94× of its initial level. Late tokens therefore require more work per sequence while being generated at lower concurrency, making them substantially more energy-intensive than early tokens. 7DEPLOYMENT CHOICES The previous sections compare different models and reasoning behaviours. We next ask how much energy can be changed through deployment choices while holding the underlying model configura- tion fixed. We consider two such choices: weight precision and hardware platform. RTX 5090 RTX PRO 6000 8x H100 all three −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 n=5n=13n=10n=28 Score Change (% Points) RTX 5090 RTX PRO 6000 8x H100 all three 0.98 0.99 1.00 1.01 1.02 n=5n=13n=10n=28 Score Change (Relative) RTX 5090 RTX PRO 6000 8x H100 all three −8 −6 −4 −2 0 2 n=5n=13n=10n=28 Energy Change (kWh) RTX 5090 RTX PRO 6000 8x H100 all three 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 n=5n=13n=10n=28 Energy Change (Relative) QwenNemotronGLMMistralIBM Granite Figure 5: Effect of replacing BF16 weights with vendor-released FP8 weights on the same hardware. Each point compares the same base model and reasoning mode in vendor- released BF16 and FP8 variants on its tier’s reference platform (28 matched pairs), coloured by lineage; boxes show quartiles by platform and pooled. Left: change in suite score, in points and as a ratio. Right: change in GPU energy consumption, in kWh and as a ratio. FP8 reduces energy by about a quarter with little change in score Across the 28 matched pairs, moving from BF16 to FP8 changes the suite score by a median of−0.15 points (mean−0.16; 95% CI−0.37 to +0.05). The corresponding GPU energy falls to 0.73× that of BF16 (geometric mean; 95% CI 0.69–0.77), with 27 of the 28 pairs using less energy. The saving varies by platform: FP8 uses 0.69× the BF16 energy on the RTX PRO 6000, 0.79× on the 8×H100, and 0.84× on the RTX 5090. The absolute saving consequently grows with model scale, from less than 0.5 kWh per suite pass for 3–4B models on the RTX 5090 to 7.8 kWh for Qwen3-235B-A22B-Thinking on the 8×H100. Overall, the energy reduction is much larger and more consistent than the change in capability. This comparison also bounds the weight-format difference in Fig. 1, where the 32 and 96 GiB panels use BF16 and the 512 GiB panel uses FP8. Hardware rescales energy but barely changes the relative efficiency of models The same serv- ing configurations have almost identical energy rankings across the four platforms: relative to the RTX PRO 6000, Spearman correlation is ρ = 0.99 on every platform, while benchmark scores dif- fer by a median of only 0.4–0.7 points. The score–energy Pareto frontier also contains the same members on every platform, to within one model. Absolute energy nevertheless changes substan- tially. Depending on weight format, the GH200 uses 0.61–0.77× the accelerator energy of the RTX PRO 6000, the GB10 0.53–0.62×, and the two RTX A6000s 1.9–2.2×. The difference is mainly 8 Preprint runtime for the GH200, PRO 6000 and A6000 pair; the GB10 instead trades much lower power for substantially longer runtime. Purchase cost dominates accelerator electricity at matched throughput At the throughput of one RTX PRO 6000, the required accelerator hardware costs roughly $24–34k at our indicative secondary-market prices, compared with $0.5–1.7k per year in accelerator electricity at $0.20/kWh (Fig. 6). This is illustrative rather than a full TCO analysis: host systems, cooling, maintenance, and infrastructure are excluded. PRO 6000GH2002x A6000GB10 0.1 0.2 0.5 1 2 5 10 20 Energy per Suite Pass kWh PRO 6000GH2002x A6000GB10 50 100 200 500 1,000 Average GPU Power W PRO 6000GH2002x A6000GB10 0.2 0.5 1 2 5 10 20 50 100 Suite Runtime h PRO 6000GH2002x A6000GB10 100 200 500 1,000 2,000 5,000 Throughput tokens/s BF16 FP8 FP4 PRO 6000GH2002x A6000GB10 0 5 10 15 20 25 30 35 40 8.6 29.0 9.3 3.0 30.7 37.9 12.6 5.7 GPU Price thousand USD launch market PRO 6000GH2002x A6000GB10 0 5 10 15 20 25 30 35 30.7 23.7 24.5 33.7 Hardware for One PRO 6000's Throughput, at Market Price thousand USD PRO 6000GH2002x A6000GB10 0 250 500 750 1,000 1,250 1,500 1,750 2,000 916 585 1,710 509 Electricity for One PRO 6000's Throughput USD per year PRO 6000GH2002x A6000GB10 0 200 400 600 800 1,000 523 334 976 290 Average GPU Power at One PRO 6000's Throughput W Figure 6: Energy, performance, and cost across the four 96 GiB platforms. Top: suite GPU energy, accelerator power, runtime, and token throughput. Bottom: accelerator price, hardware cost at matched RTX PRO 6000 throughput, and corresponding annual acceler- ator electricity cost and power. Secondary-market prices use ten eBay listings; $1,000 is deducted from the GB10 price for its non-GPU resources. Electricity assumes $0.20/kWh and continuous operation. Host and infrastructure power are excluded. 8LIMITATIONS AND SCOPE Output budgets Scores are conditional on the benchmark-specific generation caps used in our evaluation. Verbose reasoning models can be penalised when answers are truncated: increasing the cap beyond 32K recovers substantial capability for some models, but costs up to 6.1× more energy and can change their ranking (Fig. 4). We also report a clean-completion upper bound (Fig. 11). Model coverage Our largest tier uses a single 8×H100 node and includes models up to 400B parameters. Larger open-weight models were beyond our compute budget, while closed API models fall outside our direct measurement setup. Measurement and serving Our primary measurements cover accelerator energy rather than com- plete system or data-centre energy; package-level measurements for the GB10 and GH200 are re- ported in Appendix A.3. All main experiments use saturated continuous batching, the energy values describe a high-utilisation serving regime and should not be assumed to hold under lighter loads. 9CONCLUSION We directly measured the inference energy of 136 serving configurations of open-weight LLMs across seven benchmarks and six hardware platforms under saturated serving. The capability–energy trade-off varies widely: configurations off the Pareto frontier use a median of 2.6–4.8× the energy of an alternative that scores at least as well, and up to 27×. Reasoning can improve capability, but at a substantial energy cost, and its benefit depends strongly on the model lineage and output budget; in every memory tier, a non-thinking configuration comes within five points of the best 9 Preprint score for only a quarter to a sixth of its energy. Generated-token and parameter counts do not reliably recover measured energy, while FP8 reduces energy by roughly a quarter with little change in score. Changing hardware substantially rescales absolute energy but largely preserves the relative efficiency of models. These results suggest that benchmark capability should be reported together with directly measured energy under a clearly stated serving regime and output budget. AI USE STATEMENT We used generative AI tools to assist with literature discovery, parts of code development, and language editing during the preparation of this work. All cited papers were checked against their original sources, AI-assisted code was reviewed and tested by the authors, and all AI-assisted writ- ing was manually revised and verified. The authors take full responsibility for the methodology, experiments, results, and final text. ETHICS STATEMENT This work evaluates open-weight language models on existing benchmarks and does not involve hu- man subjects or the collection of personal data. The experiments consume substantial computational resources and electricity; we therefore report the energy consumed by the evaluation campaign in Appendix D. We are not aware of other significant ethical risks specific to the methodology or results presented here. REPRODUCIBILITY STATEMENT We document the model roster, sampling configurations, benchmarks, hardware and software en- vironments, serving regime, energy-measurement procedure, and scoring methodology in the main paper and Appendix A. These details are intended to support independent reproduction of the exper- iments. We plan to release the evaluation codebase and result dataset publicly upon acceptance. REFERENCES Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. In Conference on Language Modeling (COLM), 2025. Francisco Caravaca, ́ Angel Cuevas, and Rub ́ en Cuevas. From prompts to power: Measuring the energy footprint of LLM inference, 2025. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do NOT think that much for 2+3=? on the overthinking of o1-like LLMs, 2024. Jae-Won Chung, Jeff J. Ma, Ruofan Wu, Jiachen Liu, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, and Mosharaf Chowdhury. The ML.ENERGY benchmark: Toward automated inference energy mea- surement and optimization. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track, 2025. Jae-Won Chung, Ruofan Wu, Jeff J. Ma, and Mosharaf Chowdhury. Where do the joules go? diag- nosing inference energy consumption, 2026. Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency mis- nomer. In International Conference on Learning Representations (ICLR), 2022. Zheng Du, Hao Kang, Song Han, Tushar Krishna, and Ligeng Zhu. OckBench: Measuring the efficiency of LLM reasoning, 2025. Anurag Dutt, Young Won Choi, Avirup Sil, Anshul Gandhi, Aruna Balasubramanian, and Niranjan Balasubramanian. Fine-grained energy prediction for parallelized LLM inference with PIE-P, 2025. Preprint. Cooper Elsworth, Keguo Huang, David Patterson, Ian Schneider, Robert Sedivy, Savannah Good- man, Ben Townsend, Parthasarathy Ranganathan, Jeff Dean, Amin Vahdat, Ben Gomes, and James Manyika. Measuring the environmental impact of delivering AI at Google scale, 2025. 10 Preprint Luke Emberson et al. LLM responses to benchmark questions are getting longer over time. Epoch AI data insight, https://epoch.ai/data-insights/output-length, 2025. Accessed 2026-09. Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. LLMCarbon: Modeling the end-to-end carbon footprint of large language models. In Interna- tional Conference on Learning Representations (ICLR), 2024. Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell. Energy considerations of large language model inference and efficiency optimizations. In Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 32556–32569, 2025. URL https://aclanthology.org/2025. acl-long.1563. Bradley Fowler, Ryan Smith, Daniel Thi Graviet, William Myers, Joshua Greaves, Narmeen Fa- timah Oozeer, Ant ́ ıa Garc ́ ıa, Philip Quirke, Amirali Abdullah, Fazl Barez, and Shriyash Kaustubh Upadhyay. The capability frontier: Benchmarks miss 82% of model performance. In 3rd Work- shop on Data Problems in Foundation Models (DATA-FM) at ICLR 2026, 2026. Zhenxiao Fu, Fan Chen, Shan Zhou, Haitong Li, and Lei Jiang. LLMCO2: Advancing accurate carbon footprint prediction for LLM inferences. ACM SIGENERGY Energy Informatics Review, 5(2):63–68, 2025. doi: 10.1145/3757892.3757901. Yuxuan Gao, Megan Wang, and Yi Ling Yu. Token arena: A continuous benchmark unifying energy and cognition in AI inference, 2026. Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. To- wards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248):1–43, 2020. IEA. Energy and ai. Technical report, International Energy Agency, Paris, 2025. URL https: //w.iea.org/reports/energy-and-ai. Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evalua- tion of large language models for code. In International Conference on Learning Representations, volume 2025, p. 58791–58831, 2025. Yunho Jin, Gu-Yeon Wei, and David Brooks. The energy cost of reasoning: Analyzing energy usage in LLMs with test-time compute, 2025. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, p. 611–626, 2023. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, p. 39578–39601, 2024. Alexandra Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of AI deployment? In Proceedings of the ACM Conference on Fairness, Account- ability, and Transparency (FAccT ’24), 2024. Sasha Luccioni, Boris Gamazaychikov, et al. AI energy score: Standardized energy efficiency ratings for AI models. https://huggingface.github.io/AIEnergyScore/, 2025. Accessed 2026-09. Sabiya Banu Masthan Ali, Syed Muhammad Danish, Zeeshan Sattar, and Arghavan Asad. Assessing the sustainability of LLM inference through energy–accuracy analysis. In Proceedings of the 17th ACM International Conference on Future and Sustainable Energy Systems (e-Energy ’26), p. 627–635, 2026. doi: 10.1145/3744255.3811741. 11 Preprint Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand ` es, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. Felipe Oviedo, Fiodar Kazhamiaka, Esha Choukse, Allen Kim, Amy Luers, Melanie Nakagawa, Ricardo Bianchini, and Juan M. Lavista Ferres. Energy use of AI inference, efficiency pathways, and test-time scaling. Joule, 10:102430, 2026. doi: 10.1016/j.joule.2026.102430. Pratyush Patel, Esha Choukse, Chaojie Zhang, ́ I ̃ nigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. Characterizing power management opportunities for LLMs in the cloud. In Proceedings of the 29th ACM International Conference on Architectural Support for Program- ming Languages and Operating Systems (ASPLOS), 2024a. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, ́ I ̃ nigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024b. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agen- tic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. David Patterson, Joseph Gonzalez, Urs H ̈ olzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. The carbon footprint of machine learn- ing training will plateau, then shrink. Computer, 55(7):18–28, 2022. Hiari Pizzini Cavagna, Andrea Proia, Giacomo Madella, Giovanni Battista Esposito, Francesco An- tici, Daniele Cesarini, Zeynep Kiziltan, and Andrea Bartolini. SweetSpot: An analytical model for predicting energy efficiency of LLM inference. In Proceedings of the 17th ACM/SPEC International Conference on Performance Engineering (ICPE ’26), p. 83–95, 2026.doi: 10.1145/3777884.3797011. Soham Poddar, Paramita Koley, Janardan Misra, Niloy Ganguly, and Saptarshi Ghosh. Brevity is the soul of sustainability: Characterizing llm response lengths. In Findings of the Association for Computational Linguistics: ACL 2025, p. 21848–21864, 2025. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Lev- skaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling trans- former inference. In Proceedings of Machine Learning and Systems (MLSys), 2023. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. arXiv preprint arXiv:2311.12022, 2023. Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, et al. Intelligence per watt: Measuring intelligence efficiency of local AI, 2025. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AI. Communications of the ACM, 63(12):54–63, 2020. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute opti- mally can be more effective than scaling model parameters, 2024. Jovan Stojkovic, Chaojie Zhang, ́ I ̃ nigo Goiri, Josep Torrellas, and Esha Choukse. DynamoLLM: Designing LLM inference clusters for performance and energy efficiency. In IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2025. Linus Stuhlmann, Mauricio Fadel Argerich, and Jonathan F ̈ urst. Bench360: Benchmarking local LLM inference from 360 degrees, 2025. Arya Tschand, Arun Tejusve Raghunath Rajan, Sachin Idgunji, Anirban Ghosh, Jeremy Holleman, Csaba Kiraly, et al. MLPerf power: Benchmarking the energy efficiency of machine learning systems from microwatts to megawatts for sustainable AI, 2024. Presented at HPCA 2025. 12 Preprint Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024. Grant Wilkins, Srinivasan Keshav, and Richard Mortier. Offline energy-optimal LLM serving: Workload-based energy models for LLM inference on heterogeneous systems, 2024. HotCar- bon 2024. Zeyu Yang and Wesley Armour. The hidden joules: Evaluating the energy consumption of vision backbones for progress towards more efficient model inference. In Proceedings of the 42nd In- ternational Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, 2025. Zeyu Yang, Karel Adamek, and Wesley Armour. Accurate and convenient energy measurements for gpus: A detailed study of nvidia gpu’s built-in power sensor. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, p. 1–17. IEEE, 2024. Jie You, Jae-Won Chung, and Mosharaf Chowdhury. Zeus: Understanding and optimizing GPU energy consumption of DNN training. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), p. 521–538, 2022. URL https: //w.usenix.org/conference/osdi22/presentation/yu. Yifan Zhang and Math-AI Team. American invitational mathematics examination (aime) 2026, 2026. URL https://huggingface.co/datasets/math-ai/aime26. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 13 Preprint AEXPERIMENTAL SETUP A.1MODELS Tables 3–5 list the checkpoints in each memory tier. A checkpoint is assigned to the smallest tier in which its stored weights fit while retaining headroom for the KV cache. Vendor-published FP8 and NVFP4 releases are treated as separate checkpoints, so lower-precision variants can appear in a smaller tier than their higher-precision counterparts. Table 3: Roster of the 32 GiB tier (31 checkpoints, 47 study rows), served on a single NVIDIA GeForce RTX 5090. Params: billions of parameters; MoE checkpoints report to- tal / active parameters (Gemma 4 E-series: total / effective). Weights: size of the safetensors weights loaded by the server. Precision: stored weight format; gpt-oss stores its MoE ex- perts in MXFP4. Modes: I = instruct, T = thinking, NT = non-thinking; hybrid checkpoints listed as T, NT contribute two study rows. L, M, H denote documented low, medium, and high reasoning-effort settings. Released: checkpoint release date. CheckpointParams Weights Precision ModesReleased (B)(GiB) Alibaba Qwen/ Qwen3-4B-Instruct-250747.5 BF16I2025-09-17 Qwen3-4B-Instruct-2507-FP844.8 FP8I2025-09-17 Qwen3-4B-Thinking-250747.5 BF16T2025-08-06 Qwen3-4B-Thinking-2507-FP844.8 FP8T2025-08-06 Qwen3.5-0.8B0.81.6 BF16NT2026-03-09 Qwen3.5-2B24.2 BF16T, NT2026-03-09 Qwen3.5-4B48.7 BF16T, NT2026-03-09 Qwen3.5-9B918.0 BF16T, NT2026-03-09 DeepSeek deepseek-ai/ DeepSeek-R1-Distill-Qwen-1.5B1.53.3 BF16T2025-01-20 DeepSeek-R1-Distill-Qwen-7B714.2 BF16T2025-01-20 DeepSeek-R1-Distill-Llama-8B815.0 BF16T2025-01-20 Google google/ gemma-3-4b-it48.0 BF16I2025-03-12 gemma-3-12b-it1222.7 BF16I2025-03-12 gemma-4-E2B-it5 / 29.5 BF16T, NT2026-04-02 gemma-4-E4B-it8 / 414.9 BF16T, NT2026-04-02 gemma-4-12B-it1222.3 BF16T, NT2026-04-02 IBM ibm-granite/ granite-4.2-3b36.8 BF16T, NT2026-08-07 granite-4.2-8b816.4 BF16T, NT2026-08-07 granite-4.2-30b-nvfp43016.4 NVFP4T, NT2026-08-13 Meta meta-llama/ Llama-3.1-8B-Instruct815.0 BF16I2024-07-23 Microsoft microsoft/ Phi-4-mini-instruct3.87.2 BF16I2025-04-30 Mistral AI mistralai/ Ministral-3-3B-Instruct-2512-BF1637.2 BF16I2025-12-02 Ministral-3-3B-Instruct-251234.3 FP8I2025-12-02 Ministral-3-3B-Reasoning-251237.2 BF16T2025-12-02 Ministral-3-8B-Instruct-251289.7 FP8I2025-12-02 NVIDIA nvidia/ Nemotron-3-Nano-4B-BF1647.4 BF16T, NT2026-03-16 Nemotron-3-Nano-4B-FP844.9 FP8T, NT2026-03-16 Nemotron-3-Nano-30B-A3B-NVFP430 / 318.0 NVFP4T, NT2025-12-15 Nemotron-3-Nano-Omni-30B-A3B-NVFP430 / 320.9 NVFP4T, NT2026-04-28 Nemotron-3.5-Lightning-30B-A3B-NVFP430 / 320.1 NVFP4T, NT2026-08-04 OpenAI openai/ gpt-oss-20b21 / 3.612.8 MXFP4L, M, H 2025-08-05 14 Preprint Hybrid checkpoints contribute one study row per documented reasoning mode, selected through the vendor-provided chat-template controls (for example, enable thinking). gpt-oss and GLM- 5.3-Flash similarly contribute one row per documented reasoning-effort level. A model-specific reasoning parser in the serving engine removes the reasoning trace before answer scoring. Sampling follows vendor-provided settings. For 79 configurations, parameters are read from the checkpoint’s generation config.json and applied by the proxy; for another 17, the serving engine applies that configuration directly. The 15 Mistral configurations use model-card settings. For the remaining 25 configurations, for which no settings are documented, we use temperature 0.6 and top-p 0.95. Table 4: Roster of the 96 GiB tier (29 checkpoints, 47 study rows), served on all four 96 GiB platforms: GB10, RTX PRO 6000 Blackwell, GH200, and 2×RTX A6000 (TP=2). Columns are defined in Table 3. CheckpointParams Weights Precision ModesReleased (B)(GiB) Alibaba Qwen/ Qwen3-30B-A3B-Instruct-250730 / 356.9 BF16I2025-09-17 Qwen3-30B-A3B-Instruct-2507-FP830 / 329.0 FP8I2025-09-17 Qwen3-30B-A3B-Thinking-250730 / 356.9 BF16T2025-08-17 Qwen3-30B-A3B-Thinking-2507-FP830 / 329.0 FP8T2025-07-30 Qwen3.6-27B2751.8 BF16T, NT2026-04-24 Qwen3.6-27B-FP82728.8 FP8T, NT2026-04-24 Qwen3.6-35B-A3B35 / 367.0 BF16T, NT2026-04-24 Qwen3.6-35B-A3B-FP835 / 334.9 FP8T, NT2026-04-24 Qwen3.8-27B2751.8 BF16T, NT2026-08-05 Qwen3.8-27B-FP82728.8 FP8T, NT2026-08-13 DeepSeek deepseek-ai/ DeepSeek-R1-Distill-Qwen-14B1427.5 BF16T2025-01-20 DeepSeek-R1-Distill-Qwen-32B3261.0 BF16T2025-01-20 Google google/ gemma-3-27b-it2751.1 BF16I2025-03-12 gemma-4-26B-A4B-it26 / 448.1 BF16T, NT2026-04-02 gemma-4-31B-it3158.2 BF16T, NT2026-04-02 IBM ibm-granite/ granite-4.2-30b3054.5 BF16T, NT2026-08-07 granite-4.2-30b-fp83028.0 FP8T, NT2026-08-13 Mistral AI mistralai/ Ministral-3-8B-Instruct-2512-BF16816.6 BF16I2025-12-02 Ministral-3-8B-Reasoning-2512816.6 BF16T2025-12-02 Ministral-3-14B-Instruct-2512-BF161426.0 BF16I2025-12-02 Ministral-3-14B-Instruct-25121414.7 FP8I2025-12-02 Ministral-3-14B-Reasoning-25121426.0 BF16T2025-12-02 Mistral-Small-4-119B-2603-NVFP4119 / 6.566.0 NVFP4T, NT2026-03-16 NVIDIA nvidia/ Nemotron-3-Nano-30B-A3B-BF1630 / 358.8 BF16T, NT2025-12-15 Nemotron-3-Nano-30B-A3B-FP830 / 330.4 FP8T, NT2025-12-15 Nemotron-3-Super-120B-A12B-NVFP4120 / 1274.8 NVFP4T, NT2026-03-11 Nemotron-3.5-Lightning-30B-A3B-BF1630 / 361.3 BF16T, NT2026-08-01 OpenAI openai/ gpt-oss-120b117 / 5.160.8 MXFP4L, M, H 2025-08-05 Z.ai zai-org/ GLM-4.7-Flash30 / 358.2 BF16T, NT2026-01-14 15 Preprint Table 5: Roster of the 512 GiB tier (26 checkpoints, 42 study rows), served on the 8×H100 node with TP=8 and expert parallelism enabled for MoE checkpoints. Columns are defined in Table 3. CheckpointParams Weights Precision ModesReleased (B)(GiB) Alibaba Qwen/ Qwen3-235B-A22B-Instruct-2507235 / 22437.9 BF16I2025-09-17 Qwen3-235B-A22B-Instruct-2507-FP8235 / 22220.2 FP8I2025-09-17 Qwen3-235B-A22B-Thinking-2507235 / 22437.9 BF16T2025-08-17 Qwen3-235B-A22B-Thinking-2507-FP8235 / 22220.2 FP8T2025-07-30 Qwen3.5-122B-A10B122 / 10233.0 BF16T, NT2026-03-09 Qwen3.5-122B-A10B-FP8122 / 10118.4 FP8T, NT2026-03-09 Qwen3.5-397B-A17B-FP8397 / 17378.3 FP8T, NT2026-03-09 Qwen3.8-Flash-Next180 / 6335.3 BF16T, NT2026-08-24 Qwen3.8-Flash-Next-FP8180 / 6172.8 FP8T, NT2026-08-24 DeepSeek deepseek-ai/ DeepSeek-R1-Distill-Llama-70B70131.4 BF16T2025-01-20 DeepSeek-V4-Flash284 / 13148.7 FP4/FP8 T, NT2026-04-24 Meta meta-llama/ Llama-3.1-70B-Instruct70131.4 BF16I2024-07-23 Llama-4-Scout-17B-16E-Instruct109 / 17202.4 BF16I2025-04-05 Llama-4-Maverick-17B-128E-Instruct 400 / 17388.1 FP8I2025-04-05 MiniMax MiniMaxAI/ MiniMax-M2230 / 10214.3 FP8T2025-10-27 MiniMax-M2.5230 / 10214.3 FP8T2026-02-12 MiniMax-M2.7230 / 10214.3 FP8T2026-03-18 Mistral AI mistralai/ Mistral-Medium-3.5-128B128248.9 BF16T, NT2026-04-29 Mistral-Small-4-119B-2603119 / 6.5225.2 BF16T, NT2026-03-16 NVIDIA nvidia/ Nemotron-3-Super-120B-A12B-BF16120 / 12230.2 BF16T, NT2026-03-11 Nemotron-3-Super-120B-A12B-FP8120 / 12119.5 FP8T, NT2026-03-11 Z.ai zai-org/ GLM-4.5-Air106 / 12205.8 BF16T, NT2025-07-28 GLM-4.5-Air-FP8106 / 12104.8 FP8T, NT2025-07-28 GLM-4.5-FP8355 / 32336.5 FP8T, NT2025-07-28 GLM-4.7-FP8355 / 32337.2 FP8T, NT2025-12-22 GLM-5.3-Flash320 / 18305.8 FP8L, H, M 2026-08-25 16 Preprint A.2HARDWARE Table 6 gives the hardware, software, measurement, and serving details of each platform. All ac- celerators run at their default power limits. Single-accelerator platforms serve each model on one device; the A6000 pair uses tensor parallelism over NVLink, while the 8×H100 node uses tensor parallelism across eight GPUs with expert parallelism enabled for MoE models. The GB10 serving engine is limited to 96 GiB of its 128 GB unified memory. The server admission limit follows the pinned serving engine’s default: 256 concurrent sequences on the RTX 5090, A6000 pair, and GB10, and up to 1,024 on the other three platforms, with the engine reducing the limit where required by the model. Table 6: Hardware, software, measurement, and serving details for the six platforms in Table 1. GeForce RTX 5090 2× RTX A6000 GH200GB10 (DGX Spark) RTX PRO 6000 Blackwell 8× H100 SXM System Host CPURyzen 9 7900X Xeon w7-3455 Grace10× Cortex-X925 + 10× Cortex-A725 Core i5-14600K 2× Xeon Platinum 8468 Cores/Threads12C/24T24C/48T72C/72T20C/20T6P8E/20T96C/192T Host RAM64 GB DDR5 256 GB DDR5 480 GB LPDDR5X 128 GB unified LPDDR5X 128 GB DDR5 1 TB DDR5 OSUbuntu 24.04 Ubuntu 24.04 AlmaLinux 9.4 Ubuntu 24.04 Ubuntu 22.04 AlmaLinux 9.4 kernel6.86.85.146.146.85.14 Driver580.126580.65580.65580.95595.84580.65 CUDA13.013.013.013.013.213.0 Power limit (W) 575 (default)300 (each)900140 (SoC)600700 (each) Meter Scopeboard2 boards, summed GPU+HBM (module recorded) GPU rail (package recorded) board8 boards, summed Serving Parallelismsingle GPUTP=2 (NVLink) single GPUsingle GPUsingle GPUTP=8 (NVSwitch); MoE: EP=8 Memory budget 0.950.950.9596.0 GiB0.950.95 AttentionFA2FA2FA3FA2FA2FA3 FP8 kernelnativeMarlin W8A16 nativenativenativenative NVFP4 kernelnativeMarlin W4A16 Marlin W4A16 nativenativeMarlin W4A16 17 Preprint A.3ENERGY MEASUREMENT Instrumentation On discrete GPUs, board power is read from the driver’s onboard sensor and integrated over time; multi-GPU energy is summed across devices. On the GB10, we read the SoC’s cumulative energy counters and use the GPU-rail counter as the primary measurement, while also recording the CPU and package counters. On the GH200, the primary GPU reading covers the GPU and HBM3; whole-module power for the Grace Hopper superchip is recorded alongside it. Integrated superchips Figure 7 compares these primary accelerator measurements with the wider package or module boundary. On the GB10, package energy is a median 1.69× the GPU-rail energy (5th–95th percentile 1.52–1.85×): 59% is attributed to the GPU rail, 10% to the CPU cores, and 31% to the remainder of the package, including the LPDDR5X memory. On the GH200, module energy is 1.125× the GPU+HBM reading (1.11–1.19×); the remaining 11% covers the Grace CPU, LPDDR5X, and NVLink-C2C. The additional package or module draw is approximately constant at 34 W on the GB10 and 77 W on the GH200, so these shares vary little across reasoning modes, weight formats, and benchmarks. The measurement boundary matters when comparing against discrete GPUs, whose board-power measurements include device memory. For the same configurations, the GB10 consumes 0.61× the RTX PRO 6000 energy at GPU scope but 1.03× at package scope; the corresponding GH200 ratios are 0.64× and 0.73×. 59.2 % 89 kWh 10.1 % 15 kWh 30.7 % 46 kWh GB10 (DGX Spark): 150 kWh at package scope for its 89 kWh of GPU energy in Fig. 16 average power 85 W = 50 W + 9 W + 26 W GPU rail (hwmon gpu counter) CPU cores, e + p (hwmon counters) rest of the SoC package: LPDDR5X, fabric, I/O (package minus GPU minus CPU) 0 %20 %40 %60 %80 %100 % Share of the Package Energy (%) all runs non-thinking thinking BF16 FP8 FP4 IFEval MATH-500 MMLU-Pro GPQA Diamond AIME 2025 AIME 2026 LiveCodeBench BFCL v3 85 W 83 W 86 W 82 W 83 W 89 W 84 W 84 W 88 W 83 W 81 W 81 W 81 W 95 W GB10 (DGX Spark): the split by reasoning class, weight format and benchmark 59.210.130.7 59.010.230.8 59.210.130.7 57.311.331.4 58.510.431.2 60.49.430.2 58.110.931.0 58.710.430.9 60.19.929.9 59.49.830.8 57.510.432.1 57.510.532.0 59.110.230.8 63.88.927.4 88.6 % 128 kWh 11.4 % 16 kWh GH200: 144 kWh at package scope for its 128 kWh of GPU energy in Fig. 16 average power 641 W = 564 W + 77 W Hopper GPU + HBM3 (power.draw) rest of the superchip: Grace CPU, LPDDR5X, NVLink-C2C (module power minus power.draw) 0 %20 %40 %60 %80 %100 % Share of the Module Energy (%) all runs non-thinking thinking BF16 FP8 FP4 IFEval MATH-500 MMLU-Pro GPQA Diamond AIME 2025 AIME 2026 LiveCodeBench BFCL v3 641 W 614 W 665 W 652 W 621 W 639 W 595 W 679 W 711 W 665 W 598 W 595 W 634 W 654 W GH200: the split by reasoning class, weight format and benchmark 88.611.4 88.311.7 88.711.3 88.511.5 88.611.4 88.811.2 88.611.4 89.310.7 89.310.7 88.911.1 87.712.3 87.712.3 88.611.4 89.610.4 Figure 7: Energy outside the primary accelerator measurement on the two superchips. Left: energy over all metered attempts, split by component. For the GB10, the primary measurement is the GPU rail; for the GH200, it is the GPU+HBM reading. Right: the same shares by reasoning mode, weight format, and benchmark, with 95% bootstrap intervals on the primary-counter share and mean power at right. Package/module measurements still exclude power-supply losses, fans, and storage. 18 Preprint A.4SUITE AGGREGATION The suite contains seven components, with AIME 2025 and AIME 2026 combined into one compo- nent by averaging their scores and summing their energies. Suite score is the arithmetic mean of the seven component scores, while suite energy is the geometric mean of their per-attempt energies. We use the geometric mean to prevent benchmarks with more or longer generations from domi- nating the aggregate. Under a simple sum, LiveCodeBench contributes a median 25–29% of the summed suite-pass energy and is the largest component for 93 of the 134 configurations with com- plete reference-platform results, whereas IFEval contributes only 3–5% (Fig. 8). LiveCodeBench also accounts for 27–36% of the cross-configuration variation under the sum, compared with 3–7% for IFEval. Under the geometric mean, each component contributes 9–20%, much closer to the equal share of one seventh (Fig. 9). The aggregation choice has little effect on model ordering: the arithmetic-sum and geometric-mean rankings have Spearman ρ = 0.985–0.995, and dropping any one component gives ρ≥ 0.986. We use the geometric mean because changing a benchmark’s number of items or attempts then rescales all configurations equally, leaving pairwise energy ratios and rankings unchanged. 0102030405060 Share of the Model's Suite Energy (%) IFEval MATH-500 BFCL v3 GPQA Diamond MMLU-Pro AIME 2025+26 LiveCodeBench RTX 5090 (24 non-thinking, 21 thinking) 0102030405060 Share of the Model's Suite Energy (%) RTX PRO 6000 (22 non-thinking, 25 thinking) 0102030405060 Share of the Model's Suite Energy (%) 8x H100 (19 non-thinking, 23 thinking) non-thinkingthinkingequal share (1/7): the weight of a component in the geometric mean Figure 8: Benchmark shares of summed suite-pass energy. For every complete configu- ration on each reference platform, boxes show the share of total suite-pass energy contributed by each benchmark, separated into non-thinking (orange) and thinking (blue) configurations. Points show individual configurations. The dashed line marks one seventh, the equal weight assigned to each component by the geometric mean. 0510152025303540 Part of the Differences Between Models (%) IFEval MATH-500 BFCL v3 GPQA Diamond MMLU-Pro AIME 2025+26 LiveCodeBench RTX 5090 (n=45) 0510152025303540 Part of the Differences Between Models (%) RTX PRO 6000 (n=47) 0510152025303540 Part of the Differences Between Models (%) 8x H100 (n=42) arithmetic sum of the raw energiesgeometric mean: the combined energy of the frontier figures (after the log)equal share (1/7) Figure 9: Each benchmark’s contribution to cross-configuration variation in aggregate energy. For each reference platform, bars show the contribution of each benchmark under the arithmetic sum of raw energies (light) and the geometric mean used in the paper (dark, on log energies). Contributions sum to 100%; error bars are 95% bootstrap intervals over configurations. The dashed line marks an equal one-seventh contribution. 19 Preprint BADDITIONAL RESULTS B.1THE FRONTIER IN DETAIL Fig. 10 reproduces the three panels of Fig. 1 at full width with every configuration labelled. Each panel uses the majority weight format of its tier: BF16 for the 32 and 96 GiB tiers, and FP8 for the 512 GiB tier. Robustness of the fronts We recompute the fronts over 2,000 resampled draws. In each draw, benchmark scores are resampled over their items and attempts, giving a typical suite-score standard error of about 0.8 points, while component energies are perturbed by 7%, the median variation between repeated attempts of the same benchmark. Eighteen of the 21 frontier members remain on their front in at least 85% of draws. The three less-stable members are all near-ties. Qwen3.5-4B without thinking is within 0.1 score points of Nemotron-3-Nano-4B with thinking and remains on the front in 77% of draws. On the 512 GiB tier, Mistral-Medium-3.5 and GLM-4.5-Air without thinking are within 2% of the energy of neighbouring frontier configurations and remain on the front in 65% and 49% of draws, respec- tively. No off-front configuration joins a frontier in more than 40% of draws. When any one suite component is removed, 16 of the 21 frontier members remain. The clean-completion bound Fig. 11 replaces each benchmark score with its score over requests that complete within the output cap, equivalently dividing the observed score by one minus the truncated-request fraction when truncated requests contribute zero. Energy is left unchanged. The bound is deliberately optimistic: it assumes that truncated answers would be correct at the same rate as completed answers, while charging no additional energy for allowing them to finish. It therefore estimates an upper bound on the capability recoverable from a larger output budget. The effect is largest where truncation is most severe. Across the three tiers, thinking configurations gain a median of 3.5, 3.9, and 5.9 points, compared with 2.2, 0.9, and 0.6 points for non-thinking configurations. Overall rankings remain similar to the scored suite (Spearman ρ = 0.79, 0.75, and 0.85), but thinking configurations move to the top of each frontier. On the RTX 5090, Qwen3.5 thinking configurations join the frontier at scores of 78–84; on the RTX PRO 6000, Qwen3.6-35B- A3B and two Nemotron thinking configurations join at 85–88; and on the 8×H100, four thinking configurations join at 86–90. Every non-thinking frontier member remains. 20 Preprint 10 1 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 30 40 50 60 70 80 Mean Score over the Text Suite (%) 32G Tier: RTX 5090, BF16 (n=28) 1B4B12B marker area = total parameters Gemma-4-E2B Gemma-4-E4B Gemma-4-E2B Nemotron-3-Nano-4B Phi-4-mini Gemma-3-4b Qwen3.5-2B Gemma-4-E4B Ministral-3-3B-Reasoning Qwen3-4B Qwen3.5-0.8B Qwen3.5-4B Nemotron-3-Nano-4B Qwen3.5-9B Ministral-3-3B DeepSeek-R1-Distill-Qwen-7B Gemma-4-12B Granite-4.2-3b Qwen3.5-2B Llama-3.1-8B Gemma-3-12b Qwen3.5-4B Granite-4.2-3b Granite-4.2-8b Qwen3-4B-Thinking Qwen3.5-9B Gemma-4-12B Granite-4.2-8b 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 45 50 55 60 65 70 75 80 85 Mean Score over the Text Suite (%) 96G Tier: RTX PRO 6000, BF16 (n=27) 8B16B32B marker area = total parameters Nemotron-3-Nano-30B-A3B Ministral-3-8B-Reasoning Gemma-4-26B-A4B Ministral-3-14B-Reasoning Qwen3.6-35B-A3B Qwen3-30B-A3B Nemotron-3.5-Lightning GLM-4.7-Flash Ministral-3-8B Gemma-3-27b Ministral-3-14B Gemma-4-31B Qwen3.8 Qwen3.6-27B Gemma-4-26B-A4B DeepSeek-R1-Distill-Qwen-14B Nemotron-3-Nano-30B-A3B Nemotron-3.5-Lightning Qwen3-30B-A3B-Thinking Qwen3.6-35B-A3B Granite-4.2-30b Gemma-4-31B GLM-4.7-Flash Qwen3.8 DeepSeek-R1-Distill-Qwen-32B Qwen3.6-27B Granite-4.2-30b 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 60 65 70 75 80 Mean Score over the Text Suite (%) 512G Tier: 8x H100, FP8 (n=27) 100B200B400B marker area = total parameters Llama-4-Maverick Mistral-Small-4 GLM-4.5-Air Mistral-Medium-3.5 Qwen3.8-Flash-Next Qwen3.5-122B-A10B Qwen3-235B-A22B Nemotron-3-Super GLM-5.3-Flash (low) Qwen3.5-397B-A17B GLM-4.5 GLM-4.7 Qwen3.8-Flash-Next Nemotron-3-Super Mistral-Small-4 Qwen3.5-122B-A10B MiniMax-M2.5 GLM-4.5-Air MiniMax-M2.7 MiniMax-M2 GLM-5.3-Flash (high) Qwen3.5-397B-A17B Mistral-Medium-3.5 GLM-4.5 Qwen3-235B-A22B-Thinking GLM-5.3-Flash (max) GLM-4.7 Qwen DeepSeek Llama Gemma GLM IBM Granite Mistral Phi Nemotron MiniMax non-thinking / instruct thinking dense: solid, all parameters active MoE: total translucent, active solid Pareto front Figure 10: Score–energy fronts with every configuration labelled. Enlarged versions of the three panels in Fig. 1; marker encoding is unchanged. 21 Preprint 10 1 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 40 50 60 70 80 Clean-Completion Score over the Text Suite (%) 32G Tier: RTX 5090, BF16 (n=28), clean-completion bound 1B4B12B marker area = total parameters Gemma-4-E2B Gemma-4-E4B Gemma-4-E2B Nemotron-3-Nano-4B Phi-4-mini Gemma-3-4b Qwen3.5-2B Gemma-4-E4B Ministral-3-3B-Reasoning Qwen3-4B Qwen3.5-0.8B Qwen3.5-4B Nemotron-3-Nano-4B Qwen3.5-9B Ministral-3-3B DeepSeek-R1-Distill-Qwen-7B Gemma-4-12B Granite-4.2-3b Qwen3.5-2B Llama-3.1-8B Gemma-3-12b Qwen3.5-4B Granite-4.2-3b Granite-4.2-8b Qwen3-4B-Thinking Qwen3.5-9B Gemma-4-12B Granite-4.2-8b 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 50 60 70 80 90 Clean-Completion Score over the Text Suite (%) 96G Tier: RTX PRO 6000, BF16 (n=27), clean-completion bound 8B16B32B marker area = total parameters Nemotron-3-Nano-30B-A3B Ministral-3-8B-Reasoning Gemma-4-26B-A4B Ministral-3-14B-Reasoning Qwen3.6-35B-A3B Qwen3-30B-A3B Nemotron-3.5-Lightning GLM-4.7-Flash Ministral-3-8B Gemma-3-27b Ministral-3-14B Gemma-4-31B Qwen3.8 Qwen3.6-27B Gemma-4-26B-A4B DeepSeek-R1-Distill-Qwen-14B Nemotron-3-Nano-30B-A3B Nemotron-3.5-Lightning Qwen3-30B-A3B-Thinking Qwen3.6-35B-A3B Granite-4.2-30b Gemma-4-31B GLM-4.7-Flash Qwen3.8 DeepSeek-R1-Distill-Qwen-32B Qwen3.6-27B Granite-4.2-30b 10 2 10 3 Energy per Attempt (Wh, Suite Geometric Mean) 60 65 70 75 80 85 90 Clean-Completion Score over the Text Suite (%) 512G Tier: 8x H100, FP8 (n=27), clean-completion bound 100B200B400B marker area = total parameters Llama-4-Maverick Mistral-Small-4 GLM-4.5-Air Mistral-Medium-3.5 Qwen3.8-Flash-Next Qwen3.5-122B-A10B Qwen3-235B-A22B Nemotron-3-Super GLM-5.3-Flash (low) Qwen3.5-397B-A17B GLM-4.5 GLM-4.7 Qwen3.8-Flash-Next Nemotron-3-Super Mistral-Small-4 Qwen3.5-122B-A10B MiniMax-M2.5 GLM-4.5-Air MiniMax-M2.7 MiniMax-M2 GLM-5.3-Flash (high) Qwen3.5-397B-A17B Mistral-Medium-3.5 GLM-4.5 Qwen3-235B-A22B-Thinking GLM-5.3-Flash (max) GLM-4.7 Qwen DeepSeek Llama Gemma GLM IBM Granite Mistral Phi Nemotron MiniMax non-thinking / instruct thinking dense: solid, all parameters active MoE: total translucent, active solid Pareto front Figure 11: Score–energy fronts under the clean-completion bound. As in Fig. 10, but benchmark scores are recomputed using only requests that finish within the output cap. En- ergy is unchanged and the Pareto fronts are recomputed. 22 Preprint B.2ENERGY PER CORRECT ANSWER Fig. 12 reports benchmark energy divided by the number of correct answers for every configuration on its tier’s reference platform. BFCL is excluded because its headline score is a category macro- average rather than a count of correct answers. The cost of a correct AIME answer ranges from 0.54 to 815 Wh. The upper end is driven by config- urations with near-zero accuracy; excluding configurations below 5% accuracy narrows the range to 0.54–50 Wh. Across the other benchmarks, the median cost per correct answer ranges from 0.12 Wh on IFEval to 5.9 Wh on LiveCodeBench, with thinking configurations having a 1.7–4.5× higher me- dian than non-thinking configurations. Cost and accuracy are only weakly correlated, with Spearman ρ = 0.12–0.37 across benchmarks. Among the 29 configurations achieving at least 80% on AIME, a correct answer costs from 1.5 Wh (gpt-oss-120b, medium effort, RTX PRO 6000) to 50 Wh (GLM-4.7 with thinking, 8×H100), a 33× spread. Within every five-point accuracy band from 40% to 90%, the most energy-intensive configuration uses 12–36× as much energy per correct answer as the least. These comparisons span memory tiers and therefore combine model and hardware effects. 0.010.11101001,000 Energy per Correct Answer (Wh) IFEval MMLU-Pro MATH-500 GPQA Diamond LiveCodeBench AIME 2025 + 2026 Every Model on Its Tier's Reference Machine gemma-4-E2B-it [NT] Llama-3.1-8B-Instruct gpt-oss-20b [T-low] Qwen3-235B-A22B-Thinking-2507 Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 [NT] granite-4.2-8b [T] Nemotron-3-Nano-30B-A3B-FP8 [NT] Qwen3.6-27B [T] gpt-oss-20b [T-low] Qwen3-235B-A22B-Thinking-2507 gemma-4-E2B-it [NT] granite-4.2-30b [T] 020406080100 AIME Accuracy (%) 1 10 100 1,000 Energy per Correct AIME Answer (Wh) AIME 2025 + 2026: the Cost of Accuracy non-thinkingthinkingcheapest correct answer at this accuracy or better Figure 12: Energy per correct answer. All configurations are shown on their tier’s refer- ence platform; AIME 2025 and 2026 are pooled, and BFCL is excluded. Left: energy per correct answer across benchmarks; boxes summarise configurations and points denote rea- soning mode, with the extremes labelled. Right: energy per correct AIME answer against AIME accuracy. The dashed step gives the minimum energy per correct answer among con- figurations achieving at least each accuracy. 23 Preprint B.3EFFICIENCY OVER RELEASE DATE Fig. 13 places each complete configuration at its checkpoint release date. Energy per score point spans 114×, 59×, and 27× across the three reference platforms, but shows no detectable trend with release date: the fitted multiplicative change per year is 1.18×, 1.77×, and 1.07×, with all 95% intervals including one. Scores do increase with release date. Among non-thinking configurations, the fitted increase is 22– 29% per year on all three platforms, while the energy of thinking configurations increases at least as quickly on the two single-GPU platforms. The best observed energy per score point improves by 32×, 12×, and 5× relative to the oldest configurations, although the current records were already established between April 2025 and April 2026. Within lineages at matched model size, the 30 successor transitions change energy per score point by a median of 0.97×. Non-thinking successors gain 7.6 score points at approximately unchanged energy, whereas thinking successors lose 1.3 points while using 0.94× the energy. These temporal trends should be interpreted cautiously: the roster contains few releases before mid-2025 and is not a random sample of the model ecosystem. 0.1 0.2 0.5 1 2 5 10 20 Energy per Score Point (Wh per Point) RTX 5090 2B8B30B marker area = total parameters RTX PRO 6000 10B30B100B marker area = total parameters 8x H100 100B200B400B marker area = total parameters 30 40 50 60 70 80 Mean Score over the Text Suite (%) Jul 2024Jan 2025Jul 2025Jan 2026Jul 2026 Release Date 5 10 20 50 100 200 500 1000 2000 Energy per Attempt (Wh, Suite Geo Mean) Jul 2024Jan 2025Jul 2025Jan 2026Jul 2026 Release Date Jul 2024Jan 2025Jul 2025Jan 2026Jul 2026 Release Date Qwen DeepSeek Llama Gemma GLM IBM Granite Mistral Phi Nemotron MiniMax gpt-oss non-thinking / instruct thinking dense: solid, all parameters active MoE: total translucent, active solid record so far (lowest Wh per point, highest score) least-squares trend, non-thinking least-squares trend, thinking Figure 13: Efficiency, capability, and energy over release date. Each complete configura- tion is placed at its checkpoint release date on its tier’s reference platform, across all weight formats. Rows show energy per score point, suite score, and suite energy. Solid steps show the best value observed up to each date; dashed and dotted lines show log-scale least-squares trends for non-thinking and thinking configurations. Marker encoding follows Fig. 1. 24 Preprint CSERVING OUTSIDE THE SATURATED REGIME C.1BATCH SIZE 1 On the 8×H100 node, we re-served the AIME 2026 and LiveCodeBench traffic of seven configu- rations with one request in flight, preserving the prompt and generation lengths of the benchmark requests. After warm-up, each configuration was metered for 256 s. The seven configurations com- prise three MoE checkpoints in both reasoning modes and the dense DeepSeek-R1-Distill-Llama- 70B. At batch size 1, the node draws 1.7–1.9 kW for the MoE configurations and 3.2 kW for the dense model, while generating 71–190 tokens/s. For a given configuration, energy per generated token varies by at most 1.08× between the two benchmarks and increases with active parameter count, from 9.8 J at 12B active parameters to 33 J for the dense 70B model. Relative to the corresponding campaign runs, energy per generated token is 6.6× higher at batch size 1 on average (geometric mean; 2.9–18.5× across configuration–benchmark pairs). Throughput falls by 11×, while power falls by only 1.5×. These results cover seven configurations on one platform and quantify the sensitivity to batching rather than a universal batch-size penalty. 1251020 Energy per Generated Token (J) DeepSeek-R1-Distill-Llama-70B · AIME 2026 DeepSeek-R1-Distill-Llama-70B · LiveCodeBench GLM-4.7-FP8 [NT] · AIME 2026 GLM-4.7-FP8 [NT] · LiveCodeBench GLM-4.7-FP8 [T] · AIME 2026 GLM-4.7-FP8 [T] · LiveCodeBench Nemotron-3-Super-120B-A12B-FP8 [NT] · AIME 2026 Nemotron-3-Super-120B-A12B-FP8 [NT] · LiveCodeBench Nemotron-3-Super-120B-A12B-FP8 [T] · AIME 2026 Nemotron-3-Super-120B-A12B-FP8 [T] · LiveCodeBench Qwen3.5-397B-A17B-FP8 [NT] · AIME 2026 Qwen3.5-397B-A17B-FP8 [NT] · LiveCodeBench Qwen3.5-397B-A17B-FP8 [T] · AIME 2026 Qwen3.5-397B-A17B-FP8 [T] · LiveCodeBench AIME 2026, average over 7 rows LiveCodeBench, average over 7 rows DeepSeek-R1-Distill-Llama-70B, average over 2 benchmarks GLM-4.7-FP8 [NT], average over 2 benchmarks GLM-4.7-FP8 [T], average over 2 benchmarks Nemotron-3-Super-120B-A12B-FP8 [NT], average over 2 benchmarks Nemotron-3-Super-120B-A12B-FP8 [T], average over 2 benchmarks Qwen3.5-397B-A17B-FP8 [NT], average over 2 benchmarks Qwen3.5-397B-A17B-FP8 [T], average over 2 benchmarks average over all 14 rows x9.9 x18.5 x2.9 x5.0 x3.7 x6.1 x5.8 x14.4 x5.0 x13.2 x4.4 x7.2 x3.9 x6.9 x4.7 x9.1 x13.5 x3.8 x4.7 x9.1 x8.1 x5.6 x5.2 x6.6 Energy per Generated Token 1002005001,0002,0005,000 Generated Tokens per Second x11.0 x24.5 x3.9 x9.6 x5.7 x12.7 x8.9 x28.8 x7.1 x25.7 x6.7 x18.4 x6.0 x16.6 x6.7 x18.2 x16.4 x6.1 x8.5 x16.0 x13.5 x11.1 x9.9 x11.0 Throughput batch 1: one request at a timeas run: the campaign's run, the benchmark offered at oncethinking (hexagon), non-thinking (circle)average (geometric mean) Figure 14: Batch size 1 against the campaign serving regime on the 8×H100 node. En- ergy per generated token (left) and throughput (right) for seven configurations on AIME 2026 and LiveCodeBench. Open markers show one request at a time; filled markers show the cor- responding campaign runs, with multiplicative gaps annotated. Squares show geometric means by benchmark, configuration, and across all 14 pairs. Colour denotes lineage; circles are non-thinking and hexagons thinking configurations. C.2WEIGHTS OFFLOADED TO CPU MEMORY The models in the 512 GiB tier do not fit within the GH200’s 96 GiB of HBM3 under our main serving configuration. We therefore serve two of them on a single GH200 using the serving engine’s CPU-offload mechanism, which places weights that do not fit in HBM3 in Grace LPDDR5X and transfers them over NVLink-C2C during inference. The two configurations are Qwen3.5-122B- A10B (68 of 118 GiB offloaded; 57%) and Qwen3.5-397B-A17B (344 of 378 GiB; 91%), both FP8 without thinking. We evaluate MMLU-Pro, AIME 2026, and LiveCodeBench under otherwise unchanged campaign settings. For comparison, we fit a least-squares trend on log scales against active parameter count using the seven FP8 non-thinking 96 GiB configurations whose weights fit entirely in HBM3 on the same GH200 (3–30B active parameters). Relative to this trend, the offloaded runs take 7–19× longer, 25 Preprint achieve only 0.04–0.22× the throughput, and use 1.8–7.2× more energy per generated token. The energy penalty is smaller than the runtime penalty because GPU+HBM power falls to 200–250 W during the offloaded runs, compared with 290–700 W for the in-memory configurations. The result is robust to the choice of reference configurations: throughput remains 0.05–0.26× and energy per token 1.5–7.2× the fitted trend. With 14–131 requests in flight, a uniform-routing approximation implies that each decode step ac- cesses 27–97% of the experts, corresponding to roughly 180–330 GiB/s of offloaded-weight traffic over NVLink-C2C. Together with the large throughput loss and reduced accelerator power, this is consistent with offloaded-weight traffic becoming the serving bottleneck. Offloading changes serving cost much more than capability. Scores differ from the same models on the 8×H100 node by at most 1.2 points on MMLU-Pro and LiveCodeBench, and by at most 6 points on the 30-problem AIME 2026. At full module scope, including the Grace CPU and memory holding the offloaded weights, energy rises to 1.4–1.5× the primary GPU+HBM measurement, compared with 1.1–1.25× without offloading. 2 5 10 20 50 100 200 Minutes per Attempt MMLU-Pro Time to Run the Benchmark x9.7 x17.7 100 200 500 1,000 2,000 5,000 Generated Tokens per Second Throughput x0.12 x0.04 0.1 0.2 0.5 1 2 J per Generated Token (GPU rail) Energy per Generated Token x3.4 x7.2 0.2 0.5 1 2 5 10 20 50 Minutes per Attempt AIME 2026 x12.9 x17.9 50 100 200 500 1,000 2,000 Generated Tokens per Second x0.12 x0.05 0.5 1 2 5 J per Generated Token (GPU rail) x3.5 x7.0 35102030 Active Parameters (B) 2 5 10 20 50 100 200 Minutes per Attempt LiveCodeBench x7.1 x19.0 35102030 Active Parameters (B) 100 200 500 1,000 2,000 Generated Tokens per Second x0.22 x0.07 35102030 Active Parameters (B) 0.2 0.5 1 2 J per Generated Token (GPU rail) x1.8 x4.4 GH200, FP8 non-thinking models whose weights fit trend on active parameters (band: one residual s.d.) Qwen3.5-122B-A10B, 57 % of its weights in Grace memory Qwen3.5-397B-A17B, 91 % of its weights in Grace memory Figure 15: Weights offloaded to Grace memory on the GH200. Minutes per attempt (left), throughput (middle), and energy per generated token from the GPU+HBM measurement (right) against active parameter count for MMLU-Pro, AIME 2026, and LiveCodeBench. Small points show FP8 non-thinking configurations that fit entirely in HBM3; dashed lines and bands show their log-scale least-squares trends and±1 residual standard deviation. Open markers show Qwen3.5-122B-A10B (57% offloaded) and Qwen3.5-397B-A17B (91%), la- belled by their ratio to the fitted trend. 26 Preprint DENERGY CONSUMED BY THIS STUDY Fig. 16 accounts for the primary accelerator energy consumed by the completed evaluation cam- paign. Across all platforms and benchmarks, the experiments consumed approximately 1.34 MWh. This total includes all metered benchmark attempts, including side experiments and attempts whose results were later replaced or discarded. The 8×H100 node accounts for the largest share, at 443 kWh (33%), followed by the 2×RTX A6000 platform at 349 kWh (26%). The RTX PRO 6000, RTX 5090, GH200, and GB10 account for 178, 151, 128, and 89 kWh, respectively. By benchmark, LiveCodeBench and AIME 2026 each account for about one fifth of the total, followed by AIME 2025 and GPQA Diamond at about 16% each; IFEval accounts for less than 3%. These values use the primary accelerator-level measurement boundary defined in Appendix A.3. They therefore exclude power-supply losses, cooling, host power on the discrete-GPU systems, model-loading energy, and idle periods between runs. At $0.20/kWh, the measured accelerator energy corresponds to approximately $268 of electricity. 33 % 443 kWh 8x H100 26 % 349 kWh 2x RTX A6000 13 % 178 kWh RTX PRO 6000 11 % 151 kWh RTX 5090 9.5 % 128 kWh GH200 6.6 % 89 kWh GB10 21 % 281 kWh LiveCodeBench 20 % 262 kWh AIME 2026 16 % 219 kWh GPQA Diamond 16 % 218 kWh AIME 2025 9.2 % 123 kWh BFCL v3 9.0 % 121 kWh MMLU-Pro 5.8 % 78 kWh MATH-500 2.7 %, 36 kWh IFEval Figure 16: Primary accelerator energy consumed by the completed evaluation cam- paign. Energy is broken down by platform (left) and benchmark (right), totalling approxi- mately 1.34 MWh. The accounting includes all metered benchmark attempts, including side experiments and attempts later replaced. Model loading, idle periods, host power, and other infrastructure overheads are excluded. 27