Paper deep dive
Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors
Saeid Shokoufa, Mohammad Erfan Sadeghi, Mehdi Kamal, Massoud Pedram
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/10/2026, 3:05:37 AM
Summary
The paper introduces HYMELL, a hybrid three-level framework for estimating Large Language Model (LLM) inference latency and energy consumption. It combines analytical modeling for primitive operations (RMSNorm, Softmax, GEMM) with machine learning predictors for block-level (Attention, FFN) and system-level costs. Evaluated on NVIDIA H100 GPUs, HYMELL achieves high accuracy (less than 5% error for LLaMA 3 8B) and supports diverse architectures including dense and mixture-of-experts (MoE) models, enabling hardware-free design space exploration.
Entities (11)
Relation Signals (12)
HYMELL → evaluatedon → NVIDIA H100
confidence 95% · Evaluated on an NVIDIA H100 graphics processing unit (GPU)
HYMELL → predicts → energy
confidence 95% · estimating LLM inference latency and energy
HYMELL → predicts → Latency
confidence 95% · estimating LLM inference latency and energy
GEMM → modeledby → HYMELL
confidence 90% · Level 1 models primitive operators, including GEMM, Softmax, and RMSNorm
Softmax → modeledby → HYMELL
confidence 90% · Level 1 models primitive operators, including GEMM, Softmax, and RMSNorm
RMSNorm → modeledby → HYMELL
confidence 90% · Level 1 models primitive operators, including GEMM, Softmax, and RMSNorm, using analytical estimators
HYMELL → supports → Llama-3-8B
confidence 90% · notably, for LLaMA 3 8B, it attains less than 5% error
HYMELL → uses → Analytical Modeling
confidence 90% · HYMELL models LLM execution through a three-level hierarchy: analytical estimation of primitive operations
HYMELL → uses →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment and hardware-aware design. In this work, we introduce Hybrid Modeling for Energy and Latency of LLMs (HYMELL), a hybrid three-level framework for estimating LLM inference latency and energy by combining analytical modeling with machine learning (ML). HYMELL models LLM execution through a three-level hierarchy: analytical estimation of primitive operations, ML prediction of higher-level components, and an end-to-end model that captures system-level overheads across both prefill and decode phases. The framework supports diverse architectures, including dense and mixture-of-experts (MoE) feed-forward networks (FFNs), as well as multi-head attention (MHA) and grouped-query attention (GQA) mechanisms. Evaluated on an NVIDIA H100 graphics processing unit (GPU), HYMELL achieves high predictive accuracy; notably, for LLaMA 3 8B, it attains less than 5% error for both prefill and decode phases. By predicting execution costs directly from architectural parameters, it enables fast, hardware-free design space exploration and energy-efficient optimization.
Tags
Links
- Source: https://arxiv.org/abs/2608.06723v1
- Canonical: https://arxiv.org/abs/2608.06723v1
Trouble viewing inline? Open PDF directly →
Full Text
49,048 characters extracted from source content.
Expand or collapse full text
Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical–Machine-Learning Predictors Saeid Shokoufa University of Southern CaliforniaLos AngelesCaliforniaUSA shokoufa@usc.edu , Mohammad Erfan Sadeghi University of Southern CaliforniaLos AngelesCaliforniaUSA sadeghim@usc.edu , Mehdi Kamal University of Southern CaliforniaLos AngelesCaliforniaUSA mehdi.kamal@usc.edu and Massoud Pedram University of Southern CaliforniaLos AngelesCaliforniaUSA pedram@usc.edu Abstract. The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment and hardware-aware design. In this work, we introduce Hybrid Modeling for Energy and Latency of LLMs (HYMELL), a hybrid three-level framework for estimating LLM inference latency and energy by combining analytical modeling with machine learning (ML). HYMELL models LLM execution through a three-level hierarchy: analytical estimation of primitive operations, ML prediction of higher-level components, and an end-to-end model that captures system-level overheads across both prefill and decode phases. The framework supports diverse architectures, including dense and mixture-of-experts (MoE) feed-forward networks (FFNs), as well as multi-head attention (MHA) and grouped-query attention (GQA) mechanisms. Evaluated on an NVIDIA H100 graphics processing unit (GPU), HYMELL achieves high predictive accuracy; notably, for LLaMA 3 8B, it attains less than 5% error for both prefill and decode phases. By predicting execution costs directly from architectural parameters, it enables fast, hardware-free design space exploration and energy-efficient optimization. Large Language Models, Power Estimation, Energy Modeling, GPU Energy Consumption, Latency Estimation †copyright: none†ccs: Hardware Power estimation and optimization†ccs: Hardware Platform power issues†ccs: Hardware Energy metering 1. INTRODUCTION Neural networks (NNs), particularly large language models (LLMs), have become central to modern AI. Most LLMs are based on the Transformer architecture introduced in Attention Is All You Need (vaswani2017attention, ) and are commonly implemented as either dense or mixture-of-experts (MoE) models (fedus2022switch, ). Their rapid scaling and deployment, however, have introduced substantial energy costs (schwartz2019greenai, ; luccioni2024power, ); for example, training GPT-3 was estimated to consume about 1,287 MWh of electricity and produce 552 tCO2e (patterson2021carbon, ). Consequently, extensive research has focused on reducing N and LLM energy through techniques such as pruning, quantization, efficient attention, sparsity, and memory-efficient serving (han2016deepcompression, ; abbasi2026integration, ; strumolo2025data, ; frantar2022gptq, ; dettmers2022llmint8, ; dao2022flashattention, ; dao2023flashattention2fasterattentionbetter, ; frantar2023sparsegpt, ; kwon2023efficient, ). NNs, and especially LLMs, are now used across a wide range of applications, including natural-language processing, engineering, healthcare, finance, law, and education (chen2021evaluating, ; abdollahi2026unified, ; abdollahi2026hdlforgetwostagemultiagentframework, ; golkarieh2025semi, ; fayyazi2026coft, ; khezresmaeilzadeh2025preserving, ; singhal2023large, ; wu2023bloomberggpt, ; guha2023legalbench, ; kasneci2023chatgpt, ). Their rapidly growing adoption makes efficient and accurate modeling of LLM inference latency and energy increasingly important. Figure 1. General architecture of an LLM (L and NHN_H represent the number of layers and attention heads, respectively). As shown in Figure 1, despite implementation differences, most LLMs share a common architecture of stacked layers comprising attention mechanisms (multi-head attention (MHA) or grouped-query attention (GQA)) and feed-forward networks (FFNs; dense or mixture-of-experts (MoE)). At a lower level of execution, these components map to general matrix multiplications (GEMMs) and element-wise or nonlinear operations (e.g., normalization, softmax). While GEMMs dominate prefill computation, memory-bound element-wise operations account for a significant share of total energy, especially during autoregressive decoding. LLM inference consists of two phases with distinct characteristics. In the prefill phase, the full input sequence is processed in parallel, making it largely compute-bound due to large matrix multiplications. In contrast, the decode phase generates tokens autoregressively, with increasing reliance on key-value (KV) cache accesses, making it latency-sensitive and often memory-bound. These differences motivate separate modeling of performance and energy across phases. In this paper, we introduce Hybrid Modeling for Energy and Latency of LLMs (HYMELL), a hybrid model for energy and latency during LLM inference. HYMELL adopts a three-stage modeling approach: (1) Operator-level modeling: Analytical models first estimate the latency and energy costs of primitive operations, including normalization, softmax, and GEMM. (2) Block-level modeling: These estimates are then used by machine learning (ML) models to predict the cost of block-level components, including attention and feed-forward network (FFN) blocks. (3) System-level modeling: Finally, an ML model aggregates the predicted block costs together to estimate the end-to-end inference latency and energy consumption of the full LLM. Our goal is to build accurate and interpretable predictors that generalize across sequence lengths and batching regimes, enabling energy-aware deployment and design decisions for both dense and sparse MoE-based LLMs. 2. RELATED WORK A growing body of work has explored methods for predicting the runtime (latency) and energy consumption of LLMs. AMALI (Analytical Model for Accurately Modeling LLM Inference) (cao2025amali, ) proposes a detailed analytical model for LLM inference on modern graphics processing units (GPUs) by explicitly capturing low-level architectural behavior. While it achieves high accuracy, it relies on extensive manual modeling and calibration, requiring detailed per-kernel and per-workload analysis. This strong dependence on GPU microarchitectural knowledge makes the approach complex and less accessible. Moreover, adapting the model to new LLM architectures or kernels requires repeated analysis, limiting scalability. LIFE (patwari2025forecastingllminferenceperformance, ) introduces a hardware-agnostic analytical framework that models operator-level compute and memory behavior. However, it depends on explicit hardware specifications, such as compute throughput and memory bandwidth, as well as efficiency assumptions. Adapting the framework to new models or optimizations requires manual reconfiguration of its components. In addition, its reliance on a simulation pipeline and a configuration-driven workflow increases system complexity and limits deployment ease. SweetSpot (cavagna2026sweetspotanalyticalmodelpredicting, ) presents an analytical model to estimate the energy efficiency of LLM inference based on the complexity of computation and memory access. Although effective, it requires deriving detailed equations tied to transformer internals and workload structure. This limits flexibility when adapting to new models or unseen behaviors, as extending the approach requires manual reformulation. Furthermore, its structured analytical pipeline adds modeling overhead compared to more data-driven approaches. The work in (krupp2026takinglonginvestigating, ) investigates the use of inference time as a proxy to estimate energy consumption in API-based LLMs. However, this relies on the assumption that latency directly correlates with energy, which may not hold under varying hardware utilization, parallelism strategies, or system-level overheads. As a result, the method cannot capture fine-grained architectural or workload-dependent effects, providing only coarse-grained estimates. 3. METHODOLOGY Figure 2 summarizes the HYMELL framework. HYMELL uses a three-level hierarchy to predict LLM inference latency and energy. Level 1 models primitive operators, including GEMM, Softmax, and RMSNorm, using analytical estimators based on operator dimensions and regime-specific features. Level 2 feeds these primitive predictions, together with architectural parameters, into lightweight MLPs to estimate attention and FFN block costs, capturing residual overheads such as reshaping, masking, activation, and routing. Level 3 combines the block-level predictions with global model parameters, including layers, sequence length, batch size, and inference mode, to estimate end-to-end latency and energy for both prefill and decode. Figure 2. Overview of the HYMELL hierarchical prediction framework. Level 1 analytically models primitive GPU operators. Level 2 predicts block-level execution using lightweight MLPs. Level 3 combines block-level estimates with architectural parameters to predict end-to-end latency and energy. A three-level HYMELL hierarchy connecting primitive GPU operator estimators, block-level MLP predictors, and an end-to-end latency and energy predictor. 3.1. RMSNorm Estimator (Analytical) Normalization is a lightweight operation applied independently to each token in every LLM layer. Although each invocation is inexpensive, repeated use across layers and tokens incurs non-negligible latency and energy consumption, especially during autoregressive decoding. In modern transformer-based LLMs, RMSNorm has largely replaced traditional layer normalization (LayerNorm). Therefore, in this work, we model normalization using RMSNorm. Nevertheless, the proposed estimation methodology can be easily extended to other normalization variants, as they exhibit similar reduction- and element-wise execution patterns. To capture execution behavior, we adopt a two-regime estimator that distinguishes between launch-bound and memory-bound execution based on the total number of processed elements. For each regime, execution time and energy are modeled as: (1) ERMSNorm=α5EN⋅dmodel+α4EN⋅log2(dmodel) E_RMSNorm= _5^E\,N· d_model+ _4^E\,N· _2(d_model) +α3Edmodel+α2EN+α1Elog2(dmodel)+α0E + _3^E\,d_model+ _2^E\,N+ _1^E\, _2(d_model)+ _0^E (2) TRMSNorm=α5TN⋅dmodel+α4TN⋅log2(dmodel) T_RMSNorm= _5^T\,N· d_model+ _4^T\,N· _2(d_model) +α3Tdmodel+α2TN+α1Tlog2(dmodel)+α0T + _3^T\,d_model+ _2^T\,N+ _1^T\, _2(d_model)+ _0^T where N is the number of tokens and dmodeld_model is the hidden dimension, and αiE _i^E and αiT _i^T are regression coefficients. The execution regime is determined by the total workload: when N⋅dmodelN· d_model is below a threshold thRMSth_RMS, the operation is launch-bound; otherwise, it is memory-bound. This separation improves modeling accuracy, particularly for small problem sizes. Each term in the estimator captures a specific physical aspect: dmodeld_model represents per-token operations and vector-length-dependent memory accesses; N reflects the linear scaling with the sequence length; N⋅dmodelN· d_model serves as a proxy for total compute and dominant memory traffic; log2(dmodel) _2(d_model) and N⋅log2(dmodel)N· _2(d_model) model the depth and aggregate cost of tree-based reductions; and the bias accounts for fixed kernel launch and synchronization overheads. This two-regime formulation captures distinct execution behaviors without increasing model complexity, reducing error for small tensors while maintaining accuracy at scale. Overall, normalization is best characterized as a memory-bound operator with non-negligible fixed overhead. Consequently, explicitly modeling normalization as a base operator yields more accurate time and energy estimates than approximating its cost using aggregate FLOP-based metrics. 3.2. Softmax Base Estimator (Analytical) Softmax is a core operator in the self-attention mechanism that transforms attention scores into normalized attention weights. In LLMs, it is applied to an N×N× N attention matrix under a causal mask, ensuring each token attends only to previous tokens. Despite its simplicity, softmax is invoked across layers and attention heads, leading to substantial memory traffic and non-negligible latency and energy cost. The Softmax execution exhibits two distinct behaviors depending on the length of the sequence N. Similar to RMSNorm, we adopt a two-regime estimator to capture launch-bound and memory-bound execution. For each regime, execution time and energy are modeled as: (3) ESoftmax=β3EN2+β2EN+β1Elog2(N)+β0E E_Softmax= _3^E\,N^2+ _2^E\,N+ _1^E\, _2(N)+ _0^E (4) TSoftmax=β3TN2+β2TN+β1Tlog2(N)+β0T T_Softmax= _3^T\,N^2+ _2^T\,N+ _1^T\, _2(N)+ _0^T where N is the number of input tokens, and βiE _i^E and βiT _i^T denote the regression coefficients for energy and latency, respectively. The execution regime is determined by N: if N<thSoftmaxN<th_Softmax, the operation is launch-bound; otherwise, it is memory-bound. The kernel-launch-dominated regime corresponds to small N, where kernel-launch latency, masking logic, and synchronization overhead dominate execution. The memory-bound regime corresponds to larger N, where global memory reads, writes, and normalization traffic dominate both time and energy consumption. Each term in the estimator captures a distinct component of causal softmax execution: N represents the number of independent softmax normalizations (one per row); N2N^2 captures the dominant memory traffic and total computations of the attention matrix; log2(N) _2(N) models the reduction depth for normalization and associated synchronization overhead; and the bias accounts for fixed overheads such as kernel launch latency, instruction dispatch, and masking control flow, which dominate at small N. Softmax is predominantly a memory-bound operator, with energy consumption driven primarily by quadratic memory access rather than floating-point computation. Separating kernel-launch dominated and memory-bound regimes prevents systematic error at small N and captures bandwidth-driven scaling at larger N. 3.3. GEMM Estimator (Analytical) General matrix multiplication (GEMM) is a dominant building block of LLM inference, appearing in linear projections and feed-forward layers. On NVIDIA GPUs, these GEMMs are typically executed using cuBLAS (CUDA Basic Linear Algebra Subprograms), a closed-source, highly optimized library that selects among multiple internal algorithms depending on matrix dimensions, data type, and hardware constraints. Its execution time and energy consumption depend on several interacting factors, including the total number of operations, memory access patterns, arithmetic intensity, and the specific kernel implementation chosen by cuBLAS, such as different tiling strategies, tensor-core utilization, and multiplication schemes. The cost of GEMM computation for CN×M=AN×K×BK×MC^N× M=A^N× K× B^K× M scales as (MNK)O(MNK). We define the total floating-point workload as F=2MNKF=2MNK. The next factor is memory traffic, represented in bytes as Bin=(M+N)K⋅b&Bout=MN⋅bB_in=(M+N)K· b \& B_out=MN· b. where b denotes bytes per element. We also consider the arithmetic intensity (AI) defined as AI=FBin+BoutAI= FB_in+B_out, which quantifies the ratio of computation to memory traffic. GEMM performance and energy exhibit distinct regimes depending on problem size and arithmetic intensity. We therefore partition GEMMs into five execution behaviors defined by thresholds on total work F and arithmetic intensity (AI). The kernel-launch-bound regime corresponds to very small problems where F<FlaunchF<F_launch. For all other regimes (F≥FlaunchF≥ F_launch), the behavior is determined by arithmetic intensity: • kernel-launch-bound: F<FlaunchF<F_launch • memory-bound: AI<AIlowAI<AI_low • balanced (memory-dominant): AIlow≤AI<AImidAI_low <AI_mid • balanced (compute-dominant): AImid≤AI<AIhighAI_mid <AI_high • compute-bound: AI≥AIhighAI≥ AI_high This separation isolates kernel-launch overhead for small problems and distinguishes memory- and compute-dominated behavior at larger scales. The threshold parameters FlaunchF_launch, AIlowAI_low, AImidAI_mid, and AIhighAI_high are defined as part of the category definitions above. Beyond workload, memory traffic, and AI, GEMM time and energy depend strongly on achievable GPU utilization. In matrix multiplication, independent output elements can be computed in parallel in the M and N dimensions, whereas accumulation along the K dimension is inherently serial due to the operation’s structure. To capture this behavior, we define a parallelism factor as PF=MNKPF= MNK. This reflects parallel work over M,NM,N relative to serialization over K. Larger PF indicates higher parallelism and utilization, while large K increases serialization and reduces efficiency. The inverse term PF−1PF^-1 captures regimes where limited parallelism leads to higher execution cost. The selected features were obtained via ablation, retaining only those with significant predictive contribution. For each regime, GEMM time and energy are modeled as: (5) EGEMM=γ10EPF⋅F+γ9EPF⋅Bin+γ8EAI⋅Bin+γ7EF+γ6EPF E_GEMM= _10^EPF· F+ _9^EPF· B_in+ _8^EAI· B_in+ _7^EF+ _6^EPF +γ5EAI+γ4EBout+γ3EBin+γ2Elog10(F)+γ1EPF−1+γ0E + _5^EAI+ _4^EB_out+ _3^EB_in+ _2^E _10(F)+ _1^EPF^-1+ _0^E (6) TGEMM=γ10TPF⋅F+γ9TPF⋅Bin+γ8TAI⋅Bin+γ7TF+γ6TPF T_GEMM= _10^TPF· F+ _9^TPF· B_in+ _8^TAI· B_in+ _7^TF+ _6^TPF +γ5TAI+γ4TBout+γ3TBin+γ2Tlog10(F)+γ1TPF−1+γ0T + _5^TAI+ _4^TB_out+ _3^TB_in+ _2^T _10(F)+ _1^TPF^-1+ _0^T Each term in the estimator captures a distinct component of GEMM execution: • F and log10(F) _10(F): Capture total compute and small-kernel effects such as launch overhead and scheduling inefficiencies. • BinB_in and BoutB_out: Represent dominant memory traffic from reading input and writing output. • AI: Captures the balance between computation and memory, distinguishing execution regimes. • PF: Represents the available parallelism in M and N relative to the depth of reduction in K, influencing GPU utilization. • PF−1PF^-1: Captures serialization effects for limited parallelism. • Interaction terms (PF⋅FPF· F, PF⋅BinPF· B_in, AI⋅BinAI· B_in): Capture interactions between compute workload, memory traffic, and available parallelism. For example, (PF⋅F)(PF· F) reflects how additional computation scales differently depending on available parallelism. 3.4. Attention Block Estimator (ML-based) The attention block is dominated by matrix multiplications and softmax operations. We therefore estimate its execution time and energy using the GEMM and Softmax base estimators. The GEMM estimator is applied to the linear projections that generate query (Q), key (K), and value (V), as well as to QKTQK^T, the attention-weighted value Softmax(QKT)⋅VSoftmax(QK^T)· V, and the output projection. The Softmax estimator normalizes the attention score matrix. However, several operations are not explicitly captured, including tensor reshaping (e.g., head splitting and concatenation), masking, and other lightweight memory operations. While individually inexpensive, these collectively introduce non-negligible overhead. To capture these residual costs, we employ a multilayer perceptron (MLP) correction model. It takes as input the predicted time and energy from GEMM and softmax estimators, along with architectural parameters such as number of heads, head dimension, and sequence length, and predicts total attention block time and energy. This enables modeling of both dominant kernel costs and attention-specific overheads. The model supports both multi-head attention (MHA) and grouped-query attention (GQA). During autoregressive decoding, each forward pass processes a single newly generated token, so the operator-level query length is fixed at N=1N=1; however, this does not remove the dependence on context length. Let LKVL_KV denote the KV-cache length. In the QKTQK^T multiplication, the GEMM dimensions are (1,dhead)×(dhead,LKV)(1,d_head)×(d_head,L_KV), so the cache length appears as the output dimension. Similarly, for Softmax(QKT)VSoftmax(QK^T)V, the GEMM dimensions are (1,LKV)×(LKV,dhead)(1,L_KV)×(L_KV,d_head), where LKVL_KV appears as the reduction dimension. Thus, increasing context length is explicitly modeled through the GEMM dimensions and the corresponding softmax input size, even though only one new token is processed per decoding step. During prefill, the estimators use the full prompt length. The attention estimator is implemented as an MLP with LattnL_attn layers and hidden dimension DattnD_attn. 3.5. FFN Block Estimator (ML-based) The feed-forward network (FFN) block is dominated by dense matrix multiplications. In gated variants, it consists of two parallel up projections followed by a down projection. We estimate their execution time and energy using the GEMM estimator, applied to the up projections (expanding to the intermediate dimension) and the down projection (mapping back to the model dimension). Between these projections, the FFN includes lightweight operations such as activations (e.g., Gaussian error linear unit (GELU) and sigmoid linear unit (SiLU)), element-wise multiplications, and intermediate memory accesses. Although small relative to GEMMs, they introduce additional overhead not captured by the base estimators. To account for this, we use the same MLP-based correction model as in the attention block. It takes as input the predicted time and energy of the projections, along with architectural parameters such as model dimension (dmodeld_model) and intermediate dimension (dintermediated_intermediate), and predicts total FFN execution time and energy. The formulation also supports MoE architectures. Since experts share the same structure, base estimators are applied to a representative expert, while the number of experts, top-k routing, and shared expert dimension (if present) are provided as additional MLP inputs to predict total MoE FFN cost. During decoding, estimators use N=1N=1, while during prefill, they use the prompt length. The FFN estimator is implemented as an MLP with LffnL_ffn layers and hidden dimension DffnD_ffn. 3.6. End-to-End Estimator (ML-based) Using the operator- and block-level models, we first estimate the execution time and energy of the dominant components in each LLM layer: RMSNorm, attention, and FFN, which account for most computation and memory traffic during inference. These predictions are then fed into a final ML-based estimator, together with global features such as model dimension (dmodeld_model), number of layers, sequence length, batch size, inference mode (decode vs. prefill), and MoE configuration. This estimator predicts total LLM inference time and energy, capturing system-level effects not modeled at lower levels. In particular, this stage is harder to predict due to complex interactions between layers, including lightweight operations between components and runtime overheads such as kernel launches, synchronization, and cross-layer scheduling. By combining block-level predictions with global features, it provides accurate end-to-end estimates. The estimator is implemented as a lightweight MLP with LfinalL_final layers and hidden dimension DfinalD_final. 4. Applications in Optimization The proposed framework enables efficient exploration of the LLM architectural design space by directly predicting execution time and energy from architectural parameters, serving as a fast surrogate for hardware evaluation. One key application is architecture-level optimization for energy or latency. Given a model configuration, the estimator evaluates variations in parameters such as the number of attention heads, key-value heads, model dimension (dmodeld_model), feed-forward dimension (dintermediated_intermediate), number of layers, and MoE settings. By sweeping these parameters, it identifies configurations that minimize energy or runtime. As an example, we vary the number of attention heads while fixing dmodel=4096d_model=4096 and using equal query and key-value heads. For a single attention layer in prefill (N=1024N=1024), Fig. 3 shows that 6464 heads achieve the lowest energy, followed by 3232, while 1616 and 128128 remain competitive. Very large headcounts increase time and energy due to reduced parallel efficiency and higher overhead. More broadly, the estimator supports joint optimization across parameters. For instance, increasing dmodeld_model while reducing layers can preserve capacity while improving efficiency. Similarly, jointly tuning dmodeld_model, dintermediated_intermediate, attention heads, and key-value heads enables hardware-aware designs. This approach applies at both block level (attention/FFN) and full-model level for energy- or latency-optimized LLM architectures. (a) Attention runtime vs. heads (b) Attention energy vs. heads Figure 3. Effect of the number of attention heads on execution time and energy for dmodel=4096d_model=4096 and qheads=kvheadsq_heads=kv_heads. 5. EXPERIMENTS AND RESULTS 5.1. Data Collection and Measurements Time was measured using CUDA events, and energy was measured with NVML by reading the GPU energy counter before and after each measured region. To reduce noise, each operator or configuration was repeated until the cumulative execution time exceeded 10 seconds; per-execution latency and energy were then obtained by dividing the total time and energy by the number of repetitions. For large prefill workloads whose individual executions were already long, we used five repetitions, since the longer measurement window makes NVML granularity noise negligible. All experiments were conducted in bfloat16 (BF16) precision. To avoid cache reuse effects and reflect realistic memory behavior, we rotated pre-generated weight sets across runs. We constructed datasets that span diverse architectural configurations (varying model dimensions, layers, sequence lengths, batch sizes, dense/MoE FFNs, and MHA/GQA) for both prefill and decode modes. The end-to-end evaluation used a dataset of approximately 1,200 model configurations profiled with HuggingFace Transformers. We trained MLP-based regressors for attention, FFN, and full-model latency and energy prediction. Inputs and targets were standardized using training-set statistics. All estimators used 4-hidden-layer MLPs. The attention estimators used 384 hidden units and Smooth L1 loss, optimized with AdamW using a learning rate of 5×10−55× 10^-5 and weight decay of 10−510^-5 for 2,000 epochs. The FFN estimators used width 64 and MSE loss, optimized with AdamW using a learning rate of 2×10−32× 10^-3 and weight decay of 10−610^-6 with early stopping. The full-model prefill and decode estimators used width 32 and MSE loss on log-transformed latency and energy, optimized with AdamW using a learning rate of 2×10−32× 10^-3 and weight decay of 10−610^-6 for 100 epochs. We trained the full-model estimators in log-space because this objective better matches relative-error evaluation and produced the lowest MAPE in our experiments. While the current evaluation is conducted on an NVIDIA H100 NVL GPU, HYMELL’s hierarchical design is fundamentally hardware-portable. Adapting the framework to new architectures (e.g., earlier GPU generations or alternative accelerators) does not require redesigning the architecture; it only necessitates re-profiling the hardware to update the analytical regression coefficients (e.g., α,β,γα,β,γ) and fine-tuning the lightweight MLPs on the new target’s data. Because ML estimators are implemented as lightweight MLPs (LattnL_attn, LffnL_ffn, LfinalL_final) operating on low-dimensional architectural features rather than raw tensor data, their training overhead is trivial. Furthermore, generating the 1,200 profiling configurations is a one-time automated hardware cost that requires significantly less manual engineering effort than re-deriving structural equations for purely analytical models (cavagna2026sweetspotanalyticalmodelpredicting, ; patwari2025forecastingllminferenceperformance, ). We report prediction quality using mean absolute percentage error (MAPE) and the coefficient of determination (R2R^2). Table 1. Accuracy of the RMSNorm estimator. Time Energy Regime MAPE (%) R2R^2 MAPE (%) R2R^2 Launch-bound 1.288 0.240156 3.233 0.979766 Memory-bound 2.098 0.999759 3.516 0.999451 Average 1.693 0.619958 3.375 0.989609 Table 2. Accuracy of the Softmax estimator. Time Energy Regime MAPE (%) R2R^2 MAPE (%) R2R^2 Launch-bound 2.869 0.996185 2.583 0.999042 Memory-bound 3.661 0.999982 1.251 0.999970 Average 3.265 0.998084 1.917 0.999506 5.2. Evaluation Tables 1 and 2 report the accuracy of operator-level estimators across execution regimes. The estimators achieve high accuracy, with MAPE below 4%4\% for Softmax and below 4%4\% for RMSNorm, demonstrating the effectiveness of HYMELL. In the launch-bound regime of RMSNorm, execution time is dominated by kernel launch overhead and remains nearly constant across configurations. This results in a low variance in ground-truth measurements and therefore a lower R2R^2, despite a low prediction error. The regime thresholds are thSoftmax=1024th_Softmax=1024 for softmax and thRMS=2.5×106th_RMS=2.5× 10^6 for RMSNorm. Table 3. Accuracy of the GEMM estimator. Time Energy Regime MAPE (%) R2R^2 MAPE (%) R2R^2 Launch-bound 5.587 0.834 6.134 0.930 Memory-bound 6.819 0.993 5.861 0.996 Balanced (mem-dom) 5.974 0.993 5.189 0.992 Balanced (comp-dom) 9.399 0.960 7.294 0.979 Compute-bound 4.141 0.999 4.472 0.998 Average 6.384 0.956 5.790 0.979 Table 3 reports the accuracy of the GEMM estimator in execution regimes, with MAPE generally below 7%7\% for both time and energy, indicating stable performance in diverse workload characteristics. Regimes are defined by total work F and arithmetic intensity (AI): Fthr=8.39×106F_thr=8.39× 10^6 floating-point operations (FLOPs), AIlow=81.92AI_low=81.92, AImid=166.05AI_mid=166.05, and AIhigh=288.91AI_high=288.91 FLOPs/byte. Higher errors in balanced regimes stem from transitions between memory- and compute-bound behavior, where GPU utilization and scheduling vary. Additionally, cuBLAS dynamically selects kernels and tiling strategies based on input dimensions and AI, introducing performance discontinuities that make these regimes harder to model. Tables 4 and 5 report the accuracy of ML-based estimators for attention and FFN. Both achieve high accuracy, with MAPE below 3%3\% for attention and below 6%6\% for FFN. For attention, prefill exhibits higher R2R^2 since computation scales with sequence length and dominant GEMM and Softmax operations are well modeled. During decoding, each inference step computes only the newly generated token while reusing cached keys and values from previous tokens. Consequently, the computational workload outside the attention kernel remains effectively constant across decoding steps. Although attention still depends on the KV-cache length, on modern GPUs this dependence is largely memory-bandwidth dominated and grows much more slowly than the prefill workload. As a result, the measured decode latency exhibits substantially lower variance than prefill across the evaluated context lengths, which explains the lower R2R^2 despite the very low MAPE. For FFN, the estimator accurately captures both dense and MoE architectures. Slightly higher error in MoE arises from additional routing and memory overhead for expert selection and aggregation. Overall, these results show that the ML-based block estimators effectively capture residual computation and memory overhead beyond the analytical operator models. Table 4. Accuracy of the attention block estimator. Time Energy Mode MAPE (%) R2R^2 MAPE (%) R2R^2 Prefill 3.514 0.9997 4.237 0.9996 Decode 0.441 0.9276 0.599 0.9996 Average 1.978 0.9636 2.418 0.9996 Table 5. Accuracy of the FFN block estimator. Time Energy Architecture MAPE (%) R2R^2 MAPE (%) R2R^2 Dense FFN 4.063 0.9998 4.612 0.9998 MoE FFN 5.168 0.9946 6.333 0.9951 Average 4.616 0.9972 5.473 0.9975 Table 6 reports end-to-end prediction accuracy on held-out LLM architectures. The model achieves ∼ 10% MAPE for both time and energy across dense and MoE models. Higher error compared to operator- and block-level estimators is expected, as end-to-end prediction must capture additional effects such as kernel scheduling, cross-layer memory interactions, and framework overheads. Nevertheless, high R2R^2 values indicate accurate modeling of performance trends across diverse architectures. Furthermore, while memory optimizations like PageAttention (kwon2023efficientmemorymanagementlarge, ) and FlashAttention (dao2022flashattentionfastmemoryefficientexact, ; dao2023flashattention2fasterattentionbetter, ; shah2024flashattention3fastaccurateattention, ) alter memory layout and improve utilization to enable larger batch sizes, they do not fundamentally change the computational workload. Because batch size is explicitly modeled as an input feature in HYMELL, our framework naturally captures the performance and energy implications of these optimizations without requiring changes to the underlying estimators. While our evaluation focuses on uniform batch configurations, modern serving engines employ continuous batching with heterogeneous sequence lengths. In such settings, multiple requests are processed jointly within shared kernels, leading to non-trivial interactions in computation and memory access patterns. Since HYMELL models operator costs as a function of input dimensions, it can be extended to heterogeneous batches by evaluating the effective aggregated workload at the kernel level rather than simply summing independent requests. This allows the model to approximate the combined execution cost while accounting for shared computation and improved hardware utilization. Table 6. End-to-end LLM prediction accuracy. Time Energy Configuration MAPE (%) R2R^2 MAPE (%) R2R^2 Prefill (Dense) 10.179 0.9938 9.969 0.9860 Decode (Dense) 8.194 0.8968 10.771 0.8117 Prefill (MoE) 12.772 0.9527 13.091 0.9477 Decode (MoE) 12.776 0.9221 13.305 0.9101 Average 10.980 0.9414 11.784 0.9139 Table 7. HYMELL per-sequence accuracy compared with AMALI’s reported aggregate MAPE on LLaMA 3 8B with batch size 1. Mode Seq Len Time Error (%) Energy Error (%) AMALI HYMELL AMALI HYMELL Prefill 256 15.56 5.65 – 2.60 1024 15.56 4.44 – 4.03 2048 15.56 1.60 – 1.67 4096 15.56 5.09 – 3.38 MAPE 15.56 4.19 – 2.92 Decode 256 34.90 1.64 – 2.04 1024 34.90 1.16 – 1.22 2048 34.90 1.79 – 1.09 4096 34.90 1.71 – 1.00 MAPE 34.90 1.58 – 1.34 Table 7 reports HYMELL’s per-sequence prediction error for LLaMA 3 8B with batch size 1 and compares it with AMALI’s reported aggregate MAPE for prefill and decode. AMALI reports cycle-prediction MAPE of 15.56%15.56\% for prefill and 34.90%34.90\% for decode, but does not provide per-sequence errors or energy estimates. Therefore, the AMALI values are repeated across sequence lengths only as an aggregate reference baseline, not as per-sequence measurements. In contrast, HYMELL predicts measured wall-clock latency and energy for each sequence length, achieving 4.19%4.19\% time MAPE and 2.92%2.92\% energy MAPE for prefill, and 1.58%1.58\% time MAPE and 1.34%1.34\% energy MAPE for decode. Unlike AMALI’s analytical cycle-level model, HYMELL predicts measured hardware latency and energy, capturing runtime overheads, scheduling effects, and frequency behavior while also enabling system-level energy prediction. Figure 4. Normalized prediction accuracy (batch=1). Top: time (ground truth normalized to 1). Bottom: energy (ground truth normalized to 1). Figure 4 presents the normalized prediction accuracy of HYMELL for execution time and energy across three representative models—LLaMA 3 8B (grattafiori2024llama, ), Mistral 7B (jiang2023mistral7b, ), and Qwen 1.5 2.7B MoE (qwen, ) (2.7B active parameters)—over different sequence lengths with batch size 1. Results are shown for both prefill and decode stages, where ground-truth values are normalized to one, and predictions are plotted relative to them. Across all configurations, predictions closely match the ground truth, with errors below 5%5\%, demonstrating that HYMELL accurately captures both compute- and memory-dominated behaviors. The consistent accuracy across diverse architectures and workloads highlights the robustness and generalizability of the proposed model. 5.3. Generalization and Extensibility To further validate that HYMELL is not tied to a single hardware platform, we repeated the model-level evaluation on an NVIDIA RTX A6000 GPU. This experiment serves as a cross-device ablation: the same hierarchical modeling flow is retained, while the hardware-specific profiling data and learned coefficients are updated for the A6000. As shown in Table 8, HYMELL achieves strong accuracy on the new device for both dense and MoE models, with average MAPE values of 8.141%8.141\% for time and 8.622%8.622\% for energy. These results demonstrate that the proposed analytical–ML decomposition transfers well across GPU generations once the target hardware is profiled, supporting the hardware-portable design of the framework. Table 8. Cross-device ablation on an NVIDIA RTX A6000 GPU. Time Energy Configuration MAPE (%) R2R^2 MAPE (%) R2R^2 Prefill (Dense) 9.102 0.9814 8.425 0.9805 Decode (Dense) 6.718 0.9742 7.342 0.9747 Prefill (MoE) 9.366 0.9734 9.367 0.9728 Decode (MoE) 7.379 0.9813 9.352 0.9379 Average 8.141 0.9776 8.622 0.9665 We also evaluate HYMELL on a linear-attention variant to test architectural extensibility beyond the softmax-based attention blocks used in most current LLMs. Although linear attention is not the dominant attention mechanism in contemporary LLM deployments, it is an important example of a structurally different attention formulation. Table 9 shows that HYMELL can estimate both latency and energy for this variant with low error, achieving average MAPE values of 1.910%1.910\% for time and 2.278%2.278\% for energy. This confirms that new attention mechanisms can be incorporated by adding the corresponding block-level profiling data and estimator, without changing the overall multi-level modeling framework. Table 9. Architectural ablation on linear attention. Time Energy Configuration MAPE (%) R2R^2 MAPE (%) R2R^2 Prefill 3.360 0.9980 3.740 0.9958 Decode 0.460 0.9999 0.815 0.9999 Average 1.910 0.9990 2.278 0.9979 Currently, HYMELL focuses on single-GPU inference execution. However, the framework’s block-level modularity naturally lends itself to multi-GPU scaling. To model large-scale models that require tensor parallelism (TP), the framework can be extended by introducing an additional analytical estimator for collective communication primitives and by incorporating network bandwidth as a global feature in the end-to-end MLP predictor. 6. CONCLUSIONS We introduced HYMELL, a hybrid multi-level framework for predicting LLM inference latency and energy consumption. By combining regime-aware analytical modeling for GPU operators (GEMM, Softmax, RMSNorm) with machine learning predictors for higher-level components and full-system execution, HYMELL bridges low-level hardware behavior and high-level architecture. Evaluated on NVIDIA H100 GPUs, HYMELL achieves high predictive accuracy (MAPE of 1.9%–6.3% at block levels and less than 5% for Llama3, Mistral, and Qwen models) across diverse configurations, including GQA and MoE paradigms in both prefill and decode phases. By predicting execution costs directly from architectural parameters, HYMELL enables rapid, hardware-free design space exploration, serving as an adaptable foundation for energy-efficient AI deployment. References (1) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. (2) William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. (3) Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019. (4) Alexandra Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of AI deployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 85–99, 2024. (5) David Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021. (6) Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016. (7) Seyedeh Atiyeh Abbasi Jalal, Ragib Ahsan, Zezhi Wu, Mirbehrad Mousavi, and Rehan Kapadia. Integration of oscillator-based feature extraction for energy-efficient convolutional neural networks. Journal of Applied Physics, 139(23), 2026. (8) Marissa J Strumolo, Zhaohong Sun, Iliya Larimojarad, Rehan Kapadia, and Richard L Brutchey. Data driven control of defect formation in solution deposited sb2se3 thin films. Advanced Materials Interfaces, 12(24):e00798, 2025. (9) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), 2023. (10) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2022. (11) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, 2022. (12) Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. (13) Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning (ICML), 2023. (14) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In ACM SIGOPS Symposium on Operating Systems Principles (SOSP), 2023. (15) Mark Chen, Jerry Tworek, Heewoo Jun, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. (16) Armin Abdollahi, Mehdi Kamal, and Massoud Pedram. Unified LLM model for power, performance, and area prediction from hardware code. In Ninth Conference on Machine Learning and Systems, 2026. (17) Armin Abdollahi, Saeid Shokoufa, Negin Ashrafi, Mehdi Kamal, and Massoud Pedram. Hdlforge: A two-stage multi-agent framework for efficient verilog code generation with adaptive model escalation, 2026. (18) Alireza Golkarieh, Parsa Razmara, Ahmadreza Lagzian, Amirhosein Dolatabadi, and Seyed Jalaleddin Mousavirad. Semi-supervised gan with hybrid regularization and evolutionary hyperparameter tuning for accurate melanoma detection. Scientific Reports, 15(1):31977, 2025. (19) Arya Fayyazi, Mehdi Kamal, and Massoud Pedram. Coft: Counterfactual-conformal decoding for fair chain-of-thought reasoning in large language models. arXiv preprint arXiv:2605.30641, 2026. (20) Tina Khezresmaeilzadeh, Jiang Zhang, Dimitrios Andreadis, and Konstantinos Psounis. Preserving privacy and utility in llm-based product recommendations. arXiv preprint arXiv:2505.00951, 2025. (21) Karan Singhal, Shekoofeh Azizi, Tao Tu, et al. Large language models encode clinical knowledge. Nature, 620:172–180, 2023. (22) Shijie Wu, Ozan Irsoy, Steven Lu, et al. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023. (23) Neel Guha, Julian Nyarko, Daniel E. Ho, et al. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems, volume 36, 2023. (24) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, et al. ChatGPT for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274, 2023. (25) Shiheng Cao, Junmin Wu, Junshi Chen, Hong An, and Zhibin Yu. Amali: An analytical model for accurately modeling llm inference on modern gpus. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pages 1495–1508, 2025. (26) Rajeev Patwari, Ashish Sirasao, and Devleena Das. Forecasting llm inference performance via hardware-agnostic analytical modeling, 2025. (27) Hiari Pizzini Cavagna, Andrea Proia, Giacomo Madella, Giovanni B. Esposito, Francesco Antici, Daniele Cesarini, Zeynep Kiziltan, and Andrea Bartolini. Sweetspot: An analytical model for predicting energy efficiency of llm inference, 2026. (28) Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas, Vishal Banwari, Paul Lukowicz, and Jakob Karolus. This is taking too long – investigating time as a proxy for energy consumption of llms, 2026. (29) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. (30) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. (31) Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024. (32) Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. (33) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. (34) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.