Paper deep dive
SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions
Feiyang Chen, Haibo Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/20/2026, 8:15:41 AM
Summary
SMEPilot is an LLM inference engine designed to optimize execution on CPUs with Arm Scalable Matrix Extension (SME). It addresses the mismatch between SME units and conventional CPU cores through a unified roofline-based model that identifies three performance regimes: memory-bound, backend-asymmetric ridge-crossing, and compute-bound. SMEPilot employs three key techniques to bridge execution gaps: tile-level work partitioning to maximize spatial utilization, phase-aware pipeline execution to reduce temporal bubbles in attention mechanisms, and a layout-aware runtime to maintain data locality. The engine achieves up to 3.94x end-to-end speedup over llama.cpp across various platforms (Phone, PC, Server) and models like Llama-3.2-3B and Qwen3.
Entities (9)
Relation Signals (4)
SMEPilot → improvesperformanceof → Llama-3.2-3B
confidence 100% · Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B... SMEPilot improves end-to-end inference performance by up to 3.94x.
SMEPilot → outperforms → llama.cpp
confidence 100% · SMEPilot achieves up to 3.94× end-to-end speedup over llama.cpp [16]
Apple M4 → supports → Arm Scalable Matrix Extension
confidence 100% · Apple’s M4 was the first publicly available chip reported to support SME
SMEPilot → optimizes → Arm Scalable Matrix Extension
confidence 95% · SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each operator shape.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide high-throughput matrix execution within the CPU. For LLM inference, however, these units are not a universal replacement for conventional CPU cores: prefill, decode, attention, and KV-cache operations expose different arithmetic intensities, vector behavior, and layout requirements, while SME units and CPU cores still compete for shared memory bandwidth. This paper studies this mismatch through a roofline-based characterization of SME-enabled CPUs and uses the resulting model to guide operator-level execution choices. We present SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each operator shape. SMEPilot partitions matrix work across SME and CPU cores at tile granularity, overlaps SME-suitable matrix stages with CPU-suitable vector stages in attention, and maintains layout state so packed tensor representations are reused rather than repeatedly rebuilt on critical paths. Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B on phone, PC, and server platforms, SMEPilot improves end-to-end inference performance by up to 3.94$\times$.
Tags
Links
- Source: https://arxiv.org/abs/2606.16332v1
- Canonical: https://arxiv.org/abs/2606.16332v1
Trouble viewing inline? Open PDF directly →
Full Text
66,330 characters extracted from source content.
Expand or collapse full text
SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions Feiyang Chen, Haibo Chen IPADS, Shanghai Jiao Tong University Abstract Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide high- throughput matrix execution within the CPU. For LLM infer- ence, however, these units are not a universal replacement for conventional CPU cores: prefill, decode, attention, and KV-cache operations expose different arithmetic intensities, vector behavior, and layout requirements, while SME units and CPU cores still compete for shared memory bandwidth. This paper studies this mismatch through a roofline-based characterization of SME-enabled CPUs and uses the resulting model to guide operator-level execution choices. We present SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each op- erator shape. SMEPilot partitions matrix work across SME and CPU cores at tile granularity, overlaps SME-suitable matrix stages with CPU-suitable vector stages in attention, and maintains layout state so packed tensor representations are reused rather than repeatedly rebuilt on critical paths. Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B on phone, PC, and server platforms, SMEPilot improves end-to- end inference performance by up to 3.94×. 1 Introduction The rapid progress of large language models (LLMs) has made generative AI a common component of interactive ap- plications, from chat assistants and AI agents to multimodal generation [25,27,41]. As these applications span platforms from edge devices to data centers, CPUs continue to play a critical role in executing diverse AI tasks owing to their widespread availability and cost efficiency [16, 35, 36]. At the same time, the CPU itself is evolving for LLMs. Modern CPUs are adding matrix extensions, such as Arm Scalable Matrix Extension (SME) [4], that provide high- throughput matrix instructions inside a conventional CPU package. They add specialized matrix registers and instruc- tions for computing 2D tiled matrix operations, such as outer products, directly on the CPU. These extensions promise a middle ground between GPU-style accelerators and ordinary vector cores: they expose more matrix throughput without requiring the application to leave the CPU, reusing its easy and mature ecosystem. Prior work has studied how to use Scalable Matrix Exten- sions to accelerate specific kernels. For example, ARM SME has been characterized for dense matrix multiplication [12] and sparse matrix multiplication [23]. However, they do not study how to use SME systematically across the entire LLM inference process. In particular, it remains unclear how a runtime should schedule matrix extensions and CPU vector cores across inference phases such as prefill and decode, and across operators such as FFN, MoE and attention. This paper presents the first comprehensive characteri- zation on CPUs with scalable matrix extensions for LLM inference. To reason about when SME units should be used, we build a unified roofline-based model that captures both the distinct computing throughput of SME and CPU cores and their shared memory bandwidth. For each LLM operator and shape, the model estimates whether execu- tion is compute-bound, memory-bound, or in a backend- asymmetric ridge-crossing region, and uses this classification to predict whether the operator should run on SME units, CPU cores, or both. We identify that treating SME as a drop-in replacement for CPU matrix kernels is insufficient. There are two primary reasons for this. First, for large matrix-heavy operators in prefill phases, SME and CPU cores can both contribute to throughput, and using only one backend leaves useful hard- ware idle. Second, different models and input requests can shift an operator between SME-friendly and CPU-friendly regimes, which causes a single fixed kernel replacement to underperform across the inference process. To address this problem, we introduce SMEPilot, an ac- celerated inference engine for CPUs with scalable matrix extensions. SMEPilot provides a unified framework that schedules SME and CPU cores across prefill and decode phases, and across operators with different shapes and hardware affinities. For each inference request, SMEPilot uses the roofline classification to generate an execution plan that schedules CPU-only, SME-only, or mixed SME+CPU execution for each operator. After the execution plan is generated, SMEPilot provides a performant runtime to execute the plan on SME units and CPU cores. However, attaining high performance requires more than choosing the right plan: the runtime must preserve parallelism, hardware utilization, and data locality while executing the plan. We identify three execution gaps that prevent the roofline-guided plan from realizing its predicted benefit. •Spatial utilization gap. A policy that assigns each operator entirely to a single execution target can leave useful hardware idle. Large GEMM operators expose 1 arXiv:2606.16332v1 [cs.DC] 15 Jun 2026 enough arithmetic intensity to benefit from both SME and CPU cores, but assigning the entire operator to only one side wastes the throughput of the other. • Temporal bubble gap. LLM inference operators, especially attention, interleave phases with different hardware affinity. Matrix multiplication such as푄퐾 ⊤ can benefit from SME, while softmax, masking, reduc- tions, and other vector-heavy phases are better suited to CPU cores. If these phases are executed sequentially at coarse operator boundaries, one hardware unit waits while the other runs, creating bubbles that are invisible to a per-operator roofline model. • Layout compatibility gap. SME kernels require packed, tile-friendly matrix layouts to expose their throughput, whereas the rest of the LLM pipeline often consumes and produces conventional tensor layouts. Naively packing inputs before every SME invocation adds memory traffic and disrupts data locality, reducing the effective arithmetic intensity assumed by the roofline model. To bridge these gaps between the roofline-guided plan and runtime execution, SMEPilot adopts three techniques. • Tile-level work partitioning. To bridge the spa- tial utilization gap, SMEPilot partitions matrix work across SME and CPU cores at tile granularity. This allows both backends to contribute within the same operator while respecting the tile shapes required by their optimized kernels. • Phase-aware pipeline execution. To bridge the temporal bubble gap, SMEPilot pipelines SME-friendly matrix phases with CPU-friendly vector phases. This lets attention phases such as matrix multiplication and softmax/reduction overlap when their dependencies allow, reducing idle time on both sides. • Layout-aware runtime. To bridge the layout compat- ibility gap, SMEPilot tracks tensor layouts as runtime state and performs layout-aware graph conversion. Static weights are packed once off the request criti- cal path, while online activations such as KV-cache entries are packed at the producer side when down- stream SME consumers can reuse them. This avoids repeated packing and wasted memory bandwidth on SME critical paths. SMEPilot delivers state-of-the-art performance on Phone, PC, and server CPU platforms. In end-to-end evaluations, SMEPilot achieves up to 3.94×end-to-end speedup over llama.cpp [16] across the evaluated device-model-workload configurations, including both dense models and MoE mod- els. For prefill phases, SMEPilot improves FFN GEMM by 1.42–1.67×over highly optimized Apple Accelerate [3](SME) and 6.06–7.43×over GGML [15](CPU), while improves attention by 1.56–3.50×over a FlashAttention CPU [11]. These gains come from SMEPilot’s ability to use SME and CPU cores together for large matrix-heavy prefill operators. For decode phases, SMEPilot achieves up to 3.48×speedup over CPU-only inference. This demonstrates that SMEPilot can still improve hardware utilization in regimes with less computation and stronger memory bandwidth pressure. 2 Preliminaries 2.1 LLM Inference Modern LLMs are commonly built from Transformer blocks, where each layer alternates between attention and feed- forward network (FFN) computation [34]. From a system perspective, these blocks contain a mixture of operator types. Both of the attention and FFN blocks contains two kinds of operators: matrix operations such as matrix multiplication, and non-linear operations such as softmax, normalization, and activation functions. This mixture makes CPU matrix extensions promising for LLM inference, because they can accelerate the matrix-heavy parts, but also non-trivial to use, because not every phase benefits from matrix throughput. LLM inference also has two execution phases with dif- ferent shapes: prefill and decode. During prefill, the model processes all prompt tokens at once and produces KV- cache entries for them [2,22,42]. The token dimension is therefore large, which increases arithmetic intensity and exposes substantial matrix parallelism in attention and FFN operators. During decode, the model usually processes one new token per step, or a small number of tokens when using batching [39] or multi-token prediction [9,17]. These operators are therefore more memory-bound: they read large weight and KV-cache matrices but perform little computation per request [28]. Therefore, during LLM inference, differ- ent computation characteristics occurs due to the different phases and different operators, which poses challenges for utilizing matrix extensions efficiently. 2.2 Scalable Matrix Extension The rising demands of AI workloads on both cloud and edge platforms have led to matrix extensions being added to modern CPU architectures. Arm Scalable Matrix Extension (SME) is an Armv9-A architecture extension that adds matrix- oriented execution support to the CPU [4,5]. SME introduces a specialized outer-product instruction and a new matrix register array storage, significantly increasing the matrix throughput of Arm CPUs [12, 29]. SME has been supported in a range of CPUs. Apple’s M4 was the first publicly available chip reported to support SME, and M4-family CPUs are now used across Apple mobile and laptop platforms [29]. Recent flagship mobile Arm platforms are also adopting SME. For example, MediaTek’s Dimensity 9500 introduces Armv9.3 support with SME2 instructions, and Qualcomm Snapdragon 8 Elite Gen 5 uses third-generation Oryon CPU cores that also support SME. 2 Core 0 Cluster Interface SME Unit SME Unit Processing Grid Core 1 Core 2 Core 3 Z vector reg ister ZAmatrixregister Z v ector reg ister Fig. 1. The hardware design of CPU with SME. Left: an CPU cluster attaches ordinary CPU cores and an SME unit. Right: the SME unit executes outer products instruction from Z vector registers and output to ZA matrix registers. On the cloud side, Arm-based server CPUs such as newer Kunpeng 920 also support SME. 3 Roofline Characterization of SME-enabled CPUs Scalable Matrix Extensions reshape the performance struc- ture of CPU inference. In order to systematically analyze the performance of different LLM operators on the CPU with SME, we design a unified roofline model that can capture the performance characteristics of both SME and CPU cores. Based on this roofline model, we characterize the performance of different LLM operators on the CPU with SME, and derive the design principles for SMEPilot. 3.1 SME and CPU Cores: Additive Compute, Shared Bandwidth Although SME instructions are exposed through the CPU programming model, the underlying execution resources are distinct from ordinary CPU vector cores, as shown in Figure 1. SME provides specialized matrix state and outer-product instructions for high-throughput tiled matrix computation, while CPU cores continue to provide general-purpose scalar and vector execution. Two properties determine how these resources should be used for LLM inference. Near-additive matrix throughput. For matrix-heavy operators with enough independent work, SME units and CPU cores can both contribute useful computation. As shown in Figure 3, two SME workers alone reach 2.92 TFLOP/s on FP16 matrix work and eight CPU-core workers reach 1.85 TFLOP/s. Running them together reaches 4.27 TFLOP/s, which is close to the sum of the individual throughputs. This indicates that SME and CPU cores should not be treated as mutually exclusive backends for large matrix operators. Shared memory bandwidth. The same additivity does not hold for memory traffic. SME units and CPU cores share the cache hierarchy and memory controller. As shown in Figure 3, CPU cores reach 247 GB/s read bandwidth, SME reaches 224 GB/s, and mixed SME+CPU execution does not increase the memory roof. All three configurations sustain similar write bandwidth. Thus, adding SME raises the compute ceiling but does not raise the memory-bandwidth roof. These two properties make SME-enabled CPUs differ- ent from both homogeneous multicore CPUs and discrete accelerators. Compute resources can be added within one operator, but memory bandwidth remains coupled across them. 3.2 Multi-Ceiling Roofline Model The coupled structure of SME and CPU cores creates a placement problem that a conventional roofline model does not capture. A conventional roofline view can classify whether an operator is compute-bound or memory-bound on a single hardware target, but LLM inference on SME- enabled CPUs must choose among CPU-only, SME-only, and mixed SME+CPU execution. We therefore build a coupled roofline model with a shared memory-bandwidth roof and multiple compute ceilings for CPU cores, SME units, and mixed SME+CPU execution. For an operator표, let푊 표 denote its floating-point work and푄 표 denote its useful memory traffic. Its arith- metic intensity is퐼 표 = 푊 표 /푄 표 . For each execution target ℎ ∈ CPU, SME, MIX, let푃 ℎ denote the measured com- pute throughput. The mixed target represents concurrent SME+CPU execution and has a compute ceiling close to 푃 SME +푃 CPU . Since all targets share the same memory system, they use a common memory bandwidth 퐵. The attainable performance of target ℎ is modeled as 푅 ℎ (표)= min(푃 ℎ ,퐵퐼 표 ). The ridge point of target ℎ is 퐼 푟푖푑푔푒 ℎ = 푃 ℎ 퐵 . Because푃 CPU < 푃 SME < 푃 MIX while퐵is shared, the model contains one memory roof but multiple compute ceilings. As shown in Figure 2, unlike a traditional roofline model with only one ridge point, this multi-ceiling roofline model partitions the performance space into three regimes: memory-bound, compute-bound, and backend-asymmetric ridge-crossing. These three regimes correspond to three different placement plans for LLM operators on SME-enabled CPUs. Memory-bound regime. Operators below the CPU ridge point are memory-bound on all targets. Since SME does not increase memory bandwidth, moving these operators to SME cannot reduce the dominant cost. This regime includes single-token decode GEMV and decode attention, where the operator reads large weight or KV-cache matrices but performs little computation per request. 3 131030100 100 300 1k 3k 10k Arithmetic intensity (FLOP/Byte) Attainable performance (GFLOP/s) CPU cores SME SME+CPU cores memory-bound ridge-crossing Compute-bound Decode FFN Decodeattention Batch decode GEMM Prefillattention Prefill FFN Aunifiedrooflinemodel InferencePlan CPUcores SME mixed QKLinear UpLinear 4096tokens 1tokens 16tokens Softmax PVlinear Attention Down Linear FFN SwiGLU QKLinear UpLinear Softmax PVlinear Attention Down Linear FFN SwiGLU QKLinear UpLinear Softmax PVlinear Attention Down Linear FFN SwiGLU outout out Batch decode Attention Fig. 2. Left: SME+CPU roofline view of representative LLM operators. CPU cores, SME, and mixed SME+CPU execution share the same memory bandwidth but expose different compute peaks, creating different ridge points: memory-bound, compute-bound and ridge-crossing. Right: The inference plan for different input shapes. Operators are placed into different backends based on their position estimated by the unified roofline model. CPUSMECPU+SME 0 1 2 3 4 5 Throughput (TFLOP/s) (a) Computing throughput CPUSMECPU+SME 0 100 200 300 Bandwidth (GB/s) (b) Memory bandwidth Read bandwidth Write bandwidth Fig. 3. The computing and memory characteristics of SME and CPU cores. SME and CPU cores provide near-additive matrix throughput but share memory bandwidth. CPU execution is preferable for this region because we observe that SME kernels typically has lower memory uti- lization than CPU cores due to higher instruction latency and lack of speculative execution. Backend-asymmetric ridge-crossing regime. Opera- tors between the CPU and SME ridge points are compute- bound on CPU cores but memory-bound on SME. This regime is specific to a multi-ceiling roofline model. SME improves performance in this region not by reaching its peak compute throughput, but by lifting the operator above the CPU compute ceiling until it approaches the shared memory bandwidth. Batched decode GEMM and some grouped-query attention shapes fall into this region. 50 100 150 SMECPU M4ProMediaTek Dimensity 9500 0 5 10 15 20 Latency (ms) Fig. 4. SME has lower throughput for non-linear vector operations. For a 4096×4096 fastexp kernel, CPU cores are faster than SME on both Apple M4 Pro and MediaTek Dimensity 9500. SME-only execution is preferable for this region. These op- erators favor SME execution for higher computing through- put, but mixed SME+CPU execution provides limited ad- ditional benefit because memory bandwidth becomes the bottleneck. Compute-bound regime. Operators above the SME ridge point are compute-bound on both SME and CPU cores. In this region, the memory bandwidth is no longer the limiting resource, so the near-additive compute throughput of SME and CPU cores can be exploited. Large prefill FFN and attention matrix phases fall into this region. These operators benefit from mixed SME+CPU execution, provided that the runtime can utilize parallelism within the operator. Non-linear operations. Except for the linear operations, LLM inference also contains non-linear operations such as 4 softmax, normalization and activation. We always place these non-linear operations on CPU cores, because SME has lower throughput for these operations than CPU cores. As shown in Figure 4, for a 4096× 4096 fastexp kernel, SME is 4.10×slower than CPU cores on Apple M4 Pro and 3.26×slower on MediaTek Dimensity 9500 for this non-linear kernel. This is because SME has low throughput and higher latency for SIMD instructions, and non-linear operations often compose of a long sequence of SIMD instructions with strong data dependencies. To summarize, as shown in the right panel of Figure 2, this unified roofline model characterizes LLM operators across different performance regimes and guides the general placement strategy for SME-enabled CPUs. 3.3 Scheduling Insights from the Roofline Analysis The coupled roofline model not only provides placement plans for different operators, but also reveals two general insights for the runtime of SMEPilot. Insight 1: Mixed execution requires intra-operator scheduling. Operator-level placement is too coarse for SME- enabled CPUs. For operators in compute bound regime, the coupled model predicts that the best target is mixed SME+CPU execution. However, assigning the whole operator to either backend cannot realize this bound: CPU-only exe- cution misses SME throughput, while SME-only execution leaves ordinary CPU cores idle. Reaching the mixed ceiling therefore requires the runtime to explore parallelism within one operator and achieve high utilization for both hardware sides. Insight 2: Shape-dependent placement requires dy- namic scheduling. The same operator can occupy different regions of the coupled roofline as the request shape changes. For example, an FFN layer may be memory-bound during single-token decode, ridge-crossing under batched decode, and compute-bound during long-context prefill. Attention similarly shifts with sequence length, batch size, head config- uration, and KV-cache length. Therefore, we cannot attach a fixed backend choice to each operator. Instead, the runtime must generate a shape-aware execution plan online, select- ing CPU-only, SME-only, or mixed SME+CPU execution according to the operator’s current arithmetic intensity and available intra-operator parallelism. These two insights distinguish SMEPilot from a drop-in SME kernel replacement. SMEPilot does not simply decide whether an operator should use SME; it decides which tiles should use SME and how that decision changes with the current request shape. 퐴푐푡푖푣푎푡푖표푛 tokens SME CPU cores 푤푒푖푔ℎ푡 푇 SME CPU cores 퐴푐푡푖푣푎푡푖표푛 tokens SME CPU cores 푤푒푖푔ℎ푡 푇 SME CPU cores Activationpartition Weightpartition Fig. 5. Tile partition closes the spatial utilization gap for compute-bound matrix operators. The planner uses mea- sured SME and CPU rates, together with the worker budget and contention guard, to assign tiles so that both backends finish at similar times. 4 SMEPilot Design The characterization in Section 3 identifies the placement regimes for different operators. Based on this characteriza- tion, SMEPilot designs a unified runtime that turns roofline- guided placement into a concrete, performant execution plan. To realize intra-operator mixed SME+CPU execution with high hardware utilization, we design two techniques for FFN and attention operators: tile-level work partitioning (Sec- tion 4.1) and phase-aware pipeline execution (Section 4.2). To support dynamic scheduling while preserving data locality, we design a layout-aware runtime (Section 4.3) and an online planner (Section 4.4). 4.1 Tile-level Work Partitioning Problem: spatial utilization gap. Large prefill GEMMs and FFN layers expose enough independent matrix work to use both SME and CPU cores. However, a whole-operator decision can choose only one backend, while a naive split can leave the faster backend waiting for the slower one. We need a partition plan that achieve high utilization for both SME and CPU cores. Technique: partition independent output tiles. SMEPilot partitions a GEMM operator into output tiles and assigns a subset of tiles to SME workers and the remaining tiles to CPU-core workers. For a matrix multiplication퐶 푀×푁 = 퐴 푀×퐾 퐵 퐾×푁 , SMEPilot partitions only the output dimensions of퐶. This choice keeps different workers independent: each worker computes a disjoint set of output tiles, so there is no cross-worker reduction or synchronization. In contrast, partitioning the reduction dimension퐾would require partial 5 sums from different workers to be merged, adding synchro- nization and memory traffic. There are two natural output-dimension partition strate- gies, as shown in Figure 5. A token-dimension partition splits the푀dimension, so different workers process dif- ferent token rows. An output-channel partition splits the푁 dimension, so different workers compute different output channels. SMEPilot chooses the partition dimension based on the operator shape. When the token dimension is larger, it partitions along푀; when the output-channel dimension is larger, it partitions along푁. Splitting the larger output dimension avoids highly elongated regions, which improves cache locality, and makes it easier to keep each worker’s region aligned with the native tile shapes of the SME and CPU kernels. The partition ratio is determined by the online planner(describe in Section 4.4). MoE expert partitioning. Mixture-of-Experts (MoE) layers introduce another form of uneven matrix work. After the router assigns tokens to experts, the number of tokens received by each expert is often skewed: some experts receive only a few tokens, while others receive many. SMEPilot therefore treats each expert FFN as a separate matrix operator whose token dimension is determined by the routed token count. Experts with only a small number of tokens are kept on CPU cores, because their thin matrix shapes are memory-bound and cannot amortize SME setup, packing, and tile-alignment overheads. Experts with many tokens are classified by the roofline model as larger GEMM operators and can use SME, or mixed SME+CPU execution, when their arithmetic intensity and tile count are sufficient. This expert- level decision avoids forcing the same backend choice on both cold and hot experts in the same MoE layer. Tile-level work partitioning is used for compute-bound operators such as prefill FFN layers and large attention matrix phases. For memory-bound decode GEMV-like opera- tors, SMEPilot avoids mixed partitioning unless the effective token width becomes large enough to expose useful parallel matrix work. 4.2 Phase-aware Pipeline Execution Problem: temporal bubble gap. Attention is not a single homogeneous matrix operator. It contains matrix-heavy phases, such as푄퐾 ⊤ and푃푉, and vector-heavy phases, such as masking, softmax, and reductions. These phases have different hardware affinity: the matrix phases can benefit from SME throughput, whereas the vector-heavy phases are better executed by CPU cores. A coarse phase-level schedule therefore creates temporal bubbles. If the runtime waits for 푄퐾 ⊤ to finish before starting softmax, CPU cores sit idle during the matrix phase; if it waits for softmax work before starting푃푉, SME sits idle during the vector phase. These idle periods move real execution away from the performance predicted by the roofline model. t0t1t2t3t4 SME QK^T CPU cores softmax SME PV QK 0 Softmax 0 PV 0 QK 1 QK 2 Softmax 1 Softmax 2 PV 1PV 2 Fig. 6. Pipeline execution closes the temporal bubble gap in attention. Each tile moves through SME-friendly푄퐾 ⊤ , cores-friendly softmax, and SME-friendly푃푉stages. Tile- level dependencies allow different tiles to occupy different hardware-specialized stages instead of forcing an operator- wide barrier after each phase. Existing fused attention algorithms, such as FlashAtten- tion [11], reduce memory traffic by computing softmax on- line and avoiding materialized attention matrices. However, their primary goal is memory efficiency; they do not by themselves address the hardware-affinity mismatch between SME-friendly matrix phases and CPU-friendly vector phases on SME-enabled CPUs. Technique: inter-phase pipelining. SMEPilot addresses this gap with inter-phase pipelining. Instead of executing attention as three operator-wide phases, SMEPilot allows different attention tiles to occupy different phases at the same time. Each tile still follows the dependency order 푄퐾 ⊤ →softmax→ 푃푉, but independent tiles can advance through the pipeline concurrently. As shown in Figure 6, the two matrix phases are assigned to SME workers, while the softmax phase is assigned to CPU-core workers. Once the푄퐾 ⊤ result for a tile is available, its softmax stage can start without waiting for the entire operator to finish the 푄퐾 ⊤ phase. This allows SME to compute the matrix phase of later tiles while CPU cores process the vector phase of earlier tiles. Tile sizing. The tile size controls the effectiveness of the pipeline. Very small tiles expose more overlap opportunities, but they increase scheduling overhead and reduce kernel efficiency. Very large tiles improve per-kernel efficiency, but they increase pipeline prologue and epilogue time and make the stage times harder to balance. SMEPilot therefore chooses tile sizes using the offline profile and online planning: it estimates the SME time for the matrix stages and the CPU- core time for the softmax stage based on the current request shape and the profiled SME and CPU throughputs, then selects a tile shape whose stage times are small enough to reduce long stalls while still preserving kernel-aligned matrix tiles. Mask-aware matrix phases. Attention masks introduce another source of wasted work in the matrix stages of the pipeline. Causal and sliding-window masks can make entire score blocks irrelevant, but a conventional GEMM 6 still computes these masked elements before discarding them. SMEPilot uses mask-aware matrix kernels inside the pipeline to skip tiles that are fully masked. For푄퐾 ⊤ , the mask determines which score blocks are produced; for푃푉, the same mask determines which score blocks participate in the reduction. This optimization reduces the amount of matrix work entering the pipeline without changing the tile-level dependency structure. 4.3 Layout-aware Runtime Problem: layout compatibility gap. SME kernels prefer packed, tile-friendly layouts, but LLM runtimes often store tensors in standard row-major or framework-native layouts. A naive runtime therefore packs inputs immediately before each SME kernel. This conversion adds memory traffic around the useful matrix computation and places layout conversion on the critical path. For decode FFN and attention, layout conversion can take a similar amount of time as the SME kernel itself, substantially reducing the net benefit of SME execution. Technique: layout-aware graph conversion. To reduce this overhead, SMEPilot tracks tensor layout as part of the runtime state. Each task declares which layouts it can consume and which layout it produces. The runtime then inserts conversions only when a consumer requires a layout that the producer does not already provide. SMEPilot uses two strategies to keep repeated packing off SME critical paths. Off-critical-path packing for model weights. Model weights are static during inference, so their layout can be decided before any request executes. If the runtime determines that a weight tensor will be consumed by SME kernels, SMEPilot packs it once at model load time and stores the packed copy for reuse. This avoids repacking the same weights for every request and removes weight conversion from the request critical path. Producer-side packing for activation tensors. Activation tensors are generated online and cannot be packed at model load time. For these tensors, SMEPilot moves conversion to the producer side when the downstream consumer is SME. This removes an extra memory copy from the consumer path and, for reused activations such as KV-cache entries, avoids repeatedly unpacking the same data across attention invocations. For example, the key and value tensors produced by the K-projection and V-projection are written directly into a packed KV-cache layout. As shown in Figure 7, a standard CPU KV cache stores each token as a contiguous row, while SME attention prefers layouts that group tokens and feature elements according to the SME vector length and kernel tile shape. The runtime therefore stores K and V cache entries in separate SME-friendly layouts. Subsequent attention calls can then consume the packed KV cache without repacking all previous entries on every invocation. SME Kpacked SMEK Cache SME Vector length token 0 token 1 token 2 token 3 token 4 token 5 token 6 token 7 SME Vector length SME K packed token 0 token 1 token 2 token 3 token 4 token 5 token 6 token 7 SMEV Cache token 0 token 1 token 2 token 3 token 4 token 5 token 6 token 7 CPUKV Cache Fig. 7. Producer-side packing for SME-friendly KV cache layout. A standard CPU KV cache stores each token contigu- ously in row-major order. SMEPilot stores K and V entries in SME-specific packed layouts at production time, grouping elements according to the SME vector length and the K- packed tile shape required by downstream attention kernels. Offline Profiler Attention SME CPU cores z Profiling Result Online Planner Model&Req uest analysis Execution plan Inference Runtime for SME-enabled CPU Packed Weight Weight ... Packed KVCache QKVproj partition pipeline Planning CPU SME mixed Operator Placement Thread allocation Partition Shape Measured performance Operators FFN/MOE Up FFN/MOE Down Profiling Result Execution Plan Fig. 8. The overall architecture of SMEPilot. The offline profiler runs representative operators on SME and CPU cores to collect hardware-specific performance metrics. The online planner combines the profiling result with model and request analysis to choose operator placement, thread allocation, and partition shape. The resulting plan drives the inference runtime, including the partition, pipeline, and packed-layout mechanisms used by different LLM operators. 4.4 Execution Plan Generation Putting everything together, Figure 8 shows that SMEPilot generates an execution plan in two stages. First, an offline profiler measures the hardware capability of the target SME- enabled CPU. Second, an online planner combines these measurements with the current request shape and the given model to produce a tile-level execution plan. The planner is invoked at operator granularity, but the generated plan is executed at tile granularity so that one operator can use SME, CPU cores, or both. At runtime, the plan selects the partitioned, pipelined, and packed-layout mechanisms used by QKV projection, attention, and FFN/MoE operators. 7 Performance Profiler The profiler executes a set of operator kernels on CPU cores and SME to measure the performance metrics needed by the roofline-model analysis in Section 3. The performance metrics include: (1) SME metrics: the SME unit count, matrix throughput, vector throughput (2) CPU metrics: the CPU core count, per-core throughput for matrix and vector work. (3) Memory metrics: the memory bandwidth and the cache size. These metrics are only related to the hardware and are independent of the model and the request, so they can be measured offline once and reused for different requests and models. Online Planner The online planner determines the op- timal execution plan for each operator in the computation graph, given the current request and the model. It contains two processes. The first is operator placement: for each operator, the planner decides whether it should run on SME only, CPU cores only, or a mixed SME+CPU configuration. The second is tile-plan construction: the planner decides the thread counts and the partition size. Roofline-based operator placement. For an operator표, the planner first obtains its work푊 표 , useful memory traffic 푄 표 , and shape parameters from the operator graph and request state. It then evaluates the roofline lower bound in Section 3 for three execution targets: SME-only, CPU-only, and mixed SME+CPU execution. The SME-only target is preferred when the operator has enough arithmetic intensity to amortize SME setup and layout conversion, but does not expose enough independent output tiles to keep CPU work- ers useful. The CPU-only target is preferred for memory- bound or vector-heavy operators, such as narrow decode GEMV and softmax, where SME’s matrix throughput cannot reduce the critical path. The mixed target is preferred when the operator is compute-bound and has enough independent output tiles for both SME and CPU cores. This decision is shape-dependent: the same logical operator may be CPU- only during single-token decode, SME-only for a moderate- width decode shape, and mixed SME+CPU during long- context prefill. Thread allocation. After operator placement, the planner chooses the number of SME and CPU threads for the selected target. SMEPilot uses a heuristic rather than an exhaustive search because the choice is hardware-dependent, and the same request must be planned quickly. The heuristic first tries to cover the hardware units selected by the place- ment decision. It then checks whether the operator exposes enough kernel-aligned tile regions for the candidate threads. A candidate thread is dropped if its assigned region would be smaller than the native kernel tile or would force a non- aligned boundary. This rule avoids a common failure mode of over-partitioning: adding threads increases scheduling overhead and may break the tile shapes that the SME and CPU kernels were optimized for. Partition size. After choosing worker counts, the planner computes the amount of output work assigned to SME and CPU cores. Let푃 eff SME and푃 eff CPU be the effective throughputs measured by the profiler for the selected kernel family and worker counts. The ideal fraction of the output dimension assigned to SME is 휌= 푃 eff SME 푃 eff SME + 푃 eff CPU .(1) The planner first computes the ideal split point휌퐷, where 퐷is the selected output extent. It then rounds this split point to the nearest boundary aligned with both the SME and CPU kernel tile sizes. If several aligned candidates are close to the ideal ratio, the planner chooses the one that minimizes the predicted maximum of the SME-side and CPU- side completion times. As a result, the final split is not a fixed FLOP ratio; it is the closest tile-aligned shape ratio supported by the current kernel and thread allocation. 5 Evaluation 5.1 Experimental Setup We implement SMEPilot, an optimized LLM inference en- gine for CPUs with scalable matrix extensions. SMEPilot is written in C and C++ and includes SME-aware runtime com- ponents and SME kernels. The SME kernels are developed using the optimized assembly microkernels from KleidiAI [6] and Arm C Language Extensions (ACLE) SME intrinsics [5]. We evaluate the performance of SMEPilot on three platforms equipped with CPUs supporting Scalable Matrix Extension (SME): Apple M4 Pro CPU, MediaTek Dimensity 9500, and KunPeng 920 72F8 Server CPU with SME support. These are recently released CPU chips that support SME or SME2, and they represent the current state of the art for SME performance. Table 1 summarizes their CPU resources, SME- to-CPU resource ratio, and supported instruction sets. In our evaluation, we compare SMEPilot against llama.cpp [16], one of the most widely used CPU LLM inference framework. We use the default CPU backend for all platforms. We select both dense and MoE models for evaluation: Llama-3.2-3B [26], Qwen3-4B [38], and Qwen3- 30B-A3B [38]. The Qwen3-30B-A3B model uses a 4-bit weight quantization to fit into device memory. We evaluate three representative tasks: long-text processing [18], simple question answering [10], and long-text generation [7], as shown in Table 2. The Ruler row sweeps sequence lengths of 4K and 8K, while GSM8K and LongWriter-6K sweep batch size. We use 10 threads and 8 threads for Apple M4 Pro and MediaTek Dimensity 9500, respectively, which equals the number of performance cores on the two platforms. For the KunPeng 920 Server CPU, we use 10 threads for a fair comparison. 5.2 End-to-End Inference Figure 9 shows the end-to-end inference latency of SMEPilot compared to the llama.cpp baseline on the CPU platforms. 8 Table 1. CPU platforms used in the evaluation. PlatformCPU coresSME:CPU ratioSupported instruc- tion sets Apple M4 Pro CPU14 cores (10 performance + 4 efficiency); 10 workers used2:10 (1 SME unit per 5 performance cores) SME, SME2 MediaTek Dimensity 95008 cores (1 C1-Ultra + 3 C1-Premium + 4 C1-Pro)1:8SME, SME2 KunPeng 920 72F8 Server CPUserver-class multi-core CPU; 10 workers used1:1SME 4K8K 0 10 20 30 M4 Pro Llama3.2-3B 4K8K 0 10 20 30 40 50 M4 Pro Qwen3-4B 4K8K 0 20 40 60 M4 Pro Qwen3-30B-A3B 4K8K 0 100 200 300 400 Dimensity 9500 Llama3.2-3B 4K8K 0 200 400 600 800 Dimensity 9500 Qwen3-4B 4K8K 0 100 200 300 KunPeng Llama3.2-3B 4K8K 0 200 400 600 KunPeng Qwen3-30B-A3B 12346816 0 5 10 15 20 M4 Pro Llama3.2-3B 12346816 0 10 20 30 M4 Pro Qwen3-4B 12346816 0 5 10 15 20 M4 Pro Qwen3-30B-A3B 12346816 0 20 40 60 Dimensity 9500 Llama3.2-3B 12346816 0 50 100 150 Dimensity 9500 Qwen3-4B 12346816 0 20 40 60 80 100 KunPeng Llama3.2-3B 124816 0 25 50 75 100 125 KunPeng Qwen3-30B-A3B 12346810121416 0 500 1000 1500 M4 Pro Llama3.2-3B 12346816 0 1000 2000 3000 M4 Pro Qwen3-4B 12346816 0 1000 2000 3000 M4 Pro Qwen3-30B-A3B 12346 0 1000 2000 3000 4000 Dimensity 9500 Llama3.2-3B 12 0 500 1000 1500 2000 Dimensity 9500 Qwen3-4B 1248 0 1000 2000 3000 4000 KunPeng Llama3.2-3B 1248 0 2000 4000 6000 KunPeng Qwen3-30B-A3B Latency (s) Ruler Sequence length GSM8K Batch size LongWriter-6K Batch size llama.cpp prefill llama.cpp decode SMEPilot prefill SMEPilot decode Fig. 9. End-to-end latency across devices and real-world datasets. Rows correspond to Ruler, GSM8K, and LongWriter-6K. Within each row, panels compare the measured device-model pairs: Apple M4 Pro with Llama3.2-3B, Qwen3-4B, and Qwen3- 30B-A3B; MediaTek Dimensity 9500 with Llama3.2-3B and Qwen3-4B; and KunPeng Server CPU with Llama3.2-3B and Qwen3-30B-A3B. We skip some batch or model-pair because of the memory constraints of device. SMEPilot produces up to 3.94×speedup across different datasets and CPU platforms. More specifically, in the prefill stage, SMEPilot achieves 1.13×to 5.42×speedup. In the decode stage, SMEPilot produces a up to 3.48× speedup. In the long text processing task, SMEPilot produces a 1.13× to 3.56×speedup on three different CPUs. The speedup arises from both utilizing CPU cores and SME units together for compute-bound prefill operators, including the attention, FFN and MoE. 9 Table 2. End-to-end benchmark configurations. Task typeDataset Mean prefill tokens Mean decoding tokens Long-context QARuler-4k/8k3324 / 73384 Math QAGSM8K92100 Long-form generationLongWriter-6K3395424 In the simple question answering task, SMEPilot produces a 1.0×to 3.94×speedup on three different CPUs. The speedup arises from our roofline-based operator placement for prefill, decode phases and different batch sizes. For batch sizes of 1 or 2, SMEPilot produces a comparable performance with the baseline because the operators are more memory-bound and the runtime selects CPU execution. When the batch size increases, SMEPilot produces a more significant speedup. This is because larger batch sizes reach the ridge-crossing regime where SME can be more effective, which demon- strates the effectiveness of our online planner. The speedup is smaller for the MoE models. This is because tokens are routed to different experts, which reduces the effective batch size for each expert and makes the operators more memory- bound, thus reducing the benefit of using SME. In the long-text generation task, SMEPilot produces a 1×to 2.95×speedup. This long generation task has a much longer decoding phase, and SMEPilot accelerates the decode phase by using SME for operators in the ridge-crossing regime. Notably, the speedup is more significant for the MoE model in this task. This is because Qwen3-30B-A3B uses group-query-attention with larger group size, which in- creases the arithmetic intensity of attention and help gain more speedup from our SME-aware scheduling. 5.3 Operator Performance Breakdown To better analyze the end-to-end speedup and demonstrate the effectiveness of our design, we conduct a detailed tests for operators in the prefill and decode phases on the Apple M4 Pro CPU. 5.3.1 FFN. Figure 10(a) shows the prefill FFN GEMM per- formance of SMEPilot compared with Apple Accelerate [3], GGML [15] CPU, and the ARM Kleidiai SME [6], across different prefill lengths. For prefill length from 512 to 8192, SMEPilot sustains 4.10–4.44 TFLOP/s. Compared with the GGML CPU, SMEPilot achieves a 6.06–7.43×speedup across different prefill lengths, demonstrating the effectiveness of our roofline analysis. Compared with the Apple Accelerate and ARM Kleidiai SME, which uses the SME unit but does not use SMEPilot’s mixed execution plan, SMEPilot still achieves a 1.30–1.67×speedup. The performance improvement arises from using the SME unit and CPU cores together (Section 3) and from the partition technique that balances the two sides (Section 4). 5121024204840968192 Prefill length 0 5 TFLOP/s (a) FFN 5121024204840968192 Prefill length 0.0 2.5 (b) Attention SMEPilot Apple Accelerate GGML CPU ARM Kleidiai SME FlashAttention CPU Vanilla Attention w/ SME Fig. 10. Prefill operator throughput on the Apple M4 Pro CPU. (a) FFN GEMM for the퐵×3072×8192 shape, compared with Apple Accelerate, GGML CPU, and ARM Kleidiai SME baselines. (b) Attention for Llama-3.2-3B with 24 query heads, 8 KV heads, and head dimension 128, compared with FlashAttention CPU and vanilla attention with SME. 1481216 Batch size 0.0 0.5 1.0 Latency (ms) (a) FFN 123468 Decode tokens 0 2 (b) Attention SMEPilot Kleidiai CPU GGML CPU FlashAttention CPU Fig. 11. Decode operator latency on the Apple M4 Pro CPU. (a) FFN GEMV for the 2560× 푇 ×9728 shape, compared against Kleidiai CPU and GGML CPU baselines. (b) Attention with KV length 8192, 32 query heads, 8 KV heads, and head dimension 128, compared with the FlashAttention CPU baseline. 5.3.2 Prefill Attention. Figure 10(b) shows the prefill attention performance of SMEPilot compared to the CPU FlashAttention and vanilla attention with SME across dif- ferent sequence lengths. For attention with 24 query heads and 8 KV heads, SMEPilot sustains 1.50–4.38 TFLOP/s as the prefill length increases from 512 to 8192, and achieves a 1.56–3.50×speedup over the FlashAttention CPU baseline. The gain demonstrates the effectiveness of our phase-aware pipelining design in Section 4. 5.3.3 Decode FFN. Figure 11(a) reports the decode FFN result. For different decode batch sizes, SMEPilot achieves a comparable performance with the CPU baselines when the batch size is small, and achieves 1.45×and 3.95×speedup compared to the Kleidiai CPU and GGML CPU respectively when the batch size is large. This is because the runtime selects CPU execution for smaller batches, but selects SME for larger batches that reach the ridge-crossing regime. 10 Fullwo part. 0 50 Latency (ms) (a) 48.25 70.6 GEMM Fullwo pipe w 2-stage pipe 0 100 200 (b) 98.1 202.7 156.3 Attention FullNaive pack 0 1 2 pack (c) 0.52 1.71 GEMV Fig. 12. Ablation study on the Apple M4 Pro CPU. Tile par- tition, phase-aware pipelining, and layout-aware execution reduce GEMM(4096x3072x8192), prefill attention(seq 4096), and decode GEMV(16x3072x8192) latency, respectively. Table 3. End-to-end energy on Apple M4 Pro for Qwen3-4B running the Ruler-4K workload. Mode푛 Throughput (tok/s) avg W E2E energy (J) llama.cpp baseline 3 188.812± 7.787 26.050± 2.193 482.813± 23.213 SMEPilot3 338.012± 0.680 22.611± 0.296 233.931± 2.751 5.3.4 Decode Attention. Figure 11(b) shows the latency of decode attention of SMEPilot and the FlashAttention CPU baseline. SMEPilot improve decode attention latency from 1.32×to 4.75×depending on the decode token width. The benefit arises from the runtime’s ability to use SME when the decode token width is large enough and to pipeline matrix and vector phases when dependencies permit. The speedup grows with the decode token width because wider decode attention has larger matrix shapes and has higher arithmetic intensity, making it more profitable to use SME and to pipeline matrix and vector phases. This result validates the design choice of treating decode attention as a shape- dependent case instead of a purely memory-bound operator that should always run on cpu cores. 5.4 Ablation Study Our ablation study focuses on the three key design mecha- nisms in Section 4: tile-level work partitioning, phase-aware pipeline execution, and layout-aware execution. As shown in Figure 12, removing tile partitioning increases GEMM latency by 1.46×because the operator can no longer use SME and CPU cores together. Removing the attention pipeline increases prefill attention latency by 2.07×. Finally, naive on- path layout packing increases decode GEMV latency from 0.52 ms to 1.71 ms, confirming that layout conversion must be kept off repeated SME critical paths. 5.5 Power Consumption We further run a power measurement on the Mac platform using Qwen3-4B and the Ruler-4K workload. We choose this platform because macOS provides thepowermetricstool for sampling system power during inference. Table 3 reports the end-to-end energy from this measurement. SMEPilot 12346816 Batch size 0.0 2.5 5.0 7.5 Latency (s) SMEPilotGPU Fig. 13. End-to-end Llama3.2-3B GSM8K latency on Apple M4 Pro, comparing SMEPilot on the CPU with GPU inference across batch sizes. reduces the energy to 0.485×of the llama.cpp baseline. The lower energy mainly comes from finishing inference faster and better utilize SME. 5.6 GPU Inference Comparison We also compare SMEPilot with GPU inference on the Apple M4 Pro platform because Macbook Pro is equipped with a 20-core GPU with 8 TFlops. As shown in Figure 13, SMEPilot achieves 0.72–0.96×performance compared to the GPU inference. This result positions SMEPilot as a practical CPU inference engine for systems where the GPU is unavailable or reserved for other workloads, or to coordinate with GPU execution. 6 Related Work 6.1 CPU LLM Inference CPU inference has become an important deployment path for LLMs because CPUs are widely available across edge devices, personal computers, and servers. Practical runtimes such as llama.cpp [16] show that careful low-level kernels, weight quantization, and CPU-friendly memory layouts can make local LLM inference usable on commodity machines. Intel Extension for Transformers studies CPU LLM inference on Xeon processors, combining weight-only quantization, operator fusion, memory management, and runtime opti- mizations for popular LLMs [31]. ArcLight further studies lightweight LLM inference on many-core CPUs [36]. Other CPU-oriented systems target low-bit or sparse execution: T- MAC replaces mixed-precision GEMM with table lookup for edge CPUs [35], while sparse Transformer inference systems reduce CPU-side compute and memory traffic [13]. These systems demonstrate that CPU LLM inference is practical, but their main target is model compression, quantized arith- metic, or CPU-only kernel optimization. Several works further analyze CPU inference as a phase- dependent systems problem. Sandwich observes that prefill and decode have different execution characteristics and searches different CPU config- urations for the two phases [40]. Platform studies show that LLM inference stresses memory bandwidth, scheduling, and 11 distributed inference resources differently across model sizes and request regimes [8]. On Arm CPUs, prior work charac- terizes Transformer inference on many-core processors [19] and optimizes attention by exploiting data reuse across Arm multi-core CPUs [14]. Other systems use CPUs as partners in heterogeneous LLM inference. FlexGen offloads tensors across GPU, CPU, and storage [32]; PowerInfer splits work between GPU and CPU on commodity PCs [33]; PowerInfer- 2 coordinates NPU, CPU, and storage on smartphones [37]; and Toppings uses CPU computation to hide LoRA adapter loading during LLM serving [24]. SMEPilot is complementary to these CPU inference and LLM serving systems [30]. Rather than scheduling requests across devices, or optimizing a conventional vector-core runtime, SMEPilot targets CPUs that expose both matrix- extension units and ordinary CPU cores. It therefore focuses on when to use SME, when to keep work on CPU cores, and when to coordinate both within one operator. 6.2 CPU Matrix Extension and SME Modern CPUs increasingly include matrix extensions that provide accelerator-like matrix throughput inside the CPU package. Intel AMX adds tile registers and tile matrix in- structions, and prior work has used AMX to accelerate LLM inference on CPUs [21]. SparAMX combines AMX with unstructured sparsity to accelerate compressed LLM token generation on Intel CPUs [1]. LIA uses AMX-enabled CPU computation as part of a CPU-GPU-CXL cooperative inference system [20]. These works show that CPU matrix extensions can be useful for LLM workloads, especially when the software stack exposes enough matrix work to the extension. However, they primarily study Intel AMX, compressed models, or CPU-GPU offload. SMEPilot instead studies Arm SME and the scheduling problem inside an SME- enabled CPU cluster, where SME and CPU cores share cache and memory bandwidth. Arm SME introduces matrix-oriented architectural state and outer-product instructions on top of the Arm vector execution model [4]. Recent SME studies characterize the extension and build optimized kernels for dense matrix multiplication [12,29] and sparse matrix multiplication [23]. These kernel-level studies are an important foundation for understanding SME’s peak throughput and data layout requirements. SMEPilot builds on this line of work but asks a different systems question: how should an LLM runtime use SME across prefill, decode, attention, FFN, vector operations, and layout-changing operator boundaries? The answer is not to replace every kernel with an SME kernel. SMEPilot uses a roofline-guided planner to select SME-only, CPU- only, or mixed SME+CPU execution; partitions compute- bound operators across heterogeneous units; pipelines SME- friendly matrix phases with CPU-friendly vector phases; and tracks layouts to keep packing out of repeated SME critical paths. 7 Discussion and Future Work Design implications for future CPU matrix extensions. Based on our characterization in using SME for LLM in- ference, we suggest that current SME can be improved along following dimensions. First, stronger support for vector-heavy and GEMV-like phases is important. Decode, softmax, and normalization do not naturally exploit SME outer-product throughput, and direct SME execution can be slower than using CPU cores. Increasing matrix throughput alone therefore will not fully accelerate LLM inference unless the architecture or runtime also improves these non-matrix phases or makes them easier to overlap with matrix work. Second, matrix extensions need more convenient layout management. SME kernels require packed, tile-friendly lay- outs, while the surrounding LLM pipeline often uses con- ventional tensor layouts. Without architectural or runtime support for cheap conversion, layout traffic can erase much of the gain from a faster matrix kernel. Third, future systems would benefit from clearer isolation and between SME units and CPU cores. Currently, SME and CPU cores can provide near-additive compute throughput, but they still contend for cache capacity, and the CPU cores that issue SME instructions. A more decoupled SME from cpu pipeline would reduce contention. Cooperation with GPU and NPU. While this paper fo- cuses on CPU with scalable matrix extensions, future LLM inference systems will likely need to coordinate across CPU, GPU, and NPU execution. CPU matrix extensions are attractive because they are integrated into general-purpose CPUs, which are widely deployed and cost-effective across both edge and server platforms, and can execute CPU- side work without an additional device transfer. GPUs and NPUs, in contrast, provide much higher throughput for large regular matrix workloads. A future inference runtime could therefore use GPUs or NPUs together with CPU matrix extensions, exposing more overlap opportunities. Realizing this cooperation requires a broader planner that accounts for factors, such as device transfer cost, memory placement and etc. We leave such cooperation with SME to future work. 8 Conclusion This paper studies how Arm SME should be used for LLM inference. Our characterization shows that SME is not a universal substitute for CPU vector cores. SMEPilot uses this observation to select SME-only, CPU-only, or mixed execution, partition work at tile granularity, overlap matrix and vector phases, and avoid repeated layout conversion. The resulting speedups show that CPU matrix extensions are most effective when their throughput is applied selectively and amortized across suitable inference regimes. 12 References [1]Ahmed F. AbouElhamayed, Jordan Dotzel, Yash Akhauri, Chi-Chih Chang, Sameh Gobriel, J. Pablo Munoz, Vui Seng Chua, Nilesh Jain, and Mohamed S. Abdelfattah. 2025.SparAMX: Accelerat- ing Compressed LLMs Token Generation on AMX-powered CPUs. arXiv:2502.12444 [cs.LG] doi:10.48550/arXiv.2502.12444 [2]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 117–134. https://w.usenix.org/conference/osdi24/presentation/agrawal [3] Apple. 2026. Accelerate Framework. Apple Developer Documentation. https://developer.apple.com/documentation/accelerateAccessed: 2026-06-08. [4]Arm. 2024. Part 1: Arm Scalable Matrix Extension (SME) Introduction. Arm Community Blog. https://community.arm.com/arm-community- blogs/b/architectures-and-processors-blog/posts/arm-scalable- matrix-extension-introduction [5]Arm. 2026. Arm C Language Extensions. Arm Developer Documenta- tion. https://arm-software.github.io/acle/main/acle.html Accessed: 2026-06-08. [6]Arm. 2026. KleidiAI: Optimized Micro-kernels for AI Workloads on Arm CPUs. GitHub repository. https://github.com/ARM-software/ kleidiai Accessed: 2026-06-08. [7]Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. arXiv:2408.07055 [cs.CL] https://arxiv.org/abs/2408.07055 [8]Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong, Souvik Kundu, Sudarshan Srinivasan, Suvinay Subramanian, Midhilesh Elavazhagan, Madhu Kumar, and Tushar Krishna. 2024. Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models. arXiv:2406.01698 [cs.AR] doi:10.48550/arXiv.2406.01698 [9]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM In- ference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774 [cs.LG] https://arxiv.org/abs/2401.10774 [10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hee- woo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Train- ing Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG] https://arxiv.org/abs/2110.14168 [11]Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359. [12]Chencheng Deng, Weiling Yang, Jianbin Fang, and Dezun Dong. 2025. Demystifying ARM SME to Optimize General Matrix Multiplications. arXiv:2512.21473 [cs.DC] doi:10.48550/arXiv.2512.21473 [13]Ditto PS, Jithin VG, and Adarsh MS. 2024. Inference Acceleration for Large Language Models on CPUs. arXiv:2406.07553 [cs.DC] doi:10. 48550/arXiv.2406.07553 [14] Xiao Fu, Weiling Yang, Dezun Dong, and Xing Su. 2024. Optimizing Attention by Exploiting Data Reuse on ARM Multi-core CPUs. In Proceedings of the 38th ACM International Conference on Supercom- puting (ICS ’24). Association for Computing Machinery, 137–149. doi:10.1145/3650200.3656620 [15]Georgi Gerganov and contributors. 2023. GGML: Tensor Library for Machine Learning. GitHub repository. https://github.com/ggml- org/ggml [16]Georgi Gerganov and contributors. 2023. llama.cpp: LLM inference in C/C++. GitHub repository. https://github.com/ggml-org/llama.cpp [17]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez- Paz, and Gabriel Synnaeve. 2024. Better & Faster Large Language Models via Multi-token Prediction. arXiv:2404.19737 [cs.CL] https: //arxiv.org/abs/2404.19737 [18]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654 [cs.CL] https://arxiv.org/abs/2404.06654 [19]Jiazhi Jiang, Jiangsu Du, Dan Huang, Dongsheng Li, Jiang Zheng, and Yutong Lu. 2022. Characterizing and Optimizing Transformer Inference on ARM Many-core Processor. In Proceedings of the 51st International Conference on Parallel Processing (ICPP ’22). Association for Computing Machinery, 20:1–20:11. doi:10.1145/3545008.3545022 [20]Hyungyo Kim, Nachuan Wang, Qirong Xia, Jinghan Huang, Amir Yazdanbakhsh, and Nam Sung Kim. 2025. LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Associa- tion for Computing Machinery, 544–558. doi:10.1145/3695053.3731092 [21]Hyungyo Kim, Gaohan Ye, Nachuan Wang, Amir Yazdanbakhsh, and Nam Sung Kim. 2024. Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference. IEEE Computer Architec- ture Letters 23, 1 (2024), 117–120. doi:10.1109/LCA.2024.3397747 [22]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023.Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626. [23] Kelun Lei, Hailong Yang, Kaige Zhang, Kejie Ma, Yiqing Wang, Xin You, Yufan Xu, Enrique S. Quintana-Orti, Zhongzhi Luan, Yi Liu, and Depei Qian. 2025. LOw-cOst yet High-Performant Sparse Matrix-Matrix Multiplication on Arm SME Architectures. arXiv:2511.08158 [cs.DC] doi:10.48550/arXiv.2511.08158 [24]Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. 2025. Toppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Infer- ence. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, 613–629. https://w.usenix.org/conference/ atc25/presentation/li-suyi-toppings [25] Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao, Dacheng Tao, Philip S. Yu, and Ming Zhang. 2025. Large Language Model Agent: A Survey on Methodology, Applications and Challenges. arXiv:2503.21460 [cs.CL] https://arxiv.org/abs/2503.21460 [26]Meta AI. 2024. Llama 3.2: Revolutionizing Edge AI and Vision with Open, Customizable Models. Meta AI Blog. https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/ [27]Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2025. Large Language Models: A Survey. arXiv:2402.06196 [cs.CL] https://arxiv. org/abs/2402.06196 [28]Onur Mutlu, Saugata Ghose, and Rachata Ausavarungnirun. 2018.Recent Advances in Overcoming Bottlenecks in Mem- ory Systems and Managing Memory Resources in GPU Systems. arXiv:1805.06407 [cs.AR] doi:10.48550/arXiv.1805.06407 [29]Stefan Remke and Alexander Breuer. 2024. Hello SME! Generating Fast Matrix Multiplication Kernels Using the Scalable Matrix Extension. arXiv:2409.18779 [cs.DC] doi:10.48550/arXiv.2409.18779 [30]Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX 13 Feiyang Chen, Haibo Chen Association, 397–411.https://w.usenix.org/conference/atc21/ presentation/romero [31] Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, and Hengyu Meng. 2023. Efficient LLM Inference on CPUs. arXiv:2311.00502 [cs.LG] doi:10.48550/arXiv.2311.00502 [32]Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 31094–31116.https: //proceedings.mlr.press/v202/sheng23a.html [33]Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 590–606. doi:10.1145/3694715.3695964 [34]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. At- tention Is All You Need. CoRR abs/1706.03762 (2017). arXiv:1706.03762 http://arxiv.org/abs/1706.03762 [35] Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. 2025. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25). Association for Computing Machinery. doi:10.1145/3689031.3696099 [36]Yuzhuang Xu, Xu Han, Yuxuan Li, and Wanxiang Che. 2026. ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs. arXiv:2603.07770 [cs.DC] doi:10.48550/arXiv.2603.07770 ACL 2026 Demo. [37]Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv:2406.06282 [cs.AR] https://arxiv.org/abs/2406. 06282 [38]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). https://arxiv.org/abs/2505.09388 [39]Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 521–538. https://w.usenix.org/conference/osdi22/ presentation/yu [40] Juntao Zhao, Jiuru Li, and Chuan Wu. 2025. Sandwich: Joint Con- figuration Search and Hot-Switching for Efficient CPU LLM Serving. arXiv:2507.18454 [cs.AR] doi:10.48550/arXiv.2507.18454 DAC ’26. [41]Shanshan Zhao, Xinjie Zhang, Jintao Guo, Jiakui Hu, Lunhao Duan, Minghao Fu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. 2026. Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Op- portunities. arXiv:2505.02567 [cs.CV] https://arxiv.org/abs/2505.02567 [42]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 193–210. https: //w.usenix.org/conference/osdi24/presentation/zhong-yinmin