Paper deep dive
SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs
Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow, Lin Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 6:16:35 AM
Summary
The paper introduces SelectInfer, a neuron-level optimization framework for deploying Large Language Models (LLMs) on resource-constrained edge devices. It utilizes an offline profiler to identify task-specific and general-purpose neurons, enabling selective loading to reduce memory footprint and selective computation to accelerate inference. Evaluated on NVIDIA Jetson Orin Nano with Llama3.2 and Qwen2.5 models, SelectInfer achieves significant memory and speed improvements while maintaining accuracy.
Entities (13)
Relation Signals (11)
SelectInfer â evaluatedon â NVIDIA Jetson Orin Nano
confidence 95% ¡ We validate SelectInfer across three lightweight LLMs... on the NVIDIA Jetson Orin Nano
Selective Computation â reduces â Computation Overhead
confidence 95% ¡ selective computation, which dynamically computes only the most relevant neurons at runtime... further accelerating inference.
Selective Loading â reduces â Memory Footprint
confidence 95% ¡ selective loading, which reduces memory footprint by selectively loading a subset of neurons
SelectInfer â usestechnique â Selective Loading
confidence 95% ¡ SelectInfer implements two key optimizations: selective loading... and selective computation
SelectInfer â usestechnique â Selective Computation
confidence 95% ¡ SelectInfer implements two key optimizations: selective loading... and selective computation
SelectInfer â appliedto â Qwen2.5 3B
confidence 92% ¡ We validate SelectInfer across three lightweight LLMs, Llama3.2-3B... Qwen2.5-3B... on the NVIDIA Jetson Orin Nano
SelectInfer â appliedto â Llama3.2-3B
confidence 92% ¡ We validate SelectInfer across three lightweight LLMs, Llama3.2-3B... on the NVIDIA Jetson Orin Nano
SelectInfer â appliedto â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of Natural Language Processing (NLP) tasks, but their high computational and memory demands pose significant challenges for deployment on resource-constrained edge devices. Existing approaches to model compression and optimization often rely on coarse-grained pruning or quantization, which can compromise accuracy or require re-training and fine-tuning. In this work, we introduce SelectInfer, a neuron-level optimization framework that enables efficient LLM inference on edge devices through selective neuron loading and computation. By profiling and identifying both task-specific and general-purpose neurons using an offline LLM profiler, SelectInfer implements two key optimizations: selective loading, which reduces memory footprint by selectively loading a subset of neurons that were identified to be most important during the offline stage, and selective computation, which dynamically computes only the most relevant neurons at runtime. Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices
Tags
Links
- Source: https://arxiv.org/abs/2607.18081v1
- Canonical: https://arxiv.org/abs/2607.18081v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
50,359 characters extracted from source content.
Expand or collapse full text
SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs Huzaifa Shaaban Kabakibo 1â huzaifa@mail.uni-paderborn.de Eric Schniedermeyer 1â eric1@mail.uni-paderborn.de Artem Burchanow 1â artemb@mail.uni-paderborn.de Lin Wang 1 lin.wang@uni-paderborn.de 1 Department of Computer Networks, Paderborn University, Paderborn, Germany Abstract Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of Natural Language Processing (NLP) tasks, but their high computational and memory demands pose significant challenges for deployment on resource- constrained edge devices. Existing approaches to model compression and optimiza- tion often rely on coarse-grained pruning or quantization, which can compromise accuracy or require re-training and fine-tuning. In this work, we introduce SelectIn- fer, a neuron-level optimization framework that enables efficient LLM inference on edge devices through selective neuron loading and computation. By profiling and identifying both task-specific and general-purpose neurons using an offline LLM profiler, SelectInfer implements two key optimizations: selective loading, which reduces memory footprint by selectively loading a subset of neurons that were identified to be most important during the offline stage, and selective computation, which dynamically computes only the most relevant neurons at runtime. Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices. 1 Introduction Large Language Models (LLMs) have rapidly become foundational tools across a wide array of applications, including natural language writing, code generation, healthcare diagnostics, financial analysis, and many more [1â4]. Their impressive capabilities have driven their widespread adoption, predominantly through deployment in powerful data centers equipped with high-end GPUs. These centralized environments offer immense computational resources required to run LLM inference efficiently. However, growing concerns around user privacy, the need for application customization, and the demand for offline capabilities [5â8] have fueled increasing interest in enabling LLM inference directly on edge devices, such as NVIDIA Jetson [9] and Google Coral [10]. Edge deployment offers the potential for greater data privacy and responsiveness, but it also faces significant challenges. Unlike data centers, edge devices are constrained by limited memory and computational power, which complicates the direct execution of large, resource-hungry models. â Equal contribution. arXiv:2607.18081v1 [cs.LG] 20 Jul 2026 Offline PhaseOnline Phase DatasetsLLM Profiler Model- & Task- specific Neurons Fully Activated Fully Deactivated Selective Loading % Base neurons % Secondary neurons Memory Selective Computation % Static neurons % Dynamic neurons Input Data flow Layer i neurons Figure 1: SelectInfer has two phases: offline (left) and online (right). In the offline phase, we run a profiler to generate neuron files. In the online phase, selective loading reduces memory usage, while selective computation lowers computational overhead. Despite active research, existing approaches fundamentally fail to optimize the three dimensions that jointly determine whether an LLM can be practically deployed on an edge device: memory footprint, inference speed, and output accuracy. Techniques such as quantization reduce memory usage but degrade decoding throughput and offer only coarse control; pruning methods restore neither memory nor speed without costly retraining; and neuron-sparsity approaches that operate purely at runtime still require the full model to reside in memory. On resource-constrained devices, optimizing only one or two of these dimensions is insufficient: a model that fits in memory but runs too slowly is not deployable, and a fast model that cannot fit at all is equally useless. What is needed is a framework that treats all three as first-class concerns and exposes configurable trade-offs between them. Our key insight is that LLM neurons exhibit a dual structure: a stable core of model-specific neurons that activate universally across tasks, and a set of task-specific neurons that drive performance for individual inputs. By identifying these two groups before runtime, we can make precise, principled decisions about which neurons to load into memory and which to compute at runtime, treating memory and computation as two independently tunable knobs rather than a single coarse dial. We propose SelectInfer, which identifies active neurons before runtime. Following CoreInferâs observations [11], we categorize neurons into two groups: base neurons, which activate selectively depending on the inputâs task ( e.g., QA, translation, or summarization), and secondary neurons, which serve general purposes and activate broadly across inputs. Our approach is summarized in Fig 1. As shown on the left side, using an offline LLM profiler, we accurately determine these neuron groups. To address memory constraints, in the online phase (right side of Fig 1), we implement selective loading, a mechanism that loads only the relevant neurons identified by the offline profiler into memory, discarding the remaining ones. Additionally, we introduce selective computation, which selectively computes only the most relevant neurons during runtime, further accelerating inference. We validate SelectInfer across three lightweight LLMs, Llama3.2-3B [12], Llama3.2-1B [13], and Qwen2.5-3B [14], on the NVIDIA Jetson Orin Nano [9], a highly memory-constrained device with 8 GB of shared CPUâGPU memory. Including framework overhead, 3B-scale models demand roughly 8.5â9.5 GB of memory, but the chip exposes only aboutâź6.5 GB of usable memory because the remainder is occupied by the system. As a result, native inference on the chip is not feasible. With selective loading, SelectInfer enables both 3B-scale models to run by loading only the most relevant neurons identified during offline profiling. Across QA, translation, and summarization tasks, SelectInfer maintains competitive accuracy while yielding substantial speedups; for example, Llama3.2-3B reaches about 11 tokens/s, achieving 1.53Ă improvement over 4-bit quantization, and more than a 13Ăover disk-offloading. We observe similar gains for Llama3.2-1B and Qwen2.5-3B, demonstrating that SelectInfer generalizes across model families and offers a practical path to deploy LLMs on resource-constrained edge devices. 2 PiQA-WMT16 PiQA-CNN/DM WMT16-CNN/DM All tasks 0 20 40 60 80 100 Avg overlap % in top 40% neurons 67.9 72.9 71.8 59.9 Figure 2: High neurons over- lap between different tasks, proving that a set of neurons always gets activated, regard- less of the input task. PiQAWMT16CNN/DM 0 20 40 60 80 100 Avg overlap % in top 40% neurons 58.5 65.1 69.6 Figure 3: High neurons over- lap within the same task show- ing a consistent set of neurons always get activated. PiQA-WMT16 PiQA-CNN/DM WMT16-CNN/DM 0 20 40 60 80 100 Avg overlap % in top 40% neurons 22.7 23.6 24.3 Figure 4: Low cross-task neu- rons overlap. This proves that each task activates its own set of neurons. In summary, our contributions include an offline profiler that identifies model- and task-specific neurons, a novel selective loading scheme that enables deployment of models beyond memory limits on edge devices, and a selective computation mechanism that selectively processes relevant neurons to accelerate inference with negligible accuracy degradation. 2 Background and Motivation This section highlights some background and motivates the insights that shaped the development of our approach. 2.1 Sparsity in LLMs A key property of LLMs is their inherent sparsity: a small subset of neurons is strongly activated for a given input, largely determining output quality [15â17]. Thus, the modelâs full capacity is rarely used at once; instead, different parts become selectively active depending on the task or input distribution. Prior work has exploited this property at runtime: most notably, CoreInfer [11] observes that activation patterns cluster by semantic domain and identifies active neurons during the prefill phase of each prompt, achieving high throughput by skipping computation over inactive neurons. However, CoreInfer requires the full model to reside in memory and relies entirely on the current prompt for neuron selection. Building on this observation, we further analyze neuron roles across tasks, distinguishing task-specific from model-specific neurons. 2.2 Model-Specific Neurons We define model-specific neurons as neurons that are consistently activated across tasks, regardless of input type. To quantify this, we aggregate prompts from three distinct datasets: PiQA [18] for question answering, CNN/DailyMail [19,20] for summarization, and WMT16 DE-EN [21] for translation. Then we measure neuron activations across 100 prompts per task using Llama3.2-3B [12] model. We then select the top 40% of neurons (ranked by absolute activation value) for each layer and examine their overlap across tasks. We average the results across layers. We refer to §6 for more details on the setup. The results are shown in Fig 2. Each bar represents the overlap of the top 40% neurons for each task pair. We observe a significant cross-task overlap of approximately 68% to 72% within the top 40% of neurons, and approximately 60% overlap between all three datasets. This provides an evidence that a stable core of neurons is universally engaged across tasks. These neurons represent general-purpose computation in the model and may correspond to linguistic or representational features that are task-agnostic. 2.3 Task-Specific Neurons In addition to general-purpose neurons, we find strong evidence for task-specific neurons. These are neurons that consistently activate within a given task but differ across tasks. To quantify this, for each dataset, we create ten chunks of ten prompts each, identify the top 40% most active neurons per chunk, and measure the average overlap among all chunks. The results are averaged across all 3 layers. As shown in Fig 3, the overlap ranges from 58.6% for PiQA to 69.5% for CNN/DM. This indicates that for each task, a subset of neurons is consistently responsible for processing task-specific information. Furthermore, to confirm that these subsets are distinct across tasks, we pick the top 40% neurons for each task, and calculate the overlap across tasks. As shown in Fig 4, the cross-task overlap is around 24-26% which is far lower than same-task overlap. This highlights that although each task reuses some task-specific neurons, it also recruits its own specialized subset of neurons. These results show a dual structure: a set of model-specific neurons that engage universally across tasks, and a set of task-specific neurons that drive performance for individual tasks. This interplay between generality and specialization is a key property of LLM computation in SelectInfer. 3 LLM Profiler In the offline stage, we run selected datasets through the model to measure neuron activations. In Transformer Feed Forward Network (FFN) blocks (e.g.,gate_proj,up_proj,down_projin Llama3 and Qwen2.5), each neuron corresponds to a column in the gate/up projections and a row in the down projection. During profiling, we record the absolute activations at the input of the down_projlayer (i.e., the output of the activation function), which directly correspond to these FFN neurons. For each prompt, we record the absolute values of neuron activations, along with the number of input and generated tokens. Since activation functions vary across architectures, the scoring method must sometimes be adapted. For models using SwigLU, we define the following scoring function: LetTbe the set of tokens processed during scoring and leta n (t)be the activation of neuronnon tokentâT. Then, for each neuronnâ1,...,N, we defines n as the score of the neuron: s n = X tâT |a n (t)|.(1) The scores of all neurons, along with the token counts, are stored in score files. These files can be merged in different configurations by normalizing scores by token count and summing the normalized values. Task-specific neurons can be obtained by merging score files from the same task, while model-level neurons can be obtained by merging across tasks. The final output is a file containing neuron indices ranked in decreasing order of importance across the evaluated datasets. 4 Selective Loading Following the analysis conducted by the LLM Profiler, the extracted insights are used to optimize the deployment of LLMs on edge devices by reducing their computational and memory requirements. We first focus on minimizing the memory footprint, which allows larger models to be deployed on resource-constrained devices where full model loading is otherwise infeasible. The core idea is to selectively load the model into memory. We apply selective loading to the FFN, as this component accounts for the largest proportion of the total model parameters. To reduce the memory usage, we selectively load only a subset of its neurons, focusing on those considered most relevant. This means that, for every block in the FFN (e.g.,gate_proj), we select the most important neurons, and load the corresponding slices into the memory, and discard the rest. The remaining components of the model are fully loaded. This selective loading is guided by the neuron relevance profiled by the LLM Profiler during the offline phase. Our method classifies FFN neurons into two groups, base neurons and secondary neurons. As illustrated in Fig 6, we first select the topδmost relevant task-specific neurons to form the base neurons, and then add the most relevant model-specific neurons as secondary neurons until the overall loading thresholdÎłis reached. We specify the proportion of neurons to be loaded based on memory constraints and application requirements. This selection is applied to each FFN layer in the model. When loading the model and reading neurons from disk, we only load the weights corresponding to these selected neurons into memory, leading to substantial memory savings while maintaining competitive accuracy. 4 not loaded neurons base neurons ξδ static neurons dynamic neurons secondary neurons Figure 5: Process of selective computation. not loaded neurons secondary neurons neurons to load, Îł model-specific neuronstask-specific neurons, δ base neurons Figure 6: Neurons to load during selective loading. 5 Selective Computation While selective loading effectively alleviates memory constraints by keeping only a subset of neurons in memory, the computational cost of inference remains significant. To address this, we propose selective computation, a runtime mechanism that selectively computes only the most relevant loaded neurons. This strategy reduces computational overhead and accelerates inference without significantly degrading output quality. As shown in §2, a relatively small fraction of neurons is sufficient to preserve model quality if they are chosen judiciously. For example, even thoughÎłof neurons may be loaded into memory due to selective loading, we can further reduce computation by computingÎľof neurons at runtime (Îľ < Îł). This trade-off between accuracy and speed is user-configurable, enabling flexible deployment depending on application requirements. During computation, we classify neurons into two categories: static neurons and dynamic neurons. Static neurons are identical to the base neurons selected during selective loading. These task-specific neurons are always computed for every prompt, regardless of the input. Dynamic neurons are selected at runtime based on the input prompt. Their selection is adaptive and varies per prompt, allowing the computation process to focus on neurons that are most relevant for the current context. Fig 5 illustrates the relationship between static and dynamic neurons in the context of selective computation. The selection of dynamic neurons occurs during the prefill phase, inspired by Cor- eInferâs [11] methodology. Fig 7 illustrates this process. For each token, we compute activation values for all loaded neurons. We sort these activations in descending order and select the topÎą neurons for that token and record the indices of these topÎąneurons. This is repeated for all tokens in the prompt, resulting in multiple lists of top neurons. From these lists, we count the frequency of neuron occurrences and select the topβneurons with the highest frequency. Theseβneurons form the dynamic set for this prompt. In the end, we combine the static neurons with the selected dynamic neurons until the total computation ratioĎ = δ + Îľis reached. This approach ensures that computation resources concentrates on neurons most likely to contribute to the output for the given input, achieving substantial speedups while retaining accuracy. 6 Evaluation In this section, we evaluate SelectInfer across the three dimensions central to edge deployment: accuracy (§6.5), memory footprint (§6.6), and decoding speed (§6.7). We first describe the evaluation setup (§6.1), the datasets and neuron files used (§6.2), the offline profiler cost (§6.3), and our parameter configuration (§6.4). Throughout, we compare SelectInfer against CoreInfer, 4-bit bitsandbytes quantization, and disk offloading, baselines that each excel on at most one or two of the three target dimensions, to demonstrate that SelectInfer provides a balanced and tunable operating point across all three simultaneously. 6.1 Setup For our evaluation, we use the NVIDIA Jetson Orin Nano Developer Kit [22], which integrates a 6-core Arm Cortex-A78AE CPU, an NVIDIA Ampere GPU with 1,024 CUDA cores and 32 Tensor Cores, and 8 GB of LPDDR5 shared memory. We benchmark three representative models, Llama3.2-3B, Llama3.2-1B, and Qwen2.5-3B, to compare SelectInfer against two strong baselines: CoreInfer and 4-bit bitsandbytes quantization. 5 Figure 7: Dynamic neuron selection process. Table 1: Important parameter values used in our experiments for all three models. ParameterValue First layer5 Last layer-2 (Skip last 2) % of base / static neurons30% % of dynamic neurons10% % of computed neurons (sparsity)40% % of secondary neurons40% % of overall loaded neurons70% Table 2: Offline profiler cost per model and number of processed tokens. ModelTokens processed Time Throughput Peak GPU memory (M)(h)(tokens/s)(GB) Qwen2.5-3B43.95.51221416.0 Llama3.2-3B43.75.01242621.8 Llama3.2-1B8.80.5643497.6 Llama3.2-1B43.82.69451810.8 6.2 Datasets and Neuron-Files Selection We evaluate three tasks (QA, translation, and summarization) using multiple datasets within each domain. For QA, we use SQuAD [23], TriviaQA [24], MLQA [25], and PIQA [18]. For transla- tion, we use WMT16-de-en and WMT16-en-de [21], WMT14-fr-en and WMT14-en-fr [26]. For summarization, we use CNN/DailyMail [20], Samsum [27], and XSum [28]. We generate task-specific neuron files offline using all datasets within a task, ensuring that the resulting neurons reflects consistent task-specific responses. In contrast, we derive model-specific neurons by applying the same scoring method to a cross-task mixture constructed from one dataset per task, identifying neurons that generalize beyond a single task. Specifically, we use PIQA, WMT14- en-fr, and Xsum datasets to generate model-specific neurons file. We observe that neuron scores stabilize after a few datasets, indicating that this is sufficient for reliable results. Moreover, the choice of datasets for model-specific neuron calculation has little impact on scores, as long as they were drawn from different task groups. 6.3 LLM Profiler Cost Table 2 summarizes the offline profiling cost across all models. All results were obtained on an NVIDIA L40S GPU using a batch size of 16. For the 3B-parameter models Qwen2.5-3B and Llama3.2-3B, the profiler processes roughly 44M tokens in 5.0â5.5 hours, achieving throughputs of 2.2kâ2.4k tokens/s with peak GPU memory usage between 16 GB and 21.8 GB. For Llama3.2-1B on the same 43.8M-token corpus, profiling completes in 2.69 hours at 4.5k tokens/s with a peak of 10.8 GB GPU memory, while using a smaller 8.8M-token corpus reduces the wall-clock time to 0.56 hours at 4.3k tokens/s and 7.6 GB peak memory. 6.4 Parameter Configuration We find empirically that static neurons should correspond to task-specific neurons, while dynamic neurons represent model-specific behavior. Since task-specific neurons are more directly tied to the target objective, they exhibit higher stability across prompts. For instance, withδ = 0.3, Fig 3 shows that roughly 60% of task-specific neurons overlap across prompts within the same task. SelectInfer exposes several tunable parameters (see Table 1). Their values depend on model size, device constraints (e.g., memory capacity), expected speedup, and the desired accuracy. In practice, hardware limitations largely determine the feasible percentage of loaded neurons, while the allocation between base and secondary neurons and the fraction of computed neurons must be empirically validated. 6 Figure 8: Impact of different layer combinations on cross-task performance. The values are nor- malized to the maximum value of each task. Figure 9: Effect ofδon performance across dif- ferent tasks. Values are normalized to the max value of each task. Table 3: Benchmark results for SelectInfer and CoreInfer variants across three models and six datasets. ModelMethodSQuADv2MLQA enâenWMT16 deâenWMT16 roâenXSUMCNN/DailyMail (Metric)(Exact Match)(Exact Match)(BLEU)(BLEU)(Rouge)(Rouge) Llama3.2-1B SelectInfer13.750.4010.5010.850.120.15 CoreInfer6.000.352.912.870.080.13 CoreInfer + Selective Loading3.850.281.681.390.050.11 CoreInfer + Random Loading1.260.000.030.030.040.03 Llama3.2-3B SelectInfer22.550.5225.6825.960.120.17 CoreInfer17.370.534.974.980.000.15 CoreInfer + Selective Loading11.600.441.701.650.100.12 CoreInfer + Random Loading0.260.010.040.030.000.05 Qwen2.5-3B SelectInfer24.320.538.284.830.100.15 CoreInfer32.580.534.324.090.110.17 CoreInfer + Selective Loading35.690.481.550.800.100.16 CoreInfer + Random Loading29.650.340.440.160.070.13 For the loading ratioÎł, we observe thatÎł = 0.7is the maximum value at which the 3B-scale models fits on the Jetson Nano chip without running out of memory. To ensure consistent comparison across model sizes, we apply the same value to Llama3.2-1B. The percentage of computed neuronsĎfollows prior analyses from CoreInfer, which shows that computing the top40%of neurons provides a strong accuracyâefficiency trade-off. Our experiments confirmed that this setting works well across tasks and models. Layer-range selection. We evaluated multiple first/last layer combinations using the Llama3.2-3B model (see Fig 8), which has 28 layers in total, and found that modifying layers 4 through 26 yields the best cross-task performance, considering the number of skipped layers. This aligns with the common observation that early and final layers are more sensitive and have a disproportionate impact on final logits. Division between base and dynamic neurons. We tested values ofδfrom25%to35%in small increments (Fig 9) using the Llama3.2-3B model. The accuracy remains stable across this range, with δ = 30% slightly outperforming the alternatives, making it a reasonable and robust choice. Overall, we choose the parameter values presented in Table 1 based on hardware feasibility, prior validated methodology, and targeted empirical exploration rather than arbitrary selection. 6.5 Accuracy Table 3 summarizes the accuracy of our proposed SelectInfer method compared to three baselines: the original CoreInfer [11], an enhanced version incorporating our selective loading (CoreInfer + Selective Loading), and a control condition where we load neurons at random (CoreInfer + Random Loading). We evaluate these methods on six datasets across three model families: Llama3.2-1B, Llama3.2-3B, and Qwen2.5-3B. The results highlight the complementary benefits of SelectInfer, and demonstrate that its advantages generalize across model families. Superiority of SelectInfer. Across models and datasets, SelectInfer mostly achieves the highest accuracy. For both Llama models, SelectInfer reaches substantially higher scores than CoreInfer, particularly on translation tasks WMT16 DEâEN and ROâEN. On the 3B model, SelectInfer attains scores around25â26BLEU, compared to only4â5BLEU for CoreInfer. On the 1B model, SelectInfer reaches10â11BLEU, far above CoreInferâs2â3BLEU range. For Qwen2.5-3B model, the results 7 Table 4: Benchmark results using Dense and Quantization methods. For dense, we load 70% of neurons and compute all of them. For quantization, we use 4-bit bitsandbytes [29]. ModelMethodSQuADv2MLQA enâenWMT16 deâenWMT16 roâenXSUMCNN/DailyMail (Metric)(Exact Match)(Exact Match)(BLEU)(BLEU)(Rouge)(Rouge) Llama3.2-1B 70% Dense17.870.4427.3425.950.140.16 4-bit Quantized13.970.4326.7024.650.150.17 Llama3.2-3B 70% Dense26.410.5534.9133.400.210.18 4-bit Quantized30.140.5937.6435.030.220.19 Qwen2.5-3B 70% Dense31.060.5827.9720.290.140.18 4-bit Quantized34.320.5918.1811.300.120.17 further confirm this trend. SelectInfer achieves the strongest performance on both translation tasks (8.28BLEU on DEâEN and4.83BLEU on ROâEN), and remains competitive on MLQA enâen, and both summarization tasks, where it matches or closely trails CoreInfer. The only exception is SQuADv2 dataset, where SelectInfer achieves less accuracy (24.32) than CoreInfer (32.58). Notably, CoreInfer + Selective Loading (35.69) outperforms CoreInfer, eventhough it loads only 70% of neurons. Another example is in Llama3.2B model in Xsum dataset, where CoreInfer + Selective Loading, achieves 0.1, while CoreInfer achieves 0. We attribute this to the offline profiling stage acting as a beneficial pre-filter: by only loading neurons that are consistently activated across the profiling tasks, some task-irrelevant neurons may be removed, helping CoreInfer focus on a more relevant subset. Effectiveness of selective loading. Comparing CoreInfer + Selective Loading against CoreInfer + Random Loading isolates the benefit of profiling-based neuron selection. Random loading severely damages performance in every setting, often collapsing accuracy to near-zero values across all six tasks. In contrast, selective loading maintains a meaningful proportion of CoreInferâs performance, even though onlyÎł = 70%of neurons are available. This stability holds across all evaluated models. On Qwen2.5-3B, for example, CoreInfer + Selective Loading achieves BLEU scores of 1.55and0.80on the translation tasks, whereas random loading yields only0.44and0.16. Although selective loading naturally reduces performance relative to full-neuron CoreInfer, it clearly preserves more accuracy than random selection. This demonstrates that profiling is essential for identifying consistently active and task-relevant neurons. Contribution of selective computation. To understand the effect of selective computation, we compare SelectInfer with CoreInfer + Selective Loading, which shares the same set of loaded neurons but differs during computation. Across all models and nearly all datasets, SelectInfer consistently outperforms its selective-loading counterpart. The gaps are particularly large on the translation datasets: on Llama3.2-3B, SelectInfer achieves25.68BLEU on WMT16 DEâEN compared to only 1.70for CoreInfer + Selective Loading, and similar patterns appear on Llama3.2-1B and Qwen2.5-3B. These results demonstrate that profiling-guided selective computation is crucial for identifying and prioritizing the most informative neurons during inference. Comparison with dense and quantized baselines. Table 4 extends our analysis by evaluating an additional baseline, 4-bit quantized variant using bitsandbytes [29] quantization, which is commonly used to reduce inference cost and memory footprint. To make it a fair comparison, we compare it against 70% dense configuration, where only 70% of neurons are loaded using our selective loading, but all of them are fully computed. Across all three models, the 70% dense baseline proves to be a strong and reliable reference point, often outperforming or matching the 4-bit quantized version, especially on translation datasets. For Llama3.2-1B, the dense baseline dominates on WMT16 DEâEN and ROâEN, reaching27.34and25.95BLEU, whereas the quantized version achieves slightly lower values (26.70and24.65). This pattern persists in the Qwen2.5-3B model, where dense loading achieves substantially stronger translation performance (27.97BLEU on DEâEN and20.29 on ROâEN), while quantization drops to18.18and11.30BLEU. On the other hand, the quantized version performs competitively on QA tasks (e.g., MLQA and SQuADv2), and occasionally surpasses the dense version on Llama3.2-3B and Qwen2.5-3B. However, even in these settings, the differences are generally small, and the dense baseline remains competitive and stable across all tasks. Summary. These results validate the dual contribution of our mechanisms: selective loading improves memory efficiency without degrading performance, while selective computation boosts performance by focusing on the most relevant neurons. Additionally memory footprint and speedup can be 8 Table 5: End-to-end peak memory usage (in GB) for different inference approaches across three models. ModelOriginalSelectInferCoreInferQuantized (4-bit) Llama3.2-3B6.885.977.443.03 Llama3.2-1B2.862.602.861.52 Qwen2.5-3B6.485.377.522.67 controlled independently. Available memory can be fully utilized while speedup can be aligned to available computational resources and application requirements. 6.6 Memory Footprint A key motivation behind SelectInfer is reducing the memory required to load and execute LLM on resource-constrained edge devices. Unlike full-parameter loading approaches, SelectInfer stores only a subset of neurons in memory. The theoretical reduction in memory usage can be expressed as âM = N ffn Ă (1â r load )Ă r layer Ă s,(2) whereN ffn denotes the number of FFN parameters,r load is the fraction of neurons preloaded into memory,r layer is the fraction of layers where SelectInfer is applied, andsis the storage size per parameter. For example, in Llama3.2-3B, whereN ffn â 2.1B, choosingr load = 0.7,r layer = 0.75, and s = 2 bytes (float16) yields 0.945 GB, aligning with the value observed during deployment. To validate these analytical expectations, we report the end-to-end memory usage of SelectInfer against three baselines: the original model (full-parameter loading), CoreInfer, and 4-bit bitsandbytes quantization. Table 5 summarizes the results. As expected, the original model incurs full memory footprint, since all neurons are loaded in memory during inference. CoreInfer consistently exceeds the reference memory usage due to additional buffers and intermediate structures needed to support its core neurons computation. In contrast, SelectInfer achieves a lower memory footprint across all evaluated models as it loads only a fraction of neurons into memory (70% in our experiments). The reduction is most noticeable in the 3B-scale models. For Llama3.2-3B, SelectInfer reduces memory from 6.88 GB to 5.97 GB, corresponding to a savings of 13.2%, consistent with the analytical estimate in Eq 2. A similar trend appears in Qwen2.5-3B, where SelectInfer achieves a reduction of over 1.1 GB relative to the original model. For the 1B model, the improvement is smaller: from 2.86 GB to 2.6 GB. This is because we apply SelectInfer to fewer layers in Llama3.2-1B (9 of 16 layers), reducing the relative impact of selective loading compared to larger models where a higher proportion of layers are covered. Finally, 4-bit quantization unsurprisingly yields the lowest memory footprint overall, as it compresses all weights, whereas SelectInfer selectively loads fewer neurons but retains full precision for the loaded parameters. However, quantization trades off memory reduction for lower decoding throughput, as we will discuss in the next subsection. 6.7 Speedup In this section, we evaluate the impact of SelectInfer on decoding throughput (tokens/s) and task accuracy using Llama3.2-3B and Llama3.2-1B models. The decoding speed corresponding to varying proportions of loaded neurons is illustrated in Fig??. Reducing the fraction of loaded neurons leads to an increase in prefilling and decoding stage, since fewer weights are fetched from memory and computed. In our example, loading onlyÎł = 40%of the neurons improves throughput from8.65 token/s to9.75tokens/s compared to 80% loading. However, this comes at the cost of accuracy: on SQuADv2, accuracy improves from 14.87 at 40% loaded neurons to 24.70 at 80%. Comparison with other baselines. We evaluate the decoding speed of SelectInfer, against three baselines on the NVIDIA Jetson Orin Nano platform: CoreInfer, 4-bit bitsandbytes [29] quantization, and disk offloading. Fig 11 summarizes these results. For the Llama3.2-3B model, SelectInfer attains 9.85 tokens/s, outperforming 4-bit quantization (6.41 tokens/s) by roughly 1.5Ăand exceeding disk offloading (0.75 tokens/s) by more than 13Ă, the latter being heavily bottlenecked by frequent parameter transfers between disk and GPU memory. CoreInfer is excluded from the 3B comparison because it cannot fit within the memory constraints of the device. The substantial speedup of SelectInfer arises from selectively computing only 40% of neurons at runtime, in contrast to quantized 9 4050607080 Loaded neurons (%) 8.6 8.8 9.0 9.2 9.4 9.6 9.8 Speed (tokens/s) SpeedAccuracy 16 18 20 22 24 SQuADv2 accuracy (Exact Match) Figure 10: Effect of selective loading on decod- ing speed and accuracy using Llama3.2-3B. Llama3.2-3BLlama3.2-1B 0 5 10 15 20 Speed (tokens/s) 9.85 6.41 0.75 19.22 19.20 10.71 PartInfer ( = 0.7, = 0.4) CoreInfer ( = 0.4) 4-Bit Quantization Disk Offloading Figure 11: Throughput comparison between baselines. inference, which evaluates all neurons. For the Llama3.2-1B model, SelectInfer and CoreInfer achieve comparable performance (âź19.2 tokens/s), as both similarly restrict computation to 40% of neurons. Meanwhile, the quantized model reaches only 10.71 tokens/s, approximately half the speed of the previous approaches. 6.8 Summary Our evaluation shows that SelectInfer provides a balanced and tunable trade-off across accuracy, mem- ory usage, and decoding speed, metrics that existing approaches struggle to optimize simultaneously. Although 4-bit quantization achieves the smallest memory footprint (about half of SelectInferâs; Ta- ble 5), SelectInfer reaches comparable accuracy while delivering 1.5â2Ăhigher decoding throughput (Fig 11), and it avoids the hardware limitations of low-precision kernels, which are not consistently supported across GPUs [30,31]. Moreover, quantization offers only coarse control (typically 4-bit vs. 8-bit), making it difficult to fully utilize the available memory or computation of edge devices. In contrast, SelectInfer exposes fine-grained knobs that allow users to adjust the percentage of loaded and computed neurons, enabling precise memoryâspeedâaccuracy trade-offs and allowing the model to scale up or down to match device capabilities. Compared to CoreInfer, with fast decoding speed but less accurate and no memory savings, SelectInfer provides a more robust and adaptable framework. 7 Related Work Quantization and compression. Quantization reduces model size by representing weights at lower precision, and has been widely adopted to shrink LLMs for deployment [32,33]. While effective at reducing memory footprint, quantization degrades output quality and, counterintuitively, can reduce decoding throughput on hardware that lacks native support for low-precision kernels [34,30]. Furthermore, it offers only coarse granularity (typically 4-bit or 8-bit), making it difficult to precisely match a modelâs resource consumption to available device capacity. Model compression via weight sharing or knowledge distillation [35] similarly requires expensive retraining and provides limited runtime flexibility. Pruning. Structured and unstructured pruning methods [36,37] reduce the number of active pa- rameters by removing less important weights or neurons. While they can reduce both memory and computation, they typically require fine-tuning to recover lost accuracy, which is costly and task-specific. Unlike SelectInfer, pruning permanently removes weights from the model and cannot adapt to varying tasks or memory budgets at deployment time. Offloading. Offloading approaches such as DeepSpeed [38] and Medusa [39] move portions of the model to CPU or disk, alleviating GPU memory pressure. However, they are fundamentally limited by PCIe bandwidth, require powerful host CPUs, and are incompatible with unified CPUâGPU memory architectures common in edge SoCs such as NVIDIA Jetson [22] and Qualcomm Snapdragon [40]. 10 Neuron sparsity at runtime. A line of work exploits the observation that only a small fraction of neurons strongly activate for any given input [17,15,16]. PowerInfer [41] and PowerInfer-2 [6] train lightweight MLP predictors at each layer to identify active neurons dynamically, but these predictors themselves add approximately 15â20% memory overhead and impose additional computational cost. LLM-in-a-Flash [16] enables inference from flash storage by fetching only active neuron weights on demand, reducing memory pressure but remaining bottlenecked by storage I/O bandwidth. Semantics-aware sparse activation. CoreInfer [11] is the most closely related prior work. It observes that neuron activation patterns cluster by semantic domain and exploits this by identifying active neurons during the prefill phase of each prompt. This yields high decoding throughput by skipping computation over inactive neurons at runtime. However, CoreInfer requires the full model to be loaded into memory and it does not leverage any offline model-level knowledge, relying entirely on the current promptâs prefill to select neurons. As a result, CoreInfer cannot run 3B-scale models on devices with fewer thanâź7 GB of usable memory (e.g., NVIDIA Jetson Orin Nano), and its neuron selection is noisier than SelectInferâs offline-profiled approach. Positioning of SelectInfer. SelectInfer differs from all of the above by jointly addressing memory, speed, and accuracy through two complementary mechanisms: selective loading and selective computation, both guided by an offline profiler that identifies model-specific and task-specific neurons. Unlike quantization, it retains full precision for loaded weights and offers fine-grained, continuously tunable memoryâspeedâaccuracy knobs. Unlike pruning, it requires no retraining. Unlike offloading, it is compatible with unified memory architectures. Unlike runtime-only sparsity methods, it reduces memory footprint, enabling models that would otherwise not fit on a device to run successfully. 8 Conclusion This paper presents SelectInfer, a neuron-level optimization framework for deploying LLM on edge devices. It provides fine-grained control over memory footprint and computational requirements, while maintaining the accuracy of the generated outputs. We define model-specific and task-specific neurons highlighting their distinct roles. We exploit this insight to introduce selective loading to reduce the memory footprint and selective computation to lower computational demands. These optimizations are realized by model knowledge extracted during an offline phase. Evaluations demonstrate effectiveness in enabling LLM inference on resource-constrained devices. Experimental results show the effectiveness of SelectInfer on memory footprint and speedup while maintaining high accuracy. Future work will extend the current approach beyond CPU and GPU execution by incorporating support for neural processing units, such as those available on Jetson devices. In addition, further research will address the current limitation of manually identifying task-specific neurons offline. This could be addressed by integrating lightweight classifiers that allow dynamic neuron selection during inference based on the task type. Finally, to broaden the applicability of the approach, future efforts will focus on generalizing the method to a wider range of models and tasks. 11 References [1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thĂŠe Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971. [2]Daivi. 7 top large language model use cases and applications, 2024. URLhttps://w. projectpro.io/article/large-language-model-use-cases-and-applications/ 887. Accessed: 2026-04-08. [3] Anastasiya Zharovskikh. Best applications of large language models, 2023. URLhttps: //indatalabs.com/blog/large-language-model-apps. Accessed: 2026-04-08. [4]CellStrat.Real-worldusecasesforlargelanguagemod- els(llms),2023.URLhttps://cellstrat.medium.com/ real-world-use-cases-for-large-language-models-llms-d71c3a577bf2 .Ac- cessed: 2026-04-08. [5]Byung-Gon Chun, Sunghwan Ihm, Petros Maniatis, Mayur Naik, and Ashwin Patti. Clonecloud: elastic execution between mobile device and cloud. In Proceedings of the sixth conference on Computer systems, pages 301â314, 2011. [6] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. Powerinfer-2: Fast large language model inference on a smartphone, 2024. URLhttps://arxiv.org/abs/ 2406.06282. [7]Zylon by PrivateGPT. PrivateGPT, May 2023. URLhttps://github.com/zylon-ai/ private-gpt. [8] Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Christopher Leung, Jiajie Tang, and Jiebo Luo. Llm-rec: Personalized recommendation via prompting large language models, 2024. URL https://arxiv.org/abs/2307.15780. [9]NVIDIA Corporation.Jetson orin nano developer kit - 8gb.https://w. nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/ nano-super-developer-kit/, 2025. Accessed: 2026-04-08. [10]Google coral | Edge TPU / Coral Dev Board / Coral Accelerator Modules.https://coral. ai/, 2025. Accessed: 2026-04-08. [11]Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation, 2024. URL https://arxiv.org/abs/2410.18311. [12]Meta AI. Llama 3.2 - 3b model.https://huggingface.co/meta-llama/Llama-3.2-3B, 2023. Accessed: 2026-04-08. [13]Meta AI. Llama 3.2 - 1b model.https://huggingface.co/meta-llama/Llama-3.2-1B, 2023. Accessed: 2026-04-08. [14]Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps:// qwenlm.github.io/blog/qwen2.5/. [15] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher RĂŠ, and Beidi Chen. Deja vu: contextual sparsity for efficient llms at inference time. In Proceedings of the 40th International Conference on Machine Learning, ICMLâ23. JMLR.org, 2023. [16] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12562â12584, 2024. 12 [17] Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https://arxiv.org/abs/2310.04564. [18] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020. [19]Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073â1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1099. URL https://w.aclweb.org/anthology/P17-1099. [20]Karl Moritz Hermann, TomĂĄĹĄ Ko Ë ciskĂ˝, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPSâ15, page 1693â1701, Cambridge, MA, USA, 2015. MIT Press. [21] Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pages 131â198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http://w.aclweb.org/anthology/W/W16/W16-2301. [22]NVIDIACorporation.Jetsonchips.https://w.nvidia.com/en-us/ autonomous-machines/embedded-systems/, 2025. Accessed: 2026-04-08. [23]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250. [24]Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URLhttps://arxiv. org/abs/1705.03551. [25]Patrick Lewis, Barlas O Ě guz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering, 2020. URLhttps://arxiv.org/ abs/1910.07475. [26]Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12â58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URLhttp://w. aclweb.org/anthology/W/W14/W14-3302. [27]Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70â79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5409. URLhttps: //w.aclweb.org/anthology/D19-5409. [28]Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Donât give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, 2018. URL https://arxiv.org/abs/1808.08745. [29] Bitsandbytes.Bitsandbytes quantization library, 2025.URLhttps://github.com/ bitsandbytes-foundation/bitsandbytes. [30] NVIDIA Corporation.Working with quantized types.https://developer.nvidia. com/docs/drive/drive-os/7.0.3/public/drive-os-tensorrt-developer-guide/ work-quantized-types.html, 2025. Accessed: 2026-04-08. 13 [31]Jamie Dborin.Choosing the right quantization for self-hosted llms.https://w. doubleword.ai/resources/behind-the-stack-ep-7--, 2025. Accessed: 2026-04-08. [32]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314. [33]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87â100, 2024. [34]Tae-Hyeon Kim, Jaewoong Lee, Sungjoon Kim, Jinwoo Park, Byung-Gook Park, and Hyungjin Kim. 3-bit multilevel operation with accurate programming scheme in tio x/al2o3 memristor crossbar array for quantized neuromorphic system. Nanotechnology, 32(29):295201, 2021. [35] Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A compre- hensive survey on model compression and acceleration. Artificial Intelligence Review, 53(7): 5113â5155, 2020. [36]Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale, 2023. URL https://arxiv.org/abs/2212.09095. [37]Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702â21720, 2023. [38]Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC â22. IEEE Press, 2022. ISBN 9784665454445. [39]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024. URL https://arxiv.org/abs/2401.10774. [40] Ismet Dagli and Mehmet E Belviranli. Shared memory-contention-aware concurrent dnn execution for diversely heterogeneous system-on-chips. In Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pages 243â256, 2024. [41]Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 590â606, 2024. 14