Paper deep dive
LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
Forough Majidi, Mohammad Mehdi Morovati, Foutse Khomh, Heng Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/5/2026, 4:54:09 AM
Summary
This empirical study investigates the adoption of Large Language Model (LLM) serving frameworks and methods in open-source software systems. It analyzes five key frameworks (vLLM, SGLang, TensorRT-LLM, LMDeploy, FlashInfer) and identifies that vLLM is the most popular. The study highlights that while multi-framework usage is limited, developers combine serving methods like parallel computation and memory management. Adoption varies by model type, modality, and deployment context, supporting diverse applications such as RL-based reasoning and microservices.
Entities (11)
Relation Signals (6)
vLLM โ ismostpopular โ LLM Serving Frameworks
confidence 95% ยท vLLM is the most visible framework in popularity and adoption
LLM Serving Frameworks โ support โ Microservices
confidence 90% ยท LLM serving frameworks support applications and architectures, including ... microservices
FlashInfer โ specializesin โ Kernel Fusion
confidence 88% ยท framework-specific patterns such as the stronger role of kernel fusion in FlashInfer repositories
LLM Serving Frameworks โ support โ Reinforcement Learning
confidence 88% ยท LLM serving frameworks support applications and architectures, including Reinforcement Learning (RL)-based reasoning
vLLM โ hashighestcousage โ FlashInfer
confidence 85% ยท In multi-framework repositories, vLLM has the largest absolute co-usage count, while FlashInfer has the highest co-usage ratio
Developers โ prefer โ Single Framework
confidence 85% ยท Multi-framework usage is limited, suggesting that developers rely on a single serving framework
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are integrated into software systems and AI services, making efficient LLM serving a concern for software engineering. Serving LLMs is challenging because inference requires computation, memory, GPU resources, and execution while maintaining latency and throughput. Although prior research has proposed LLM inference, optimization, and serving techniques and frameworks, little is known about how they are adopted in practice. In this study, we investigate the use of LLM serving frameworks and serving methods in open-source software systems. We identify and analyze five LLM-specific frameworks: vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. We examine how these frameworks and techniques are adopted individually and in combination, how adoption varies across categories of LLMs, and how repositories differ in intent, focus, use case, and architectural design. Our results show that vLLM is the most visible framework in popularity and adoption, while parallel computation, memory management, and network pruning are the most frequently used serving-method categories. Multi-framework usage is limited, suggesting that developers rely on a single serving framework; however, combined frameworks connect complementary capabilities across the serving stack. Framework adoption varies across model families, modalities, model sizes, domain specializations, and deployment settings. Repository-level analysis shows that LLM serving frameworks support applications and architectures, including Reinforcement Learning (RL)-based reasoning, multimodal generation and understanding, microservices, and cloud infrastructure. Overall, this study provides a large-scale empirical characterization of LLM serving framework adoption in practice and offers insights for researchers, framework maintainers, and practitioners working on LLM systems.
Tags
Links
- Source: https://arxiv.org/abs/2608.03036v1
- Canonical: https://arxiv.org/abs/2608.03036v1
Trouble viewing inline? Open PDF directly โ
Full Text
167,260 characters extracted from source content.
Expand or collapse full text
โ 11institutetext: F. Majidi 22institutetext: M. Morovati 33institutetext: F. Khomh 44institutetext: H. Li 55institutetext: SWAT Lab., Polytechnique Montrรฉal, Montrรฉal, Canada 55email: forough.majidi@polymtl.ca, mehdi.morovati@polymtl.ca, foutse.khomh@polymtl.ca, heng.li@polymtl.ca LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs Forough Majidi Mohammad Mehdi Morovati Foutse Khome Heng Li (Received: date / Accepted: date) Abstract Large Language Models (LLMs) are integrated into software systems and AI services, making efficient LLM serving a concern for software engineering. Serving LLMs is challenging because inference requires computation, memory, GPU resources, and execution while maintaining latency and throughput. Although prior research has proposed LLM inference, optimization, and serving techniques and frameworks, little is known about how they are adopted in practice. In this study, we investigate the use of LLM serving frameworks and serving methods in open-source software systems. We identify and analyze five LLM-specific frameworks: vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. We examine how these frameworks and techniques are adopted individually and in combination, how adoption varies across categories of LLMs, and how repositories differ in intent, focus, use case, and architectural design. Our results show that vLLM is the most visible framework in popularity and adoption, while parallel computation, memory management, and network pruning are the most frequently used serving-method categories. Multi-framework usage is limited, suggesting that developers rely on a single serving framework; however, combined frameworks connect complementary capabilities across the serving stack. Framework adoption varies across model families, modalities, model sizes, domain specializations, and deployment settings, indicating differences in support for operational requirements. Repository-level analysis shows that LLM serving frameworks support applications and architectures, including Reinforcement Learning (RL)-based reasoning, multimodal generation and understanding, microservices, and cloud infrastructure. Overall, this study provides a large-scale empirical characterization of LLM serving framework adoption in practice and offers insights for researchers, framework maintainers, and practitioners working on LLM systems. 1 Introduction Large Language Models (LLMs) are increasingly being integrated into software systems and user-facing AI services, driving rapid growth in applications such as conversational agents, code assistants, content generation platforms, and decision-support systems Pan and Li (2025); Chang et al. (2024). As these models transition from research prototypes to production systems, the efficiency of model deployment and serving has become a critical software-engineering concern. Serving LLMs requires managing substantial computational and memory demands while meeting strict latency, throughput, scalability, and cost requirements Miao et al. (2025). Recent surveys highlight that efficient LLM serving depends not only on model execution but also on system-level mechanisms such as request scheduling, memory management, kernel optimization, and resource orchestration Pan and Li (2025). Consequently, LLM serving has emerged as a key systems challenge that directly influences the feasibility, performance, and cost-effectiveness of real-world AI applications. These challenges are further amplified by the diversity of deployment environments. Modern LLM services must operate across heterogeneous hardware platforms, cloud infrastructures, and workload characteristics, often under stringent resource and cost constraints Jiang et al. (2025b). During inference, memory consumption can become a major bottleneck, limiting throughput and increasing operational costs when not managed efficiently Kwon et al. (2023). As a result, scalable LLM deployment increasingly relies on specialized serving frameworks that coordinate model execution, request processing, memory allocation, scheduling, and hardware-specific optimizations. To address these challenges, researchers and practitioners have proposed a growing ecosystem of LLM serving frameworks and optimization techniques. Frameworks such as vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer encapsulate a wide range of capabilities, including memory-efficient attention mechanisms, scheduling strategies, kernel optimizations, model compression techniques, and hardware-aware execution engines Kwon et al. (2023); Zheng et al. (2024); TensorRT ; LMDeploy ; FlashInfer . These frameworks are rapidly becoming foundational infrastructure for deploying production-grade LLM applications. Despite the rapid evolution of LLM serving technologies, our understanding of their adoption in practice remains limited. Prior work has extensively surveyed efficient inference techniques, model compression methods, deployment optimizations, and serving architectures Bai et al. (2024); Wan et al. (2023); Xu et al. (2024); Zhou et al. (2024a); Pan and Li (2025); Miao et al. (2025). Other studies have proposed and evaluated specific algorithmic and systems-level optimizations, including quantization, speculative decoding, GPU-serving architectures, and production-serving infrastructures Xia et al. (2024); Wang et al. (2024); Zhu et al. (2024a); Ding et al. (2024); Gao et al. (2024a); Jiang et al. (2025b); Xia et al. (2023); Beck et al. (2025); Team et al. (2025). However, little empirical evidence exists on how these frameworks and serving methods are actually adopted, combined, and deployed in real software systems. In particular, it remains unclear which serving frameworks have gained traction among developers, how efficient serving methods are used in practice, how frameworks and optimization techniques are combined, which frameworks are preferred for different categories of LLMs, and what types of software systems rely on these technologies. Addressing this gap is important for both research and practice. Understanding real-world adoption patterns can reveal which innovations successfully transition from research into deployment, identify underexplored opportunities for framework development, and help practitioners make informed decisions about serving architectures for production systems. It can also provide framework maintainers with empirical insights into ecosystem needs, integration patterns, and emerging deployment requirements. To address this gap, we conduct an empirical study of LLM serving framework adoption in open-source GitHub repositories. We investigate the adoption of serving frameworks, the efficient serving methods associated with them, framework and method co-usage patterns, framework adoption across different categories of LLMs, and the characteristics of software systems that rely on these frameworks. By connecting serving technologies with repository-level system characteristics, our study provides a comprehensive view of how efficient LLM serving is realized in practice and offers actionable insights for researchers, framework maintainers, and practitioners building production-grade LLM systems. To guide this empirical study, we formulate three research questions. The first question examines which LLM serving frameworks and efficient serving methods are most popular and most adopted in practice. The second question investigates how efficient serving methods and serving frameworks are used in combination, as well as which serving frameworks developers adopt for different types of LLM. The third question examines the types of LLM-based software systems that adopt serving frameworks and how this adoption differs across repository intent, technical focus, primary use case, and system design. RQ1. What are the most popular and widely adopted LLM serving frameworks and efficient serving methods? โข What are the most popular LLM serving frameworks? โข What are the most adopted LLM serving frameworks? โข What are the most used efficient LLM serving methods? RQ2. What are the most used combinations of efficient LLM serving methods and serving frameworks, and which serving frameworks are adopted for different types of LLM? โข What are the most used combinations of efficient LLM serving methods? โข What are the most used combinations of LLM serving frameworks? โข What serving frameworks are adopted by developers for each type of LLMs? RQ3. How are LLM serving frameworks used across different types of LLM-based software systems? โข What types of LLM-based software systems, in terms of repository intent, technical focus, use case, and system design, adopt LLM serving frameworks? โข Which types of LLM-based software systems most commonly adopt each LLM serving frameworks, and how does this adoption vary across repository intent, technical focus, primary use case, and system design? The results show that LLM serving is used unevenly across frameworks, methods, model contexts, and repository types. vLLM is the most visible framework in both popularity and adoption, while parallel computation, memory management, and network pruning are the most frequently observed serving-method categories, with framework-specific patterns such as the stronger role of kernel fusion in FlashInfer repositories. The results also show that developers sometimes combine multiple serving methods within the same framework, whereas multi-framework usage remains limited. In multi-framework repositories, vLLM has the largest absolute co-usage count, while FlashInfer has the highest co-usage ratio, suggesting that it is more often adopted alongside other serving frameworks. These combinations appear to connect complementary capabilities across the serving stack, including scheduling, memory management, parallel execution, and attention- or kernel-level optimization. For model usage, frameworks are associated with different model families, sizes, specializations, and deployment contexts, with vLLM and LMDeploy used with broader model-family ranges than some other frameworks. Finally, repository-level analysis shows that LLM serving frameworks are adopted across diverse contexts, including Mathematical Reasoning with RL, Speech, OCR, and Multimedia Processing Pipelines, Multimodal Generation and Understanding, and Microservices and Cloud Infrastructure. This study makes the following contributions: โข We provide an empirical characterization of LLM serving framework popularity and adoption in open-source GitHub repositories, focusing on LLM-specific serving frameworks selected from literature and GitHub sources. โข We identify and analyze the efficient serving methods used across the studied frameworks, showing how different categories of serving methods, such as parallel computation, memory management, network pruning, request scheduling, quantization, and kernel optimization, are used in open-source repositories. โข We examine how LLM serving is used in combination by analyzing efficient serving method combinations within frameworks, serving-framework combinations across repositories, and the serving frameworks adopted for different types of LLM, including model families, modalities, sizes, specializations, and deployment contexts. โข We characterize the types of LLM-based software systems that adopt serving frameworks by clustering repository summaries and analyzing repository intent, technical focus, primary use case, and system design. โข We provide a replication package containing the materials and datasets used in the study to support transparency and reproducibility Majidi et al. (2026b). The remainder of this paper is organized as follows. Section 2 provides background on LLM and efficient LLM serving. Section 3 describes the methodology used to identify serving frameworks and methods, extract GitHub repositories, and characterize repository-level system designs. Section 4 presents the empirical results for the research questions, including framework and method adoption, framework and method combinations, framework usage across LLM types, and repository-level system designs. Section 6 presents the implications of our findings for researchers, framework maintainers, and developers. Section 5 discusses related work on efficient LLM inference, serving methods, and deployment infrastructure. Section 7 discusses threats to validity. Finally, Section 8 concludes the paper and outlines future work. 2 Background 2.1 Large Language Models AI broadly refers to computational methods that allow machines to carry out tasks commonly associated with human capabilities, including learning, reasoning, language understanding, and decision making IBM (2026b); Russell and Norvig (2020); Gignac and Szodorai (2024). Within AI, generative AI has recently become a prominent area, focusing on the creation of new content such as text, code, images, audio, and video from user prompts Feuerriegel et al. (2024); Banh and Strobel (2023); IBM (2026c). Large Language Models (LLMs) represent one of the main classes of generative AI models. These models are generally trained on large collections of text and code and commonly rely on transformer-based architectures to understand, process, and generate natural language and other types of content Naveed et al. (2023); Banh and Strobel (2023); IBM (2026a). As a result, LLMs have been applied to a wide range of natural language processing tasks, including text generation, summarization, machine translation, question answering, and code generation Qin et al. (2026); Chang et al. (2024); Naveed et al. (2023); Minaee et al. (2024). More recent multimodal extensions further connect language models with visual inputs, supporting applications that combine natural language processing and computer vision Yin et al. (2024); Li et al. (2025b). Beyond general-purpose applications, LLMs are also being adopted across a growing range of domain-specific settings. In sectors such as healthcare, finance, autonomous driving, insurance, and actuarial services, they are being explored to support complex tasks including medical question answering, clinical documentation and summarization, financial analysis, decision support, claims processing, and workflow automation Wang and Zhang (2024); Nie et al. (2024a); Zhu et al. (2024b); Balona (2024); Nie et al. (2024b). Although LLMs have achieved strong results in many language- and software-related tasks, using them in real applications is still challenging. In comparison with many earlier task-specific ML and DL systems, LLMs generally involve larger model sizes, higher memory demands, and greater computation during inference, making their serving costly and resource-intensive Zhou et al. (2024b); Miao et al. (2025). These issues are more critical in production environments, where systems need to handle many users while maintaining low latency and high throughput. For instance, LLM serving often relies on expensive hardware accelerators such as GPUs, and limited hardware capacity can make it difficult for service providers to satisfy performance requirements Jiang et al. (2025b). Memory management is another important bottleneck, since the key-value cache used during generation can increase dynamically for each request and limit how many requests can be processed together Kwon et al. (2023). Therefore, efficient LLM serving depends on methods and frameworks that can reduce resource usage, improve inference efficiency, and support scalable deployment in practical software systems Pan and Li (2025); Miao et al. (2025). 2.2 Efficient Serving Efficient LLM serving refers to deploying large language models in a way that reduces inference latency, lowers computational and memory overhead, and enables scalable operation in real-world applications. This is particularly important because LLM deployment is often resource-intensive, especially in settings that require serving many concurrent users while maintaining low latency and high throughput Miao et al. (2025); Pan and Li (2025); Majidi et al. (2026a). In practice, efficient serving improves response times for end users, enables more effective utilization of hardware resources, and reduces the operational cost of LLM-based systems Kwon et al. (2023); Miao et al. (2025); Majidi et al. (2026a). A range of techniques have been proposed to improve the efficiency of LLM serving. One prominent direction focuses on memory optimization, where system designs more carefully manage activation and KV-cache memory during generation, allowing more concurrent requests to be handled on the same hardware infrastructure Kwon et al. (2023). Another direction focuses on improving the execution of inference requests, such as optimizing request scheduling, batching strategies, and the underlying organization of model computations to better utilize available hardware resources Pan and Li (2025); Miao et al. (2025). These approaches are particularly valuable because they enhance the practical efficiency of LLM systems without altering the functionality or intended behavior of the applications themselves. Beyond these system-level improvements, efficient serving has important broader implications. For end users, it enables faster and more reliable responses in applications such as chatbots, search engines, code assistants, and other LLM-powered services. For organizations, it reduces the infrastructure burden associated with large-scale deployment, particularly when relying on expensive accelerators such as GPUs Miao et al. (2025); Kwon et al. (2023). From an environmental standpoint, inference efficiency is increasingly recognized as a key concern, since LLM serving contributes to significant energy consumption and associated carbon emissions. Recent studies have therefore begun to explore strategies for improving the carbon efficiency of LLM inference workloads Li et al. (2024b); Argerich and Patiรฑo-Martรญnez (2024). Overall, efficient LLM serving is not only a systems optimization problem, but also a practical requirement for deploying LLM-based software in a scalable, cost-effective, and environmentally responsible manner. 3 Methodology This section presents the methodology adopted in this study. Fig. 1 provides a high-level view of the overall research process. All materials and datasets used in this study are publicly available in the replication package Majidi et al. (2026b). Figure 1: High level view of the methodology used for this study 3.1 Extracting LLM Serving Frameworks This study relies on two primary sources to identify LLM serving frameworks: scientific literature and GitHub repositories. To collect relevant research papers, we adopt a systematic search process commonly used in prior studies Majidi et al. (2022). Specifically, we query Google Scholar111https://scholar.google.ca and Engineering Village222https://w.engineeringvillage.com databases using a set of keywords related to cost efficiency and serving LLMs. It is worth noting that we limit our search to papers published after January 1, 2023, since cost efficiency and serving of LLMs only start to receive attention after that time. The keyword that we use in this step is (โlarge language modelโ OR โLLMโ OR โfoundation modelโ OR โfoundationโ) AND (โmodel servingโ OR โmodel deploymentโ OR โinference servingโ OR โservingโ OR โdeploymentโ OR โinferenceโ OR โproductionโ) AND (โcost optimizationโ OR โcost reductionโ OR โcost efficientโ OR โcost effectiveโ OR โcost awareโ OR โresource optimizationโ OR โresource reductionโ OR โefficiency improvementโ). Next, we examine the top 100 results paper from each database, ranked by relevance. We read the whole text of retrieved papers and extract LLM serving or inference frameworks mentioned in the studied papers. Through this process, we identify 46 distinct frameworks reported in the literature for supporting LLM serving and efficiency experiments. The complete list of the identified frameworks from the research papers exist in the replication package of this paper Majidi et al. (2026b). To identify additional LLM serving frameworks that may not capture through our literature review, we conduct a complementary search on GitHub. Specifically, we use combinations of the keywords โLLMโ, โlarge language modelโ, โservingโ, and โcost efficiencyโ to retrieve relevant repositories. This process is conducted using the GitHub REST API v3 developer guideline documentation (2021). As a result, we initially identify 439 repositories. Following prior studies Taraghi et al. (2026); Shah et al. (2026), we filter out repositories with fewer than 500 stars to ensure dataset quality and exclude unpopular projects (based on the literature Li et al. (2025a)), resulting in 26 repositories. Then, we manually examine the remaining repositories and exclude those that meet at least one of the following exclusion criteria. โข Repositories with the descriptions in languages other than English. โข Repositories include only tutorials, examples, and training materials. โข Repositories without any implementation regarding efficiency and serving LLMs. 3.2 Extracting LLM Serving Methods To extract LLM serving methods, we manually review the full text of the research papers collected in the previous step (Subsection 3.1) and identify the techniques proposed to improve inference efficiency and serving performance. During this review, we record the serving methods described in each paper and map them to a common classification framework. Specifically, we adopt the taxonomy proposed by Miao et al. Miao et al. (2025), which provides a comprehensive overview of existing approaches for enhancing LLM serving efficiency. As illustrated in Fig. 2, this taxonomy organizes serving techniques into two broad categories: (1) algorithmic innovations and (2) system-level optimizations Miao et al. (2025). We use this taxonomy as a coding scheme to systematically classify the methods extracted from the literature and to facilitate a consistent comparison of approaches across studies. In the subsequent step, we examine the documentation and technical materials of the serving frameworks identified earlier to determine which of the extracted serving methods are implemented by each framework. This allows us to analyze the adoption of different efficiency-enhancing techniques in existing LLM serving tools. Figure 2: Taxonomy of existing methods on improving the LLM serving efficiency Miao et al. (2025). 3.3 Extracting Repositories In order to analyze the adoption of LLM serving frameworks and serving methods identified in the previous steps (Section 3.1 and 3.2), we collect open-source software systems that use these frameworks in practice. Following prior studies on software ecosystems and ML-based systems Shah et al. (2026); Morovati et al. (2024b), we use GitHub as the primary source for identifying open-source projects. Since Python is the dominant programming language for developing LLM-based applications Morovati et al. (2024a); Taraghi et al. (2026), our analysis focuses exclusively on Python repositories. To identify repositories that use the selected LLM serving frameworks, we adopt a methodology similar to those used in previous large-scale repository mining studies Morovati et al. (2023, 2024a). First, for each framework, we review its official documentation and identify the Python APIs and import statements required to access its functionality. For example, developers typically use the statement import vllm to interact with the vLLM framework vLLM . We then use the GitHub Search API developer guideline documentation (2021) to retrieve Python source files containing these framework-specific API usages. Because the GitHub Search API returns at most 1,000 results per query, a single query may not capture all relevant files for popular frameworks. To overcome this limitation, we partition the search space using the file-size criterion (i.e., size:<min>..<max>), generating multiple non-overlapping queries that each return fewer than 1,000 results. As no relevant files exceeding 500 KB are identified during our preliminary investigation, we restrict our search to files within the 1 byte and 500 KB size range. Next, we divide this range into 500 bytes intervals, yielding approximately 1,000 queries per framework. This systematic querying strategy ensures the comprehensive collection of Python scripts that employ the targeted serving frameworks. After collecting the candidate files, we identify the unique repositories associated with them and apply a set of exclusion criteria, following previous repository-mining studies Morovati et al. (2024b, 2023), to remove repositories that are unlikely to represent mature and actively maintained software projects. Specifically, we exclude: โข Unpopular repositories: repositories with fewer than 10 stars and fewer than 10 forks. โข Inactive repositories: repositories with no recorded activity during the previous year. โข Personal repositories: repositories with only one contributor. โข Repositories with trivial history: repositories containing fewer than 100 commits. To study the adoption of specific serving methods, we further analyze the Python scripts extracted from the selected repositories. Because a single LLM serving framework may support multiple serving techniques, we first identify, from the official documentation of each framework , the APIs associated with the serving methods in our taxonomy. For example, โfrom tensorrt_llm.quantization import โฆโ is an example of APIs that we extracted for Low-bit Quantization method of TensorRT_LLM. We then use a combination of Python regular expressions and Abstract Syntax Tree (AST) analysis Wikipedia ; Majidi et al. (2022) to detect occurrences of these APIs within the collected scripts. Specifically, for each efficient serving method, we first constructed a method-to-API mapping from the official documentation of the selected frameworks, and then used AST analysis to detect relevant imports and parameters in Python scripts, complemented by regular expressions to capture method-related parameter names. By mapping API usages to their corresponding serving methods, we can determine which techniques are adopted in practice and quantify their prevalence across open-source LLM-based software systems. 3.4 Clustering Repositories After identifying the repositories that adopt LLM serving frameworks, we further analyze their repository-level characteristics to understand what types of software systems use these frameworks in practice. To better understand the diversity of LLM serving-related repositories and identify higher-level patterns across their functionalities, we cluster the collected repositories into semantically coherent groups. This enables us to move beyond individual repository-level analysis and instead characterize broader themes, such as deployment-oriented systems, optimization tools, or benchmarking frameworks. The clustering process consists of two main steps. First, we construct structured textual representations of each repository. Second, we apply a topic-driven clustering pipeline over these representations to identify groups of repositories with similar characteristics. 3.4.1 Repository Summary Generation To construct a consistent representation of each repository, we generate structured summaries based on three complementary metadata sources: the README, the about section, and repository keywords. These sources are selected because they capture different levels of abstraction about a project. The README typically provides detailed descriptions of functionality, usage, and design rationale GitHub (2026); Wang et al. (2023), while the about section offers a concise high-level description. In addition, repository keywords (or topics) provide compact semantic labels describing the repositoryโs purpose, domain, or technology stack GitHub . We leverage these complementary signals to generate a unified structured summary for each repository using GPT-4o mini OpenAI (2026). The prompt instructs the model to extract four key dimensions: โข Repository intent: the high-level objective of the repository (e.g., serving, training, benchmarking, or deployment). โข Technical focus: the main techniques or mechanisms implemented. โข Primary use case: the application scenarios supported by the repository. โข System design: the architectural or system-level organization of the solution. These dimensions are designed to provide complementary perspectives on each repository. Repository intent captures the overall goal, which is often implicit or inconsistently described in raw documentation GitHub (2026); Wang et al. (2023); Hellman et al. (2021). Technical focus distinguishes repositories that share similar goals but rely on different underlying methods, such as quantization, parallelism, or scheduling strategies in LLM serving systems Miao et al. (2025); Pan and Li (2025). Primary use case connects the repository to concrete application scenarios such as question answering, code generation, or retrieval-augmented workflows Pan and Li (2025). Finally, system design captures how the repository organizes and deploys its components in practice, which is essential for understanding real-world serving architectures Miao et al. (2025); Pan and Li (2025); Soliman et al. (2025). Overall, these structured summaries provide a normalized and semantically enriched representation of repositories, enabling systematic comparison and downstream clustering. 3.4.2 Summary Clustering We cluster repositories based on the generated summaries in order to identify recurring themes in LLM serving-related software systems. To ensure robustness of the results, we evaluate multiple text preprocessing strategies and compare their impact on clustering quality. Text preprocessing. Following prior work Kumi et al. (2024); Jafari et al. (2025), we construct three versions of the dataset: (i) raw summaries, (i) lightly preprocessed summaries, and (i) fully preprocessed summaries. All versions first undergo a common initial cleaning stage, which includes removing repeated template phrases (e.g., repository intent, technical focus, primary use case, system design), normalizing whitespace, converting text to lowercase, expanding contractions, removing null entries, and eliminating duplicates. In the light preprocessing variant, we tokenize the text using a regex-based tokenizer retaining only alphabetic tokens, remove standard and domain-specific stopwords, and reconstruct the cleaned tokens into text form. In the full preprocessing variant, we further apply part-of-speech filtering (retaining only nouns and verbs), followed by lemmatization, and removal of non-alphabetic or residual stopword tokens. Clustering and topic modeling pipeline. We adopt a topic-driven clustering approach based on semantic text embeddings. First, each repository summary is encoded using the SentenceTransformer model all-MiniLM-L6-v2 Reimers and Gurevych (2019); Janssens et al. (2025), producing dense semantic embeddings that capture contextual similarity between repositories. Next, we reduce the dimensionality of these embeddings using UMAP Kumi et al. (2024); Taraghi et al. (2026), which preserves local semantic structure while improving clustering efficiency. The reduced embeddings are then clustered using HDBSCAN, a density-based clustering algorithm that does not require pre-specifying the number of clusters and can identify outliers as noise Kumi et al. (2024). Finally, we apply BERTopic Kumi et al. (2024); Janssens et al. (2025) to extract interpretable topic representations for each cluster by identifying the most representative terms within each group. In this setup, HDBSCAN is used as the underlying clustering component of BERTopic, while UMAP and sentence embeddings provide the semantic representation space. Model selection. To select the best preprocessing configuration, we evaluate the resulting topic models using two coherence metrics: cvc_v and uโ_โmโaโsโsu\_mass. The cvc_v metric measures the semantic coherence of top topic words (higher values indicate more coherent and interpretable topics), while uโ_โmโaโsโsu\_mass captures statistical co-occurrence patterns in the corpus Kumi et al. (2024) (values closer to zero indicate better coherence). We select the configuration that achieves the best overall balance between these two metrics. This entire pipeline is applied separately to the three preprocessing variants, and the resulting topic structures are compared to assess robustness of the clustering outcomes. 4 Result This section presents the findings of our study by addressing the research questions introduced earlier. 4.1 RQ1. What are the most popular and widely adopted LLM serving frameworks and efficient serving methods? Our analysis of LLM serving frameworks shows that they provide a diverse set of serving methods and functionalities, while attracting varying levels of attention from developers. Furthermore, the results indicate that developers tend to favor certain LLM serving methods over others. 4.1.1 What are the most popular LLM serving frameworks? To identify the most popular LLM serving frameworks, we collect detailed information from the GitHub repositories associated with each tool identified in Section 3.1. Following prior studies Schreiber and Tippe (2025); Das et al. (2022), we use GitHub repository popularity metrics (the number of stars and forks) to rank these tools. We then manually examine all the collected frameworks and filter them based on the following exclusion criteria: โข Frameworks that are not specifically designed for LLM serving but instead provide general-purpose distributed computing infrastructures (e.g., Ray Moritz et al. (2018)) without implementing LLM-specific serving optimizations. โข Frameworks that do not provide a clear description or documentation of the cost-efficient LLM serving methods they implement (e.g., OpenLLM OpenLLM ). โข Frameworks that primarily target proprietary or closed-source LLMs (e.g., RouteLLM RouteLLM ), limiting their applicability and reproducibility within the scope of our study. After applying these criteria, we identified five representative and widely used LLM serving frameworks: vLLM vLLM , SGLang sglang , TensorRT-LLM TensorRT , LMDeploy LMDeploy , and FlashInfer FlashInfer . Table 1 presents the ten most popular frameworks based on GitHub stars and forks as of November 2025. The upper portion of the table contains the frameworks retained for further analysis, whereas the lower portion lists those excluded during the filtering process. The remainder of this paper focuses on the five selected frameworks as a representative set of open-source LLM-serving solutions. Table 1: Top 10 LLM serving frameworks ranked by GitHub stars, with the five selected frameworks for subsequent RQs Framework No. of stars No. of Forks Selected vllm vLLM 56.3K 9.6K SGLang sglang 17.2K 2.7K TensorRT-LLM TensorRT 13.2K 2.2K LMDeploy LMDeploy 6.9K 599 FlashInfer FlashInfer 3.6K 459 Filtered out Ray Ray 38.6K 6.7 OpenLLM OpenLLM 12.3K 805 RouteLLM RouteLLM 4.2K 330 LighLLM LightLLM 3.5K 276 FastDeploy FastDeploy 3.4K 604 4.1.2 What are the most adopted LLM serving frameworks? To assess framework adoption, we analyzed the extracted serving scripts and associated repositories that invoke each framework, as described in Section 3.3. Table 2 reports the number of unique repositories using each framework before and after filtering out unpopular, inactive, personal, and immature repositories according to the criteria defined in Section 3.3. The results indicate that vLLM is by far the most widely adopted framework, with 1,821 repositories remaining after filtering. Its dominant adoption may be partially explained by its early emergence as one of the first frameworks specifically designed to improve the efficiency and cost-effectiveness of LLM serving. Interestingly, popularity and adoption do not always align. For example, FlashInfer ranks last among the selected frameworks in terms of GitHub popularity, yet it is the third most adopted framework. One possible explanation is its relatively recent release, which has limited the time available to accumulate stars and forks despite growing practical usage. Conversely, TensorRT-LLM ranks third in popularity but exhibits the lowest adoption among the selected frameworks. Although it is widely recognized as a high-performance serving solution, its reliance on specialized NVIDIA hardware TensorRT may restrict its accessibility and consequently limit its adoption across open-source repositories which requires further investigation. Overall, these findings suggest that popularity metrics alone do not necessarily reflect real-world adoption. Factors such as hardware requirements, ease of integration, maturity, and ecosystem support appear to play an important role in determining the extent to which a serving framework is adopted in practice. Table 2: Top five LLM serving frameworks ranked by the number of repositories adopting each framework. framework Repos (Before Filtering) Repos (After Filtering) vLLM 7,057 1,821 SGLang 315 54 FlashInfer 371 52 LMDeploy 463 45 TensorRT-LLM 70 15 4.1.3 What are the most used efficient LLM serving methods? To identify the most widely used LLM serving methods, we first extracted the serving techniques supported by the five selected frameworks following the procedure described in Section 3.2. Table 3 summarizes the serving methods implemented by each framework and their adoption across repositories. When mapping framework-specific methods to the taxonomy presented in Fig. 2, we observed that several methods could be assigned to existing subcategories but did not fit any of the predefined sub-subcategories mentioned in Figure 2. To avoid forcing these methods into wrong classes , we introduced three additional sub-subcategories: Additional Decoding Algorithms, Additional Parallel Computation Methods, and Additional Kernel Optimization Methods. Specifically, Additional Parallel Computation Methods include Tensor Parallelism, Pipeline Parallelism, Expert Parallelism, Data Parallelism, and Model Parallelism; Additional Decoding Algorithms include Guided Decoding; and Additional Kernel Optimization Methods include CUDA/HIP Graph Execution, FlashAttention-based Execution, and High-Performance CUDA Kernels. To assess method adoption, we searched the extracted Python scripts for APIs associated with each serving method, following the approach described in Section 3.3. Table 3 reports both the number and percentage of repositories using each method. The total number of repositories associated with each framework is shown in the table header, while the two most frequently used methods within each framework are highlighted in bold. Table 3: Number and percentage of unique repositories using each LLM serving methods. For each framework, Method Support indicates whether a method is supported or implemented by the framework, No. of repo reports the number of unique repositories using that method, and Ratio% reports the percentage relative to all unique repositories using that framework. The numbers in parentheses beside each framework name denote the total number of unique repositories that used that framework. Repos means repositories. The two most commonly used serving methods for each framework are shown in bold text. Method vLLM (1,821 Repos) SGLang (54 Repos) TensorRT-LLM (15 Repos) LMDeploy (45 Repos) FlashInfer (52 Repos) Total Method Support # Repos (Ratio %) Method Support # Repos (Ratio %) Method Support # Repos (Ratio %) Method Support # Repos (Ratio %) Method Support # Repos (Ratio %) # Repo Non-autoregressive Decoding - - - - - - - - - - - Speculative Decoding โ 9 (0.49%) โ 1 (1.85%) โ 1 (6.6%) - - - - 11 Early Exiting โ 1 (0.05%) - - - - - - - - !10 1 Cascade Inference โ - - - - - - - โ 12 (23%) 12 Additional Decoding Algorithms - - - - โ 1 (6.6%) - - - - !10 1 Config Downsizing - - - - - - - - - - - Attention Simplification - - - - - - - - - โ 1 (1.9%) !10 1 Recurrent Unit - - - - - - - - - - - Activation Sharing - - - - - - - - - - - Conditional Computing - - - - - - - - โ 10 ( 19.2%) 10 Knowledge Distillation - - - - - - - - - - - Network Pruning โ 202 (11%) - - โ 1 (6.6%) - - - - !15 203 Low-bit Quantization methods โ 140 (7%) โ 4 (7.4%) โ 2 (13.3%) โ โ 15 (28%) 161 Cloud Scaling - - - - - - - - - - - Decentralized Inference - - - - - - - - - - - Additional Parallel Computation methods โ 977 (53%) โ 22 (40%) โ 3 (20%) โ - โ 8 (15%) !15 1010 Memory Management methods โ 396 (21%) โ 6 (11%) โ 8 (53%) โ 7 (15%) โ 34 (65%) !15 451 Kernel Fusion โ - - - - - - - โ 46 (88%) 46 Tailored Attention - - - - - - - - โ 27 (51%) 27 Variable Sequence Length - - - - - - - - - - - Automatic Compilation - - - - - - - - - - - Additional Kernel Optimizations methods โ 22 (1.2%) - - - - โ - โ 2 (3.8%) 24 Request Scheduling methods โ 121 (6.6%) โ 1 (1.85%) โ 7 (46%) โ - - - 129 The results reveal that Additional Parallel Computation Methods (1,010 repositories), Memory Management Methods (451 repositories), and Network Pruning (203 repositories) are the most widely adopted serving-method categories overall. These findings suggest that developers primarily focus on addressing the two dominant bottlenecks of LLM serving: computational scalability and memory efficiency. However, adoption patterns vary considerably across frameworks. For example, repositories using FlashInfer most frequently employ Kernel Fusion, even though the framework also supports parallel computation and memory-management techniques. This observation is consistent with FlashInferโs design objective of providing highly optimized GPU kernels for inference acceleration Ye et al. (2025). Consequently, developers appear to adopt FlashInfer primarily for its kernel-level optimizations rather than its broader serving capabilities. The prominence of parallel computation and memory-management techniques also helps explain the widespread adoption of vLLM. Prior studies identify computational parallelism and memory consumption as the primary performance bottlenecks in LLM serving Kwon et al. (2023); Qianli et al. (2025); Chen et al. (2026). vLLM was explicitly designed to address these challenges through efficient batching mechanisms and KV-cache management vLLM . By directly targeting the most critical efficiency concerns, vLLM provides capabilities that align closely with practitionersโ needs, which likely contributes to its dominant adoption. In contrast, methods such as early exiting, speculative decoding, attention simplification, and other architecture-level optimizations exhibit very limited adoption. These techniques typically require modifications to model architectures or inference procedures and often necessitate retraining or fine-tuning Elhoushi et al. (2024); Li et al. (2024a). Moreover, many of these approaches remain primarily research-oriented and have not yet been widely integrated into production-ready serving frameworks Miao et al. (2025). Their adoption may also be hindered by compatibility constraints with pre-trained models and potential trade-offs between efficiency gains and output quality Pope et al. (2023); Shi and Ding (2025). Consequently, practitioners appear to favor optimization techniques that are easier to deploy and better supported by existing serving infrastructures. Interestingly, some serving methods are implemented in existing frameworks but are rarely or never observed in practice. For example, although LMDeploy supports LoRA-based serving, we found no evidence of its use in the analyzed repositories. One possible explanation is that LoRA primarily benefits adapter-based fine-tuning scenarios Dettmers et al. (2023), whereas many applications deploy pre-trained models without additional adaptation. Furthermore, LoRA serving introduces additional operational complexity, including the management of base models and adapter weights, as well as compatibility considerations Hu et al. (2022). As a result, LoRA-based serving remains relatively uncommon in production deployments Mao et al. (2025). Finally, several serving methods identified in the literature; including non-autoregressive decoding, configuration downsizing, recurrent architectures, and activation sharing, are not currently implemented in any of the studied frameworks. This absence likely reflects practical deployment considerations. Many of these methods require substantial departures from the transformer-based autoregressive paradigm that underpins most modern LLMs Sun et al. (2023); Peng et al. (2023). Integrating such approaches would therefore require significant changes to both model architectures and serving infrastructures Gu et al. (2017). In addition, these techniques are often evaluated under specialized research settings and may depend on customized training procedures, limiting their applicability to existing pre-trained models Miao et al. (2025). In contrast, serving frameworks tend to prioritize general-purpose optimizations, such as parallel computation and memory management, that can be broadly applied across models and workloads while preserving output quality and compatibility. Finding 1. Among the selected LLM-specific serving frameworks, vLLM is the most visible framework in terms of GitHub popularity and the most adopted framework in the filtered repositories. Across the studied repositories, Additional Parallel Computation, Memory Management, and Network Pruning are the most frequently observed serving-method categories, while method adoption remains framework-specific, with repositories that use FlashInfer showing a strong concentration around Kernel Fusion. 4.2 RQ2. What are the most used combinations of LLM serving frameworks, efficient serving methods, and LLMs? This section examines the extent to which developers combine LLM serving methods, either within a single serving framework or across multiple frameworks, to improve LLM serving efficiency. Our analysis of repositories that use LLM serving frameworks reveals that developers frequently employ multiple serving methods provided by the same framework and, in some cases, integrate several frameworks and their associated optimization techniques within a single system. These combinations are used to address different performance bottlenecks and enhance serving efficiency. Furthermore, this section identifies the most common combinations of serving methods and highlights the LLMs most frequently deployed with each serving framework. 4.2.1 What are the most used combinations of efficient LLM serving methods? In this section, we analyze how serving methods are combined within each studied framework. Specifically, we examine the combinations of methods that are most frequently used together to improve LLM serving efficiency. Figures 3a, 3b, 3c, 3d present the most common method combinations observed in vLLM, SGLang, TensorRT-LLM, and FlashInfer, respectively. LMDeploy is not included in these figures because, among the repositories analyzed, only Memory Management methods were used, and no combinations involving other LMDeploy-supported methods were observed. Consequently, the absence of an LMDeploy plot reflects the usage patterns found in the studied repositories rather than any limitation of LMDeployโs documented capabilities. (a) vLLM (b) SGLang (c) TensorRT-LLM (d) FlashInfer Figure 3: UpSet plot of the most frequent method combinations of each of the vLLM, SGLang, TensorRT-LLM, and FlashInfer frameworks, where bars show usage ratio (%) w.r.t. the total number of repositories using each framework and connected dots indicate the methods in each combination. The patterns shown in Figure 3 provide insights into how LLM serving methods are combined in practice across open-source repositories. Rather than indicating which combinations are the most effective, the results reveal which combinations are most commonly adopted by developers. Consequently, the observed combinations should be interpreted as usage patterns rather than as evidence of superior serving performance. These findings may help developers of LLM serving frameworks identify combinations of methods that are frequently used together in practice. Framework developers can use this information to improve documentation, APIs, tutorials, and deployment examples by considering commonly adopted combinations in addition to describing individual optimization methods in isolation. The results may also assist developers of LLM-based applications during framework configuration and deployment. Instead of selecting serving optimizations independently, practitioners can use the observed combinations as reference points when exploring configuration choices. For example, repositories using vLLM and SGLang frequently combine Memory Management methods with Additional Parallel Computation methods, whereas FlashInfer repositories more often combine Tailored Attention, Memory Management, and Kernel Fusion methods. These observations do not imply that such combinations are optimal; rather, they indicate that these combinations have been adopted by developers in existing repositories and may therefore warrant further evaluation in new deployments. More broadly, the findings highlight the importance of studying interactions among serving methods. While prior work often evaluates individual optimization techniques in isolation, our results suggest that real-world deployments frequently combine multiple techniques simultaneously. Understanding how these methods interact, complement one another, or potentially introduce trade-offs remains an open research question. Finally, the long-tail distribution of observed combinations may be equally informative. Frequently occurring combinations reveal common deployment practices, whereas rarely observed combinations may indicate underexplored design spaces, limited framework support, higher implementation complexity, or simply a lack of adoption within the repositories studied. Future work could investigate the reasons behind both frequent and infrequent combinations and empirically evaluate their impact on serving efficiency, scalability, and resource utilization. A possible interpretation of the combinations in the Figure 3 is that some developers combine serving methods that address different bottlenecks in LLM inference. For vLLM, the combination of Memory Management methods with Additional Parallel Computation methods suggests that repositories may use memory-efficient KV-cache management together with distributed execution to support larger models and improve serving capacity under GPU memory constraints. This interpretation could be supported by the vLLM paper, which states that vLLM uses โblock-level memory management and preemptive request schedulingโ together with PagedAttention, and also supports models whose sizes exceed the memory capacity of a single GPU Kwon et al. (2023). It could also be supported by the vLLM documentation, which states that vLLM supports distributed tensor-parallel and pipeline-parallel inference and serving vLLM (2026a). Therefore, this combination can be interpreted as bringing together two complementary capabilities: reducing KV-cache memory waste and scaling inference across GPU resources Kwon et al. (2023); vLLM (2026a). The combinations involving Request Scheduling methods further suggest that vLLM users may try to improve the latency-throughput trade-off. For example, vLLM documentation explains that chunked prefill can improve inter-token latency by prioritizing decode requests and can improve GPU utilization by batching compute-bound prefill requests with memory-bound decode requests vLLM (2026b). Thus, the three-way combination of Request Scheduling, Memory Management, and Additional Parallel Computation could be interpreted as a possible practical configuration for jointly managing GPU memory, scheduling mixed prefill/decode workloads, and scaling execution across GPU resources Kwon et al. (2023); vLLM (2026a, b). For SGLang, the combination of Memory Management methods with Additional Parallel Computation methods could be interpreted through the interaction between RadixAttention, batching, and parallel execution. The SGLang paper Zheng et al. (2024) explains that RadixAttention enables KV-cache reuse by retaining the cache in a radix tree, and that this design supports efficient prefix search, reuse, insertion, and eviction. The same paper also states that RadixAttention is compatible with continuous batching, paged attention, and tensor parallelism Zheng et al. (2024). Therefore, when repositories combine Memory Management methods with Additional Parallel Computation methods, the combination can be interpreted as bringing together prefix/KV-cache reuse and parallel execution. This may help repositories reduce redundant computation and memory usage for requests with shared prompt prefixes, while still allowing the system to use tensor parallelism for larger models on multiple GPUs Zheng et al. (2024). The additional appearance of Low-bit Quantization methods suggests a complementary optimization path: quantization can reduce the computational and memory costs of inference by representing weights and activations with lower-precision data types Face . For TensorRT-LLM, Figure 3c shows that Memory Management methods appear in 11 of the 12 observed method combinations. This may suggest that memory-related optimization is a recurring part of the observed TensorRT-LLM combinations. TensorRT-LLM lists In-Flight Batching and Paged Attention together with KV Cache Management as advanced optimization features TensorRT-LLM (2026b). In-flight batching improves GPU utilization and reduces latency, while paged KV cache manages the KV cache using blocks assigned to different requests TensorRT-LLM (2026b, a). Therefore, the combination of Memory Management methods with Request Scheduling methods could be interpreted as bringing together request handling and cache management. Combinations with Additional Parallel Computation methods may also be useful when a model cannot fit in a single GPUโs memory or when a single GPU cannot provide the desired performance, since these are cases where multi-GPU parallelism becomes necessary TensorRT-LLM (2026c). Finally, combinations with Low-bit Quantization methods suggest an additional efficiency direction, because quantization reduces memory footprint and computational cost TensorRT-LLM . Overall, these combinations could be bring together memory management, scheduling, parallel execution, and reduced precision. For FlashInfer, Figure 3d shows that the observed combinations are concentrated around Tailored Attention, Memory Management, and Kernel Fusion. This pattern can be interpreted as an emphasis on attention- and kernel-level efficiency, since efficient GPU attention kernels are important for high-throughput and low-latency LLM inference serving Ye et al. (2025). The combination of Tailored Attention with Memory Management can be interpreted as bringing together attention execution and memory-efficient KV-cache handling, since FlashInfer addresses KV-cache storage heterogeneity using block-sparse and composable formats to improve memory access and reduce redundancy Ye et al. (2025). The combination of Tailored Attention with Kernel Fusion can also be interpreted as targeting latency reduction, since the FlashInfer paper reports latency reduction in long-context inference settings Ye et al. (2025). The findings of Figure 3 can help developers of LLM serving frameworks understand what efficient seving methods are combined in practice by other repositories and developers. The findings of Figure 3 will help framework developers improve APIs, documentation, and examples for commonly used method combinations along with documenting each method as an isolated feature. The results can also help developers of LLM-based applications choose and configure serving frameworks more effectively by trying the common combinations of methods and check whether they perform better regarding efficiency. Instead of selecting serving methods independently, developers can use these figures to identify combinations already used in existing GitHub repositories. For example, vLLM and SGLang users may focus on combinations involving Memory Management and Parallel Computation, while FlashInfer users may focus on combinations involving Tailored Attention, Memory Management, and Kernel Fusion to check whether the efficiency improves. The findings can also guide researchers in LLM systems and software engineering. The results show which serving-method combinations are common in practice and which combinations are less visible in open-source repositories. This can motivate future research on how serving methods interact when they are used together, rather than studying each method only in isolation. 4.2.2 What are the most used combinations of LLM serving frameworks? In this section, we examine whether different LLM serving frameworks are used together within the same GitHub repository. While Section 4.2.1 investigated whether multiple serving methods of a single framework are used in combination to further improve the efficiency of LLM serving, it remains unclear whether different frameworksโand their corresponding method categoriesโare also combined in practice. Therefore, this section explores whether GitHub repositories use multiple LLM serving frameworks together and, if so, which methods from these frameworks are used within the same repository. Table 4 reports the count and percentage of repositories that combine each serving framework with other frameworks, showing that framework co-usage differs substantially across the studied frameworks. For example, vLLM has the largest number of repositories that combine it with at least one other framework, with 28 repositories. However, because vLLM is also by far the most frequently used framework in the dataset, this represents only 1.54% of all repositories using vLLM, which is the lowest co-usage ratio in the table. This suggests that although vLLM is widely adopted, most vLLM repositories use it without combining it with another studied serving framework. One possible reason could be that vLLM is designed as a high-throughput distributed LLM serving engine based on PagedAttention, block-level KV-cache memory management, and preemptive request scheduling addressing the main bottleneck of efficient LLM serving, such as high memory usage. Therefore, it could be used as an standalone serving engine Kwon et al. (2023). In contrast, FlashInfer has the highest co-usage ratio, with 23 out of 52 repositories, or 44.23%, also using at least one other studied serving framework A possible interpretation is that FlashInfer is often adopted as a specialized attention- and kernel-level component within a broader LLM serving stack. Also, Ye et al. describe FlashInfer as an efficient and customizable attention engine for LLM serving and report its integration with other serving frameworks Ye et al. (2025). Since FlashInfer also supports KV-cache storage heterogeneity, customizable attention templates, and dynamic scheduling Ye et al. (2025), its higher co-usage ratio may reflect its role as a specialized optimization component that can complement broader serving frameworks. Table 4: The number and the ratio of repositories that use a combination of LLM serving frameworks. โno of reposโ represents the total number of repositories that adopted each serving frameworks. โNum of repos using combined frameworksโ also indicates the number of repositories that combine related LLM serving framework with other frameworks. Framework (num of repos) Num of repos using combined frameworks vLLM (1,821) 28 (1.54%) SGLang (54) 11 (20.37%) TensorRT-LLM (15) 4 (26.67%) LMDeploy (45) 2 (4.44%) FlashInfer (52) 23 (44.23%) Although Table 4 provides an overview of the extent to which each LLM serving framework is used in combination with other frameworks, it does not explain which framework combinations occur or which serving methods are used in those multi-framework repositories. Therefore, we conducted a more detailed analysis on the repositories that use multiple LLM serving frameworks. Table 5 summarizes the framework combinations observed in these repositories. To provide a more fine-grained view, Table LABEL:tab:repo-framework-methods-cleaned presents the repository-level results for all repositories that use multiple LLM serving frameworks. Each row in Table LABEL:tab:repo-framework-methods-cleaned corresponds to one repository. For each repository, the table reports the frameworks used together, the detected APIs or parameters related to serving methods in the โAPIs or parameters usedโ column, and the mapped categories of those APIs or parameters in the โMethod category usedโ column. To focus on recurring framework co-usage patterns, Tables 4 and 5 reports only the framework combinations that appear in at least four repositories. Combinations with fewer than four occurrences provide limited evidence for drawing meaningful conclusions about common co-usage practices, as they may reflect repository-specific design decisions rather than recurring patterns across projects. This threshold helps keep the analysis focused on combinations with stronger empirical support and reduces the risk of over-interpreting sparse observations. Table 5: Co-occurrence of LLM serving frameworks and method categories in multi-framework repositories. Each row corresponds to one framework combination. The โFramework combinationโ column lists the frameworks used together. The โMethods category used from each frameworkโ column reports the method categories observed for each framework in that combination and the โ# Reposโ column reports the number of repositories in which each combination appears. Bold method names indicate method categories implemented by one framework but not by the other frameworks in the same combination. Framework combination Method Category used from each framework # Repos FlashInfer (34.62%), vLLM (0.99%) FlashInfer: Tailored Attention, Cascade Inference, Additional Parallel Computation methods, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods, Attention Simplification vLLM: Additional Parallel Computation methods, Request Scheduling methods, Memory Management methods, Low-bit Quantization methods, Speculative Decoding methods, Network Pruning 18 SGLang (9.26%), vLLM (0.27%) SGLang: Memory Management methods, Additional Parallel Computation methods vLLM: Network Pruning, Additional Parallel Computation methods 5 FlashInfer (7.69%), SGLang (7.41%) FlashInfer: Tailored Attention, Cascade Inference, Additional Parallel Computation methods, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods SGLang: Request Scheduling methods, Low-bit Quantization methods, Additional Parallel Computation methods, Memory Management methods 4 Table 6: Repository-level co-usage of LLM serving frameworks, where each row represents one repository and shows the APIs or parameters used for each framework combination and their corresponding mapped method categories. Framework combination APIs or parameters used Method category used FlashInfer, vLLM FlashInfer: Continuous batching, FlashInfer Attention Kernels, cascade, comm, flashinfer.gemm, fused_moe, norm, page, quantization, sampling, topk vLLM: Chunked prefill, KVCaching, Multi-LoRA, Quantization, Speculative decoding, parallelism FlashInfer: Tailored Attention, Cascade Inference, Parallel Computation methods, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods vLLM: Request Scheduling methods, Memory Management methods, Network Pruning, Quantization, Speculative Decoding, Parallel Computation methods FlashInfer, vLLM FlashInfer: Continuous batching, FlashInfer Attention Kernels, cascade, comm, flashinfer.gemm, fused_moe, norm, page, quantization, sampling, topk vLLM: Chunked prefill, KVCaching, Multi-LoRA, Quantization, parallelism FlashInfer: Tailored Attention, Cascade Inference, Parallel Computation methods, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods vLLM: Request Scheduling methods, Memory Management methods, Network Pruning, Quantization, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, norm, page, sampling vLLM: CUDA, KVCaching, Multi-LoRA, Quantization, parallelism FlashInfer: Tailored Attention, Kernel Fusion, Memory Management methods vLLM: Kernel Optimizations methods, Memory Management methods, Network Pruning, Quantization, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, logits_processor, norm, quantization, sampling vLLM: Quantization, Speculative decoding, parallelism FlashInfer: Tailored Attention, Kernel Fusion, Low-bit Quantization methods vLLM: Quantization, Speculative Decoding, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, norm, page, sampling vLLM: CUDA, KVCaching, Multi-LoRA, Quantization, parallelism FlashInfer: Tailored Attention, Kernel Fusion, Memory Management methods vLLM: Kernel Optimizations methods, Memory Management methods, Network Pruning, Quantization, Parallel Computation methods FlashInfer, vLLM FlashInfer: flashinfer.gemm, fused_moe, quantization, topk vLLM: KVCaching, parallelism FlashInfer: Kernel Fusion, Conditional Computing, Low-bit Quantization methods vLLM: Memory Management methods, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, cascade, comm, norm, page, sampling, topk vLLM: KVCaching, Multi-LoRA, Quantization, Speculative decoding, parallelism FlashInfer: Tailored Attention, Cascade Inference, Parallel Computation methods, Kernel Fusion, Memory Management methods vLLM: Memory Management methods, Network Pruning, Quantization, Speculative Decoding, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, page, quantization vLLM: Multi-LoRA, parallelism FlashInfer: Tailored Attention, Memory Management methods, Low-bit Quantization methods vLLM: Network Pruning, Parallel Computation methods FlashInfer, vLLM FlashInfer: norm vLLM: Multi-LoRA FlashInfer: Kernel Fusion vLLM: Network Pruning FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, green_ctx vLLM: Multi-LoRA FlashInfer: Tailored Attention, Attention Simplification vLLM: Network Pruning FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, cascade, norm, page, sampling vLLM: KVCaching, Multi-LoRA, parallelism FlashInfer: Tailored Attention, Cascade Inference, Kernel Fusion, Memory Management methods vLLM: Memory Management methods, Network Pruning, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, cascade, comm, norm, page, sampling, topk vLLM: CUDA, Chunked prefill, KVCaching, Multi-LoRA, Quantization, Speculative decoding, parallelism FlashInfer: Tailored Attention, Cascade Inference, Parallel Computation methods, Kernel Fusion, Memory Management methods vLLM: Kernel Optimizations methods, Request Scheduling methods, Memory Management methods, Network Pruning, Quantization, Speculative Decoding, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, page, sampling vLLM: CUDA, Chunked prefill, KVCaching, Multi-LoRA, Quantization, Speculative decoding, parallelism FlashInfer: Tailored Attention, Memory Management methods vLLM: Kernel Optimizations methods, Request Scheduling methods, Memory Management methods, Network Pruning, Quantization, Speculative Decoding, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, page, quantization vLLM: Multi-LoRA, Prefix caching, parallelism FlashInfer: Tailored Attention, Memory Management methods, Low-bit Quantization methods vLLM: Network Pruning, Memory Management methods, Parallel Computation methods FlashInfer, vLLM FlashInfer: page, sparse vLLM: KVCaching, Prefix caching, parallelism FlashInfer: Memory Management methods, Attention Simplification vLLM: Memory Management methods, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, page, sampling vLLM: CUDA, Chunked prefill, Quantization, parallelism FlashInfer: Tailored Attention, Memory Management methods vLLM: Kernel Optimizations methods, Request Scheduling methods, Quantization, Parallel Computation methods FlashInfer, vLLM FlashInfer: Continuous batching, FlashInfer Attention Kernels, cascade, comm, flashinfer.gemm, fused_moe, norm, page, quantization, sampling, topk vLLM: CUDA, Chunked prefill, KVCaching, Multi-LoRA, Quantization, parallelism FlashInfer: Tailored Attention, Cascade Inference, Parallel Computation methods, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods vLLM: Kernel Optimizations methods, Request Scheduling methods, Memory Management methods, Network Pruning, Quantization, Parallel Computation methods FlashInfer, vLLM FlashInfer: FlashInfer Attention Kernels, page, sampling vLLM: CUDA, Chunked prefill, KVCaching, Multi-LoRA, Prefix caching, Quantization, parallelism FlashInfer: Tailored Attention, Memory Management methods vLLM: Kernel Optimizations methods, Request Scheduling methods, Memory Management methods, Network Pruning, Quantization, Parallel Computation methods SGLang, vLLM SGLang: all parallelism, data parallelism, pipeline parallelism, tensor parallelism vLLM: Multi-LoRA, parallelism SGLang: Parallel Computation methods vLLM: Network Pruning, Parallel Computation methods SGLang, vLLM SGLang: parallelism vLLM: parallelism SGLang: Parallel Computation methods vLLM: Parallel Computation methods SGLang, vLLM SGLang: all parallelism, data parallelism, pipeline parallelism, tensor parallelism vLLM: parallelism SGLang: Parallel Computation methods vLLM: Parallel Computation methods SGLang, vLLM SGLang: all parallelism, data parallelism, pipeline parallelism, tensor parallelism vLLM: parallelism SGLang: Parallel Computation methods vLLM: Parallel Computation methods SGLang, vLLM SGLang: Continuous batching, parallelism vLLM: parallelism SGLang: Memory Management methods, Parallel Computation methods vLLM: Parallel Computation methods FlashInfer, SGLang FlashInfer: Continuous batching, FlashInfer Attention Kernels, cascade, comm, flashinfer.gemm, fused_moe, norm, page, quantization, topk SGLang: Multi-LoRA, all parallelism, kv_cache FlashInfer: Tailored Attention, Cascade Inference, Parallel Computation methods, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods SGLang: Request Scheduling methods, Parallel Computation methods, Memory Management methods FlashInfer, SGLang FlashInfer: FlashInfer Attention Kernels, cascade, flashinfer.gemm, fused_moe, norm, page, quantization, topk SGLang: Quantization, all parallelism, kv_cache FlashInfer: Tailored Attention, Cascade Inference, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods SGLang: Low-bit Quantization methods, Parallel Computation methods, Memory Management methods FlashInfer, SGLang FlashInfer: FlashInfer Attention Kernels, cascade, flashinfer.gemm, fused_moe, norm, page, quantization, topk SGLang: Quantization, all parallelism, kv_cache FlashInfer: Tailored Attention, Cascade Inference, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods SGLang: Low-bit Quantization methods, Parallel Computation methods, Memory Management methods FlashInfer, SGLang FlashInfer: FlashInfer Attention Kernels, cascade, flashinfer.gemm, fused_moe, norm, page, quantization, topk SGLang: Quantization, all parallelism, kv_cache FlashInfer: Tailored Attention, Cascade Inference, Kernel Fusion, Conditional Computing, Memory Management methods, Low-bit Quantization methods SGLang: Low-bit Quantization methods, Parallel Computation methods, Memory Management methods For the 18 repositories that combine FlashInfer and vLLM in Table 4 and 5, the main pattern is a cross-layer co-usage of FlashInferโs kernel- and attention-oriented methods with vLLMโs serving methods. As shown in Table 5, FlashInfer APIs such as Attention Kernels, cascade, and fused-moe appear together with vLLM APIs or parameters such as KVCache, Chunked prefill, Prefix caching, Multi-LoRA, Quantization, Speculative decoding, and parallelism. The pattern of repositories that combine FlashInfer and vLLM suggests that these repositories use vLLM for serving-level execution, memory management, scheduling, and parallel inference, while relying on FlashInfer for lower-level attention and GPU-kernel optimization. This interpretation is consistent with the vLLMโs design as a high-throughput distributed serving engine based on PagedAttention, block-level KV-cache memory management, and preemptive request scheduling vLLM ; Kwon et al. (2023) and FlashInferโs design as a customizable attention engine which is designed to integrate with serving frameworks such as vLLM Kwon et al. (2023); Ye et al. (2025). The highlighted methods in Table 4 suggest that the co-usage of FlashInfer and vLLM is partly driven by complementary capabilities. On the FlashInfer side, Tailored Attention appears in 15 of the 18 repositories and is frequently used with vLLMโs Parallel Computation methods in 14 repositories, Network Pruning in 13 repositories, and Memory Management methods and Quantization in 11 repositories each. Because each repository may use one or several serving methods at the same time, these counts are not mutually exclusive. Cascade Inference appears in 6 repositories and is always used with vLLMโs Memory Management methods, Network Pruning, and Parallel Computation methods. Similarly, Kernel Fusion appears in 11 repositories and is mainly combined with vLLMโs Parallel Computation methods in 10 repositories, Memory Management methods and Network Pruning in 9 repositories each, and Quantization in 8 repositories. Conditional Computing appears in 4 repositories and is always used with vLLMโs Memory Management methods and Parallel Computation methods. Repositories may have used one or more than one method at the same time . These results show that FlashInfer-specific methods are often paired with vLLM methods that support memory-aware, parallel, and optimized serving execution. The highlighted vLLM methods including Request Scheduling, Network Pruning, and Speculative Decoding, also show a clear pairing with FlashInferโs attention- and kernel-level methods. Request Scheduling appears in 7 repositories and is always used with FlashInferโs Tailored Attention and Memory Management methods, while also appearing with FlashInferโs Cascade Inference, Parallel Computation methods, and Kernel Fusion in 4 repositories each. Network Pruning appears in 14 repositories and is mostly combined with FlashInferโs Tailored Attention in 13 repositories, Memory Management methods in 12 repositories, and Kernel Fusion in 9 repositories. Speculative Decoding appears in 6 repositories and is always used with FlashInferโs Tailored Attention, while also appearing with FlashInferโs Kernel Fusion and Memory Management methods in 5 repositories each. Finally, vLLMโs Parallel Computation methods appear in 16 repositories and are frequently used with FlashInferโs Tailored Attention and Memory Management methods in 14 repositories each and Kernel Fusion in 10 repositories. Overall, this pattern suggests that repositories combining FlashInfer and vLLM may use vLLM for serving-level orchestration, such as scheduling, memory management, pruning, speculative decoding, and parallel execution, while using FlashInfer for specialized attention, cascade inference, kernel fusion, and conditional-computation primitives. This interpretation could be due to vLLMโs design as a high-throughput serving engine based on PagedAttention, KV-cache memory management, and request scheduling, and with FlashInferโs role as an attention and GPU-kernel optimization engine for LLM inference serving Kwon et al. (2023); Ye et al. (2025); vLLM ; FlashInfer . Based on Tables 4 and 5, across the five repositories that combine SGLang and vLLM, the main recurring pattern is the joint use of Parallel Computation methods: all five repositories use vLLMโs โparallelismโ, and all five also use SGLang parallelism-related APIs or parameters. This pattern may suggest that the SGLangโvLLM combination is mainly used in repositories focused on scalable and distributed serving execution, where SGLang provides explicit parallelism controls and vLLM contributes high-throughput serving capabilities based on efficient memory management, scheduling, and batching Zheng et al. (2024); Kwon et al. (2023). Table LABEL:tab:repo-framework-methods-cleaned also shows two additional patterns: one repository combines SGLangโs Continuous batching with vLLMโs parallelism, and one repository combines vLLMโs Network Pruning, with SGLangโs distributed parallelism methods. Overall, these combinations may unlock more efficient serving for workloads that require both distributed parallel execution and support for multiple specialized versions of the same model, rather than only scaling a single base model Zheng et al. (2024); Kwon et al. (2023); vLLM ; sglang . Across the four repositories that combine FlashInfer and SGLang, FlashInferโs Tailored Attention, Cascade Inference, Kernel Fusion, and Conditional Computing methods appear in all four repositories. In each of these repositories, these FlashInfer-specific methods are used together with SGLangโs Memory Management methods, mainly kv-cache, and SGLangโs Parallel Computation methods. In three of the four repositories, these FlashInfer-specific methods are also used with SGLangโs Low-bit Quantization methods. This pattern suggests that the FlashInferโSGLang combination is mostly associated with using FlashInferโs attention- and kernel-level optimization methods alongside SGLangโs runtime-level capabilities for cache management, parallel execution, and, in most cases, quantized serving. One important finding of this section is that the studied LLM serving frameworks are not frequently used together within the same repositories. This suggests that, in practice, developers often rely on a single serving framework rather than combining multiple frameworks and their methods. However, the reasons behind this limited co-usage remain unclear and should be investigated in future studies. Possible explanations that need to be investigated in the future research include the complexity of integrating different serving frameworks, compatibility constraints between specific frameworks and LLMs, and the lack of clear documentation or examples showing how different frameworks and their serving methods can be combined in practice. Therefore, future work should examine the technical and practical barriers that may prevent developers from adopting multi-framework serving configurations when separate tools provide different complementary efficient serving methods. The findings of Section 4.2.2 show that when frameworks are combined, they are often used to connect complementary capabilities across different layers of the serving stack. For example, the FlashInfer and vLLM combination links vLLMโs serving-level methods, such as memory management, scheduling, and parallel execution, with FlashInferโs attention- and kernel-level optimizations. These findings can help practitioners better understand which framework combinations are used in practice and can guide framework maintainers to provide clearer integration support, examples, and documentation. They also motivate future survey research on why multi-framework adoption remains uncommon. 4.2.3 What serving frameworks are adopted by developers for each type of LLMs? In this section, we analyze the LLMs associated with each serving framework along four model-level dimensions: model family, model parameter-size category, model specialization, and model deployment context. These dimensions help characterize which types of LLMs are observed with each framework in the studied GitHub repositories, including the model family they belong to, their reported parameter scale, the task or domain for which they are specialized, and the practical context in which they are deployed. Table 7 summarizes these categories, together with their descriptions and subcategories. Furthermore, Table 8 presents detailed information on the categories and subcategories of LLMs used with each studied framework. Table 7: LLM model categories, subcategories, and their descriptions Category Description Subcategories Model Family Identifies the model family or provider lineage associated with the LLM Baichuan, InternVL, LLaMA, Qwen, ChatGLM, DeepSeek, InternLM, LLaVA, Vicuna Size Captures the reported parameter scale of the LLM Small models upto 2B, Medium models 3B to 9B, Large models 10B to 34B, Very large models 70B plus, Mixture of Experts (MoE) models Specialization Describes the main task, domain, or capability for which the LLM is designed or commonly used General chat assistants, Instruction following, Vision language models, Code generation, Mathematics reasoning, Embedding and retrieval, Grounding and long context, General base models Deployment Context Indicates the practical setting in which the LLM is used Production ready chat models, Instruction serving and agents, Multimodal application models, Research and base models, Edge and resource constrained, Domain specific pipelines Table 8: Detailed information on the LLMs used with the LLM serving frameworks. Category vLLM SGLang TensorRT-LLM LMDeploy FlashInfer Model Family LLaMA family Qwen family GLM family Vicuna family InternLM family LLaMA family Qwen family LLaMA family LLaVA family Vicuna family Qwen family InternLM family LLaMA family Baichuan family InternVL family ChatGLM family Vicuna family DeepSeek family LLaMA family Size Medium models 3B to 30B Small models upto 3B Large models 30B to 100B Very large models 70B plus Medium models 3B to 9B Small models upto 2B MoE models Medium models 3B to 9B Medium models 3B to 9B Large models 10B to 34B Small models upto 2B Very large models 70B plus MoE models MoE models Specialization Instruction following General chat assistants Vision language models General base models Code generation Instruction following Vision language models General chat assistants General chat assistants Instruction following Vision language models General chat assistants Vision language models General base models Instruction following Embedding and retrieval Mathematics reasoning Grounding and long context Code generation Instruction following Deployment Context Instruction serving and agents Production ready chat models Multimodal application models Research and base models Domain specific pipelines Instruction serving and agents Multimodal application models Production ready chat models Production ready chat models Instruction serving and agents Multimodal application models Production ready chat models Multimodal application models Edge and resource constrained Research and base models Instruction serving and agents Domain specific pipelines Instruction serving and agents Identifying the Model Family helps determine whether a serving framework is adopted across a diverse set of LLM ecosystems or is mainly associated with a narrower group of model families. As the Model Family category shows, vLLM and LMDeploy are used with a broader range of model families, whereas FlashInfer is used only with the LLaMA family. This result suggests that vLLM and LMDeploy currently demonstrate broader popularity and compatibility across diverse LLM families. Some LLMs are used exclusively with specific frameworks. For instance, Baichuan, ChatGLM, DeepSeek, and InternVL are used only with LMDeploy, while LLaVA is used only with TensorRT-LLM. In contrast, several LLMs are adopted across multiple frameworks. Notably, LLaMA is used with all frameworks, Qwen is used with vLLM, SGLang, and LMDeploy, InternLM is used with vLLM and LMDeploy, and Vicuna is used with vLLM, TensorRT-LLM, and LMDeploy. The size category is useful because model size is related to compute demand and deployment difficulty Sheng et al. (2023); Gao et al. (2024b). Knowing whether frameworks are used with small, medium, large, very large, or MoE models can help explain which frameworks are adopted for lightweight deployment versus larger and more resource-demanding LLM serving. The description of each subcategory in the category size is provided in what follows. Small models up to 2B have up to about 2B parameters and are typically designed for lightweight or resource-efficient use. Medium models 3B to 9B contain about 3B to 9B parameters and usually balance capability with deployment efficiency. Large models 10B to 34B contain about 10B to 34B parameters and are generally aimed at stronger performance on complex tasks. Very large models 70B plus have 70B or more parameters and are typically used when maximum capability is prioritized over cost. Finally, MoE models use a mixture-of-experts architecture, where only part of the model is activated for each input to improve scaling efficiency. As shown in the size category of Table 8, large and very large LLMs are used only with vLLM and LMDeploy. This finding suggests that developers aiming to serve large or very large LLMs may consider vLLM and LMDeploy as two strong candidate frameworks when productionizing their systems. Category Specialization helps connect each model to the main task, domain, or capability it is intended or commonly documented to support, such as dialogue, instruction following, vision-language understanding, code-related tasks, mathematical reasoning, embedding and retrieval, grounding with external or long context, or downstream adaptation. General chat assistants refer to models designed for dialogue or assistant-style use case. Instruction following models refer to models trained or aligned to follow user intent and task-oriented prompts. Vision language models refer to models that connect visual and language inputs for multimodal understanding. Code generation models refer to models designed for code-related tasks such as program synthesis, code completion, debugging, and code infilling. Mathematics reasoning models refer to models designed or evaluated for mathematical and quantitative problem solving with step-by-step solutions. Embedding and retrieval models refer to models that encode text into vector representations for tasks such as retrieval, reranking, clustering, and classification. Grounding and long context models refer to models or systems that use retrieved external knowledge or extended input contexts to support response generation. Finally, General base models refer to broadly trained foundation models that can be adapted to downstream tasks through fine-tuning or other adaptation methods. Category specialization of Table 8 shows that LMDeploy and vLLM are used with a broader variety of LLMs specialized for different domains, including mathematical reasoning, code generation, and vision-language tasks. In contrast, FlashInfer is used only with LLMs specialized for instruction following. Paleyes et al. (2022); Jouini et al. (2024) . This helps interpret framework usage from a software-engineering perspective, rather than only from the perspective of serving-method usage. The description of each subcategory in the category Deployment Context is provided in what follows. Production ready chat models are models typically used in deployed chat assistants and user-facing conversational applications. Instruction serving and agents are models typically used in systems that execute explicit instructions or power agent-style workflows. Multimodal application models are models typically used in applications that process both text and visual inputs, such as images, documents, or charts. Research and base models are models typically used as base or experimental models for further fine-tuning, evaluation, or research. Edge and resource constrained models are typically used in low-memory or limited-compute environments where efficient deployment is important. Finally, Domain specific pipelines are models typically used in specialized pipelines targeting particular tasks such as coding, math, retrieval, or grounded generation. As shown in the deployment context category of Table 8, LMDeploy is the only framework used with edge and resource-constrained LLMs. This suggests that developers aiming to serve LLMs designed for low-memory or limited-compute environments, where deployment efficiency is particularly important, may consider LMDeploy a strong candidate. In addition, only vLLM and LMDeploy are used with research and base models. Therefore, researchers seeking serving frameworks for such models may consider these two tools as strong candidate choices. The findings of Table 8 show that only LMDeploy is associated with highly specialized use cases, such as embedding retrieval, math reasoning, and specialized grounding. Furthermore, vLLM appears to balance breadth and practicality, as it is used with several model families, multiple specializations, several sizes, and multiple deployment contexts. In contrast, SGLang appears to be more concentrated around mainstream instruction/chat workloads and small- to medium-sized LLM adoption. The findings of Table 8 can benefit several stakeholder groups. Developers of LLM-based applications can use these results to identify which serving frameworks are used with specific model families, model sizes, specializations, and deployment contexts, which may help narrow the set of frameworks to evaluate. Developers and maintainers of LLM serving frameworks can use the findings to understand where their frameworks show broad or limited observed coverage and to improve support, documentation, and examples for underrepresented model categories. MLOps, platform, and infrastructure engineers can also use the results to plan deployment environments based on the types of models served with each framework, especially for large models, multimodal models, MoE models, and edge or resource-constrained settings. Finally, researchers in software engineering and AI systems can use these findings to study how model characteristics are associated with framework adoption in open-source LLM-serving projects. Finding 2. Developers combine multiple serving methods within the same framework to address different serving bottlenecks together, whereas multi-framework usage remains limited. vLLM has the largest absolute co-usage count, while FlashInferโs highest co-usage ratio indicates that it is more often adopted in multi-framework settings. Recurring framework combinations connect complementary capabilities across the serving stack, including scheduling, memory management, parallel execution, and attention- or kernel-level optimization. Framework adoption also varies across LLM types, with vLLM and LMDeploy used across broader model-family ranges than other frameworks. 4.3 RQ3. How are LLM serving frameworks used across different types of LLM-based software systems? This section presents the characteristics and topics of repositories that use serving frameworks. 4.3.1 What types of LLM-based software systems adopt LLM serving frameworks in terms of repository intent, technical focus, use case, and system design? This section presents the results of clustering and topic modeling conducted to identify the main topics of repositories that use LLM serving frameworks from four perspectives: Repository Intent, Technical Focus, Primary Use Cases, and System Design, as described in Section 3.4.2. Tables 9, 10, 11, and 12 presents the topics identified for Repository Intent, Technical Focus, Primary Use Cases, and System Design, respectively, together with a description of each topic in the Description column. As Tables 9, 10, 11, and 12 show, the numbers of identified topics for Repository Intent, Technical Focus, Primary Use Cases, and System Design are 10, 7, 11, and 9, respectively. Table 9 shows the intent of repositories that use the studied serving frameworks. The results presented in Table 9 reveal that repositories using LLM serving frameworks span a diverse set of intents. The discovered repository-intent topics cover high-performance serving infrastructure, RL workflows, retrieval and RAG systems, multimodal understanding, chatbot and agent systems, speech applications, and visual grounding tasks. These findings suggest that LLM serving frameworks are used in general-purpose components for building and deploying complex AI systems. Table 9: Topic modeling results for the Repository Intent summary category. Topic No. Topic Name Description 0 High-Performance GPU Serving Repos focused on high-performance GPU-based serving, optimization, and distributed deployment of DL systems. 1 RL Frameworks Repos providing flexible tools, libraries, and agent-based frameworks for RL and post-training workflows. 2 Mathematical Reasoning with RL Repos centered on RL for mathematical reasoning, including dataset preparation and self-training or verification processes. 3 Retrieval and RAG Systems Repos aimed at information retrieval, question answering, and benchmarking using evaluation metrics and RAG-based approaches. 4 Speech and Voice Applications Repos developing speech and voice systems, including recognition, text-to-speech, and real-time multimodal interaction. 5 General ML and Generative AI Projects Repos showcasing or implementing machine learning and DL projects, including optimization and generative AI applications. 6 Multimodal Vision Understanding Repos focused on multimodal video and image understanding, including benchmarking, tuning, and generation tasks. 7 Chatbot and Agent Systems Repos designed for chatbot systems, enabling interaction, agent functionality, personalization, and automation outputs. 8 Data Generation and Training Pipelines Repos emphasizing code, datasets, and pipelines for data generation, preprocessing, and reproducible training workflows. 9 Visual Grounding and Reasoning Repos targeting visual grounding and reasoning in vision-language systems, including image-based understanding and learning. Topic High-Performance GPU Serving in Table 9 highlights the importance of efficient serving, distributed deployment, and GPU optimization in LLM-based software systems. The existence of a distinct topic around GPU serving suggests that infrastructure optimization is important for developers adopting LLM serving frameworks. Another observation of Table 9 is the presence of reinforcement-learning-related repositories. The identified topics include both general RL frameworks and repositories specifically targeting mathematical reasoning with RL. These findings indicate that LLM serving frameworks are integrated into post-trainingvan Niekerk et al. (2025) and reasoning-oriented workflows rather than being used only for inference. In particular, the existence of a dedicated topic for Mathematical Reasoning with RL suggests interest in using RL to improve reasoning capabilities and self-verification processes in LLM systemsWang et al. (2025). The results of Table 9 also show that retrieval-oriented systems represent a category of repositories using LLM serving frameworks. The Retrieval and RAG Systems topic includes repositories focused on information retrieval, question answering, benchmarking, and retrieval-augmented generation approaches. This finding suggests that integrating external knowledge sources and retrieval mechanisms has occured in LLM-based systems. Furthermore, several identified topics in Table 9 involve multimodal understanding, speech interaction, visual grounding, and vision-language reasoning. In particular, the topics Speech and Voice Applications, Multimodal Vision Understanding, and Visual Grounding and Reasoning indicate that repositories support systems operating across multiple modalities, including text, image, audio, and video. These findings suggest that LLM-based systems are evolving beyond text-only interaction toward more general multimodal AI systems. Another interesting finding of Table 9 is the presence of a dedicated topic for Chatbot and Agent Systems. Repositories within this topic focus on interaction, personalization, automation, and agent functionality. This result indicates that agent-oriented applications form a category of systems using LLM serving frameworks. It also suggests that developers combine LLM serving capabilities with autonomous functionalities. The identified topic Data Generation and Training Pipelines of Table 9 further suggests that repositories using LLM serving frameworks integrate end-to-end engineering workflows. These repositories emphasize dataset preparation, preprocessing pipelines, reproducible training workflows, and data generation processes. This observation indicates that LLM serving frameworks are used as components within broader AI development pipelines rather than as isolated serving modules. In summary, the identified topics listed in Table 9 demonstrate that repositories using LLM serving frameworks combine both infrastructure oriented and application oriented concerns. Some topics focus primarily on deployment efficiency, distributed serving, and GPU optimization, while others focus on application-level functionalities such as retrieval systems, multimodal interaction, chatbot agents, and reasoning systems. This suggests that LLM-based software systems that use the studied serving frameworks span multiple abstraction layers, ranging from low-level serving infrastructure to high-level intelligent applications. Overall, the results of Table 9 suggest that the ecosystem of repositories using LLM serving frameworks is highly heterogeneous and multidisciplinary. The findings of Table 9 can benefit multiple stakeholders, including serving-framework developers, researchers, and technology decision-makers. Developers of LLM serving frameworks can use these findings to better understand the practical contexts in which their frameworks are used, and to improve support for those contexts. Besides, framework builders such as orchestration-tool developers, and middleware providers can use these findings to improve compatibility and integration across the broader LLM stack, especially because the results suggest that serving frameworks are used as part of larger end-to-end AI pipelines rather than as isolated components. Researchers in software engineering and AI systems can use these findings to understand how LLM serving frameworks are being adopted in practice, to identify emerging application domains, and to motivate future research on system support for multimodal, retrieval-enhanced, agent-oriented, and reasoning-intensive workflows. Technology decision-makers can use these findings to better understand the practical landscape of LLM-based software systems, which can help them prioritize investments, evaluate technical directions, and make more informed decisions about infrastructure. Table 10 shows the technical focus of repositories that use the studied serving frameworks. The results in Table 10 show that the technical focus of GitHub repositories using LLM serving frameworks is broad. The identified topics span both model-centric concerns, such as training, tuning, and reinforcement-learning-related workflows, and systems-centric concerns, such as CUDA-based optimization, API deployment, containerization, and Kubernetes based infrastructure. This indicates that repositories using LLM serving frameworks do not focus only on inference execution, but rather operate across the broader technical stack of LLM-based software systems. Table 10: Topic modeling results for the Technical Focus summary category. Topic No. Topic Name Description 0 Model Training and Optimization Repos focused on training, evaluation, optimization, tuning, and performance measurement for learning and generation systems. 1 CUDA Parallelism and Kernel Optimization Repos centered on CUDA-based parallelism techniques, including tensor parallelism, pipeline parallelism, batching, kernel optimization, memory management, and attention computation. 2 Containerized API Deployment Repos focused on Python-based API and containerized deployment workflows, including Docker, PyTorch, FastAPI, OpenAI integration, image handling, and CUDA support. 3 Python Environment and Serving Workflows Repos concerned with Python environments, dependencies, installation, build processes, scripting, evaluation, and serving workflows. 4 Speech, OCR, and Multimedia Processing Pipelines Repos focused on audio, speech, and voice processing, along with object detection, OCR-related components, and transport or connector-based processing pipelines. 5 Hugging Face and RL Integration Repos centered on Hugging Face integration, device mapping, resharding, and RL-related actor or GRPO workflows. 6 Kubernetes-Based AI Deployment Repos focused on Kubernetes-based AI deployment and configuration, including provisioning, generation services, metric collection, FAISS integration, and environment setup. One finding of Table 10 is that the topics Model Training and Optimization and CUDA Parallelism and Kernel Optimization show that repositories emphasize training efficiency, batching, tensor parallelism, pipeline parallelism, kernel optimization, memory management, and attention computation. This suggests that performance and optimization are technical concerns in repositories using LLM serving frameworks. Another important finding of Table 10 is that deployment engineering is a distinct technical focus. The topics Containerized API Deployment and Kubernetes-Based AI Deployment indicate that repositories include practical deployment workflows based on Docker, FastAPI. This suggests that LLM serving frameworks are also used in repositories targeting deployable and operational AI systems. The identified topic Python Environment and Serving Workflows in Table 10 further shows that software environment management is itself a technical concern. The emphasis on dependencies, installation, build processes, scripting, evaluation, and serving workflows suggests that repositories using LLM serving frameworks require engineering effort for execution management. Besides, Table 10 shows that the technical focus of these repositories extends beyond text-only pipelines. The topic Speech, OCR, and Multimedia Processing Pipelines indicates that repositories integrate audio, speech, OCR, and other multimedia components into their processing workflows. This suggests that LLM serving frameworks are used within multimodal AI systems. Another result of Table 10 is the presence of a topic on Hugging Face and RL Integration which suggests that LLM serving frameworks are not used in isolation, but are sometimes embedded within broader post-training pipelinesvan Niekerk et al. (2025). Taken together, the identified topics listed in Table 10 suggest that the technical focus of repositories using LLM serving frameworks combines optimization, deployment, environment management, ecosystem integration, and multimodal processing. In other words, these repositories reflect not only low-level serving concerns, but also the end-to-end technical requirements of building, deploying, and maintaining practical LLM-based systems. A further implication of the Table 10 findings is that repositories using LLM serving frameworks span multiple layers of abstraction. Some topics focus on low-level computational efficiency, such as CUDA kernels, parallelism, and memory management, whereas others focus on higher-level engineering concerns, such as containerization. This suggests that modern LLM serving frameworks function as part of a larger technical ecosystem rather than as standalone inference utilities. The findings of Table 10 can benefit the developers of LLM-based applications choose frameworks that better match their technical needs. Also, it can benefit the developers of LLM serving frameworks prioritize features and integrations that better reflect practical usage contexts. Since the identified topics include CUDA-based optimization, containerized deployment, Kubernetes-based deployment, Hugging Face integration, RL-related workflows, and multimedia pipelines, the results can guide framework developers in improving support for various identified end-to-end engineering needs rather than focusing only on core inference performance. For MLOps engineers, these findings can help them improve deployment pipelines and operational support for LLM systems. The presence of topics related to containerized APIs, Kubernetes-based deployment, dependency management, and execution workflows suggests that LLM systems require robust deployment, orchestration, and reproducibility support. These results can therefore help such stakeholders improve system maintainability, and deployment efficiency. For framework builders and platform providers, the findings can help them improve interoperability across the broader LLM engineering stack. The identified topic on Hugging Face and RL integration, together with topics on deployment workflows and multimedia pipelines, suggests that serving frameworks are used together with other model-development and deployment ecosystems. This can help framework providers identify where better compatibility, integration support, and modular interfaces may be most useful. For researchers, the findings can help them better understand the practical technical landscape of repositories using LLM serving frameworks. In particular, the results provide empirical evidence that modern repositories combine optimization, deployment, environment management, post-training integration, and multimodal processing. This can help researchers identify underexplored technical challenges and motivate future work on end-to-end LLM software engineering across different applications. Table 11 shows the primary use cases of repositories that use the studied serving frameworks. The topics of Table 11 indicate that repositories using LLM serving frameworks support several practical AI use cases rather than a single narrow use case. Table 11: Topic modeling results for the Primary Use Cases summary category. Topic No. Topic Name Description 0 Multimodal Generation and Understanding Repos focused on multimodal image, text, and video generation, understanding, and real-time recognition tasks. 1 Efficient LLM Serving Repos designed for running and serving LLMs efficiently, including multimodal LLM applications. 2 Machine Learning Training and Usage Repos supporting machine learning models and datasets for learning, training, and model usage. 3 Cloud-Based AI Deployment Repos focused on AI model training, inference, and deployment across cloud-based systems and environments. 4 Long-Context QA and Benchmarking Repos centered on language models for long-context understanding, question answering, and benchmarking tasks. 5 Mathematical Reasoning Systems Repos targeting mathematical reasoning, problem solving, and evaluation of reasoning tasks. 6 Model Benchmarking and Fine-Tuning Repos focused on model benchmarking, tuning, fine-tuning, evaluation, and inference across systems. 7 RL Workflows Repos implementing RL algorithms, including training and fine-tuning workflows. 8 Agent-Based Systems Repos designed for agent-based systems, including tools, tool-calling, and handling complex tasks. 9 Retrieval and Recommendation Systems Repos focused on information retrieval, document search, text extraction, and recommendation systems. 10 GPU-Optimized Model Deployment Repos focused on GPU-based model deployment and optimized execution, including kernel-level and single-node serving systems. One finding of Table 11 is that efficient LLM serving itself forms a use-case category. This suggests that efficient serving of LLMs constitute an application goal in their own. The presence of a separate topic for GPU-optimized model deployment further reinforces that optimized execution, is an important practical use case for repositories adopting LLM serving frameworks. Another result of Table 11 is the presence of multimodal use cases. The topic Multimodal Generation and Understanding covers image, text, and video generation, understanding, and real-time recognition tasks. This indicates that repositories using LLM serving frameworks are not limited to text-only systems, but are used to support multimodal applications that integrate multiple input and output modalities. In the Table 11, the topics Machine Learning Training and Usage and Model Benchmarking and Fine-Tuning indicate that there are repositories supporting training, benchmarking, evaluation, and fine-tuning activities. This suggests that LLM serving frameworks are also used within end-to-end AI workflows that connect model development, evaluation, and operational use, rather than being limited to final-stage inference alone. Furthermore, the identified topics of Table 11 also highlight the importance of knowledge-intensive and retrieval-oriented applications. The topics Long-Context QA and Benchmarking and Retrieval and Recommendation Systems show that repositories support question answering, document search, text extraction, benchmarking, and recommendation tasks. This suggests that some repositories using LLM serving frameworks build systems that rely on long-context processing, retrieval mechanisms, and information-access workflows. Another result of Table 11 is the presence of use-case categories for mathematical reasoning and RL workflows. The topics Mathematical Reasoning Systems and RL Workflows indicate there are repositories that use LLM serving frameworks for reasoning-oriented tasks and RL-based training or post-training van Niekerk et al. (2025). The topic Agent-Based Systems in Table 11 provides evidence that the studied frameworks are also used in interactive systems. Since this topic includes tools, tool-calling, and handling complex tasks, it indicates that there are repositories that employ LLM serving frameworks as part of agent-oriented systems. For developers of LLM serving frameworks can benefit from the results of the Table 11 to prioritize improvements for the use cases that appear in practice and broaden framework support beyond standalone inference. Also, the results can guide framework developers toward improving support for diverse downstream workloads and end-to-end usage settings. For researchers, the findings of Table 11 can help them understand the practical use-case landscape of repositories using LLM serving frameworks and identify important directions for future empirical and technical research. Table 12 shows the system design of repositories that use the studied serving frameworks. Table 12: Topic modeling results for the System Design summary category. Topic No. Topic Name Description 0 Scalable Serving Backends Repos focused on scalable backend system design for serving and execution, including API integration, resource management, and performance optimization strategies. 1 Containerized GPU Deployment Repos designed around containerized and GPU-enabled deployment using Docker, cloud services, multi-GPU setups, and service interfaces. 2 Python/Conda Pipeline Organization Repos structured around Python and Conda environments with organized directories, scripts, and pipelines for training, dataset handling, and evaluation. 3 Dependency and Script Management Repos emphasizing dependency management, documentation, community contributions, and command-based script execution workflows. 4 Microservices and Cloud Infrastructure Repos implementing microservices architectures with containerization, orchestration, load balancing, fault tolerance, and scalable cloud infrastructure. 5 Experiment Tracking and GPU Resource Management Repos focused on experiment tracking, GPU resource utilization, scalable mapping, and seamless integration policies for managing experiments. 6 CUDA and Tensor Parallelism Frameworks Repos centered on tensor operations, CUDA kernels, parallelism, and efficient memory management within frameworks like PyTorch. 7 Modular Training Pipelines Repos organized as modular training pipelines with script execution, evaluation integration, and codebase compatibility handling. 8 Kubernetes-Based Cloud Infrastructure Repos designed for Kubernetes-based cloud infrastructure, including cluster scaling, job management, Helm usage, and key management. The topics scalable serving backends and containerized GPU deployment in Table 12 show that there are repositories that emphasize API integration, resource management, performance optimization, Docker-based deployment, cloud services, and multi-GPU execution. This suggests that LLM systems require architectural support for scalable and efficient serving in real deployment environments. The topics Python/Conda pipeline organization, dependency and script management, and modular training pipelines in the Table 12 highlight the importance of organized environments, documentation, and evaluation integration. This indicates that repositories using LLM serving frameworks require engineering structure to support maintainability. A further finding of Table 12 is that resource management is embedded in system design. The topics experiment tracking and GPU resource management and CUDA and Tensor parallelism frameworks show that repositories integrate experiment management, GPU utilization, CUDA kernels, parallelism, and efficient memory management into their architectures. This suggests that memory and resource management are important part of efficient LLM serving. This result further strengthen our findings of the usage of Memory management and parallelization methods presented on Table 3, and Figure 3. Overall, the identified topics of Table 12 suggest that repositories using LLM serving frameworks span multiple architectural layers, from low-level GPU execution to high-level orchestration and cloud deployment. The findings of Table 12 can benefit several stakeholder groups. Developers of LLM serving frameworks can use them to improve support for containerized GPU deployment, scalable cloud infrastructure. MLOps and infrastructure engineers can use them to improve deployment pipelines, and resource management. Finally, researchers can use these findings to better understand the broader architectural role of LLM serving frameworks in practical software systems and to identify important system-level challenges for future work. 4.3.2 Which types of LLM-based software systems most commonly adopt each LLM serving frameworks, and how does this adoption vary across repository intent, technical focus, primary use case, and system design? This section presents the popularity of the topics among the repositories that use each of the studied serving frameworks. Table 13 presents the topics identified in repositories that use vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. As mentioned in Section 4.3.1, the summary categories are Repository Intent, Technical Focus, Primary Use Cases, and System Design. For each topic within each summary category, the table reports the topic name, topic number, and the number of repositories associated with each framework. Therefore, the table provides a comparative overview of how the studied serving frameworks are used in practice, what repository intents are associated with their use, what technical areas they are connected to, what application scenarios they support, and what system design patterns appear in repositories that adopt them. Table 13: Summary categories and topic numbers across vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. Summary categories Topic Name Topic No. vLLM SGLang TensorRT-LLM LMDeploy FlashInfer Repository Intent High-Performance GPU Serving 0 129 53 34 9 36 RL Frameworks 1 60 44 2 1 2 Mathematical Reasoning with RL 2 31 16 2 1 1 Retrieval and RAG Systems 3 32 8 15 2 - Speech and Voice Applications 4 24 6 4 1 - General ML and Generative AI Projects 5 20 4 1 3 - Multimodal Vision Understanding 6 13 11 - 1 - Chatbot and Agent Systems 7 13 3 - 1 - Data Generation and Training Pipelines 8 14 2 - - - Visual Grounding and Reasoning 9 11 7 - - - Technical Focus Model Training and Optimization 0 301 121 15 21 14 CUDA Parallelism and Kernel Optimization 1 69 36 19 7 35 Containerized API Deployment 2 59 17 11 2 7 Python Environment and Serving Workflows 3 21 5 3 2 1 Speech, OCR, and Multimedia Processing Pipelines 4 17 5 11 1 - Hugging Face and RL Integration 5 16 14 - - - Kubernetes-Based AI Deployment 6 14 4 - - - Primary Use Cases Multimodal Generation and Understanding 0 69 31 17 11 3 Efficient LLM Serving 1 60 24 4 4 17 Machine Learning Training and Usage 2 51 17 5 6 5 Cloud-Based AI Deployment 3 44 10 12 1 3 Long-Context QA and Benchmarking 4 36 7 7 4 1 Mathematical Reasoning Systems 5 38 16 2 1 1 Model Benchmarking and Fine-Tuning 6 23 11 2 1 2 RL Workflows 7 24 18 2 1 1 Agent-Based Systems 8 20 5 - - 4 Retrieval and Recommendation Systems 9 15 6 - - - GPU-Optimized Model Deployment 10 13 7 - - - System Design Scalable Serving Backends 0 77 22 5 10 13 Containerized GPU Deployment 1 72 29 20 1 11 Python/Conda Pipeline Organization 2 47 17 5 3 2 Dependency and Script Management 3 35 13 1 1 10 Microservices and Cloud Infrastructure 4 26 4 4 3 1 Experiment Tracking and GPU Resource Management 5 21 20 4 1 - CUDA and Tensor Parallelism Frameworks 6 9 6 - - - Modular Training Pipelines 7 11 7 - - - Kubernetes-Based Cloud Infrastructure 8 12 6 - - - Framework-level analysis: vLLM appears in all identified topics in the Table 13 and has the highest count in all of topics compared to other frameworks. Its most topic occurrences are in Model Training and Optimization, High-Performance GPU Serving, Scalable Serving Backends, Containerized GPU Deployment, CUDA Parallelism and Kernel Optimization, Multimodal Generation and Understanding, and Efficient LLM Serving. One possible reason for this pattern is design of VLLM as a high-throughput and memory-efficient LLM inference and serving engine vLLM . Its documentation highlights memory management, optimized kernels, quantization, parallelism, OpenAI-compatible APIs, tool calling in AI Agents, reasoning parsers, and support for multimodal, embedding, retrieval, and reward models vLLM . Therefore, the wide distribution of vLLM topics in Table 13 may reflect its role as a general-purpose serving infrastructure that is used both in low-level performance-oriented repositories and in higher-level application repositories. SGLang also shows broad coverage across Table 13, although its counts are generally lower than vLLM , which may be partly explained by the lower number of repositories adopting it. Its strongest topics include Model Training and Optimization, High-Performance GPU Serving, RL Frameworks, CUDA Parallelism and Kernel Optimization, Multimodal Generation and Understanding, Containerized GPU Deployment, Efficient LLM Serving, and Scalable Serving Backends. One possible reason for this results could be the focus of SGLang on low-latency and high-throughput serving for large language and multimodal models, using methods such as memory management, parallelism, quantization, and request scheduling sglang . Besides, one possible reason for the presence of SGLang in reinforcement-learning, reasoning, agent topics could be the support by SGLang designed as a system for executing complex language-model programs involving multiple generation calls, control flow, agent control, logical reasoning, RAG pipelinesZheng et al. (2024). TensorRT-LLM shows a more concentrated pattern than vLLM and SGLang across Table 13. Its highest occurrences appear in High-Performance GPU Serving, Containerized GPU Deployment, CUDA Parallelism and Kernel Optimization, Multimodal Generation and Understanding, Retrieval and RAG Systems, Model Training and Optimization, Cloud-Based AI Deployment, and Speech, OCR, and Multimedia Processing Pipelines. One possible reasons for this concentration is its support as an NVIDIA GPU-oriented inference framework for request scheduling, memory management, quantization TensorRT , Docker containers, online serving, and multimodal applications. LMDeploy has lower counts than vLLM, SGLang, and TensorRT-LLM across Table 13, but it appears across several application and deployment topics. Its strongest occurrences are in Model Training and Optimization, Multimodal Generation and Understanding, Scalable Serving Backends, High-Performance GPU Serving, CUDA Parallelism and Kernel Optimization, Machine Learning Training and Usage, Efficient LLM Serving, and Long-Context QA and Benchmarking. One possible reason for the occurrence of these topics is LMDeployโs support for compressing, deploying, and serving LLMs by providing a variety of memory management, parallelism, high-performance CUDA kernels, quantization, and request scheduling methods for resource-efficient LLM serving LMDeploy . LMDeploy also documents support for many LLM and VLM families, including Llama, InternLM, Qwen, Baichuan, ChatGLM, and DeepSeek variants LMDeploy , which support its appearance in multimodal and model-oriented topics. This may also explain why a broad range of LLMs are used with LMDeploy, as shown in the results of Section 4.2.3. FlashInfer shows a specialized pattern in Table 13. Its highest occurrences are in High-Performance GPU Serving, CUDA Parallelism and Kernel Optimization, Efficient LLM Serving, Model Training and Optimization, Scalable Serving Backends, Containerized GPU Deployment, and Dependency and Script Management. One possible reason is of this concentrations is the FlashInferโs design as a framework for kernel generator rather than an end-to-end application framework FlashInfer . Efficient GPU attention kernels are essential for high-throughput and low-latency inference Ye et al. (2025). Therefore, its stronger presence in GPU-serving and kernel-optimization topics, and weaker presence in application-level topics such as RAG, agents, speech, and recommendation, could be explained by its role as a low-level performance component. Summary category level and topic level analysis: Within the Repository Intent category in Table 13, vLLM has the highest count across all topics, while SGLang is the second most frequently used framework in 90% of the intent topics. Both frameworks show their highest counts in the High-Performance GPU Serving, RL Frameworks, and Mathematical Reasoning with RL repository intents. TensorRT-LLM is mainly associated with GPU serving and RAG-related repositories, whereas FlashInfer appears mostly in performance-oriented GPU serving topics, reflecting its role as a kernel-level serving framework rather than a full application framework. Also, LMDeploy has lower counts than vLLM and SGLang, but it appears across several intent topics, suggesting a smaller but relatively diverse observed usage. Besides, Repository Intent High-Performance GPU Serving is the most frequent topic across all studied frameworks. This topic has the highest count for every framework which suggests that one important intent behind a considerable number of repositories is to improve or support efficient LLM inference on GPU-based infrastructures. Within the Technical Focus category in Table 13, Model Training and Optimization is the dominant topic for vLLM, SGLang, TensorRT-LLM, and LMDeploy, while FlashInfer is most strongly associated with CUDA Parallelism and Kernel Optimization. vLLM leads all topics, SGLang is broadly represented and close to vLLM in Hugging Face and RL Integration, TensorRT-LLM is concentrated in CUDA parallelism and kernel-optimized topic, and FlashInferโs pattern reflects its GPU-kernel focus FlashInfer by mostly being used in CUDA Parallelism and Kernel Optimization topic. Within the Primary Use Cases category in Table 13, vLLM again has the highest count across all topics. TensorRT-LLM shows stronger occurrences in Multimodal Generation and Understanding and Cloud-Based AI Deployment, which could possibly be explained by its multimodal and deployment-oriented support TensorRT . In the System Design category in Table 13, vLLM has the highest occurrence in all topics, especially in Scalable Serving Backends and Containerized GPU Deployment, this may indicate its association with scalable and GPU-based deployment architectures. TensorRT-LLM is mainly concentrated in Containerized GPU Deployment. Also, LMDeploy appears with highest counts across Scalable Serving Backends. FlashInfer is mostly associated with Scalable Serving Backends, Containerized GPU Deployment, and Dependency and Script Management. Overall, the results show that scalable backend design and containerized GPU deployment are the dominant system-design patterns, while the more specialized topics, including CUDA and Tensor Parallelism Frameworks, Modular Training Pipelines, and Kubernetes-Based Cloud Infrastructure, appear only for vLLM and SGLang in the studied repositories. Table 13 can benefit several stakeholder groups because it connects each studied serving framework with the types of repositories in which it appears. Developers and maintainers of LLM serving frameworks can use the results of Table 13 to better understand the practical contexts in which their frameworks are adopted. For example, the table can help them identify whether their framework is mostly used for high-performance GPU serving, reinforcement-learning workflows, multimodal applications, cloud deployment, or scalable backend systems. This information can guide framework developers in improving APIs, documentation, examples, and integration support for the repository types in which their frameworks are most frequently used. It can also help them identify application or system-design contexts where their framework has limited observed adoption and may require better support. Application developers who use LLM serving frameworks can also benefit from these results. Table 13 provides empirical evidence about which frameworks are commonly associated with specific application scenarios, such as multimodal generation and understanding, long-context question answering, reinforcement-learning workflows, and so on. Therefore, developers can use the Table 13 as a practical reference when selecting a serving framework for their target application. This can help reduce the effort required to compare multiple frameworks and can support more informed framework selection. MLOps, and infrastructure engineers can use the results of Table 13 to identify the system-level requirements that commonly appear in repositories using LLM serving frameworks. These commonly used system-level topics suggest that LLM serving frameworks are often embedded in broader deployment and infrastructure pipelines rather than used only as isolated inference tools. As a result, infrastructure engineers can use these findings to improve deployment pipelines, resource management, containerization support, and cloud orchestration for LLM-based systems. Researchers in software engineering and AI systems can use the findings of Table 13 to identify important research directions such as system design challenges that the developers may face when using LLM serving frameworks. Finally, technology decision-makers adopting LLM systems can use Table 13 to better understand the practical landscape of LLM serving frameworks. The table can support decisions about which frameworks to evaluate, which infrastructure capabilities to prioritize, and which engineering skills may be needed for deployment. In this way, the results can help organizations make more evidence-based decisions about LLM serving infrastructure. Finding 3. Repositories using LLM serving frameworks cover diverse software-system types, including high-performance serving, RL, RAG, and multimodal systems. Their designs span multiple architectural layers, from low-level GPU execution and resource management to scalable backends, containerized deployment, and cloud orchestration. Topic adoption is uneven: vLLM has the broadest adoption across all four dimensions, while SGLang is also broadly adopted across several contexts. TensorRT-LLM is concentrated in GPU-oriented and deployment-heavy contexts. LMDeploy shows smaller but diverse adoption across training, efficient-serving, long-context, and scalable-backend topics. FlashInfer is mainly specialized around GPU-serving, kernel optimization, and serving-backend infrastructure. 5 Related Works This section presents and discusses the related literature. 5.1 Efficient LLM Inference and Serving Efficient inference and serving of LLMs are important for understanding how LLM-based systems can be deployed with practical latency, throughput, memory use, and deployment efficiency Miao et al. (2025); Pan and Li (2025). Prior studies have examined efficient and resource-efficient LLMs from broad technical perspectives. Bai et al.Bai et al. (2024) review resource-efficient LLM techniques across architecture design, pre-training, fine-tuning, inference, and system design, while Wan et al. Wan et al. (2023) organize efficient LLM research into model-centric, data-centric, and framework-centric directions. In a broader foundation-model context, Xu et al. Xu et al. (2024) study resource-efficient architectures, algorithms, and systems, whereas Zhou et al. Zhou et al. (2024a) focus on efficient LLM inference through data-level, model-level, and system-level optimization. Furthermore, Miao et al. Miao et al. (2025) review efficient LLM serving methods from both algorithmic and system-level perspectives and compare representative open-source GPU-based LLM serving systems. Pan et al. Pan and Li (2025) review LLM inference systems through the inference stack, including request processing, model execution, batching, scheduling, kernel design, and memory management. These studies provide useful background for understanding efficient LLM methods, inference, frameworks, system design, and the technical space of LLM serving. However, their main focus is to review, classify, organize, and conceptually compare existing techniques and systems, rather than empirically examine how LLM serving frameworks, serving methods, framework combinations, and system designs are used in open-source GitHub repositories. Our study complements this line of work by empirically analyzing these aspects in practice. 5.2 Algorithmic Techniques for Efficient LLM Inference Algorithmic techniques are important part of efficient LLM inference because they aim to reduce memory use and computation cost during model execution Wang et al. (2024); Zhu et al. (2024a). Prior work has examined algorithmic and model-level methods for efficient LLM inference. Wang et al. Wang et al. (2024) study LLM compression and inference acceleration from an algorithmic perspective that reduce memory and computation costs during inference. Also, Zhu et al. Zhu et al. (2024a) review LLM model compression techniques, including quantization, network pruning, together with related metrics, benchmarks, and open challenges. Furthermore, Xia et al. Xia et al. (2024) study speculative decoding, which aims to accelerate autoregressive LLM inference, by organizing existing methods into a taxonomy, and comparing representative approaches for inference acceleration. The paper also highlights open challenges such as balancing accuracy and latency, improving batched speculative decoding, and integrating speculative decoding with other efficient inference frameworks and techniques such as vLLM and continuous batching. Both surveys show that efficient inference is closely connected to reducing model size, memory demand, and computational cost while preserving the general capabilities of LLMs, and speculative decoding is one of the efficient inference methods that may be implemented or exposed by LLM serving frameworks. Although Wang et al. (2024); Zhu et al. (2024a) provide useful background for our study because several of these techniques can appear as serving methods supported by LLM serving frameworks, based on the provided summaries, these works mainly organize algorithmic and compression methods, rather than empirically examining how such methods are adopted, combined, or used in open-source GitHub repositories. Similarly, Xia et al. Xia et al. (2024) focus on the method itself, while our study complements these works by analyzing how LLM serving methods, including speculative decoding, framework combinations, and system designs are used in open-source GitHub repositories. 5.3 System-Level Optimization for Efficient LLM Serving System-level optimization is important for practical LLM serving because inference cost and performance can be affected by runtime routing, memory management, hardware allocation, and framework-level execution strategies Ding et al. (2024); Gao et al. (2024a); Jiang et al. (2025a); Xia et al. (2023). Ding et al. (2024); Gao et al. (2024a); Jiang et al. (2025a); Xia et al. (2023) illustrated how such system-level decisions can improve serving efficiency from different perspectives, including query routing, KV-cache reuse, heterogeneous GPU deployment, and sparse GPU-based inference Ding et al. (2024); Gao et al. (2024a); Jiang et al. (2025a); Xia et al. (2023). System-level methods and approaches have also been proposed to improve the cost efficiency of LLM serving at runtime. One approach is query routing, where Ding et al. Ding et al. (2024) study how to reduce inference cost by sending easier queries to a smaller model and harder queries to a larger model while keeping response quality close to that of the larger model. Another approach is memory-aware serving for multi-turn conversations, where Gao et al. Gao et al. (2024a) propose CachedAttention to reuse historical KV caches instead of recomputing them for each new turn. Other system-level studies focus on hardware-aware and framework-level efficiency for LLM serving. Jiang et al. Jiang et al. (2025a) examine cost-efficient serving over heterogeneous cloud GPUs and jointly optimizes GPU composition, deployment configuration, and workload assignment under budget and availability constraints. Xia et al. Xia et al. (2023) introduce Flash-LLM, a GPU-based library that improves large generative model inference by supporting sparse matrix multiplication on tensor cores. These works address different runtime inefficiencies and show that serving efficiency can depend on reducing unnecessary computation, deployment decisions, GPU resources, memory access, and framework-level execution strategies. They are related to our study because they represent concrete serving methods and highlight system-level concerns that may appear in LLM serving frameworks or open-source systems. However, they study specific optimization methods rather than how such methods are adopted, combined, and used across GitHub repositories, and they do not empirically analyze open-source GitHub adoption patterns, framework combinations, or repository-level system designs, which are the main focus of our study. 5.4 LLM Serving Frameworks and Deployment Infrastructure Serving frameworks and deployment infrastructure are important for moving model-serving systems from experimental settings to practical production environments Beck et al. (2025); Team et al. (2025). Beck et al. (2025) evaluates open-source model serving tools through feature comparison, runtime experiments, and experience from real-world ML projects. The study shows that serving tools differ in performance, setup difficulty, framework compatibility, scalability, and production requirements, and that no single tool is best for all serving scenarios. This work also treats model serving as an important part of production ML systems and provides practical guidance for selecting serving environments. More recent work has moved toward production-grade LLM-serving infrastructure. Shan et al. Team et al. (2025) introduce AIBrix as a cloud-oriented LLM inference infrastructure designed to improve large-scale deployment, performance, scalability, and cost efficiency. The framework integrates several serving-related components, including LoRA management, autoscaling, routing, unified runtime support, distributed KV cache, hybrid orchestration, heterogeneous GPU optimization, and diagnostic tools. This work presents a LLM-serving framework and discusses several serving methods used in production-oriented settings. However, Beck et al. Beck et al. (2025) focuse mainly on general-purpose ML model-serving tools and do not examine how LLM serving frameworks and serving methods are adopted or combined across open-source GitHub repositories, while Shan et al. Team et al. (2025) study one framework rather than empirically analyzing how LLM serving frameworks, serving methods, framework combinations, and system designs are used across open-source GitHub repositories. 6 Discussion and Implications Our findings show that efficient serving in open-source repositories is shaped by the interaction between framework capabilities, serving-method availability, types, and repository-level system goals. Developers frequently rely on methods that address common serving bottlenecks, particularly memory management and parallel execution, while framework-specific patterns show more specialized uses, such as the stronger role of kernel- and attention-level optimization in FlashInfer repositories. The results also show that multi-framework usage remains limited, suggesting that most repositories rely on a single studied serving framework. However, when frameworks are combined, recurring combinations such as FlashInfer + vLLM, SGLang + vLLM, and FlashInfer + SGLang connect complementary capabilities across the serving stack, including orchestration, scheduling, memory management, parallel execution, and attention- or kernel-level optimization. Our results provide more specific implications for framework maintainers, developers, and researchers. The RQ1 results show that Additional Parallel Computation, Memory Management, and Network Pruning are the most frequently observed serving-method categories, while FlashInfer repositories show a stronger concentration around Kernel Fusion. This suggests that maintainers can improve documentation and examples around frequently used optimization paths, such as memory-aware serving, parallel execution, and kernel-level optimization. The RQ2 results further show that some repositories combine multiple methods within the same framework and that recurring framework combinations, including FlashInfer + vLLM, SGLang + vLLM, and FlashInfer + SGLang, connect complementary capabilities across scheduling, memory management, parallel execution, and attention- or kernel-level optimization. Therefore, maintainers could provide integration examples, compatibility notes, and configuration templates for common multi-method and cross-framework settings rather than documenting each method only in isolation. The RQ3 results also show that LLM serving frameworks are used across diverse repository-level contexts, such as mathematical reasoning with RL, multimodal generation, multimedia processing pipelines, and microservice-based infrastructure, suggesting that developers and researchers can use these results to select and study serving frameworks with respect to both model characteristics and system-level deployment goals. 7 Threats to Validity This section discusses the main threats to the validity of our study and the strategies used to reduce their possible effects. We organize the threats into construct, internal, and external validity. 7.1 Construct Validity One possible construct validity threat is related to the classification of LLM serving methods. We adopted an established taxonomy of LLM serving methods and studied the official documentation of each selected framework to identify the APIs associated with each method. However, some framework-specific methods may not exactly match the existing taxonomy categories. To mitigate this threat, we carefully reviewed the framework documentation and, when a method did not directly fit the taxonomy, we added additional sub-subcategories instead of forcing it into an inaccurate class. A further construct validity threat is related to the representation of repository characteristics for clustering and topic modeling. Repository descriptions may be incomplete, and generated summaries may not fully capture the actual intent, technical focus, use case, or system design of each repository. To address this threat, we generated repository summaries using three complementary metadata sources: README files, about sections, and repository keywords. 7.2 Internal Validity One possible threat concerns possible errors during the aggregation of scripts into repository-level results. A single repository may contain multiple Python scripts, and each script may include several framework or method APIs. If raw script occurrences were counted directly, the adoption of some frameworks or methods could be overestimated. To reduce this threat, we counted unique repositories rather than raw script occurrences when reporting framework adoption, method usage, and framework combinations. This helped ensure that repositories with repeated API occurrences did not dominate the results. 7.3 External Validity One possible threat is related to the focus on Python scripts. Prior studies identify Python as the most used programming language for developing LLM-based software systems Twist et al. (2026). Thus, we believe that the results of this study are applicable to a considerable portion of LLM-serving systems. Another threat is the focus of this study on open-source GitHub repositories. We reduced this threat by clearly defining the scope of the study as open-source repositories and by following prior empirical software-engineering studies that use GitHub as a source of software systems Kalliamvakou et al. (2014). 8 Conclusion and Future Work As LLM are increasingly integrated into practical software systems, understanding how they are served efficiently in practice has become an important software engineering concern. However, empirical evidence on how serving frameworks and efficient serving methods are adopted and combined in open-source repositories remains limited. To address this gap, this study analyzed five serving frameworks, their supported efficient serving methods, method and framework combinations, the serving frameworks adopted for different types of LLM, and the repository-level characteristics of software systems that adopt these frameworks in terms of repository intent, technical focus, primary use case, and system design. The results show that vLLM is the most popular and adopted framework, while parallel computation, memory management, and network pruning are among the most frequently observed efficient serving-method categories. The results also show that developers combine multiple serving methods within frameworks, whereas multi-framework usage remains limited. In multi-framework repositories, vLLM has the largest absolute co-usage count, while FlashInfer has the highest co-usage ratio, indicating that it is more often adopted alongside other serving frameworks. These recurring combinations connect complementary capabilities across the serving stack, including scheduling, memory management, parallel execution, and attention- or kernel-level optimization. Furthermore, the findings show that serving-framework adoption differs across model families, modalities, sizes, specializations, and deployment contexts, and that serving frameworks are used across diverse repository-level contexts, including mathematical reasoning with RL, multimedia processing pipelines, multimodal generation, and microservice-based infrastructure. Overall, these findings provide empirical evidence for researchers, developers, and framework maintainers about how serving frameworks and efficient serving methods are used in real open-source software systems. Future work can extend this study by examining how AI-agent systems that rely on LLM can be served efficiently, and by conducting empirical studies to identify the practical challenges developers face when efficiently deploying and serving LLM in real-world software systems. 9 Declaration Ethical approval: Not applicable. This study does not involve any human participants or animals. Therefore, we do not require ethical approval. Informed consent: Not applicable. This study does not involve a survey or human participants. Therefore, we do not require informed consent. Author contributions: Forough Majidi and Mehdi Morovati proposed the study idea and approach, revised the methodology, interpreted the results, developed the Python scripts, and contributed to drafting and revising the manuscript, with Forough Majidi taking the lead across these activities. Foutse Khomh and Heng Li contributed to refining the study idea, approach, and methods. They also improved the manuscript by restructuring the paper, identifying and addressing writing and presentation issues, and proofreading the manuscript. Data availability: All data used for analysis, as well as the scripts and the analysis results, are provided in Majidi et al. (2026b). Conflict of interest: The authors declared that they have no conflict of interest. Clinical Trial Number: Not applicable References M. F. Argerich and M. Patiรฑo-Martรญnez (2024) Measuring and improving the energy efficiency of large language models inference. IEEE Access 12, p. 80194โ80207. Cited by: ยง2.2. G. Bai, Z. Chai, C. Ling, S. Wang, J. Lu, N. Zhang, T. Shi, Z. Yu, M. Zhu, Y. Zhang, et al. (2024) Beyond efficiency: a systematic survey of resource-efficient large language models. arXiv preprint arXiv:2401.00625. Cited by: ยง1, ยง5.1. C. Balona (2024) ActuaryGPT: applications of large language models to insurance and actuarial work. British Actuarial Journal 29, p. e15. Cited by: ยง2.1. L. Banh and G. Strobel (2023) Generative artificial intelligence. Electronic Markets 33, p. 63. External Links: Document Cited by: ยง2.1. N. Beck, B. J. Stein, L. Helmer, and D. Wegener (2025) Evaluation of tools and frameworks for machine learning model serving. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), p. 80โ90. Cited by: ยง1, ยง5.4. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. (2024) A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15 (3), p. 1โ45. Cited by: ยง1, ยง2.1. Z. Chen, T. Bu, C. Song, X. Lu, Y. Ye, and Z. Zhou (2026) A universal load balancing principle and its application to large language model serving. arXiv preprint arXiv:2601.17855. Cited by: ยง4.1.3. A. Das, G. Uddin, and G. Ruhe (2022) An empirical study of blockchain repositories in github. In Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering, p. 211โ220. Cited by: ยง4.1.1. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) Qlora: efficient finetuning of quantized llms. Advances in neural information processing systems 36, p. 10088โ10115. Cited by: ยง4.1.3. G. developer guideline documentation (2021) GitHub rest api. Note: ://developer.github.com/v3/Accessed: 2025-09-05 Cited by: ยง3.1, ยง3.3. D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. Lakshmanan, and A. H. Awadallah (2024) Hybrid llm: cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618. Cited by: ยง1, ยง5.3. M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, et al. (2024) Layerskip: enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12622โ12642. Cited by: ยง4.1.3. [13] H. Face Quantization. Note: ://huggingface.co/docs/optimum/en/concept_guides/quantizationAccessed: 2026-05-11 Cited by: ยง4.2.1. [14] FastDeploy () Official document of fastdeploy. Note: ://github.com/PaddlePaddle/FastDeployAccessed: 2026-04-01 Cited by: Table 1. S. Feuerriegel, J. Hartmann, C. Janiesch, and P. Zschech (2024) Generative ai. Business & Information Systems Engineering 66, p. 111โ126. External Links: Document Cited by: ยง2.1. [16] FlashInfer () Official document of flashinfer. Note: ://github.com/flashinfer-ai/flashinferAccessed: 2026-04-01 Cited by: ยง1, ยง4.1.1, ยง4.2.2, ยง4.3.2, ยง4.3.2, Table 1. B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo (2024a) \cost-Efficient\ large language model serving for multi-turn conversations with \cachedattention\. In 2024 USENIX annual technical conference (USENIX ATC 24), p. 111โ126. Cited by: ยง1, ยง5.3. S. Gao, C. Lin, T. Hua, Z. Tang, Y. Shen, H. Jin, and Y. Hsu (2024b) DISP-LLM: dimension-independent structural pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: ยง4.2.3. G. E. Gignac and E. T. Szodorai (2024) Defining intelligence: bridging the gap between human and artificial perspectives. Intelligence 104, p. 101832. External Links: Document Cited by: ยง2.1. [20] GitHub () Classifying your repository with topics. Note: ://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/classifying-your-repository-with-topicsAccessed: 2026-04-28 Cited by: ยง3.4.1. GitHub (2026) About readmes. Note: ://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-readmesAccessed: 2026-04-28 Cited by: ยง3.4.1, ยง3.4.1. J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher (2017) Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281. Cited by: ยง4.1.3. J. Hellman, E. Jang, C. Treude, C. Huang, and J. L. Guo (2021) Generating github repository descriptions: a comparison of manual and automated approaches. arXiv preprint arXiv:2110.13283. Cited by: ยง3.4.1. E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: ยง4.1.3. IBM (2026a) What are large language models (llms)?. Note: ://w.ibm.com/think/topics/large-language-modelsAccessed: 2026-04-30 Cited by: ยง2.1. IBM (2026b) What is artificial intelligence (ai)?. Note: ://w.ibm.com/think/topics/artificial-intelligenceAccessed: 2026-04-30 Cited by: ยง2.1. IBM (2026c) What is generative ai?. Note: ://w.ibm.com/think/topics/generative-aiAccessed: 2026-04-30 Cited by: ยง2.1. M. Jafari, F. Majidi, and A. Heydarnoori (2025) Prioritizing app reviews for developer responses on google play. arXiv preprint arXiv:2502.01520. Cited by: ยง3.4.2. W. Janssens, M. Bogaert, and D. V. d. Poel (2025) LLM-assisted topic reduction for bertopic on social media data. arXiv preprint arXiv:2509.19365. Cited by: ยง3.4.2, ยง3.4.2. Y. Jiang, F. Fu, X. Yao, G. He, X. Miao, A. Klimovic, B. Cui, B. Yuan, and E. Yoneki (2025a) Demystifying cost-efficiency in llm serving over heterogeneous gpus. arXiv preprint arXiv:2502.00722. Cited by: ยง5.3. Y. Jiang, F. Fu, X. Yao, T. Wang, B. Cui, A. Klimovic, and E. Yoneki (2025b) Thunderserve: high-performance and cost-efficient llm serving in cloud environments. Proceedings of Machine Learning and Systems 7. Cited by: ยง1, ยง1, ยง2.1. O. Jouini, K. Sethom, A. Namoun, N. Aljohani, M. H. Alanazi, and M. N. Alanazi (2024) A survey of machine learning in edge computing: techniques, frameworks, applications, issues, and research directions. Technologies 12 (6), p. 81. External Links: Document, Link Cited by: ยง4.2.3. E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian (2014) The promises and perils of mining github. In Proceedings of the 11th Working Conference on Mining Software Repositories, MSR 2014, New York, NY, USA, p. 92โ101. External Links: Document Cited by: ยง7.3. S. Kumi, C. Snow, R. K. Lomotey, and R. Deters (2024) Uncovering concerns of citizens through machine learning and social network sentiment analysis. Ieee Access 12, p. 94885โ94913. Cited by: ยง3.4.2, ยง3.4.2, ยง3.4.2, ยง3.4.2. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611โ626. Cited by: ยง1, ยง2.1, ยง2.2, ยง2.2, ยง2.2, ยง4.1.3, ยง4.2.1, ยง4.2.2, ยง4.2.2, ยง4.2.2, ยง4.2.2, ยง4.2.2. B. Li, Y. Jiang, V. Gadepally, and D. Tiwari (2024a) Llm inference serving: survey of recent advances and opportunities. In 2024 IEEE High Performance Extreme Computing Conference (HPEC), p. 1โ8. Cited by: ยง4.1.3. B. Li, Y. Jiang, V. Gadepally, and D. Tiwari (2024b) Sprout: green generative ai with carbon-efficient llm inference. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 21799โ21813. Cited by: ยง2.2. H. Li, H. Zhang, and A. E. Hassan (2025a) The rise of ai teammates in software engineering (se) 3.0: how autonomous coding agents are reshaping software engineering. arXiv preprint arXiv:2507.15003. Cited by: ยง3.1. Z. Li, X. Wu, H. Du, F. Liu, H. Nghiem, and G. Shi (2025b) A survey of state of the art large vision language models: benchmark evaluations and challenges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, p. 1587โ1606. Cited by: ยง2.1. [40] LightLLM () Official document of lightllm. Note: ://github.com/ModelTC/LightLLMAccessed: 2026-04-01 Cited by: Table 1. [41] LMDeploy () Official document of lmdeploy. Note: ://github.com/InternLM/lmdeployAccessed: 2026-04-01 Cited by: ยง1, ยง4.1.1, ยง4.3.2, Table 1. F. Majidi, F. Khomh, H. Li, and A. Nikanjam (2026a) An efficient model maintenance approach for mlops. Empirical Software Engineering 31 (1), p. 21. Cited by: ยง2.2. F. Majidi, M. M. Morovati, and F. Khomh (2026b) External Links: Link Cited by: 5th item, ยง3.1, ยง3, ยง9. F. Majidi, M. Openja, F. Khomh, and H. Li (2022) An empirical study on the usage of automated machine learning tools. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), p. 59โ70. Cited by: ยง3.1, ยง3.3. Y. Mao, Y. Ge, Y. Fan, W. Xu, Y. Mi, Z. Hu, and Y. Gao (2025) A survey on lora of large language models. Frontiers of Computer Science 19 (7), p. 197605. Cited by: ยง4.1.3. X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia (2025) Towards efficient generative large language model serving: a survey from algorithms to systems. ACM Computing Surveys 58 (1), p. 1โ37. Cited by: ยง1, ยง1, ยง2.1, ยง2.2, ยง2.2, ยง2.2, Figure 2, ยง3.2, ยง3.4.1, ยง4.1.3, ยง4.1.3, ยง5.1. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao (2024) Large language models: a survey. arXiv preprint arXiv:2402.06196. External Links: Document, 2402.06196 Cited by: ยง2.1. P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, et al. (2018) Ray: a distributed framework for emerging \ai\ applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), p. 561โ577. Cited by: 1st item. M. M. Morovati, A. Nikanjam, F. Khomh, and Z. M. Jiang (2023) Bugs in machine learning-based systems: a faultload benchmark. Empirical Software Engineering 28 (3), p. 62. Cited by: ยง3.3, ยง3.3. M. M. Morovati, A. Nikanjam, and F. Khomh (2024a) Fault localization in deep learning-based software: a system-level approach. arXiv preprint arXiv:2411.08172. Cited by: ยง3.3, ยง3.3. M. M. Morovati, A. Nikanjam, F. Tambon, F. Khomh, and Z. M. Jiang (2024b) Bug characterization in machine learning-based systems. Empirical Software Engineering 29 (1), p. 14. Cited by: ยง3.3, ยง3.3. H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian (2023) A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435. External Links: Document, 2307.06435 Cited by: ยง2.1. Y. Nie, Y. Kong, X. Dong, J. M. Mulvey, H. V. Poor, Q. Wen, and S. Zohren (2024a) A survey of large language models for financial applications: progress, prospects and challenges. arXiv preprint arXiv:2406.11903. Cited by: ยง2.1. Y. Nie, Y. Kong, X. Dong, J. M. Mulvey, H. V. Poor, Q. Wen, and S. Zohren (2024b) A survey of large language models for financial applications: progress, prospects and challenges. arXiv preprint arXiv:2406.11903. External Links: Document, 2406.11903 Cited by: ยง2.1. OpenAI (2026) Note: Accessed: 2026-03-18 External Links: Link Cited by: ยง3.4.1. [56] OpenLLM () Official document of openllm. Note: ://github.com/bentoml/OpenLLMAccessed: 2026-04-01 Cited by: 2nd item, Table 1. A. Paleyes, R. Urma, and N. D. Lawrence (2022) Challenges in deploying machine learning: a survey of case studies. ACM Computing Surveys 55 (6), p. 1โ29. External Links: Document, Link Cited by: ยง4.2.3. J. Pan and G. Li (2025) A survey of llm inference systems. arXiv preprint arXiv:2506.21901. Cited by: ยง1, ยง1, ยง2.1, ยง2.2, ยง2.2, ยง3.4.1, ยง5.1. B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, L. Derczynski, et al. (2023) Rwkv: reinventing rnns for the transformer era. In Findings of the association for computational linguistics: EMNLP 2023, p. 14048โ14077. Cited by: ยง4.1.3. R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean (2023) Efficiently scaling transformer inference. Proceedings of machine learning and systems 5, p. 606โ624. Cited by: ยง4.1.3. L. Qianli, H. Zicong, C. Fahao, L. Peng, and G. Song (2025) Mell: memory-efficient large language model serving via multi-gpu kv cache management. arXiv preprint arXiv:2501.06709. Cited by: ยง4.1.3. L. Qin, Q. Chen, X. Feng, Y. Wu, Y. Zhang, Y. Li, M. Li, W. Che, and P. S. Yu (2026) Large language models meet nlp: a survey. Frontiers of Computer Science 20 (11), p. 2011361. Cited by: ยง2.1. [63] Ray () Official document of ray. Note: ://github.com/ray-project/rayAccessed: 2026-04-01 Cited by: Table 1. N. Reimers and I. Gurevych (2019) Sentence-bert: sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 3982โ3992. Cited by: ยง3.4.2. [65] RouteLLM () Official document of routellm. Note: ://github.com/lm-sys/RouteLLMAccessed: 2026-04-01 Cited by: 3rd item, Table 1. S. J. Russell and P. Norvig (2020) Artificial intelligence: a modern approach. 4 edition, Pearson. External Links: ISBN 9780134610993, Link Cited by: ยง2.1. M. Schreiber and P. Tippe (2025) Security vulnerabilities in ai-generated code: a large-scale analysis of public github repositories. In International Conference on Information and Communications Security, p. 153โ172. Cited by: ยง4.1.1. [68] sglang () Official document of sglang. Note: ://github.com/sgl-project/sglangAccessed: 2026-04-01 Cited by: ยง4.1.1, ยง4.2.2, ยง4.3.2, Table 1. M. B. Shah, M. M. Morovati, M. M. Rahman, and F. Khomh (2026) Characterizing faults in agentic ai: a taxonomy of types, symptoms, and root causes. arXiv preprint arXiv:2603.06847. Cited by: ยง3.1, ยง3.3. Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Re, I. Stoica, and C. Zhang (2023) FlexGen: high-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 31094โ31116. External Links: Link Cited by: ยง4.2.3. T. Shi and Y. Ding (2025) Systematic characterization of llm quantization: a performance, energy, and quality perspective. arXiv preprint arXiv:2508.16712. Cited by: ยง4.1.3. M. Soliman, M. Albonico, I. Malavolta, and A. Wortmann (2025) Mining software repositories for software architectureโa systematic mapping study. Information and Software Technology 181, p. 107677. Cited by: ยง3.4.1. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei (2023) Retentive network: a successor to transformer for large language models. arXiv preprint arXiv:2307.08621. Cited by: ยง4.1.3. M. Taraghi, M. M. Morovati, and F. Khomh (2026) Real faults in model context protocol (mcp) software: a comprehensive taxonomy. arXiv preprint arXiv:2603.05637. Cited by: ยง3.1, ยง3.3, ยง3.4.2. T. A. Team, J. Shan, V. Gupta, L. Xu, H. Shi, J. Zhang, N. Wang, L. Xu, R. Kang, T. Liu, et al. (2025) AIBrix: towards scalable, cost-effective large language model inference infrastructure. arXiv preprint arXiv:2504.03648. Cited by: ยง1, ยง5.4. [76] N. TensorRT-LLM Quantization. Note: ://nvidia.github.io/TensorRT-LLM/latest/features/quantization.htmlAccessed: 2026-05-11 Cited by: ยง4.2.1. N. TensorRT-LLM (2026a) Multi-head, multi-query, and group-query attention. Note: ://nvidia.github.io/TensorRT-LLM/advanced/gpt-attention.htmlAccessed: 2026-05-1 Cited by: ยง4.2.1. N. TensorRT-LLM (2026b) Overview. Note: ://nvidia.github.io/TensorRT-LLM/overview.htmlAccessed: 2026-05-1 Cited by: ยง4.2.1. N. TensorRT-LLM (2026c) Parallelism in tensorrt llm. Note: ://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/parallel-strategy.mdAccessed: 2026-05-1 Cited by: ยง4.2.1. [80] TensorRT () Official document of tensorrt. Note: ://developer.nvidia.com/tensorrtAccessed: 2026-04-01 Cited by: ยง1, ยง4.1.1, ยง4.1.2, ยง4.3.2, ยง4.3.2, Table 1. L. Twist, J. M. Zhang, M. Harman, D. Syme, J. Noppen, H. Yannakoudakis, and D. Nauck (2026) A study of llmsโ preferences for libraries and programming languages. In Findings of the Association for Computational Linguistics: ACL 2026, p. 331โ351. Cited by: ยง7.3. C. van Niekerk, R. Vukovic, B. M. Ruppik, H. Lin, and M. Gaลกiฤ (2025) Post-training large language models via reinforcement learning from self-feedback. arXiv preprint arXiv:2507.21931. Cited by: ยง4.3.1, ยง4.3.1, ยง4.3.1. [83] vLLM () Official document of vllm. Note: ://docs.vllm.ai/en/latest/Accessed: 2026-04-01 Cited by: ยง3.3, ยง4.1.1, ยง4.1.3, ยง4.2.2, ยง4.2.2, ยง4.2.2, ยง4.3.2, Table 1. vLLM (2026a) Parallelism and scaling. Note: ://docs.vllm.ai/en/stable/serving/parallelism_scaling/Accessed: 2026-05-11 Cited by: ยง4.2.1. vLLM (2026b) Performance and tuning: chunked prefill. Note: ://docs.vllm.ai/en/v0.4.2/models/performance.htmlAccessed: 2026-05-11 Cited by: ยง4.2.1. Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, et al. (2023) Efficient large language models: a survey. arXiv preprint arXiv:2312.03863. Cited by: ยง1, ยง5.1. D. Wang and S. Zhang (2024) Large language models in medical and healthcare fields: applications, advances, and challenges. Artificial intelligence review 57 (11), p. 299. Cited by: ยง2.1. T. Wang, S. Wang, and T. P. Chen (2023) Study the correlation between the readme file of github projects and their popularity. Journal of Systems and Software 205, p. 111806. Cited by: ยง3.4.1, ยง3.4.1. W. Wang, W. Chen, Y. Luo, Y. Long, Z. Lin, L. Zhang, B. Lin, D. Cai, and X. He (2024) Model compression and efficient inference for large language models: a survey. arXiv preprint arXiv:2402.09748. Cited by: ยง1, ยง5.2. Y. Wang, Q. Yang, Z. Zeng, L. Ren, L. Liu, B. Peng, H. Cheng, X. He, K. Wang, J. Gao, et al. (2025) Reinforcement learning for reasoning in large language models with one training example, 2025. URL https://arxiv. org/abs/2504.20571. Cited by: ยง4.3.1. [91] Wikipedia () Abstract syntax tree. Note: ://en.wikipedia.org/wiki/Abstract_syntax_treeAccessed: 2026-04-28 Cited by: ยง3.3. H. Xia, Z. Zheng, Y. Li, D. Zhuang, Z. Zhou, X. Qiu, Y. Li, W. Lin, and S. L. Song (2023) Flash-llm: enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309.10285. Cited by: ยง1, ยง5.3. H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui (2024) Unlocking efficiency in large language model inference: a comprehensive survey of speculative decoding. Findings of the Association for Computational Linguistics: ACL 2024, p. 7655โ7671. Cited by: ยง1, ยง5.2. M. Xu, W. Yin, D. Cai, R. Yi, D. Xu, Q. Wang, B. Wu, Y. Zhao, C. Yang, S. Wang, et al. (2024) A survey of resource-efficient llm and multimodal foundation models. arXiv preprint arXiv:2401.08092. Cited by: ยง1, ยง5.1. Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthy, et al. (2025) Flashinfer: efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Systems 7. Cited by: ยง4.1.3, ยง4.2.1, ยง4.2.1, ยง4.2.2, ยง4.2.2, ยง4.2.2, ยง4.3.2. S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2024) A survey on multimodal large language models. National Science Review 11 (12), p. nwae403. Cited by: ยง2.1. L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, et al. (2024) Sglang: efficient execution of structured language model programs. Advances in neural information processing systems 37, p. 62557โ62583. Cited by: ยง1, ยง4.2.1, ยง4.2.2, ยง4.2.2, ยง4.3.2. Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li, et al. (2024a) A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294. Cited by: ยง1, ยง5.1. Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li, et al. (2024b) A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294. Cited by: ยง2.1. X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang (2024a) A survey on model compression for large language models. Transactions of the Association for Computational Linguistics 12, p. 1556โ1577. Cited by: ยง1, ยง5.2. Y. Zhu, S. Wang, W. Zhong, N. Shen, Y. Li, S. Wang, Z. Li, C. Wu, Z. He, and L. Li (2024b) A survey on large language model-empowered autonomous driving. arXiv preprint arXiv:2409.14165. Cited by: ยง2.1.