Paper deep dive
Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection
Sabrina Kaniewski, Tim Krämer, Julius Bächle, Markus Enzweiler, Michael Menth, Tobias Heer
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Retrieval-Augmented Generation (RAG) is increasingly used to enhance Large Language Model (LLM)-based software vulnerability detection by grounding predictions in retrieved vulnerability knowledge, such as vulnerability reports. However, existing RAG-based software vulnerability detection (RAG4SVD) systems are often evaluated using proprietary models, which challenges open science and reproducibility. Further, studies use different datasets, custom knowledge bases, different backbone models, and diverse metrics, which hinders meaningful cross-system comparison. In this work, we study six open-source RAG4SVD systems and address these reproducibility and comparability challenges through (i) reproduction of their experimental settings under an open-weight setting, and (ii) a unified benchmark using a common dataset, metric suite, and pool of open-weight models. Further, RAG4SVD systems typically consist of multiple components, yet are often evaluated only as a whole system, i.e., end-to-end. Therefore, we perform (iii) a component-level analysis that decomposes representative RAG4SVD pipelines into input abstraction, knowledge retrieval, and detection. Our results demonstrate that reproducibility varies substantially across systems. Under the presented unified benchmark, published RAG4SVD performance does not transfer under a controlled open-weight evaluation and depends strongly on the used model. The component analysis shows that effective RAG4SVD depends on the alignment between pipeline stages. For example, oracle knowledge raises retrieval to near-optimal, yet performance remains low (0.51 pairwise accuracy), demonstrating that retrieval effectiveness alone is insufficient for reliable detection. These findings motivate evaluating RAG4SVD not only end-to-end, but at the level of pipeline components, and provide a basis for more standardized, RAG-aware evaluation practices.
Tags
Links
- Source: https://arxiv.org/abs/2609.37669v1
- Canonical: https://arxiv.org/abs/2609.37669v1
Trouble viewing inline? Open PDF directly →
Full Text
81,712 characters extracted from source content.
Expand or collapse full text
Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection Sabrina Kaniewski email: sabrina.kaniewski@hs-esslingen.de Affiliation: Institute for Secure Networked Systems, Esslingen University, Esslingen, Germany , Tim Krämer email: tikrit01@hs-esslingen.de Affiliation: Esslingen University, Esslingen, Germany , Julius Bächle email: julius.baechle@hs-esslingen.de Affiliation: Institute for Intelligent Systems, Esslingen University, Esslingen, Germany , Markus Enzweiler email: markus.enzweiler@hs-esslingen.de Affiliation: Institute for Intelligent Systems, Esslingen University, Esslingen, Germany , Michael Menth email: menth@uni-tuebingen.de Affiliation: Chair of Communication Networks, University of Tübingen, Tübingen, Germany and Tobias Heer email: tobias.heer@hs-esslingen.de Affiliation: Institute for Secure Networked Systems, Esslingen University, Esslingen, Germany Abstract. Retrieval-Augmented Generation (RAG) is increasingly used to enhance Large Language Model (LLM)-based software vulnerability detection by grounding predictions in retrieved vulnerability knowledge, such as vulnerability reports. However, existing RAG-based software vulnerability detection (RAG4SVD) systems are often evaluated using proprietary models, which challenges open science and reproducibility. Further, studies use different datasets, custom knowledge bases, different backbone models, and diverse metrics, which hinders meaningful cross-system comparison. In this work, we study six open-source RAG4SVD systems and address these reproducibility and comparability challenges through (i) reproduction of their experimental settings under an open-weight setting, and (i) a unified benchmark using a common dataset, metric suite, and pool of open-weight models. Further, RAG4SVD systems typically consist of multiple components, yet are often evaluated only as a whole system, i.e., end-to-end. Therefore, we perform (i) a component-level analysis that decomposes representative RAG4SVD pipelines into input abstraction, knowledge retrieval, and detection. Our results demonstrate that reproducibility varies substantially across systems. Under the presented unified benchmark, published RAG4SVD performance does not transfer under a controlled open-weight evaluation and depends strongly on the used model. The component analysis shows that effective RAG4SVD depends on the alignment between pipeline stages. For example, oracle knowledge raises retrieval to near-optimal, yet performance remains low (0.510.51 pairwise accuracy), demonstrating that retrieval effectiveness alone is insufficient for reliable detection. These findings motivate evaluating RAG4SVD not only end-to-end, but at the level of pipeline components, and provide a basis for more standardized, RAG-aware evaluation practices. Keywords: software vulnerability detection, software security, retrieval-augmented generation, large language models, reproducibility, benchmark 1. Introduction Large language models (LLMs) are increasingly applied to software vulnerability detection by leveraging their code-understanding and reasoning capabilities to discover insecure code (Kaniewski et al., 2026a). However, an LLM’s internal knowledge is inherently limited by its training data and respective cutoff date. Thus, newly disclosed vulnerabilities, emerging attack patterns, and project-specific security knowledge that were published after the training cutoff date may be unavailable to the model at inference time. Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) addresses this limitation by dynamically retrieving external vulnerability knowledge, e.g., Common Weakness Enumeration (CWE) descriptions, specific Common Vulnerabilities and Exposures (CVE) reports, or similar vulnerable code examples, from a knowledge base and incorporating it into the prompt (Chen et al., 2026). Unlike costly retraining or fine-tuning, the knowledge base can be updated independently of the model to add more specific and up-to-date information and, thus, enhance detection performance. Diagram of a RAG4SVD pipeline from input source code through input abstraction, knowledge retrieval, and vulnerability detection to a vulnerable/non-vulnerable prediction. Nested outlines show how RQ1 evaluates reproducibility, RQ2 evaluates end-to-end detection, and RQ3 separately evaluates abstraction quality, retrieval effectiveness, and detection performance. Figure 1. Overview of a typical RAG4SVD pipeline and the evaluation roadmap of this work.Diagram of a RAG4SVD pipeline from input source code through input abstraction, knowledge retrieval, and vulnerability detection to a vulnerable/non-vulnerable prediction. Nested outlines show how RQ1 evaluates reproducibility, RQ2 evaluates end-to-end detection, and RQ3 separately evaluates abstraction quality, retrieval effectiveness, and detection performance. The adoption of RAG for LLM-based software vulnerability detection (RAG4SVD) has grown rapidly, with proposed RAG systems reporting improvements over non-retrieval baselines (Du et al., 2025; Kaniewski et al., 2026a; Chen et al., 2026). RAG4SVD systems typically follow a common pipeline, cf. Fig. 1: The code under analysis is first transformed via input abstraction into the representation used to query the knowledge base, e.g., the raw code itself or a semantic summary of the code. This query is passed to a knowledge retrieval mechanism, e.g., based on embedding similarity, lexical matching, structured lookup, or a hybrid combination thereof, which retrieves relevant entries. Finally, the retrieved knowledge is combined with the code under analysis and passed to the LLM for detection, producing a vulnerability prediction, e.g., a binary vulnerable/non-vulnerable label. Yet, it remains unclear how generalizable reported gains are. First, results are rarely independently verified (Kaniewski et al., 2026b). Incomplete public artifacts, undocumented preprocessing and dependencies, non-deterministic inference, and reliance on proprietary models or APIs further complicate reproducing LLM-based systems (Sallou et al., 2024; HS Esslingen IT Security, 2026; Kaniewski et al., 2026b). Model providers may discontinue or modify their models without notice, which renders experiments non-reproducible. Second, systems differ substantially in their datasets, knowledge bases, query representations, retrieval mechanisms, and backbone models (Kaniewski et al., 2026a; Chen et al., 2026). Consequently, results across individual studies do not provide a basis for comparing different systems. Third, despite the fact that these systems consist of multiple components, they are commonly evaluated only through their final detection accuracy, i.e., end-to-end. It is often unclear if improvements stem from the quality and coverage of the knowledge base, the abstraction used to formulate a retrieval query, the retrieval mechanism, or the capabilities of the detection model used. We summarize these observations as three evaluation gaps in the current literature: Reproducibility Gap: The results of published RAG4SVD systems have rarely been independently reproduced. It remains uncertain whether reported performance can be recovered. Comparability Gap: Systems are evaluated using different datasets, custom knowledge bases, different models, and diverse metrics, preventing meaningful cross-system comparison. Attribution Gap: RAG systems are predominantly evaluated end-to-end; consequently, it remains unclear which pipeline components drive observed performance improvements. To address these gaps, we present, to the best of our knowledge, the first study in RAG-based software vulnerability detection to jointly assess reproducibility, cross-system performance, and individual pipeline components. The contributions of this work are threefold: (i) Considering six open-source RAG4SVD systems, i.e., SVD-Bench (Zhang et al., 2025), Llama-VD11 1 We introduce the abbreviation Llama-VD in this work for brevity. (Ouchebara and Dupont, 2025), GRACE (Lu et al., 2024), LLM4Vuln (Sun et al., 2025b), Vul-RAG (Du et al., 2025), and VulTriage (Tang et al., 2026), we first reproduce systems using open-weight models and quantify deviations from their reported results. (i) Second, we analyze how the considered RAG4SVD systems compare when re-implemented under a common dataset, metric suite, and pool of heterogeneous open-weight models, enabling controlled comparison across independently proposed systems. (i) Third, we move beyond end-to-end evaluation and dissect representative multi-stage RAG4SVD pipelines into input abstraction, knowledge retrieval, and detection, proposing a component-aware evaluation of RAG4SVD systems. Specifically, we evaluate whether generated abstractions preserve information useful for retrieval, whether relevant knowledge is actually retrieved, and whether oracle knowledge improves the detection performance. Together, these analyses provide a RAG-aware evaluation that examines not whether RAG improves vulnerability detection, but which components drive performance. The replication package for this study is publicly available at https://github.com/hs-esslingen-it-security/RAG4SVD. The remainder of this work is structured as follows. Section 2 reviews related work. Section 3 describes the experimental setup, i.e., the selection of RAG4SVD systems, datasets, metrics, models, and implementation details. Section 4 presents the evaluation along the identified gaps and discusses the findings. We address threats to validity in Sect. 5 and conclude this work in Sect. 6. 2. Related Work Research on LLM-based software vulnerability detection has expanded rapidly in recent years (Kaniewski et al., 2026a). In this section, we review prior work on (i) benchmarking LLM-based software vulnerability detection, (i) benchmarking RAG-based software vulnerability detection approaches, and (i) component-level evaluation of RAG pipelines, positioning the contributions of this work. 2.1. Benchmarking LLM-based Vulnerability Detection Several works benchmark how different prompting and tuning strategies affect vulnerability detection performance (Zhang et al., 2025; Steenhoek et al., 2025; Tamberg and Bahsi, 2025; Ullah et al., 2024). For example, Steenhoek et al. (Steenhoek et al., 2025) systematically compare prompting strategies, including in-context learning, few-shot, and Chain-of-Thought prompting, and Zhang et al. (Zhang et al., 2025) compare prompt engineering, instruction tuning, and sequence-classification fine-tuning across programming languages and model sizes. However, none of these efforts extend to different RAG-based systems, which introduce an additional layer of design choices, such as knowledge base construction or retrieval strategy, on top of existing evaluation challenges. Recent works further question whether reported performance reflects genuine detection capability of LLMs or is instead an artifact of the datasets and evaluation protocols used (Ding et al., 2025; Steenhoek et al., 2025; Ullah et al., 2024). Ding et al. (Ding et al., 2025) show that established benchmarks such as BigVul (Fan et al., 2020) overestimate model performance due to, e.g., noisy labels and high duplication rates (Guo and Bettaieb, 2023). To address these issues, they introduce an improved vulnerability dataset PrimeVul, which combines rigorous de-duplication and human-verified labeling with a stricter pairwise evaluation protocol. Under this more realistic setting, reported F1 scores for state-of-the-art code language models dropped to as low as 0.030.03, motivating a shift toward PrimeVul as the de facto benchmark for LLM-based software vulnerability detection, which we also adopt as a common benchmark dataset. Further, we assess multiple detection models across systems, allowing us to distinguish gains attributable to model capacity from gains attributable to the RAG system architectures. In doing so, we extend prior LLM benchmarking from model-centric evaluation to system-level evaluation of RAG-based vulnerability detection. 2.2. Fragmented Evaluation of Retrieval-Augmented Vulnerability Detection Existing RAG-based software vulnerability detection systems are predominantly evaluated in isolation, often using custom evaluation protocols, making direct comparisons difficult (Kaniewski et al., 2026a). Kaniewski et al. (Kaniewski et al., 2026b) provide an initial reproducibility and replicability study of Vul-RAG (Du et al., 2025). Motivated by the dependence of many LLM-based approaches on proprietary models, the authors adapt the Vul-RAG pipeline to an open-weight setting and first reproduce its reported open-weight baselines before evaluating a broader set of LLMs. However, the study scope is intentionally restricted to a single RAG4SVD system and primarily investigates reproducibility and model replicability; it does not compare multiple proposed RAG architectures, transfer them to a new dataset, or evaluate individual pipeline components. Where multi-system comparisons do exist, they are scoped differently: For example, VulInstruct (Zhu et al., 2026) and RASM-Vul (Zhao et al., 2026) each compare their detection performance only against Vul-RAG (Du et al., 2025); and VulTriage (Tang et al., 2026) ablates pipeline components to quantify their individual contribution. However, comparisons via final detection performance and ablations do not reveal which RAG design choices generalize, nor whether reported gains reproduce outside their evaluation setting. This fragmentation is further highlighted by Chen et al. (Chen et al., 2026). Surveying 30 RAG-based studies across the software vulnerability lifecycle, they highlight that evaluation practices remain fragmented across datasets, baselines, models, and metrics. We address this gap: to the best of our knowledge, we present the first reproducibility assessment and benchmark for RAG4SVD, applying a common dataset, metric suite, and model pool across systems to make reported gains comparable. 2.3. RAG Evaluation Methodology and Component Analysis RAG evaluation frameworks such as RAGAs (Es et al., 2024) and ARES (Saad-Falcon et al., 2024) automate faithfulness and relevance scoring via an LLM-as-a-Judge. However, these frameworks assume free-text question-answer tasks and do not address the semantics of software vulnerability detection, where retrieval is based on input code and retrieved artifacts consist of code fragments or structured vulnerability knowledge. For the software vulnerability lifecycle, Chen et al. (Chen et al., 2026) further highlight that RAG-specific properties such as retrieval quality, grounding, and faithfulness remain insufficiently evaluated. Component-wise evaluation of RAG pipelines has been explored for selected software engineering tasks. Ke et al. (Ke et al., 2026) present a component-wise empirical study that systematically isolates and evaluates four query processing techniques, seven retrieval models, four context refinement methods, and six generators across the code generation, summarization, and repair tasks. However, their evaluation focuses on extrinsic performance: every component is judged solely by its effect on final task performance, but never assessed individually (e.g., whether retrieved knowledge is relevant). We address this gap by expanding extrinsic evaluation with intrinsic evaluation. Rather than varying components across systems, we evaluate the components of representative RAG4SVD systems in isolation alongside standard detection metrics. 3. Experimental Setup Our experimental design is built around the three evaluation gaps identified in Sect. 1 and visualized in Fig. 1. To address the Reproducibility Gap, we first reproduce considered open-weight systems and assess whether their reported results can be recovered (RQ1). To address the Comparability Gap, we evaluate the systems using a common dataset, metric suite, and open-weight model pool, enabling a controlled comparison of their performance (RQ2). Finally, to address the Attribution Gap, we dissect representative RAG4SVD pipelines into input abstraction, knowledge retrieval, and detection, and evaluate these components individually to determine how they contribute to overall performance (RQ3). Accordingly, we formulate the following research questions: RQ1 To what extent are published open-source RAG4SVD systems reproducible? RQ2 How do open-source RAG4SVD systems compare under a unified benchmark? RQ3 Which component-level factors drive detection in representative RAG4SVD pipelines? 3.1. RAG4SVD System Selection 31RAG4SVD studies (HS Esslingen IT Security, 2026)12studies with public artifacts6executable systemsartifact availabilitycompleteness &documentation Figure 2. RAG4SVD system screening and selection.Flow diagram of the study-selection process. From 31 identified RAG4SVD studies, 12 provide publicly accessible code or artifacts. Of these, 6 provide a sufficiently complete and documented setup, including the pipeline, data, and experimental configuration, to support reproduction. We consider RAG as the dynamic selection of external context at inference-time, conditioned on the code under analysis. Within RAG4SVD, we consider example-retrieval RAG, where semantically or lexically similar labeled code examples are selected for few-shot prompting, and knowledge-retrieval RAG, where the system queries a dedicated vulnerability knowledge base. A continuously updated survey artifact by Kaniewski et al. (HS Esslingen IT Security, 2026) tracks LLM-based software vulnerability detection literature. As of August 2026, it lists 3131 studies implementing RAG within the RAG4SVD scope. For this study, we require (i) publicly accessible implementation code or equivalent executable artifacts and (i) a sufficiently complete and documented setup to reconstruct the RAG pipeline and its evaluation, including the required data, configuration, and reported experimental settings. Applying these criteria leaves 1212 (39%39\%) studies with public artifacts, of which only 66 provide a complete and executable setup for reproduction. Fig. 2 summarizes the selection process. We provide study-specific inclusion and exclusion rationale in the replication package. Tab. 1 summarizes the selected systems along the RAG4SVD pipeline stages. Table 1. Overview of representative RAG4SVD systems considered in this study, characterized by their retrieval query, retrieval mechanism, retrieved knowledge, and coverage in the research questions RQ1–RQ3. System Query Retrieval Retrieved Knowledge RQ1 RQ2 RQ3 SVD-Bench (Zhang et al., 2025) Code Dense (simcse-bert-base) Code examples ✓ ✓ Llama-VD (Ouchebara and Dupont, 2025) Code Dense (codebert-base) Code examples ✓ ✓ GRACE (Lu et al., 2024) Code Dense (CodeT5) + re-ranking Code examples ✓ LLM4Vuln (Sun et al., 2025b) Semantic abstr. Dense (text-embedding-ada-002) Summarized CWE reports ✓ ✓ Vul-RAG (Du et al., 2025) Code + semantic abstr. Sparse (BM25) + re-ranking Functionality + root causes + patches ✓ ✓ ✓ VulTriage (Tang et al., 2026) Vulnerability abstr. Hybrid (dense + sparse; bge-m3) CWE descriptions + code examples ✓ SVD-Bench (Zhang et al., 2025) studies RAG as one of several prompting techniques. It uses dense retrieval, which embeds the function under analysis and labeled training functions with the simcse-bert-base model and ranks them by cosine similarity to retrieve the top-n most similar examples. Llama-VD (Ouchebara and Dupont, 2025) investigates different prompting and fine-tuning strategies for adapting Llama-3.1-8B to vulnerability detection. It applies dense retrieval using codebert-base, and the 66 nearest training examples are retrieved using Euclidean distance. The study additionally explores using RAG-selected examples in test-time tuning, thereby combining retrieval with parameter adaptation. GRACE (Lu et al., 2024) uses dense retrieval, embedding the function under analysis and training functions with CodeT5 to retrieve semantically similar candidates. These candidates are subsequently re-ranked using lexical token overlap and syntactic similarity derived from Abstract Syntax Tree (AST) representations to select a single example. GRACE further extracts structural program information from the code and supplies it alongside the code and retrieved example in the prompt. LLM4Vuln (Sun et al., 2025b) presents a modular evaluation framework that separates an LLM’s intrinsic vulnerability reasoning from external enhancements, including knowledge retrieval, program-context supplementation, and prompting. For knowledge retrieval, it uses dense retrieval with text-embedding-ada-002, supporting either the raw source code or an LLM-generated functionality summary (semantic abstraction) as the query representation. The top-33 most similar entries are retrieved, and their associated raw or summarized vulnerability knowledge, including vulnerability descriptions and root causes, is supplied to the LLM for detection. Vul-RAG (Du et al., 2025) retrieves abstract, vulnerability-specific knowledge distilled from vulnerable-patched code pairs. Its knowledge base represents each vulnerability through functional semantics, vulnerability causes, and corresponding patches. At inference, an LLM generates an abstract purpose and detailed behavioral description of the code under analysis; Vul-RAG uses these together with the raw code as three retrieval queries. It applies sparse BM25 retrieval, which ranks entries by lexical term matching, and combines the resulting candidate lists using reciprocal rank fusion. The retrieved items are then examined sequentially by the LLM to determine whether the code exhibits a retrieved vulnerability cause without the corresponding patch behavior. VulTriage (Tang et al., 2026) augments vulnerability detection through three complementary context paths targeting program structure, program semantics, and vulnerability knowledge. For knowledge retrieval, it generates up to two coarse natural-language vulnerability predictions of the code under analysis in a first step and uses these predictions as queries to a CWE-derived knowledge base. It performs hybrid retrieval by combining dense semantic similarity using bge-m3 embeddings with sparse lexical matching. The retrieved CWE descriptions and code examples are then combined with the code under analysis, structural context, and semantic summary for the final prediction. 3.2. Datasets In the evaluation, we distinguish between the original datasets used by each vulnerability detection system and a common dataset used for cross-system comparison. For RQ1, we retain each system’s original dataset and data split (including the system-specific knowledge base) to reproduce its reported performance. For the unified benchmark (RQ2), we use PrimeVul (Ding et al., 2025) as the common dataset. PrimeVul was introduced to address limitations of earlier vulnerability datasets, particularly noisy labels, duplicate samples, and data leakage across training and evaluation sets (Ding et al., 2025). The full dataset (v0.1) contains 6,968 vulnerable and 228,800 benign functions spanning 94 CWEs and employs a chronological train, validation, and test split. For the benchmark, we specifically use the paired subset of PrimeVul, i.e., PrimeVul Paired: each vulnerable function is matched with its corresponding patch, resulting in 5480 pairs across 88 CWEs, making it particularly suitable for evaluating whether a model can correctly distinguish vulnerable code from its patched counterpart. We use the train split as the basis for the knowledge base and the test split for testing. For the component analysis in RQ3, specifically the knowledge retrieval, we additionally construct test set probes, i.e., randomly sample from the test sets (see Appendix A). 3.3. Metrics Detection Performance. We assess detection performance using the standard classification metrics: Accuracy, Precision, Recall, and F1-score. While these metrics quantify performance at the level of individual functions, they do not capture whether a model can distinguish a vulnerable function from its patched non-vulnerable counterpart. Following the pairwise evaluation protocol introduced for PrimeVul (Ding et al., 2025), we report Pairwise Accuracy (Du et al., 2025). A vulnerability-patch pair is counted as correctly classified only if the vulnerable instance is predicted as vulnerable and its corresponding patched instance is predicted as non-vulnerable: Pair.Acc.=1N∑i=1N(y^ivuln=1∧y^ipatch=0)Pair.\,Acc.= 1N _i=1^N1( y^vuln_i=1 y^patch_i=0), where N denotes the number of pairs. Since paired instances differ primarily in the security-relevant changes introduced by the patch, this metric provides a stricter test of whether a model correctly distinguishes vulnerable from patched code. Retrieval Effectiveness. For evaluating retrieval effectiveness, we adopt Hit@k and Mean Reciprocal Rank (MRR) (Gan et al., 2025), treating a retrieved item as a successful match if it shares the queried code’s ground truth CWE, considering both exact matches and hierarchical CWE parent/child relaxations (cf. Tamberg and Bahsi (Tamberg and Bahsi, 2025)). Hit@k measures the proportion of queries for which at least one successful match occurs within the top-k retrieved items: Hit@k=1N∑i=1N(ri≤k)Hit@k= 1N _i=1^N1(r_i≤ k), where N is the number of queries and rir_i denotes the rank of the first successful match for query i. MRR captures how high the first successful match is ranked, averaged across all queries: MRR=1N∑i=1N1riMRR= 1N _i=1^N 1r_i. 3.4. Model Selection Several of the considered RAG4SVD systems originally rely on proprietary models, e.g., GPT-4 (OpenAI et al., 2024), or use open-weight models of scales typically requiring API services, e.g., DeepSeek-V3 (DeepSeek-AI et al., 2025b) with 685685B parameters. For the unified evaluation, we select a common pool of 2222 open-weight models from 77 model families: Qwen (Qwen et al., 2025; Hui et al., 2024; Yang et al., 2025; Qwen, 2026a; Qwen, 2026b; Qwen, 2025), Llama (Grattafiori et al., 2024), Phi (Abdin et al., 2024; Microsoft et al., 2025), Gemma (Team et al., 2025), NextCoder (Aggarwal et al., 2025), DeepSeek (DeepSeek-AI et al., 2025a), and Mistral (Jiang et al., 2023). The pool spans parameter scales from approximately 33B to 3232B, deliberately focusing on scales that remain amenable to practical local and on-premises deployment. The selection further includes both general-purpose and code-specialized models. Where available, we include multiple parameter scales from the same family, allowing us to additionally examine the effect of model scale. Unless otherwise stated, we use the full model pool consistently across experiments. The complete model lineup and checkpoints are reported in Appendix B. 3.5. Implementation Details To enable a consistent evaluation across systems, we integrate all considered RAG systems into a unified local inference pipeline using Hugging Face22 2 https://huggingface.co, accessed 11-09-2026. and the Transformers library. Systems that originally relied on API-based models were adapted by replacing their API calls with a common interface for locally hosted open-weight models. We preserve all other system logic, prompts, retrieval configurations, and system-specific hyperparameters. Minor implementation changes were necessary to accommodate differences between proprietary and open-weight models. In particular, some open-weight models do not always adhere as reliably to requested output formats, e.g., requiring <result>YES/NO< > tags (Du et al., 2025), which requires more robust parsing of generated responses. We therefore hardened system-specific output parsers where necessary to tolerate formatting variations without changing the semantic interpretation of model outputs. Detailed system-specific changes are documented in the replication package. All retrieval and detection experiments with locally hosted open-weight models were executed on a high-performance computing cluster equipped with NVIDIA L40S 48 GB and NVIDIA H100 80 GB GPUs. The only exception to local inference is the LLM-as-a-Judge analysis in RQ3. Following the judge quality assessment by Li et al. (Li et al., 2026), we use GPT-oss-120B (OpenAI et al., 2025) as the judge model and access it via the Blablador API33 3 The Blablador API, provided by Jülich Supercomputing Centre, is accessible to researchers: https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_api_access/, accessed 11-09-2026. due to its computational requirements. The judge is used only for evaluating generated abstractions and does not participate in retrieval or vulnerability detection. 4. Evaluation In this section, we report the empirical evaluation of the considered RAG4SVD systems, structured around the derived research questions RQ1-RQ3. We first determine whether reported results can be reproduced (RQ1), then compare reproducible systems using a unified benchmark (RQ2), and finally isolate individual RAG components (RQ3). 4.1. RQ1 Reproducibility Reproducibility is a prerequisite for meaningful benchmarking. If systems cannot be faithfully reproduced, differences observed in subsequent comparisons may reflect implementation artifacts rather than underlying methods. For RQ1, we focus on systems that report results for at least one open-weight model. This criterion includes SVD-Bench, Llama-VD, LLM4Vuln, and Vul-RAG, while excluding GRACE and VulTriage, whose evaluations rely exclusively on proprietary models. For the considered systems, we assess whether the reported results can be reproduced in a local open-weight inference setting. Where a study evaluates multiple datasets or models, we reproduce these settings separately. We repeat each experiment three times to account for inference variability; results are averaged across runs. Following Nong et al. (Nong et al., 2023), we quantify reproducibility using the absolute relative deviation between reproduced and reported results. Deviations below 1%1\%, between 1%1\% and 5%5\%, and above 5%5\% are classified as reproducible, weakly reproducible, and not reproducible, respectively. Fig. 3 summarizes the results across all considered systems. Dumbbell plots compare originally reported and reproduced performance for SVD-Bench, Llama-VD, LLM4Vuln, and Vul-RAG across accuracy, precision, recall, and F1. Relative deviations are annotated and categorized as reproducible, weakly reproducible, or not reproducible. Figure 3. Reproducibility of considered RAG4SVD systems. Absolute relative deviation between ∘ reported and ∙ reproduced results for each metric reported, averaged across all evaluated open-weight models. Lower values indicate higher reproducibility; deviations below 1%1\% are considered reproducible, 11-5%5\% weakly reproducible, and above 5%5\% not reproducible. See Appendix C for the absolute differences.Dumbbell plots compare originally reported and reproduced performance for SVD-Bench, Llama-VD, LLM4Vuln, and Vul-RAG across accuracy, precision, recall, and F1. Relative deviations are annotated and categorized as reproducible, weakly reproducible, or not reproducible. SVD-Bench. We reproduce the results of SVD-Bench across its Python, Java, and JavaScript evaluation settings. Reproducibility does not hold consistently across this language coverage: deviations exceed 5%5\% for Python and Java, while selected JavaScript metrics fall within the weakly reproducible range. The largest deviation occurs for recall on the Python dataset (58.2%58.2\%), indicating that the reproduced implementation does not recover the reported performance. Llama-VD. We reproduce the results of Llama-VD for Llama-3.1-8B on both originally evaluated datasets, Big-Vul and PrimeVul. Reproducibility is consistent across these datasets, with nearly all reported metrics deviating by less than 1%1\%; only recall marginally exceeds this threshold (1.01.0-1.1%1.1\%). LLM4Vuln. LLM4Vuln reports results for different detection models, including three open-weight models, but still relies on GPT-4.1 for summary generation and output evaluation and, thus, cannot be reproduced with its original configuration in a fully open-weight reproduction setting. We therefore substitute these components with open-weight alternatives. We chose the substitutes (summary: Qwen3-4B, judge: Gemma-3-27B) through an ablation of candidate models based on their error rate and ability to produce correctly parseable outputs (cf. replication package). Despite using the closest-performing substitutes, the reproduced results deviate by more than 5%5\% from the reported results, rendering LLM4Vuln not reproducible under a fully open-weight setting. Unlike the other considered systems, this deviation is not attributable to, e.g., parsing or inference variance alone, since LLM4Vuln depends on unavailable proprietary components. The reported results here reflect a model substitution rather than a reproduction of the original configuration, making a deviation from the reported results expected. Vul-RAG. We reproduce the results of Vul-RAG for Qwen2.5-Coder-32B on the PairVul dataset. The reproduced results closely match the reported performance, with deviations below 1%1\% for accuracy and recall; precision exhibits a small deviation of 2.5%2.5\%, i.e., is weakly reproducible. Finding #1: Reproducibility varies substantially across the considered RAG4SVD systems, with potential contributing factors including parser hardening and inference non-determinism. For systems that require substituting proprietary models, reproducibility in an open-weight setting is unattainable. 4.2. RQ2 Unified Benchmark Table 2. Performance comparison across RAG4SVD systems on the PrimeVul Paired dataset. We report a cell only if at least 95% of pairs yielded a correctly parsed prediction. Best and worst pairwise accuracy per system highlighted. For the best configuration per system, we repeat inference 3×3× and report mean ± std. The system average is computed over the intersection of models (marked with *) that yielded a prediction for all systems, and over the top-55 performing models per system. SVD-Bench GRACE Llama-VD Vul-RAG VulTriage LLM Pair. Acc. F1 Pair. Acc. F1 Pair. Acc. F1 Pair. Acc. F1 Pair. Acc. F1 Qwen2.5-Coder-3B 0.07 0.33 0.04 0.62 0.07 0.45 - - - - Qwen2.5-Coder-14B* 0.09 0.48 0.02 0.39 0.07 0.46 0.22 ± 0.01 0.55 0.07 0.14 Qwen2.5-Coder-32B* 0.09 0.43 0.04 0.39 0.11 0.48 0.16 0.53 0.21 ± 0.00 0.37 Qwen2.5-3B 0.13 0.49 0.10 0.56 0.10 0.46 0.12 0.51 - - Qwen2.5-14B 0.12 ± 0.01 0.53 0.04 0.46 0.09 0.49 0.18 0.55 - - Qwen2.5-32B 0.12 0.56 0.03 0.44 0.13 0.48 0.16 0.56 - - Qwen3-4B 0.10 0.47 0.10 0.26 0.09 0.47 0.17 0.54 - - Qwen3.5-9B - - 0.05 0.17 0.15 ± 0.00 0.50 - - - - Qwen3.6-27B - - 0.01 0.01 0.06 0.42 0.14 0.56 - - QwQ-32B 0.12 0.58 0.17 ± 0.00 0.62 0.06 0.43 0.21 0.57 - - Llama-3.2-3B 0.09 0.37 0.01 0.67 0.12 0.48 - - - - Llama-3.1-8B 0.12 0.46 0.00 0.67 0.12 0.49 0.12 0.53 - - Phi-4-mini 0.11 0.51 0.09 0.55 0.09 0.46 - - - - Phi-4 0.10 0.36 0.04 0.42 0.11 0.49 0.14 0.56 - - Gemma-3-4B 0.10 0.47 0.07 0.64 0.15 0.51 0.19 0.50 - - Gemma-3-12B* 0.08 0.36 0.04 0.49 0.09 0.48 0.20 0.54 0.06 0.63 Gemma-3-27B* 0.07 0.38 0.03 0.58 0.08 0.48 0.20 0.55 0.21 0.70 NextCoder-14B* 0.08 0.43 0.02 0.40 0.07 0.46 0.22 0.56 0.07 0.13 NextCoder-32B 0.12 0.51 0.06 0.40 0.07 0.48 0.19 0.55 - - DeepSeek-R1-8B 0.12 0.48 0.14 0.58 0.09 0.46 0.21 0.51 - - DeepSeek-R1-32B 0.10 0.50 0.16 0.51 0.10 0.49 0.19 0.53 - - Mistral-v0.3-7B 0.09 0.41 0.06 0.57 0.14 0.49 - - - - Average (n=5n=5, ∗*intersection) 0.08 0.41 0.03 0.45 0.08 0.47 0.20 0.55 0.12 0.40 Average (n=5n=5, best) 0.12 0.53 0.13 0.51 0.14 0.49 0.21 0.55 0.12 0.40 We evaluate the considered systems under a common dataset, metric suite, and model pool to enable a controlled cross-system comparison. Notably, we are not estimating each system’s best possible performance; we assess how the systems transfer under a controlled open-weight environment. Accordingly, we exclude LLM4Vuln from the unified benchmark. LLM4Vuln’s modular pipeline relies on GPT-4.1 for summarization and judge-based evaluation such that evaluating it in an open-weight inference setting would require model substitutions, which introduces confounds beyond the choice of detection backbone. To further separate detection performance from parsing failures and prompt-specific instruction-following issues, we report a configuration only if at least 95% of pairs yield a correctly parsed prediction. System Ranking. Table 2 summarizes the performance across systems and backbone models. Averaged over the intersection of models that yielded a prediction for all systems, Vul-RAG achieves the highest mean pairwise accuracy (0.200.20), followed by VulTriage (0.120.12), SVD-Bench and Llama-VD (0.080.08), and GRACE (0.030.03). Notably, SVD-Bench and Llama-VD, which both implement example-retrieval RAG using dense retrieval (with different embedding models), converge to the same mean pairwise accuracy. Considering the top-55 best performing models per system, the ordering slightly shifts: Vul-RAG reaches 0.210.21 pairwise accuracy, followed by Llama-VD (0.140.14), GRACE (0.130.13), SVD-Bench (0.120.12), and finally VulTriage (0.120.12). Considering the strongest individual configurations, we repeat each configuration three times to assess inference variability and report the mean pairwise accuracy ± standard deviation. Vul-RAG reaches 0.22±0.010.22± 0.01, compared with 0.21±0.000.21± 0.00 for VulTriage, 0.17±0.000.17± 0.00 for GRACE, 0.15±0.000.15± 0.00 for Llama-VD, and 0.12±0.010.12± 0.01 for SVD-Bench. The low standard deviations indicate limited run-to-run inference variability; we therefore report single-run results for the remaining benchmark configurations. Pairwise accuracy also reveals failure modes that are obscured by F1 score alone. For example, Llama-3.2-3B and Llama-3.1-8B under GRACE achieve F1 scores of 0.670.67, while their pairwise accuracy is only 0.010.01 and 0.000.00, respectively. For the pairwise evaluation setting, this combination is attributable to almost always predicting vulnerable. Despite a seemingly strong F1 score, the system fails to distinguish vulnerable functions from their patched counterparts. Backbone Sensitivity. Performance varies markedly with the underlying detection model, but we observe no consistent advantage through model scale or code specialization where model families match. Within the Qwen2.5 family, for example, performance does not increase monotonically with parameter scale across systems, and the corresponding coder variants do not systematically outperform their general-purpose counterparts. Instead, the strongest backbone is pipeline-specific. Parsing Robustness. The benchmark additionally exposes substantial differences in output-format robustness across systems. GRACE and Llama-VD yield sufficiently parseable predictions across the evaluated model pool. In contrast, VulTriage, which reports SOTA pairwise accuracy of 0.340.34 using GPT-4o on PrimeVul Paired (Tang et al., 2026), meets the 95% parsing threshold for only 55 tested open-weight models, indicating limited transferability of its prompting and output-format beyond the original proprietary-model setting. Instruction following and detection performance should therefore be treated as distinct system properties: a pipeline may perform well for a compatible backbone while remaining difficult to evaluate reliably with others due to prompt or output-format sensitivity. Finding #2: Using a common benchmark, RAG4SVD systems differ substantially in detection performance, backbone sensitivity, and parsing robustness. Vul-RAG achieves the highest mean pairwise accuracy, while reported SOTA rankings under the systems’ original evaluation settings do not transfer to a common open-weight setting. These results show that system-level gains strongly depend on the backbone model and evaluation setting used. 4.3. RQ3 Component Analysis RAG4SVD pipelines combine multiple components, e.g., input abstraction, knowledge retrieval, and retrieval-augmented detection, as shown in Fig. 1. However, prior work typically reports only end-to-end detection performance, leaving open which components drive the observed improvements. We complement the extrinsic evaluation in RQ2 with an intrinsic analysis that isolates and evaluates individual pipeline components. For this analysis, we focus on Vul-RAG (Du et al., 2025) and LLM4Vuln (Sun et al., 2025b), as both systems expose separable abstraction, retrieval, and detection stages and use retrieved knowledge as primary external augmentation, which facilitates component-level attribution. 4.3.1. Input Abstraction We first evaluate the quality of input abstractions, i.e., whether using summaries of code functionality preserves information necessary for retrieval. Following prior work on LLM-based code summarization evaluation (Wu et al., 2025; Wang et al., 2025; Sun et al., 2025a; Song et al., 2024; Crupi et al., 2025), we use an LLM-as-a-Judge and assess each summary along five dimensions. Four of these dimensions capture established summary-quality properties: Faithfulness (support by the input code), Functional Coverage (coverage of the code’s primary behavior), Conciseness (informativeness without unnecessary detail), and Clarity (readability and unambiguity). For the RAG setting, we additionally introduce Retrieval Utility, which assesses whether the summary preserves discriminative semantic information for retrieving relevant knowledge. Each dimension is scored from 1 to 5. Consistent with the focus on open-weight evaluation and following Li et al. (Li et al., 2026), we use GPT-oss-120B as the judge. (a) Vul-RAG (b) LLM4Vuln55 5 Due to dependency conflicts in the LLM4Vuln environment, we excluded Qwen3.5-9B and Qwen3.6-27B from its model pool. As a robustness check, we recomputed all means with and without these models and observed no material differences. Figure 4. Summary-quality evaluation for system-specific and PrimeVul test probes, with mean scores averaged over the model pool. Marker color encodes model family and marker size increasing parameter count. Two sets of five plots compare LLM-as-a-Judge scores for Faithfulness, Functional Coverage, Retrieval Utility, Conciseness, and Clarity across system-specific and PrimeVul probes for Vul-RAG and LLM4Vuln. Table 3. Inter-judge agreement and score sensitivity for the abstraction-quality evaluation. (a) Ordinal Krippendorff’s α and mean pairwise absolute deviation (MAD) across the judges. (b) Mean scores by judge. (a) Inter-judge agreement. Vul-RAG LLM4Vuln Dimension α MAD α MAD Faithfulness 0.19 0.81 0.42 0.99 Functional Coverage 0.24 0.69 0.63 0.73 Retrieval Utility 0.19 0.84 0.59 0.77 Conciseness 0.16 0.79 0.28 0.82 Clarity 0.38 0.48 0.36 0.59 System-specific dataset (PairVul for Vul-RAG, UniVul for LLM4Vuln) and PrimeVul probes. (b) Mean judge scores. GPT-oss-120B Qwen2.5-32B QwQ-32B Dimension System PrimeVul System PrimeVul System PrimeVul Vul-RAG Faithfulness 4.07 4.12 3.92 3.90 4.58 4.58 Functional Coverage 3.98 3.97 3.98 3.95 4.63 4.63 Retrieval Utility 4.12 4.15 3.40 3.43 4.39 4.28 Conciseness 3.73 3.78 3.81 3.85 4.58 4.56 Clarity 4.39 4.41 4.40 4.44 4.82 4.79 LLM4Vuln Faithfulness 3.50 3.73 2.92 2.90 3.71 3.77 Functional Coverage 2.91 2.88 2.70 2.61 3.45 3.35 Retrieval Utility 3.16 3.09 2.40 2.34 3.17 3.07 Conciseness 4.10 4.03 4.28 4.37 4.06 3.99 Clarity 4.47 4.43 4.28 4.45 4.46 4.45 For the evaluation, we retain each system’s abstraction prompt and evaluate summaries generated by the different models in the pool for test codes sampled from the original and PrimeVul test sets (cf. Appendix A). Vul-RAG asks the model to extract the purpose and functionality of the code in a specific format, whereas LLM4Vuln requests a short summary (less than three sentences) intended to support search over the vulnerability-knowledge database. Figure 4 visualizes judge-assessed summary quality across the model pool. Vul-RAG summaries score consistently high in Faithfulness, Functional Coverage, and Retrieval Utility, but lower in Conciseness. By contrast, LLM4Vuln summaries are generally more concise and clear, but provide less Functional Coverage and Retrieval Utility. This difference reflects the prompt objectives: Vul-RAG emphasizes purpose and functionality, whereas LLM4Vuln explicitly favors short summaries. Within each system, mean scores remain largely stable across dimensions and test sets. These results suggest that judged abstraction quality is driven more strongly by the system-specific summarization prompt than by the code to be summarized. Furthermore, several models, including QwQ-32B, Gemma-3-12B, and Qwen2.5-Coder-32B, perform strongly under both systems, indicating that some models perform consistently well across abstraction tasks. At the same time, several models exhibit pronounced failure cases. For example, DeepSeek-R1-8B under Vul-RAG produces verbose reasoning traces that primarily reduce Conciseness and Clarity. Under LLM4Vuln, Phi-4 represents a more severe failure case, scoring poorly in Faithfulness, Functional Coverage, and Retrieval Utility; manual inspection shows that it returns "Code snippet" for every evaluated query. Finding #3: Abstraction quality depends on the system-specific abstraction prompt, with no clear trend attributable to model capabilities; robustness must be verified per pipeline. Judge Sensitivity. To assess whether the results depend on the selected LLM judge, we repeat the evaluation with Qwen2.5-32B and QwQ-32B. Table 3 reports ordinal Krippendorff’s α and mean pairwise absolute deviation (MAD) across the judges. Higher α indicates stronger agreement beyond chance, with α=1α=1 representing perfect agreement. MAD computes the average absolute score difference between judge pairs on the 1-5 scale. Vul-RAG’s α ranges from 0.160.16 to 0.380.38, which indicates high judge sensitivity. LLM4Vuln has higher agreement in Functional Coverage (α=0.63α=0.63) and Retrieval Utility (α=0.59α=0.59), but the other dimensions remain judge-sensitive. Despite differences in absolute scores, the judges preserve the LLM4Vuln abstraction profile, consistently rating Clarity and Conciseness highest, see Tab. 3 (b). For Vul-RAG, judge choice has a stronger effect on individual scores, particularly Retrieval Utility, but the overall ranking of dimensions remains broadly similar. 4.3.2. Knowledge Retrieval As a high-quality code summary can read well without necessarily helping vulnerability knowledge retrieval, we examine retrieval in practice. Therefore, we evaluate each knowledge base from two complementary perspectives: Knowledge Coverage and Retrieval Effectiveness. Knowledge coverage assesses whether the knowledge base contains knowledge relevant to the vulnerable codes represented in the test set, e.g., CWE type alignment. Retrieval effectiveness measures whether the retriever can successfully identify this relevant knowledge, quantified using Hit@k and MRR. Separating these two dimensions allows us to identify retrieval failures caused by missing knowledge alignment and those caused by ineffective retrieval. Knowledge Coverage. Vul-RAG constructs the knowledge base from historical CVEs, extracting functional semantics, vulnerability causes, and fixing solutions using GPT-3.5 (Du et al., 2025). The system targets 1010 CWEs from the Linux kernel and maintains a dedicated knowledge base and test set for each CWE. By construction, the knowledge base and test data are aligned. Vul-RAG also allows for simple regeneration of the knowledge base. LLM4Vuln uses a generic knowledge base generated from CWE descriptions (Sun et al., 2025b): GPT-4.1 generates 1010 code examples for each of 929266 6 The original LLM4Vuln paper reports 8686 CWE categories. Using the publicly available code and configuration, we identified 9292 distinct CWE categories. CWEs, and further summarizes functionality and root cause. While this design aims to provide generic vulnerability coverage, the knowledge base is weakly aligned with the curated test data. Specifically, only 66 of the 9292 CWEs present in the knowledge base overlap with the 2323 CWEs covered in the UniVul test set, which align with only 6060 of 920920 entries (less than 7%7\%). (a) Mean Reciprocal Rank (MRR). (b) Hit@1 and Hit@3 performance. Figure 5. Retrieval effectiveness across knowledge base and test probe combinations for LLM4Vuln and Vul-RAG. Knowledge bases are generated from the training sets using the model indicated in parentheses, while marker shape and color identify the test set probe. Large filled markers show the mean across input-abstraction models; highlighted points indicate the minimum and maximum observed values; whiskers cover 1.5×1.5×IQR. For Hit@k, the line connects the mean under exact (∙ ) with the mean under relaxed CWE matching (∘ ). Retrieval performance for LLM4Vuln and Vul-RAG across different knowledge bases and test probes. The upper panel reports Mean Reciprocal Rank, while the lower panel reports Hit@1 and Hit@3 under exact and relaxed CWE matching. Retrieval Effectiveness. For each system, we execute the original retrieval pipeline on summaries generated for the sampled test functions. We record the ranked knowledge items returned for each query and count a retrieval as relevant when the retrieved item matches the ground-truth CWE of the query function. To account for the hierarchical structure of CWE, we also consider a relaxed match in which parent or child CWE categories are treated as relevant. Hit@k therefore measures whether at least one relevant item appears among the top-k results, while MRR captures the rank of the first such match. Figure 5 visualizes the results. For Vul-RAG, retrieval performs moderately well for the PairVul test probe on the original PairVul knowledge base (mean MRR 0.540.54, Hit@3 0.660.66), consistent with the alignment analysis above, but substantially worse for the PrimeVul test probe (mean MRR 0.350.35, Hit@3 0.430.43). For LLM4Vuln, retrieval is uniformly weak across both UniVul and PrimeVul test probes (MRR 0.070.07, Hit@3 0.120.12 for PrimeVul), in line with its limited alignment between the knowledge base and the test sets. To establish an upper bound on retrieval, we construct an oracle setting in which the knowledge base is generated from the test set, guaranteeing that a ground-truth item is present in the retrieval pool. For Vul-RAG, oracle MRR is near-ceiling when the knowledge base and test set are aligned (0.930.93 for PairVul, 0.950.95 for PrimeVul), i.e., the retrieval mechanism can recover relevant knowledge effectively when sufficient task-aligned knowledge is available. Finding #4: Across the analyzed pipelines, knowledge base coverage and alignment with the target distribution are crucial for effective retrieval. Under oracle knowledge, when relevant knowledge is guaranteed to be present, Vul-RAG retrieval reaches near-ceiling effectiveness. For Vul-RAG, we further compare knowledge bases regenerated with three models of increasing scale (Qwen2.5-Coder-3B/14B/32B) against the original GPT-3.5-generated PairVul knowledge base. On the sampled PairVul test set, mean retrieval improves only slightly with increasing generation model scale (MRR: 0.56→0.580.56→ 0.58; Hit@3: 0.66→0.680.66→ 0.68). This trend does not hold on the sampled PrimeVul test set, where retrieval remains flat. Thus, within the tested models and a fixed knowledge base construction setting, scaling the generation model yields no generalizable additional benefit. In contrast, retrieval varies substantially more with the model used for input abstraction. For example, for the original PairVul knowledge base, MRR on the PairVul probe ranges from 0.370.37 to 0.600.60 across models. Although the generator can itself affect the quality and alignment of the resulting knowledge base, these results indicate that, among the tested generators, retrieval is more sensitive to how the query is formulated than to which model generated the knowledge base. Finding #5: Within a fixed knowledge base construction setting, retrieval effectiveness is more sensitive to the query abstraction model than to the knowledge base generation model. 4.3.3. Vulnerability Detection We isolate the detection stage to determine whether the LLM can exploit relevant vulnerability knowledge and translate it into a correct vulnerability prediction. We conduct two complementary experiments. First, we fix Qwen2.5-Coder-14B, which performed best for Vul-RAG in RQ2, as the detection model and vary the Vul-RAG knowledge base to isolate how much of detection performance is determined by the available knowledge. Second, we fix an oracle knowledge base and vary the detection model to assess how model capabilities bound performance. Sensitivity to the Supplied Knowledge. Figure 6(a) visualizes the results. Sub-sampling the original Vul-RAG knowledge base to 25%/50%/75% of its size yields a consistent upward trend (PairVul: 0.20 → 0.24 → 0.26; PrimeVul: 0.17 → 0.19 → 0.23), but accuracy then drops at 100% size; i.e., the 75% subset outperforms the full knowledge base. More knowledge is not uniformly beneficial; we hypothesize this trend reflects either noise from near-duplicate or low-relevance entries diluting top-k retrieval precision, or a higher share of lower-quality entries being surfaced at full scale. Regenerating the knowledge base with Qwen2.5-Coder-14B improves pairwise accuracy from 0.230.23 to 0.360.36, a 52.57%52.57\% increase. This gain, however, is not monotonic in generator scale (14B: 0.360.36 >> 32B: 0.320.32 >> 3B: 0.280.28 on PairVul). One possible explanation is an implicit alignment between the knowledge generator and the downstream detector: knowledge generated by the same model may better match the preferred representations or abstraction. Knowledge bases generated from the PrimeVul training set achieve 0.22–0.24 pairwise accuracy regardless of which test set they are evaluated against. This observation contrasts with the PairVul-derived knowledge bases, which show a clear domain-alignment effect (e.g., pairwise accuracy of 0.360.36 on its native test set vs. 0.260.26 cross-dataset). Together with the higher generation error rate observed for the PrimeVul-based knowledge bases (cf. Appendix A), the results suggest that the current generation procedure produces less useful knowledge for PrimeVul, possibly due to the longer functions or different vulnerability distribution (Kaniewski et al., 2026a). (a) Different knowledge bases. (b) Oracle. Figure 6. Vul-RAG detection stage. (a) Pairwise Accuracy across knowledge bases with Qwen2.5-Coder-14B fixed as the detector. QC = Qwen2.5-Coder. (b) Pairwise Accuracy when oracle knowledge is supplied (for those models that parsed at least 95% of pairs for both test sets, which excludes 55 models).Two panels analyze pairwise accuracy in the Vul-RAG detection stage. The left panel compares different knowledge-base sizes, generators, and source datasets with Qwen2.5-Coder-14B fixed as detector; the right panel compares detector models under oracle knowledge for PairVul and PrimeVul, showing substantial model-dependent variation even with favorable retrieval conditions. Finding #6: In the Vul-RAG pipeline, detection performance is sensitive to the knowledge base, but does not improve monotonically with knowledge base size or generator scale. The best performance is obtained with Qwen2.5-Coder-14B used for both knowledge generation and vulnerability detection, which points to a generator-detector alignment as an opportunity for improving detection. Detection under Oracle Knowledge. To approximate best-case upstream conditions, we use the test set-specific oracle knowledge bases. Detection is performed on the same test set, with retrieval restricted to knowledge derived from that test set. This oracle setting intentionally uses test-derived knowledge and is therefore diagnostic rather than deployable; it estimates detector performance when upstream knowledge availability and retrieval are made near-optimal. Figure 6(b) visualizes the results. Mean pairwise accuracy reaches 0.510.51 on PrimeVul (0.660.66 for Qwen2.5-Coder-14B), substantially above Vul-RAG under the unified benchmark in RQ2, where Qwen2.5-Coder-14B reaches the best result of 0.220.22 pairwise accuracy. The strongest oracle detectors are larger Qwen-family models, with Qwen3.6-27B reaching 0.730.73 on PrimeVul and 0.720.72 on PairVul, whereas Llama3.2-3B remains at only 0.130.13 pairwise accuracy on PrimeVul and 0.150.15 on PairVul despite operating under the same favorable retrieval conditions. The large spread under oracle knowledge shows that detector capability remains an independent bottleneck. Finding #7: Oracle retrieval does not solve vulnerability detection. Even under near-optimal oracle conditions, mean pairwise accuracy remains only 0.510.51, with the strongest detector reaching 0.730.73. Thus, correctly applying retrieved knowledge remains a substantial bottleneck. Across all three analyzed components, the results suggest that improving RAG4SVD requires optimizing the alignment between pipeline stages, rather than simply scaling models or knowledge bases. Abstraction quality is driven primarily by the system-specific summarization objective (Finding #3); retrieval effectiveness depends on query formulation and knowledge base alignment with the target distribution (Findings #4 and #5); and detection performance is influenced more by the alignment of the supplied knowledge than by knowledge base size or generator scale (Finding #6). Even where scaling does help, particularly at the detection stage, it closes only part of the gap: mean pairwise accuracy remains near 0.500.50 even under oracle knowledge (Finding #7). 5. Threats to Validity Internal Validity. We re-implemented and adapted existing RAG4SVD systems into a unified benchmark framework. Although we preserve system logic, prompts, and hyperparameters where possible, implementation differences, parser hardening, and model substitutions may affect reproduced results. We mitigate these risks by documenting all adaptations in the replication package and repeating experiments where applicable. All RAG4SVD inference pipelines use locally hosted open-weight models; the only remote dependency is the open-weight GPT-oss-120B judge used for the abstraction-quality analysis in RQ3. Construct Validity. Pairwise accuracy provides a stricter measure of vulnerable-patched discrimination than instance-level metrics, while excluding configurations with fewer than 95% parseable predictions may favor systems with more robust output formats. We therefore report parsing success in the replication package. For retrieval, we define relevance by ground-truth CWE agreement, including parent/child relations. As CWE agreement does not necessarily imply semantic or vulnerability-specific relevance, we interpret these metrics as measuring knowledge alignment rather than actionable knowledge quality. The component analysis further relies on sampled test-set probes due to the computational cost of evaluating all model-knowledge base combinations, such that small differences may reflect sampling noise. External Validity. The findings and conclusions of this work are limited to the systems, datasets, and models considered. We study six artifact-complete RAG4SVD systems, use PrimeVul Paired as the common benchmark, and evaluate a documented pool of 3B-32B open-weight models suitable for on-premises deployment. Results may differ for proprietary models, other programming languages, or vulnerability data distributions. Moreover, the component-level analysis focuses on two representative multi-stage pipelines and should therefore be interpreted as evidence for the examined systems rather than as a universal characterization of all RAG4SVD systems. 6. Conclusion In this work, we examined RAG-based software vulnerability detection from the perspectives of reproducibility, cross-system comparability, and component-level attribution. Across six open-source RAG4SVD systems, we find that reported results do not consistently transfer to open-weight evaluation settings, and that system rankings can change substantially under a unified benchmark. In the presented component analysis of representative RAG4SVD pipelines, we further dissected input abstraction, knowledge retrieval, and detection to identify where future RAG4SVD systems should focus their engineering effort. As abstraction quality, e.g., code summarization, is shaped by the system-specific prompting objective, input abstractions should be optimized for retrieval utility rather than conciseness. As retrieval effectiveness depends on knowledge base coverage and alignment with the target distribution, these properties should be established beforehand, since poor retrieval can limit detection performance. Finally, oracle retrieval does not solve vulnerability detection: oracle knowledge containing ground truth information achieves MRR of 0.930.93–0.950.95, yet mean pairwise accuracy remains only 0.510.51. Overall, these findings suggest that progress in RAG4SVD is unlikely to come from scaling individual components, such as models or knowledge bases, alone. Performance is governed by the alignment across component interfaces, e.g., abstraction to retrieval, knowledge base to target distribution, and retrieved knowledge to detector. Hence, future evaluations should complement end-to-end detection metrics with component-level evaluations of abstraction quality, knowledge base coverage, retrieval effectiveness, parsing robustness, and downstream knowledge use. Acknowledgements. This work has been funded by the Federal Ministry of Research, Technology and Space (BMFTR) and the state of Baden-Württemberg (Program: Forschung an HAW, Grant No. 13HAW26PX4), and in part by the German Research Foundation (DFG) under project-ID 528745080 - FIP 68. The authors alone are responsible for the content of this paper. We thank the DACHS data analysis cluster, hosted at Hochschule Esslingen and co-funded by the MWK within the DFG’s "Großgeräte der Länder" program, for providing the computational resources necessary for this research. We also thank Fabian Schmidt for the insightful discussions throughout the experimental process. Appendix A Dataset Statistics and Sampling For the component-level analysis in RQ3, we construct sampled test sets that match the CWE distributions of the system-specific reference datasets. For Vul-RAG, we sample PrimeVul according to the ten CWEs represented in PairVul, selecting five instances per CWE where available. As PrimeVul contains only one CWE-264 instance, the resulting sample PrimeVul*PairVul_PairVul contains 46 vulnerable functions. For LLM4Vuln, we use the full UniVul test set as the reference distribution and sample PrimeVul to match its per-CWE counts where possible. This yields PrimeVul*UniVul_UniVul with 42 vulnerable functions across 18 overlapping CWEs. Tab. 4 summarizes the dataset and probe statistics as well as the knowledge base generation statistics. Table 4. Dataset statistics. (a) Train and test datasets, reported with their number of vulnerable functions and CWEs. (b) Test set probes. (c) Vul-RAG knowledge base generation statistics (PairVul-specific CWE subset). (a) Dataset statistics. Dataset Train Test #Vuln. #CWEs #Vuln. #CWEs PrimeVul 3789 111 435 62 PairVul 2317 10 586 10 UniVul 920 92 50 23 (b) Test set probes. System Probe #Vuln. #CWEs Vul-RAG PairVul* 46 10 Vul-RAG PrimeVul*PairVul_PairVul 46 10 LLM4Vuln UniVul 50 23 LLM4Vuln PrimeVul*UniVul_UniVul 42 18 (c) Knowledge base generation. Source Generator Expected Generated Errors Training-derived knowledge bases PairVul GPT-3.5 (original) 2317 – – PairVul Qwen2.5-Coder-3B 2317 2312 5 PairVul Qwen2.5-Coder-14B 2317 2316 1 PairVul Qwen2.5-Coder-32B 2317 2317 0 PrimeVul Qwen2.5-Coder-3B 2398 2347 51 PrimeVul Qwen2.5-Coder-14B 2398 2370 28 PrimeVul Qwen2.5-Coder-32B 2398 2374 24 Oracle knowledge bases (test-derived) PairVul Qwen2.5-Coder-14B 586 585 1 PrimeVul Qwen2.5-Coder-14B 247 244 3 Appendix B Model Lineup Table 5 lists the 22 open-weight models with their checkpoints used in the evaluation. The model pool spans multiple model families, parameter scales, and specializations. Table 5. Pool of pen-weight models used throughout the benchmark and component analysis. Family Model (Paper Abbreviation) Scale Specialization Full Hugging Face Checkpoint Context Size Qwen Qwen2.5-3B (Qwen et al., 2025) 3.1B General Qwen/Qwen2.5-3B-Instruct 32K Qwen2.5-14B (Qwen et al., 2025) 14.7B General Qwen/Qwen2.5-14B-Instruct 32K Qwen2.5-32B (Qwen et al., 2025) 32.5B General Qwen/Qwen2.5-32B-Instruct 32K Qwen2.5-Coder-3B (Hui et al., 2024) 3.1B Code Qwen/Qwen2.5-Coder-3B-Instruct 32K Qwen2.5-Coder-14.7B (Hui et al., 2024) 14B Code Qwen/Qwen2.5-Coder-14B-Instruct 32K Qwen2.5-Coder-32.5B (Hui et al., 2024) 32B Code Qwen/Qwen2.5-Coder-32B-Instruct 32K Qwen3-4B (Yang et al., 2025) 4.0B General Qwen/Qwen3-4B 32K Qwen3.5-9B (Qwen, 2026a) 9.0B General Qwen/Qwen3.5-9B 256K Qwen3.6-27B (Qwen, 2026b) 27.0B General Qwen/Qwen3.6-27B 256K QwQ-32B (Qwen, 2025) 32.5B General Qwen/QwQ-32B 128K Llama Llama-3.2-3B (Grattafiori et al., 2024) 3.2B General meta-llama/Llama-3.2-3B-Instruct 128K Llama-3.1-8B (Grattafiori et al., 2024) 8.0B General meta-llama/Llama-3.1-8B-Instruct 128K Phi Phi-4-Mini (Microsoft et al., 2025) 3.8B General microsoft/Phi-4-mini-instruct 128K Phi-4 (Abdin et al., 2024) 14.7B General microsoft/phi-4 16K Gemma Gemma-3-4B (Team et al., 2025) 4.0B General google/gemma-3-4b-it 128K Gemma-3-12B (Team et al., 2025) 12.0B General google/gemma-3-12b-it 128K Gemma-3-27B (Team et al., 2025) 27.0B General google/gemma-3-27b-it 128K NextCoder NextCoder-14B (Aggarwal et al., 2025) 14.7B Code microsoft/NextCoder-14B 32K NextCoder-32B (Aggarwal et al., 2025) 32.5B Code microsoft/NextCoder-32B 32K DeepSeek DeepSeek-R1-8B (DeepSeek-AI et al., 2025a) 8.0B General deepseek-ai/DeepSeek-R1-Distill-Llama-8B 128K DeepSeek-R1-32B (DeepSeek-AI et al., 2025a) 32.5B General deepseek-ai/DeepSeek-R1-Distill-Qwen-32B 128K Mistral Mistral-7B (Jiang et al., 2023) 7.3B General mistralai/Mistral-7B-Instruct-v0.3 32K Appendix C Full Reproduction Results Tab. 6 reports the absolute values underlying the RQ1 reproducibility analysis in Sect. 4.1. For further details, e.g., the calculation of deviations, we refer to the replication package. Table 6. Absolute reproduction results for the systems evaluated in RQ1. ∘ denotes the reported result, ∙i _i the iith reproduction run, and ∙¯ the mean across runs. SVD-Bench provides a seed and yields identical results across all three runs. For Llama-VD, precision, recall, and F1 refer to the vulnerable class. SVD-Bench (Zhang et al., 2025) Python Java JavaScript Model Prec. Rec. F1 Prec. Rec. F1 Prec. Rec. F1 CodeQwen1.5-7B ∘ 0.11 0.52 0.18 0.11 0.48 0.18 0.19 0.56 0.29 ∙ 0.08 0.20 0.11 0.08 0.25 0.12 0.20 0.50 0.28 DeepSeek-Coder-7B ∘ 0.11 0.32 0.16 0.14 0.31 0.19 0.19 0.38 0.25 ∙ 0.08 0.15 0.11 0.10 0.17 0.13 0.19 0.34 0.25 CodeGemma-7B ∘ 0.11 0.26 0.15 0.13 0.41 0.19 0.18 0.54 0.27 ∙ 0.08 0.14 0.10 0.11 0.28 0.15 0.18 0.48 0.26 StarCoder2-7B ∘ 0.12 0.29 0.17 0.16 0.19 0.18 0.21 0.40 0.28 ∙ 0.14 0.07 0.09 0.16 0.16 0.16 0.22 0.37 0.27 CodeLlama-7B ∘ 0.12 0.29 0.17 0.12 0.45 0.18 0.19 0.66 0.30 ∙ 0.09 0.14 0.11 0.09 0.24 0.13 0.20 0.58 0.29 Llama-VD (Ouchebara and Dupont, 2025) - Llama-3.1-8B Big-Vul PrimeVul Acc. Prec. Rec. F1 Acc. Prec. Rec. F1 ∙1 _1 0.69 0.70 0.65 0.68 0.66 0.68 0.63 0.65 ∙2 _2 0.70 0.71 0.67 0.69 0.67 0.68 0.64 0.66 ∙3 _3 0.70 0.71 0.67 0.69 0.67 0.68 0.63 0.66 ∘ 0.70 0.71 0.67 0.69 0.67 0.68 0.64 0.66 ∙¯ 0.70 0.71 0.66 0.69 0.67 0.68 0.63 0.66 Vul-RAG (Du et al., 2025) - Qwen2.5-Coder-32B PairVul Pair. Acc. Rec. Prec. ∙1 _1 0.25 0.58 0.58 ∙2 _2 0.25 0.59 0.60 ∙3 _3 0.27 0.60 0.60 ∘ 0.26 0.59 0.61 ∙¯ 0.26 0.59 0.59 LLM4Vuln (Sun et al., 2025b) C/C++ Java Model Prec. Rec. F1 Prec. Rec. F1 QwQ-32B ∙1 _1 0.21 0.19 0.14 0.08 0.02 0.11 ∙2 _2 0.18 0.16 0.13 0.03 0.01 0.04 ∙3 _3 0.19 0.17 0.13 0.04 0.01 0.06 ∘ 0.26 0.33 0.29 0.14 0.12 0.13 ∙¯ 0.19 0.17 0.13 0.05 0.01 0.07 Llama-3.1-8B ∙1 _1 0.12 0.12 0.11 0.04 0.03 0.03 ∙2 _2 0.19 0.19 0.17 0.07 0.06 0.05 ∙3 _3 0.15 0.15 0.14 0.05 0.04 0.04 ∘ 0.09 0.17 0.12 0.05 0.09 0.06 ∙¯ 0.15 0.15 0.14 0.05 0.04 0.04 Phi-3-Mini-128K ∙1 _1 0.21 0.17 0.21 0.08 0.08 0.08 ∙2 _2 0.17 0.15 0.18 0.08 0.08 0.08 ∙3 _3 0.19 0.16 0.19 0.08 0.08 0.08 ∘ 0.10 0.20 0.13 0.07 0.14 0.10 ∙¯ 0.19 0.16 0.19 0.08 0.08 0.08 References Abdin et al. (2024) M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al. Phi-4 Technical Report. External Links: 2412.08905 Cited by: Table 5, §3.4. Aggarwal et al. (2025) T. Aggarwal, S. Singh, A. Awasthi, A. Kanade, and N. Natarajan Robust learning of diverse code edits. External Links: 2503.03656 Cited by: Table 5, Table 5, §3.4. Chen et al. (2026) Z. Chen, L. Xu, Z. Lin, X. Ju, and X. Chen A Systematic Literature Review of Retrieval-Augmented Generation Methods for the Software Vulnerability Lifecycle. External Links: 7113581 Cited by: §1, §1, §1, §2.2, §2.3. Crupi et al. (2025) G. Crupi, R. Tufano, A. Velasco, A. Mastropaolo, D. Poshyvanyk, and G. Bavota On the Effectiveness of LLM-as-a-Judge for Code Generation and Summarization. IEEE Transactions on Software Engineering (TSE) 51 (8), p. 2329–2345. External Links: Document Cited by: §4.3.1. DeepSeek-AI et al. (2025a) DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, et al. DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. External Links: 2501.12948 Cited by: Table 5, Table 5, §3.4. DeepSeek-AI et al. (2025b) DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, et al. DeepSeek-V3 Technical Report. External Links: 2412.19437 Cited by: §3.4. Ding et al. (2025) Y. Ding, Y. Fu, O. Ibrahim, C. Sitawarin, X. Chen, B. Alomair, D. Wagner, B. Ray, and Y. Chen Vulnerability Detection with Code Language Models: How Far are We?. In International Conference on Software Engineering (ICSE), Ottawa, Ontario, Canada, p. 1729–1741. External Links: Document Cited by: §2.1, §3.2, §3.3. Du et al. (2025) X. Du, G. Zheng, K. Wang, Y. Zou, Y. Wang, W. Deng, J. Feng, M. Liu, B. Chen, X. Peng, T. Ma, and Y. Lou Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG. External Links: 2406.11147 Cited by: Table 6, §1, §1, §2.2, §2.2, §3.1, §3.3, §3.5, Table 1, §4.3.2, §4.3. Es et al. (2024) S. Es, J. James, L. Espinosa Anke, and S. Schockaert RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, St. Julians, Malta, p. 150–158. External Links: Document Cited by: §2.3. Fan et al. (2020) J. Fan, Y. Li, S. Wang, and T. N. Nguyen A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In International Conference on Mining Software Repositories (MSR), Seoul, South Korea, p. 508–512. External Links: Document Cited by: §2.1. Gan et al. (2025) A. Gan, H. Yu, K. Zhang, Q. Liu, W. Yan, Z. Huang, S. Tong, and G. Hu Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey. External Links: 2504.14891 Cited by: §3.3. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 Herd of Models. External Links: 2407.21783 Cited by: Table 5, Table 5, §3.4. Guo and Bettaieb (2023) Y. Guo and S. Bettaieb An Investigation of Quality Issues in Vulnerability Detection Datasets. In European Symposium on Security and Privacy Workshops (EuroS&PW), Delft, Netherlands, p. 29–33. External Links: Document Cited by: §2.1. HS Esslingen IT Security (2026) HS Esslingen IT SecurityAwesome llm4svd: llm4svd taxonomy(Website) External Links: Link Cited by: §1, Figure 2, §3.1. Hui et al. (2024) B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. Qwen2.5-Coder Technical Report. External Links: 2409.12186 Cited by: Table 5, Table 5, Table 5, §3.4. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7B. External Links: 2310.06825 Cited by: Table 5, §3.4. Kaniewski et al. (2026a) S. Kaniewski, F. Schmidt, M. Enzweiler, M. Menth, and T. Heer A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models. ACM Transactions on Software Engineering and Methodology (TOSEM) (), p. . Note: Just Accepted External Links: Document Cited by: §1, §1, §1, §2.2, §2, §4.3.3. Kaniewski et al. (2026b) S. Kaniewski, F. Schmidt, and T. Heer Revisiting vul-rag: reproducibility and replicability of rag-based vulnerability detection with open-weight models. In Availability, Reliability and Security. ARES 2026 International Workshops: ARES 2026 International Workshops, Linköping, Sweden, p. 76–95. External Links: Document Cited by: §1, §2.2. Ke et al. (2026) Q. Ke, Y. Zhao, H. Leng, S. Zhao, and H. Wang Not All RAGs Are Created Equal: A Component-Wise Empirical Study for Software Engineering Tasks. Proceedings of the ACM on Software Engineering 3 (FSE). External Links: Document Cited by: §2.3. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, Virtual-only, p. 9459–9474. Cited by: §1. Li et al. (2026) Y. Li, F. Yu, and X. Wang From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection. External Links: 2602.14012 Cited by: §3.5, §4.3.1. Lu et al. (2024) G. Lu, X. Ju, X. Chen, W. Pei, and Z. Cai GRACE: Empowering LLM-based Software Vulnerability Detection with Graph Structure and In-Context Learning. Journal of Systems and Software (JSS) 212 (C), p. 112031. External Links: Document Cited by: §1, §3.1, Table 1. Microsoft et al. (2025) Microsoft, A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, et al. Phi-4-mini technical report: compact yet powerful multimodal language models via mixture-of-loras. External Links: 2503.01743 Cited by: Table 5, §3.4. Nong et al. (2023) Y. Nong, R. Sharma, A. Hamou-Lhadj, X. Luo, and H. Cai Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability Detection. IEEE Transactions on Software Engineering 49 (4), p. 1983–2005. External Links: Document Cited by: §4.1. OpenAI et al. (2024) OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, et al. GPT-4 Technical Report. External Links: 2303.08774 Cited by: §3.4. OpenAI et al. (2025) OpenAI, S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, et al. gpt-oss-120b & gpt-oss-20b Model Card. External Links: 2508.10925 Cited by: §3.5. Ouchebara and Dupont (2025) D. S. Ouchebara and S. Dupont Llama-Based Source Code Vulnerability Detection: Prompt Engineering vs Fine Tuning. In European Symposium on Research in Computer Security (ESORICS), Toulouse, France, p. 289–308. External Links: Document Cited by: Table 6, §1, §3.1, Table 1. Qwen et al. (2025) Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, et al. Qwen2.5 Technical Report. External Links: 2412.15115 Cited by: Table 5, Table 5, Table 5, §3.4. Qwen (2025) Qwen QwQ-32B: Embracing the Power of Reinforcement Learning. Note: https://qwen.ai/blog?id=qwq-32bAccessed 2026-09-11 Cited by: Table 5, §3.4. Qwen (2026a) Qwen Qwen3.5: Towards Native Multimodal Agents. Note: https://qwen.ai/blog?id=qwen3.5Accessed 2026-09-11 Cited by: Table 5, §3.4. Qwen (2026b) Qwen Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model. Note: https://qwen.ai/blog?id=qwen3.6-27bAccessed 2026-09-11 Cited by: Table 5, §3.4. Saad-Falcon et al. (2024) J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Mexico City, Mexico, p. 338–354. External Links: Document Cited by: §2.3. Sallou et al. (2024) J. Sallou, T. Durieux, and A. Panichella Breaking the Silence: the Threats of Using LLMs in Software Engineering. In ACM/IEEE International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), Lisbon, Portugal, p. 102–106. External Links: Document Cited by: §1. Song et al. (2024) H. Song, H. Su, I. Shalyminov, J. Cai, and S. Mansour FineSurE: Fine-grained Summarization Evaluation using LLMs. In Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand, p. 906–922. External Links: Document Cited by: §4.3.1. Steenhoek et al. (2025) B. Steenhoek, M. M. Rahman, M. K. Roy, M. S. Alam, H. Tong, S. Das, E. T. Barr, and W. Le To Err is Machine: Vulnerability Detection Challenges LLM Reasoning. External Links: 2403.17218 Cited by: §2.1, §2.1. Sun et al. (2025a) W. Sun, Y. Miao, Y. Li, H. Zhang, C. Fang, Y. Liu, G. Deng, Y. Liu, and Z. Chen Source Code Summarization in the Era of Large Language Models. In IEEE/ACM International Conference on Software Engineering (ICSE), Vol. , Ottawa, ON, Canada, p. 1882–1894. External Links: Document Cited by: §4.3.1. Sun et al. (2025b) Y. Sun, D. Wu, Y. Xue, H. Liu, W. Ma, L. Zhang, Y. Liu, and Y. Li LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning. External Links: 2401.16185 Cited by: Table 6, §1, §3.1, Table 1, §4.3.2, §4.3. Tamberg and Bahsi (2025) K. Tamberg and H. Bahsi Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study. IEEE Access 13, p. 29698–29717. External Links: Document Cited by: §2.1, §3.3. Tang et al. (2026) W. Tang, X. Zhang, J. Liu, J. Xiao, X. Xiao, J. Yang, Y. Ma, Z. Liu, Z. Li, Z. Wang, W. Luo, Q. Li, L. Wang, and P. Xiangli VulTriage: Triple-Path Context Augmentation for LLM-Based Vulnerability Detection. External Links: 2605.09461 Cited by: §1, §2.2, §3.1, Table 1, §4.2. Team et al. (2025) G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. External Links: 2503.19786 Cited by: Table 5, Table 5, Table 5, §3.4. Ullah et al. (2024) S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks. In Symposium on Security and Privacy (SP), San Francisco, CA, USA, p. 862–880. External Links: Document Cited by: §2.1, §2.1. Wang et al. (2025) R. Wang, J. Guo, C. Gao, G. Fan, C. Y. Chong, and X. Xia Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering. Proceedings of the ACM on Software Engineering (PACMSE) 2 (ISSTA). External Links: Document Cited by: §4.3.1. Wu et al. (2025) Y. Wu, Y. Wan, Z. Chu, W. Zhao, Y. Liu, H. Zhang, X. Shi, H. Jin, and P. S. Yu Can Large Language Models Serve as Evaluators for Code Summarization?. IEEE Transactions on Software Engineering (TSE) 51 (12), p. 3205–3217. External Links: Document Cited by: §4.3.1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 Technical Report. External Links: 2505.09388 Cited by: Table 5, §3.4. Zhang et al. (2025) T. Zhang, C. Yang, Y. Su, M. Weyssow, H. Nguyen, T. Bui, H. J. Kang, Y. Li, E. L. Ouh, L. K. Shar, and D. Lo Benchmarking Large Language Models for Multi-Language Software Vulnerability Detection. External Links: 2503.01449 Cited by: Table 6, §1, §2.1, §3.1, Table 1. Zhao et al. (2026) T. Zhao, C. Ma, L. Zhang, J. Yang, and L. Nie Retrieval-Augmented Semantic Mapping for Vulnerability Detection via Multi-View Code Similarity. Electronics 15 (3), p. . External Links: Document Cited by: §2.2. Zhu et al. (2026) H. Zhu, J. Li, C. Gao, J. Qian, Y. Dong, H. Liu, L. Wang, Z. Wang, X. Hu, and G. Li VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications. Proceedings of the ACM on Software Engineering 3 (FSE). External Links: Document Cited by: §2.2.