Paper deep dive
Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning
Ronghao Ni, Mihai Christodorescu, Limin Jia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/26/2026, 5:15:11 PM
Summary
The paper introduces LLMVD.js, a multi-stage ReAct-based LLM agent pipeline designed to detect and confirm taint-style vulnerabilities (such as command injection and prototype pollution) in Node.js packages. Unlike traditional rule-based program analysis tools (e.g., FAST, NodeMedic-FINE, Explode.js) that struggle with JavaScript's dynamic nature and complex dependencies, LLMVD.js leverages the reasoning capabilities of LLMs and lightweight execution oracles to generate and validate proof-of-concept (PoC) exploits. The study demonstrates that LLMVD.js significantly outperforms prior tools and hybrid approaches, confirming 84% of vulnerabilities in public benchmarks and discovering 36 previously undocumented vulnerabilities in recently released npm packages.
Entities (8)
Relation Signals (4)
LLMVD.js → implements → ReAct
confidence 100% · we design and implement LLMVD.js, a ReAct-based agent
Taint-style Vulnerability → includes → OS Command Injection
confidence 100% · These tools primarily target taint-style vulnerabilities, including OS command injection, code injection, prototype pollution, and path traversal.
LLMVD.js → outperforms → FAST
confidence 90% · LLMVD.js confirms 84% of the vulnerabilities, compared to less than 22% for prior program analysis tools.
LLMVD.js → outperforms → PoCGen
confidence 90% · It also outperforms a prior LLM–program-analysis hybrid approach [47]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The rapidly evolving Node$.$js ecosystem currently includes millions of packages and is a critical part of modern software supply chains, making vulnerability detection of Node$.$js packages increasingly important. However, traditional program analysis struggles in this setting because of dynamic JavaScript features and the large number of package dependencies. Recent advances in large language models (LLMs) and the emerging paradigm of LLM-based agents offer an alternative to handcrafted program models. This raises the question of whether an LLM-centric, tool-augmented approach can effectively detect and confirm taint-style vulnerabilities (e.g., arbitrary command injection) in Node$.$js packages. We implement LLMVD$.$js, a multi-stage agent pipeline to scan code, propose vulnerabilities, generate proof-of-concept exploits, and validate them through lightweight execution oracles; and systematically evaluate its effectiveness in taint-style vulnerability detection and confirmation in Node$.$js packages without dedicated static/dynamic analysis engines for path derivation. For packages from public benchmarks, LLMVD$.$js confirms 84% of the vulnerabilities, compared to less than 22% for prior program analysis tools. It also outperforms a prior LLM-program-analysis hybrid approach while requiring neither vulnerability annotations nor prior vulnerability reports. When evaluated on a set of 260 recently released packages (without vulnerability groundtruth information), traditional tools produce validated exploits for few ($\leq 2$) packages, while LLMVD$.$js generates validated exploits for 36 packages.
Tags
Links
- Source: https://arxiv.org/abs/2604.20179v1
- Canonical: https://arxiv.org/abs/2604.20179v1
Trouble viewing inline? Open PDF directly →
Full Text
97,841 characters extracted from source content.
Expand or collapse full text
Taint-Style Vulnerability Detection and Confirmation forNode.js Packages Using LLM Agent Reasoning Ronghao Ni ronghaon@andrew.cmu.edu Carnegie Mellon University Mihai Christodorescu christodorescu@google.com Google Limin Jia liminjia@andrew.cmu.edu Carnegie Mellon University Abstract The rapidly evolvingNode.jsecosystem currently includes mil- lions of packages and is a critical part of modern software supply chains, making vulnerability detection ofNode.jspackages increas- ingly important. However, traditional program analysis struggles in this setting because of dynamic JavaScript features and the large number of package dependencies. Recent advances in large lan- guage models (LLMs) and the emerging paradigm of LLM-based agents offer an alternative to handcrafted program models. This raises the question of whether an LLM-centric, tool-augmented approach can effectively detect and confirm taint-style vulnerabil- ities (e.g., arbitrary command injection) inNode.jspackages. We implementLLMVD.js, a multi-stage agent pipeline to scan code, propose vulnerabilities, generate proof-of-concept exploits, and validate them through lightweight execution oracles; and systemat- ically evaluate its effectiveness in taint-style vulnerability detection and confirmation inNode.jspackages without dedicated static/- dynamic analysis engines for path derivation. For packages from public benchmarks,LLMVD.jsconfirms 84% of the vulnerabilities, compared to less than 22% for prior program analysis tools. It also outperforms a prior LLM–program-analysis hybrid approach while requiring neither vulnerability annotations nor prior vulnerability reports. When evaluated on a set of 260 recently released packages (without vulnerability groundtruth information), traditional tools produce validated exploits for few (≤2) packages, whileLLMVD.js generates validated exploits for 36 packages. Keywords Automatic Vulnerability Detection,Node.js, Large Language Mod- els, ReAct Agents, LLM Agents, Exploit Generation, Vulnerability Confirmation 1 Introduction TheNode.jsecosystem comprises millions of JavaScript packages and is among the most widely used software platforms today. How- ever, numerous studies have shown that a substantial fraction of these packages contain security vulnerabilities [15,58] and have been exploited in software supply-chain attacks [30,40]. To ensure application security, it is critical to be able to identify vulnerabil- ities within this ecosystem and in recent years researchers have proposed tools to automatically detect and confirm vulnerabilities inNode.jspackages [9,10,28,35]. These tools primarily target taint-style vulnerabilities, including OS command injection, code injection, prototype pollution, and path traversal. These tools use a wide range of analysis techniques, such as dynamic taint analysis, Preprint. code-property-graph-based static analysis, symbolic execution, and constraint-based synthesis. While successful in uncovering many real-world vulnerabilities, these tools share several fundamental challenges stemming from inherent limitations of the underlying analysis techniques. First, JavaScript’s highly dynamic nature makes accurate modeling of language semantics difficult. The lack of precise type information further complicates analysis. Second,Node.jsnative (built-in) func- tions are implemented in C++, requiring either instrumentation of the V8 engine or manually constructed abstractions. Third, these tools rely on external JavaScript analysis infrastructure, such as parsers [2], transpilers [1], and instrumentation tools [44]. These dependencies are inherently brittle due to JavaScript’s complex lan- guage features and evolving standards. Finally, many approaches depend on satisfiability modulo theories (SMT) solvers, which often fail to handle constraints involving string operations and regular- expression matching. Large Language Models (LLMs) have shown strong performance on coding-related tasks, including code generation and code com- prehension [11]. Rather than relying on purpose-fit abstractions, LLMs leverage extensive pre-trained knowledge and reasoning abil- ities to comprehend complex code patterns and dynamic program behaviors. Furthermore, LLM agents can iteratively refine their outputs by incorporating feedback from previous unsuccessful at- tempts. These capabilities suggest LLMs may be able to overcome the limitations of traditional program analysis techniques. This paper aims to answer the following question: Can an LLM- centric, tool-augmented workflow effectively detect and confirm vul- nerabilities in npm packages without dedicated static/dynamic anal- ysis engines? To answer this question, we design and implement LLMVD.js, a ReAct-based [54] agent that leverages large language models to perform taint-style vulnerability detection and confirma- tion forNode.jspackages. We evaluateLLMVD.json three datasets (existing public benchmarks, one private benchmark, and a set of recently released npm packages) and show that it confirms 84% of public benchmark vulnerabilities with valid exploits, substantially outperforming prior program-analysis tools and also outperform- ing an LLM+program-analysis hybrid system [47] while requiring significantly less prior information, and further discovers 36 vul- nerabilities in recently released packages. For the rest of this paper, we use rule-based program analysis to refer to analysis techniques such as symbolic execution and taint analysis that do not rely on machine learning components. We use LLM-centric reasoning to refer to LLM-agent reasoning over raw source code with lightweight tooling (e.g., search, execution, and oracles), but without dedicated static/dynamic analysis engines for taint/path derivation. Our contributions are as follows: 1 arXiv:2604.20179v1 [cs.CR] 22 Apr 2026 Ronghao Ni, Mihai Christodorescu, and Limin Jia •A systematic evaluation of a multi-stage ReAct-style LLM agent framework for taint-style vulnerability detection and confirmation inNode.jspackages, with direct com- parison against state-of-the-art program-analysis tools and a program-analysis–aided LLM approach. •A multi-dataset evaluation setup for LLM-agent vulnerabil- ity research: combining standard public benchmarks, trans- formed benchmark variants for memorization robustness checks, a private real-world dataset without CVEs/public exploits, and recently released npm packages to assess gen- eralizability under realistic settings. • LLMVD.jsidentified 36 previously undocumented vulnera- bilities in recently released Node.js packages. Ethical Considerations Our work raises inherent dual-use con- cerns due toLLMVD.js’s ability to automatically detect vulnerabili- ties and generate exploits; however, we believe that the defensive benefits outweigh the associated risks. All experiments were con- ducted in sandboxed environments; no production systems or exter- nal servers were targeted. We analyze only open-source packages. We reported all 36 previously unreported validated vulnerabili- ties identified in newly released packages to maintainers and have received acknowledgments from 3 maintainers. 2 Background and Related Work Vulnerability detection is the task of identifying security-relevant flaws in software that may be exploited by adversaries [12,31, 42,43]. Vulnerability detection tools typically only report poten- tial vulnerabilities and a separate confirmation step is needed to remove false positives. To reduce the costly manual confirma- tion effort, researchers have developed automated vulnerability confirmation methods that synthesize proof-of-concept (PoC) ex- ploits [5,9,10,35]. We review most recent work on vulnerability detection and confirmation ofNode.jspackages and applying LLMs to vulnerability detection. 2.1 Node.js Vulnerability Detection and Confirmation Node.js Taint-style Vulnerability Detection. Recent vulnerability detection tools forNode.jspackages [9,10,28,29,35,39] focus on taint-style vulnerabilities, partly because they have easy to detect code patterns and partly because they can lead to serious con- sequences such as allowing attackers to inject arbitrary code or execute arbitrary commands. For detection, these tools need to iden- tify tainted paths from attacker controlled inputs to arguments of a sink. In the case of command injection, code injection, and path tra- versal, the sinks are known APIs such asexec,Function,File.Write. For prototype pollution, the vulnerability pattern involves two tainted paths and specific object field accesses. To compare against LLM-centric agent reasoning without dedicated static/dynamic analysis engines, we use FAST [28], NodeMedic-FINE [9], andExplode.js[35] as representative exam- ples of rule-based program analysis tools. FAST andExplode.js use code-property-graph (CPG) based methods for detection. They generate (their own custom) graphs representing information such as dependency, object relationship, and key operations; then Listing 1: Code snippet of a vulnerable API 1 const exec = require('child_process'); 2 Arpping.prototype.ping = function(range) 3 ... 4 return new Promise((resolve, reject) => 5 range.forEach(ip => 6 exec(`ping $flag $this.timeout $ip`, ( err, stdout, stderr) => 7 ... 8 ););); 9 Listing 2: PoC exploit 1 const Arpping = require('./index'); 2 (async () => 3 const a = new Arpping( timeout: 1 ); 4 const payload = ['127.0.0.1; touch /tmp/ os_cmd_success']; 5 await a.ping(payload); 6 )(); Figure 1: The vulnerable npm packagearpping@2.0.0(Snyk ID: SNYK-JS-ARPPING-1060047). vulnerability detection is reduced to a graph query. NodeMedic- FINE on the other hand, instruments JavaScript at the source level to implement dynamic taint tracking for vulnerability detection. The current implementation of NodeMedic-FINE only detects command injection and code injection vulnerabilities. Vulnerability confirmation via exploit generation. To generate PoC exploits, the tools need to generate a driver (testing harness) that can trigger a call to the vulnerable API (sink) and find inputs that can not only reach the sink, but also deliver the desired attack payload (e.g., a command of attacker’s choice for command injection vulnerabilities). For example, Listing 1 shows one vulnerable API in the arpping package (Snyk ID: SNYK-JS-ARPPING-1060047). The pingfunction constructs a command string using attacker-controlled inputrangeand passes it to theexecfunction, leading to an OS com- mand injection vulnerability. To confirm this vulnerability, the tool needs to generate a driver (Listing 2) that calls thepingfunction with an input that injects an attacker-controlled command. For inputs, FAST andExplode.jsuse symbolic execution to iden- tify path constraints to reach the vulnerable API. NodeMedic-FINE synthesizes input constraints from the taint provenance graph, which is an output from the dynamic taint analysis and documents all the operations that the tainted inputs underwent before reach- ing the sink. OnlyExplode.jsis capable of generating drivers that can chain multiple API calls to reach the sink. It does so by query- ing its custom code property graph to identify a linear call chain. NodeMedic-FINE and FAST, instead, use a fix template to directly call the vulnerable API. All tools rely on SMT solvers to resolve constraints. 2 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning 2.2 LLMs in Software Vulnerability Detection LLM-assisted Security Audit. Recent work has explored using LLMs as assistants for security auditing, either as standalone vul- nerability detectors or as components that augment traditional pro- gram analysis workflows [26,46,52,56]. Several studies systemati- cally evaluate the capability of LLMs on vulnerability detection and code analysis tasks. For example, Fang et al. analyze the strengths and failure modes of LLMs for code reasoning in security-relevant settings [19], while Lin et al. conduct a large-scale comparative eval- uation of LLM configurations across multiple datasets and program- ming languages [34]. Similar evaluation efforts further characterize prompt sensitivity, model scale effects, and generalization limits in vulnerability detection [37,41]. LLMs have been increasingly inte- grated into end-to-end auditing pipelines that combine detection, testing, and repair. Prior work demonstrates LLM-guided fuzzing and protocol testing, where models infer grammars and message se- quences to improve coverage for stateful implementations [36,49]. Other systems leverage LLMs to assist patch generation and auto- mated repair under realistic constraints [18, 38]. LLMs have also been integrated intoNode.jsvulnerability anal- ysis [24,32,33]. Most closely related to our work, PoCGen utilizes LLMs alongside static and dynamic analyses to interpret vulner- ability reports, draft candidate exploits, and iteratively validate and refine them for npm vulnerabilities [47]. PoCGen relies on CodeQL-based static taint analysis to identify candidate vulnerable functions and input-to-sink paths, and employs dynamic analysis by executing generated exploits in a sandboxedNode.jsenviron- ment with vulnerability-specific runtime oracles to validate exploit success and guide refinement. PoCGen demonstrates the effective- ness of tightly integrating LLMs with program analysis and argues that plain LLM-based agents are insufficient for this task [47]; in contrast, we investigate whether a carefully designed LLM reason- ing–only agent can achieve competitive performance. Accordingly, we include PoCGen as a baseline in our evaluation. LLM Agents in Security Auditing. A parallel line of work studies LLMs as agents that can plan, use tools, and iteratively refine hy- potheses, moving beyond single-pass vulnerability labeling toward multi-step auditing [22,25,45,57]. PentestGPT formalizes an LLM- driven penetration-testing workflow with modular agent roles that decompose high-level objectives into actionable testing steps and tool interactions [16]. More broadly, recent studies demonstrate that agentic LLM systems can exploit real-world known vulnerabili- ties from vulnerability descriptions, highlighting both the potential and the risks of autonomous offensive capability when paired with tool use and structured memory [20]. However, to our knowledge, no prior work has studied the performance and limitations of a carefully designed LLM agent for end-to-end taint-style vulnerabil- ity detection and confirmation inNode.jspackages, which is the focus of this paper. 3 Motivation In this section, we discuss fundamental challenges that rule-based program analysis tools face when detecting and confirming taint- style vulnerabilities inNode.jspackages and outline why the capa- bilities of LLMs in a ReAct-based agent design framework suit this task. Table 1: Limitations of three representative program- analysis-based tools forNode.jstaint-style vulnerability de- tection and confirmation: FAST [28], NodeMedic-FINE [9], Explode.js [35] ChallengesWhy challenging C1Generating dri- vers for PoC Drivers may include complex interactions that make multiple API calls, set up global environ- ment correctly, and construct and invoke call- backs. Tools FASTDoes not generate drivers. Limits NodeMedic- FINE Uses fixed driver templates and cannot handle complex interactions. Explode.jsSupports only linear call chains C2Hard to analyze code units Imported dependencies significantly increase the complexity of the analysis. Native operations are managed internally by the JavaScript engine and therefore need delicate custom handling. Tools FASTNeeded manual modeling of native functions. Current implementation is missing significant (> 90%) support. Limits NodeMedic- FINE Applies over-approximated tainting policy. Explode.js Needs manually crafted symbolic summaries for imported APIs, resulting in low detection rate in Node.js packages in the wild. C3Lackingtype info. Analysis needs arguments of the correct type and object structure. Tools FASTIgnore types Limits NodeMedic- FINE Algorithms for reconstructing types, neither sound nor complete Explode.jsQuerying the CPG to reconstruct types, neither sound nor complete C4Relianton other JavaScript analysis infras- tructure Analysis tools need another set of complex tools such as parsers, transpilers, and instrumentation tools, to implement their custom analysis. These tools are brittle due to JavaScript’s standard evo- lutions and complex features. Tools FASTEsprima [2] parsing errors and call-edge issues. Limits NodeMedic- FINE Jalangi2’s [44] lack of support for ES6+ Explode.jsGraph.js [21] exits with errors for many pack- ages. C5Reliant on SMTThe analysis generates constraints on string op- erations and regular expression matching, which are difficult to handle for SMT solvers and is an active area of research. Tools FASTZ3 [14] timeout when solving path constraints. Limits NodeMedic- FINE SMT solver timeout when solving constraints on input. Explode.jsSMT solver timeout when solving path and in- put constraints. 3.1Challenges in Rule-based Program Analysis Recall from Section 2.1 state-of-the-art rule-based tools [9,28,35] implement dynamic taint tracking and code-property-graph-based static analysis for detection; and leverages symbolic execution and 3 Ronghao Ni, Mihai Christodorescu, and Limin Jia constraint-based synthesis for generating inputs for PoCs. Some use [35] CPG-based static analysis for generating drivers that in- clude complex code patterns (i.e., not directly call the vulnerable API). Despite substantial improvements, these tools continue to face challenges arising from the highly dynamic nature of JavaScript, the ongoing evolution of the JavaScript language, large and complex dependencies, and their reliance on other sophisticated analysis infrastructures. These challenges are common across existing tools and stem from the fundamental limitations of the underlying tech- niques, rather than from limitations of individual implementation choices. We summarize these challenges in Table 1 and explain how each tool partially addresses them. In practice, these limitations often lead to missed vulnerabilities or failures in exploit confir- mation. Although these tools will continue to improve, as long as the same core techniques are employed, progress is likely to be incremental [7,8]. Alternatively, tools may be increasingly special- ized, exploring different trade-off spaces to achieve high efficiency for specific vulnerability classes or to target different application domains (e.g., Mini apps [48,55], React web apps [23], and Electron apps [4, 27]). 3.2 Advantages of LLM Agents Reasoning beyond handcrafted program models. As summarized in Table 1, many challenges faced by rule-based tools are from the need to construct and maintain accurate program models, which is often at odds with scalability and requires substantial manual effort and domain expertise. Moreover, there is a steep increase in the effort required for improving the analysis to cover addi- tional features, once core behaviors have been modeled. In contrast, learning-based approaches, especially large language models in the current era, provide a promising alternative by utilizing their extensive pre-trained knowledge and reasoning abilities to compre- hend complex code patterns, library usages, and dynamic behaviors without requiring exhaustive manual modeling. As a result, they can adapt more readily to evolving programming practices and soft- ware ecosystems, where manually maintaining precise program models becomes increasingly impractical. Oracle-Guided Iterative Reasoning. Another important factor that makes LLM agents well-suited for taint-style vulnerability detection and confirmation tasks inNode.jspackages is that LLM agents can iteratively refine their answers based on feedback from previous unsuccessful attempts. In contrast, traditional program-analysis techniques typically perform one-shot reasoning over a fixed pro- gram representation. While iterative counter-example guided ab- straction refinement (CEGAR) methods [13] have been applied to domains such as model checking, custom algorithm design and significant engineering effort is needed for a specific tool to benefit from CEGAR. Moreover, it is easy to design and implement a test- ing oracle for taint-style vulnerabilities by observing side effects, commonly used in previous work [9,28,35]. LLM agents can then generate candidate inputs, execute them against the target package, and refine their reasoning based on observed outcomes. Favorable code size distribution in the npm ecosystem. Despite the strong reasoning capabilities of large language models, current LLMs remain constrained by finite context window sizes, which 10 2 10 3 10 4 10 5 10 6 Token Count (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative Probability 90% = 32,448 25% 50% 75% 90% Code Injection OS Command Injection Path Traversal Prototype Pollution Total (All Types) (a) SecBench.js & VulcaN datasets 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 7 Token Count (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative Probability 90% = 147,025 25% 50% 75% 90% All Packages Total (All Types) (b) Recently crawled npm packages (17,151 pack- ages) Figure 2: Cumulative distribution function (CDF) of token counts using the gpt-5-mini tokenizer. Blue dashed lines mark the 90th percentile for the combined datasets: 32,448 tokens for SecBench.js & VulcaN and 147,025 tokens for re- cently crawled npm packages. Only JavaScript files (with ex- tensions .js, .jsx, .mjs, and .cjs) are included in the count. We exclude TypeScript files because most published npm pack- ages distribute transpiled JavaScript artifacts for execution, and our analysis focuses on code that is directly executed in production. limit the amount of code that can be processed effectively in a single pass [11,17,51]. However, when considering the natural distribu- tion of real-world npm packages, we observe that in widely used benchmarks and empirical datasets, most npm packages are suffi- ciently small that they do not stress the context limits of modern LLMs. For instance, as shown in Figure 2, the majority of packages in the VulcaN and SecBench.js datasets contain relatively small amounts of code, with 90th percentile token counts below 32,449. Similarly, in the recently crawled npm packages (which will be discussed in Section 5.1.1), the cumulative distribution over token counts exhibits a comparable trend, where 90th percentile token counts are below 147,026. This measures the total sizes of the code- base, but in reality, an LLM agent will not load the entire codebase into its context window since it can focus on specific files and functions relevant to the vulnerability detection task. However, even considering this, the 90th percentile token counts are still well within the context window sizes of recent LLM models, such as 4 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning <package>@<version> Finder Environment Setup Judge Judge Judge Constraints Constraints Inferencer Testing Oracles Exploit Exploiter Rule-based Program LLM Agent Per-finding validation Iterative exploit attempts LLM Agents "vuln_type": "prototype_pollution", "location": "file": "utils.js", "line": 34 , "description": "User input is merged into an object without filtering special keys, allowing prototype pollution.", "evidence": "Object.assign(, defaults, userInput)" "vuln_type": "path_traversal", "location": "file": "server.js", "line": 87 , "description": "User-controlled path is joined with the base directory without validation.", "evidence": "fs.readFile(path.join(root, req.url))" "vuln_type": "os_command_injection", "location": "file": "index.js", "line": 12 , "description": "User input is concatenated into a shell command.", "evidence": "exec(\"ls \" + name)" Findings "finding": "vuln_type": "path_traversal", "location": "file": "server.js", "line": 87 , "description": "User-controlled path is joined with the base directory without validation.", "evidence": "fs.readFile(path.join(root, req.url))" , "constraint": "Payload must include ../ "finding": "vuln_type": "os_command_injection", "location": "file": "index.js", "line": 12 , "description": "User input is concatenated into a shell command.", "evidence": "exec(\"ls \" + name)" , "constraint": "Payload must be a valid filename and ...” Constraints "findings": [ ... ], "verdicts": [ ... ], "constraints": [ ... ], "exploits": [ "finding": ... , "all_runs": [ "code": "const exec = require(...); ...”, "exit_code": 0 ], "success": true, "successful_run": "code": "const exec = require(...); ...”, "exit_code": 0 ], Final Report Figure 3: Overview of LLMVD.js. OpenAI’s GPT-5 series (400K), Gemini-3 series (1M), and Claude Sonnet 4.5 (200K by default, 1M experimental). This does not imply that large or complex packages are unim- portant; rather, it reflects the natural size distribution of the npm ecosystem, where most packages are relatively small. As a result, LLM-based approaches are well suited for reasoning about a sub- stantial fraction of real-world packages. Even when packages are larger, an LLM agent can iteratively construct task-relevant context through pattern-based search and package-structure understand- ing, without requiring dedicated static/dynamic analysis engines for taint/path derivation. 4 LLMVD.js Design and Implementation In this section, we present the design and implementation details of our proposed multi-stage LLM-based agent framework for detecting and confirming vulnerabilities in Node.js packages. 4.1 System Overview Figure 3 illustrates the architecture ofLLMVD.js, a multi-stage framework for vulnerability detection and exploit confirmation inNode.jspackages. In practice, end-to-end vulnerability confir- mation requires reasoning about candidate locations, construct- ing executable drivers, and validating exploitability with reliable, automated signals. To make this process tractable and auditable, LLMVD.jsdecomposes the pipeline into a small number of stages with distinct objectives: the initial finding stage prioritizes high recall, aiming to identify as many potentially vulnerable locations as possible, while the subsequent stages focus on precision by vali- dating exploitability and eliminating false positives. Accordingly, LLMVD.jsorganizes analysis as a staged workflow in which candi- date findings are first enumerated, then filtered for exploitability, then augmented with exploitation conditions, and finally validated through execution-based verification. The final stage uses auto- mated execution oracles that determine success based on concrete side effects. 4.2 Target Resolution and Execution Context The pipeline accepts either a local project path or an npm identi- fier inpackage@versionformat and operates on a fixed snapshot of the target package. We support four taint-style vulnerability classes that are commonly supported by prior program analysis tools, including command injection, code injection, path traversal, and prototype pollution. Each class is registered with (i) a natural language vulnerability specification, (i) goal-oriented exploitation criteria, and (i) a class-specific execution oracle. Success is deter- mined by vulnerability-class–specific side effects. These uniform success predicates enable automated validation across heteroge- neous vulnerability types. 4.3 Multi-Stage Vulnerability Reasoning Pipeline Candidate Enumeration (Finder). The Finder stage performs hypothesis generation by enumerating candidate vulnerabilities through lightweight codebase exploration, including directory tra- versal, pattern-based search, and source inspection. Each candidate is summarized as a structured hypothesis consisting of a vulner- ability type, precise source location, supporting code evidence, and a set of potentially reachable APIs. This stage intentionally favors over-approximation and prioritizes coverage over precision through prompt design, deferring exploitability assessment and confirmation to subsequent refinement stages. Each candidate is then processed independently through the remainder of the pipeline. Exploitability Filtering (Judge). The Judge stage filters infeasi- ble hypotheses through focused code inspection and lightweight data-flow reasoning, primarily as a reachability check without solv- ing path constraints. Exported APIs are conservatively treated as externally reachable to avoid prematurely discarding viable attack surfaces. For each candidate, the stage produces a structured verdict consisting of a binary exploitability label and a concise justification. 5 Ronghao Ni, Mihai Christodorescu, and Limin Jia Only candidate findings deemed potentially exploitable proceed to constraint inference, thereby eliminating false positives early and reducing unnecessary exploration in later stages. Constraint Inference (Constraints Inferencer). Given a validated hypothesis, the Constraints Inferencer stage derives a compact set of actionable exploitation conditions, including likely entry points, required parameters, payload structure, and relevant bypass con- siderations. These constraints summarize the minimal conditions necessary to propagate attacker-controlled input to the vulnera- ble sink and serve as an explicit interface between exploitability reasoning and exploit synthesis. By representing exploitation conditions explicitly as structured constraints, this stage provides a clear, structured interface for subsequent exploit synthesis. Execution-Coupled Exploit Synthesis (Exploiter). The Exploiter stage performs execution-coupled synthesis by generating exploits that instantiate the inferred constraints and executing them within the target package environment. Each attempt imports the vul- nerable module, executes the payload underNode.js, and records structured results together with full stdout and stderr traces. Exploit attempts are bounded by iteration limits, and all failed attempts are retained for post hoc analysis. Successful executions are immedi- ately validated by the class-specific oracle (discussed in Section 4.4) to provide confirmation of exploitability. 4.4 Oracle-Guided Execution and Automatic Probing A central design choice inLLMVD.jsis the use of oracle-guided exploit confirmation to avoid reliance on model self-reporting and manual inspection. For each vulnerability class, we define an ex- ecution oracle that evaluates concrete side effects produced by exploit attempts. The execution harness augments each payload with vulnerability-specific probing logic. For example, code injec- tion exploits must trigger a predefined marker function; prototype pollution exploits are validated by probing polluted object prop- erties; and path traversal exploits must read and print a prepared sentinel file. For OS command injection, sentinel artifacts are re- moved before each attempt to prevent contamination across runs. These class-specific probes provide uniform, automated success predicates and enable execution-driven refinement of exploit hy- potheses without explicit symbolic constraint solving. 4.5 Tooling and Coordination Infrastructure We organize the supporting toolset into three categories aligned with the stages of vulnerability reasoning. First, exploration and inspection tools enable rapid understanding of package structure and relevant logic through directory navigation, source reading, and pattern-based search. These tools are shared across all reason- ing stages. Second, execution and environment interaction tools are restricted to the exploit synthesis stage and provide controlled Node.jsand shell execution, vulnerability-specific harness integra- tion, auxiliary side-effect checks, and optional background process management for long-running services such as web server appli- cations. Finally, structured reporting and coordination utilities en- force typed submissions for hypotheses, verdicts, constraints, and exploit results, enabling stage-level auditing, failure attribution, and systematic analysis of intermediate reasoning behavior. 4.6 Implementation Each conceptual stage described above is implemented as a dedi- cated LLM agent. Each agent operates with an isolated context and is responsible for a specific task within the pipeline. LLM Model Selection. Since our evaluation involves vulnerability detection and confirmation on unrevealed or unpublished vulnera- bilities, we use APIs that have a non-training policy to minimize the risk of data leakage and the possibility of further training LLMs on unrevealed vulnerabilities. To balance performance and cost, we select OpenAI’s GPT-5-mini model (gpt-5-mini-2025-08-07) as our LLM backbone. The model is accessed via OpenAI’s API platform. Agent Framework. We implement our multi-stage agent frame- work based on LangChain 1 , a popular framework for developing LLM-powered applications. LangChain provides modular compo- nents for building complex agent workflows. We use a recursion limit of 54 for the agents to prevent infinite or excessive loops during tool usage. Considering the non-determinism ofLLMVD.js, we allow up to three attempts when no successful exploit is gener- ated. Here, success is defined as observable side effects detected by automated oracles, but not by manual verification. We adopt a custom multi-stage architecture rather than ex- isting open-source frameworks such as OpenHands [50] and SWE-agent [53], to provide explicit stage separation, enabling fine- grained auditing, debugging, and the integration of domain-specific verification components. While these frameworks are effective for general software engineering tasks, they lack native support for stage-level logging and interfaces for vulnerability validation, exploit execution, and oracle-based verification required for our analysis.LLMVD.jsenables our study of LLM agent behavior for npm vulnerability detection and exploit generation in a transparent and controllable environment. Prompt Design. We create custom prompts for each agent stage to guide the LLM’s reasoning and tool usage. The prompts include clear instructions, the exploitation goal, and an output structure formatted in JSON that contains the expected information for each stage. We refine the prompts iteratively based on initial experiments to improve agent performance, while ensuring no information leak- age related to the evaluated packages. Full prompt templates are provided in Appendix C. 5 Evaluation We evaluateLLMVD.json a variety of datasets to answer the fol- lowing research questions: • RQ1: How effective isLLMVD.jsin detecting and confirming vulnerabilities in Node.js packages? • RQ2: What is the cost of using LLMVD.js? •RQ3: What are the limitations and failure modes ofLLMVD.js, and how can they inform future improvements? 1 https://python.langchain.com/en/latest/index.html 6 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning Table 2: Overview of the VulcaN, SecBench.js, NodeMedic, and the newly crawled Wild datasets. “Raw” refers to the initial number of vulnerable instances per dataset as used in previous work (VulcaN and SecBench.js) or as received (NodeMedic and Wild). “Valid” denotes packages still available on the npm registry at collection time. “Used” is the sampled subset. “Total” is the count of valid packages and sampled packages. “Dist. (%)” indicates the distribution of each vulnerability type across the combined datasets. Vulnerability Type CWE VulcaNSecBench.js NodeMedicWild Total Dist. (%) Raw Valid Raw Valid Raw Used Raw Used Path TraversalCWE-225316115600916522421.62% Command InjectionCWE-78665882781,0221301,0046533131.95% Code InjectionCWE-94222121212281294526523622.78% Prototype PollutionCWE-13216762120118001046524523.65% Total1601443843731,2502591,6512601,036100.00% 5.1 Experiment Setup 5.1.1 Datasets. Our evaluation utilizes four types of datasets: 1) public benchmarks commonly used in vulnerability detection re- search, 2) a private dataset of real-worldNode.jspackages with known vulnerable data paths but no associated CVEs or public exploits, 3) a transformed dataset derived from public benchmarks to assess generalizability, and 4) a crawled dataset of recently re- leasedNode.jspackages from the npm registry to further evaluate performance on unseen data. Public benchmarks. Following common practice, we use two widely recognized public benchmarks: VulcaN [7] and SecBench.js [6]. These datasets collected vulnerableNode.js packages from the npm registry based on reports from GitHub Advisory, Snyk, Huntr.dev, and the CVE database. For a fair com- parison with past work, we select four vulnerability types that have been studied previously: path traversal (CWE-22), OS command injection (CWE-78), code injection (CWE-94), and prototype pollu- tion (CWE-1321). We use the same set of packages evaluated in the most recent prior work (Explode.js[35]), excluding packages that are no longer available on the npm registry. Private dataset. We obtained access to the NodeMedic-FINE pri- vate dataset [9], which contains real-worldNode.jspackages with vulnerable data paths but without associated CVEs or public ex- ploits. Although all packages contain vulnerable code paths, no CVEs or other security advisories have been assigned for reasons like low download count or the developers’ assumption that the parent package should sanitize the input before calling the vulnera- ble API. This dataset can only be used to evaluate our framework on code injection and OS command injection vulnerabilities, the only types supported by NodeMedic-FINE. Considering cost and time constraints, we randomly sample 129 packages with code-injection vulnerabilities and 130 with command-injection vulnerabilities for evaluation. These sample sizes were chosen based on the average number of vulnerable instances per vulnerability type in the VulcaN and SecBench.js datasets after filtering. This dataset is especially useful for assessing our framework’s capability to identify vulner- abilities that are not publicly documented, thus creating a more realistic scenario for vulnerability detection, particularly regarding the issues of LLMs memorizing codes and exploits instead of en- gaging in genuine reasoning. In this work, we refer to this dataset as NodeMedic. Transformed dataset. Since the public benchmarks used in this work were released before the knowledge cutoff date of the LLM we chose (GPT-5-mini: May 31, 2024), there is a risk that memorization in the LLM affects the performance and thus it is crucial to evaluate how well our framework generalizes to unseen data. To achieve this, we create a transformed dataset by selecting up to 20 vulnerable instances per CWE from each public dataset, and applying code transformations such as renaming variables and functions, remov- ing comments, and changing formatting. Package names, versions, and links in manifest files are also anonymized. The aim is to gen- erate code that maintains the original semantics and vulnerabilities while being sufficiently different from the LLMs’ training data, thus reducing the chances of memorization. For transforming JavaScript code, we useterser[3], a toolkit for mangling and compressing JavaScript. Crawled dataset. To further evaluate our framework on unseen data, we crawled recently releasedNode.jspackages from the npm registry that were published in December 2025. We consider only newly released or updated packages and collected 17,151 packages during this period. We designed regular expressions to identify potential vulnerable code patterns. The full regex set is provided in Appendix A. For example, we use (?:eval|Function) *\(to identify potential code-injection vulnerabilities. Considering cost and time constraints, we randomly sample 65 packages per vul- nerability type that were flagged by the regex patterns. To ensure diversity, for each vulnerability class we discretize three structural metrics (code size, dependency count, and number of files) into coarse buckets and stratify packages by the resulting bucket combi- nations. When applicable, we additionally ensure that both minified and non-minified artifacts are represented in the sample. The de- tailed stratified sampling procedure is described in Appendix B. Table 2 provides an overview of the datasets used in our evalua- tion, including vulnerability counts from VulcaN, SecBench.js, and NodeMedic. 7 Ronghao Ni, Mihai Christodorescu, and Limin Jia Table 3: Performance comparison ofLLMVD.jsagainst state-of-the-art tools on standard benchmarks. “NM-FINE” = NodeMedic- FINE. “Det.” = detected, “Expl.” = exploited, “Val.” = valid.Explode.jsis evaluated in two modes: “File” and “Pkg”. The total number of packages and the number of exploits for each tool, both by dataset and overall, are in bold for easier comparison. DatasetVulnerability Type Total FASTNM-FINEExplode.jsLLMVD.js FilePkg Det. Expl. Det. Expl. Det. Expl. Det. Expl. Det. Expl. Val. SecBench.js Path Traversal1561056--88795149155155149 Command Injection7865603731564111777776 Code Injection2182517431202018 Prototype Pollution11800--53484211311389 Total373178684232204 1725953365365 332 VulcaN Path Traversal310--2111333 Command Injection5846381593116103545348 Code Injection211354110330141412 Prototype Pollution6200--333164585538 Total144604319107651208129125101 Overall Total517238 1116142280 2237961494490 433 5.1.2 Baseline tools. We compareLLMVD.jswith three state-of- the-art program-analysis tools forNode.jspackage vulnerability detection: 1) NodeMedic-FINE [9], 2)Explode.js[35], and 3) FAST [28]. Since our work is the first to utilize LLMs for the complete pipeline of vulnerability detection and confirmation (including PoC generation) inNode.jspackages, we include one LLM-based baseline that is not directly comparable to our method: PoCGen [47]. PoCGen does not perform vulnerability detection; it generates PoCs based on existing CVE reports by integrating program analysis techniques with LLM reasoning. We aim to assess whether this combination is necessary or if an LLM-centric, tool-augmented workflow can effectively detect and confirm vulnerabilities. We configure PoCGen to use the same backend LLM model (GPT-5-mini) asLLMVD.jsto ensure a fair comparison. In addition, we run PoCGen on each package up to three times upon failure, matching the maximum number of attempts allowed for LLMVD.js. 5.1.3 PoC Validation. Even though bothLLMVD.jsand the base- line method PoCGen include mechanisms to validate generated PoCs and eliminate false positives, situations may still arise in which the generated PoCs are invalid. In particular, LLM-generated PoCs may trigger the desired side effects while failing to truly ex- ploit the intended vulnerability. To address this issue, we apply an additional manual validation step to all PoCs generated by both LLMVD.jsand PoCGen in order to ensure their accuracy. Con- cretely, among the PoCs that pass automated validation, we further filter out those that exhibit the following behaviors (which we mark as false positives or FPs): FP1The PoC introduces a new vulnerability to the runtime en- vironment instead of exploiting an existing one, such as re- defining a built-in function to create specific side effects. We mainly observe this behavior in packages with vulnerabili- ties that exist only on certain operating systems or relies on certain dependencies, and the LLM attempts to emulate such environments by replacing key built-in functions with stubs. FP2The PoC assumes that certain files with specific names exist, where the file names either match the payload or contain the payload. Or, the PoC modifies the environment variables. This is too strong of an assumption to make in practice. This occurs when the package checks for the existence of certain files to decide whether to execute specific code paths. FP3The PoC does not use any public APIs of the package and instead relies on internal code paths, dependencies, test code, or example scripts. FP4For prototype pollution vulnerabilities, the PoC directly usesObjectorObject.prototypeas part of the arguments passed to the vulnerable package APIs, which is unrealistic in practice because an attacker typically cannot supply these built-in objects as inputs. FP5External tools like web browsers are necessary to trigger the vulnerability. The PoC may simulate this process by directly calling internal functions that these external tools would typically invoke. For a rigorous evaluation of our framework, we classify such PoCs as invalid. 5.2 RQ1: Effectiveness 5.2.1 Comparison with Traditional Program Analysis Tools. Ta- ble 3 shows the performance comparison ofLLMVD.jsagainst three leading program-analysis tools (NodeMedic-FINE, FAST, and Explode.js) on the SecBench.js and VulcaN datasets. It’s impor- tant to note thatExplode.jsoperates in two modes: "File" mode, where the tool analyzes a specific file, and "Pkg" mode, where the tool examines the entire package without prior knowledge of which file contains the vulnerability. We include both modes in our comparison because "File" mode is used in the originalExplode.js paper [35], while "Pkg" mode is more realistic for comprehensive vulnerability detection. 8 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning 37280 Explode.js LLMVD.js (a) Path Traversal 49 18 35 22 NodeMedic-FINE Explode.js LLMVD.js (b) Command Injection 1 3 0 26 0 3 1 NodeMedic-FINE Explode.js LLMVD.js (c) Code Injection 257354 Explode.js LLMVD.js (d) Prototype Pollution Figure 4: Venn diagram overlaps for vulnerability types. Table 4: Performance comparison ofLLMVD.jsagainst PoCGen on 299 overlapping SecBench.js packages. “Expl.” = exploited, “Val.” = valid exploit, “Avg Cost” = average LLM API cost per package. Vulnerability Type Total PoCGenLLMVD.js Expl. Val. Avg Cost Expl. Val. Avg Cost Path Traversal117112108$0.089116112$0.050 Command Injection676362$0.1246766$0.068 Code Injection12109$0.1721211$0.099 Prototype Pollution1039374$0.0799979$0.135 Total299278253$0.097294268$0.085 LLMVD.jsdemonstrates significant advantages over all base- lines in both datasets across all types of vulnerabilities. Notably, LLMVD.jssuccessfully detects and generates valid exploits for 433 out of 517 vulnerable packages in both datasets, achieving an over- all confirmation rate of 83.75%. In contrast, the best-performing baseline, which is not realistic in an end-to-end detection and con- firmation setting,Explode.jsin "File" mode, manages to confirm only 223 vulnerabilities, resulting in a confirmation rate of 43.13%. Examining the distribution of successful exploit generations by various vulnerability types, Figure 4 shows that whileLLMVD.js significantly outperforms the baselines, there are still packages where the baselines succeed whileLLMVD.jsdoes not. This sug- gests that rule-based tools can offer complementary strengths in specific situations, which we discuss further in Section 5.4. 5.2.2 Comparison with PoCGen. As a comparison with a prior LLM–program analysis hybrid method, we investigate whether a carefully designed pipeline with program-analysis components can still improve LLM reasoning in terms of both performance and cost, given the rapid advancement of LLMs. This comparison with PoCGen [47] is favorable to PoCGen. PoCGen is provided with a CVE report when generating exploits, whereasLLMVD.js operates as an end-to-end framework that detects and confirms vulnerabilities without being provided any prior knowledge of the target packages beyond their source code. Nevertheless, PoCGen is the closest available LLM-based work that targets a partially overlapping problem setting. For a fair comparison, we only evaluate the overlapping packages from the PoCGen [47] and our SecBench.js [6] datasets. We also removed any packages whose vulnerability reports were deleted at the time of our evaluation, as PoCGen cannot initiate the pipeline without the report. This results in a total of 299 packages. Table 4 presents the comparison results.LLMVD.jsoutperforms PoCGen in terms of valid exploit generation across all vulnerability types and incurs lower LLM API costs in three of the four vulnerability types and achieves lower overall cost. We investigate why the LLM-program analysis hybrid design in PoCGen does not result in better performance or, at the very least, lower costs. We summarize our observations in the following points: (1) CodeQL AST/locations weren’t converted to compact facts, so the LLM still tried to resolve references itself. (2) LLMs can infer taint flows and definitions directly from raw code, so verbose CodeQL snippets offered little additional, non-redundant signal. (3) when a refinement fails, the refiner generates multiple slightly different prompt variants and sends each to the model, so one failure becomes many near-duplicate LLM requests, increasing API calls and token usage; (4) single prompts often embed overlapping sections (examples, descriptions, snippets) that inflate tokens per call beyond a concise agent prompt. These emphasize the need to better design program analysis components that can effectively assist LLM in reasoning for this task. We discuss potential future directions based on the failure modes ofLLMVD.jsthat we observed in Section 5.4. 5.2.3 Transformed dataset. We compared the number of packages thatLLMVD.jscould successfully exploit before and after trans- formation on the two sampled public benchmarks (VulcaN and SecBench.js), counting only exploits with manually verified valid PoCs. Among the 143 sampled packages,LLMVD.jssuccessfully 9 Ronghao Ni, Mihai Christodorescu, and Limin Jia Table 5: Performance comparison ofLLMVD.jsagainst state-of-the-art tools on recently releasedNode.jspackages from the npm registry and the private NodeMedic dataset. “NM-FINE” = NodeMedic-FINE, “Det.” = detected, “Expl.” = exploited, “Val.” = valid. “-” indicates that reporting is not applicable for the NodeMedic dataset. The total number of packages and the number of exploits for each tool, both by dataset and overall, are in bold for easier comparison. DatasetVulnerability Type Total FASTNM-FINE Explode.jsLLMVD.js Det. Expl. Det. Expl. Det. Expl. Det. Expl. Val. NodeMedic Command Injection130847813071105129128120 Code Injection129782912956229128128124 Total259162 107259 1273214257256 244 Wild Path Traversal6500--0044376 Command Injection65320000282617 Code Injection65100000201712 Prototype Pollution6500--002041 Total26042000011284 36 Overall Total519166 109259 1273214369340 280 generated valid PoCs for 108 packages on the original (untrans- formed) datasets. On the transformed dataset,LLMVD.jsgenerated valid PoCs for 107 packages, missing one previously successful package (lodash@4.17.15) and yielding no additional successful exploits. A detailed case study of this package is provided in Sec- tion 5.4. This result indicates thatLLMVD.jsgeneralizes well to unseen data that are syntactically different from the LLM train- ing data, demonstrating robustness against potential memorization effects. 5.2.4 Private NodeMedic dataset. Table 5 shows a comparison of LLMVD.jswith state-of-the-art tools on the private NodeMedic dataset [9] (Section 5.1.1).LLMVD.jsgenerated valid PoCs for 244 out of a total of 259 vulnerable packages, achieving a valid PoC generation rate of 94.2%. In contrast, NodeMedic-FINE produced working PoCs for only 127 (49.0%). Despite the significant perfor- mance gap,LLMVD.jsmissed 5 vulnerable packages (3 Code Injec- tion and 2 Command Injection) that were successfully exploited by NodeMedic-FINE (more in Section 5.4.3). 5.2.5 Crawled npm packages in the wild. This dataset includes ran- domly sampled 65 packages per vulnerability type (path traversal, OS command injection, code injection, and prototype pollution) that were flagged by our regex patterns, resulting in a total of 260 packages (Section 5.1.1). Among the rule-based program analysis tools, only FAST detected 4 vulnerable packages (3 command in- jection and 1 code injection) and successfully exploited 2 of them (2 command injection). NodeMedic-FINE andExplode.jsdid not detect any vulnerabilities in these packages. In contrast,LLMVD.jsdetected 112 packages that it identified as potentially vulnerable and successfully exploited 84 of them to produce the required side effects. Among the 84 exploited packages, 36 generated proofs of concept (PoCs) that were deemed valid after manual inspection. The detailed results are presented in Table 5. We analyze the non-validated cases by categorizing them into two groups: (1) environment-dependent exploits that require spe- cific system configurations or dependencies, where the LLM at- tempts to emulate the environment by introducing proxies (e.g., stubs or redefined built-ins); and (2) executions through internal code paths such as tests or examples that are not part of the public API. Our manual validation adopts a conservative policy that ex- cludes these categories, which likely underestimates the number of true vulnerabilities. We are currently evaluating cases where the ex- ploits were not deemed valid to determine whether they correspond to real vulnerabilities. These results highlight a limitation of current LLM-based agents: while they can synthesize executable exploits, they often rely on overly permissive assumptions about entry points and execution conditions. Without explicit guidance, the agent may use internal code paths instead of public APIs or emulate missing environments beyond the intended exploit boundary. These issues point to the need for prompt tuning to clearly define the detection boundary, including public APIs and environment and use-case assumptions. We have reported all 36 validated vulnerabilities to the respective package maintainers and are in the process of responsible disclosure. So far, we have received acknowledgments from 3 maintainers. Table 6: Summary of file-level unmatched rates by dataset (all vulnerability types combined) for Finder and Judge stages. MetricSecBench.jsVulcaNAll Finder Findings Unmatched22.94%28.80%24.77% Judge Findings Unmatched20.89%29.15%23.32% Finder GT Unmatched4.29%10.60%6.11% Judge GT Unmatched4.83%14.57%7.63% 5.2.6 Fine-Grained Analysis of Detection Results.LLMVD.jsmay report multiple findings per package, and some reports do not match any benchmark-labeled vulnerable file. Because benchmark 10 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning 10 2 10 3 10 4 10 5 Package Tokens 0.0 0.1 0.2 0.3 0.4 0.5 LLM Cost (USD) Exploited Not exploited Smoothed median (a) LLM cost vs. package size 10 2 10 1 LLM Cost (USD) Not exploited Exploited Exploited Not exploited (b) Exploit success vs. LLM cost 10 2 10 3 10 4 10 5 Package Tokens Not exploited Exploited Exploited Not exploited (c) Exploit success vs. package size Figure 5: Cost, package size, and exploit success trade-offs. (a) LLM API cost increases with package token count, with the red curve showing the smoothed median. 16 packages with extreme large token counts are omitted for better visualization. (b) Distribution of exploited and non-exploited cases across different LLM cost levels. (c) Distribution of exploited and non- exploited cases across package token counts. labels are not guaranteed to be exhaustive, we do not automatically treat unmatched reports as false positives. Instead, we quantify mismatch behavior with file-level coverage metrics. Here, “file-level” means the benchmark ground truth includes the file location for each vulnerability, andLLMVD.jsoutput includes a predicted vulnerable file path for each finding; we count a match only when these file locations agree. At a high level, we measure mismatch from two perspectives at both Finder and Judge stages: (i) report-side mismatch (captured by the Finder/Judge Findings Unmatched rows in Table 6), i.e., what fraction of tool-reported findings cannot be matched to benchmark-labeled vulnerable files, and (i) ground-truth-side miss rate (captured by the Finder/Judge GT Unmatched rows in Table 6), i.e., what fraction of benchmark vulnerable files are not covered by any reported finding. These metrics are formally defined in Appendix D. We report these four percentages for each dataset in Table 6. The detailed file-level un- matched results by dataset and vulnerability type are presented in Table 8 in Appendix D. Overall, the file-level mismatch rates indicate a favorable tradeoff between over-reporting and missed detections. While LLMVD.jsproduces a non-trivial fraction of unmatched reports (approximately one quarter), the ground-truth miss rate remains low (around 6–8%), suggesting that most benchmark-labeled vul- nerabilities are successfully covered. We emphasize that unmatched findings arise from two sources: incomplete benchmark labeling and the tool’s over-approximation of potential vulnerabilities. 5.3 RQ2: Cost We compute the average LLM API cost incurred byLLMVD.jsacross different vulnerability types. The average cost per package is $0.051 for path traversal, $0.068 for OS command injection, $0.107 for code injection, and $0.136 for prototype pollution, with an overall average cost of $0.089 per package across all samples. When re- stricting to successfully exploited packages,LLMVD.jsspends an average of $0.084 per valid exploit. BecauseLLMVD.jsmay gener- ate multiple exploit candidates for a single package by targeting different vulnerable locations, we additionally report an amortized cost per valid exploit of $0.050, computed by dividing the total LLM cost by the number of valid exploits. These results indicate that LLMVD.jsachieves effective exploit generation with modest and well-controlled LLM usage costs across vulnerability types. We further analyze the relationship between LLM cost and ex- ploit success using the per-package distributions. Figure 5a shows that LLM cost increases with package token count, and that suc- cessful exploits are observed across a wide range of token usage levels. Figure 5b relates exploit outcomes to the incurred LLM cost and shows no clear monotonic relationship between exploit success and LLM cost, as both exploited and non-exploited packages are distributed across similar cost ranges. 5.4 RQ3: Limitations and Failure Modes 5.4.1 Impact of Package Size. A common thought is that larger packages are harder to exploit because they tend to involve longer and more complex code paths, which can make it more difficult to 11 Ronghao Ni, Mihai Christodorescu, and Limin Jia 1 var baseSet = require('./_baseSet'); 1 var baseSet = require("./_baseSet"); 2 2 3 /** . 4 * This method is like `_.set` except that it accepts `c . . ustomizer` which is . 5 * invoked to produce the objects of `path`. If `custom . . izer` returns `undefined` . 6 * path creation is handled by the method instead. The ` . . customizer` is invoked . 7 * with three arguments: (nsValue, key, nsObject). . 8 * . 9 * **Note:** This method mutates `object`. . 10 ... . 11 * _.setWith(object, '[0][1]', 'a', Object); . 12 * // => '0': '1': 'a' . 13 */ . 14 function setWith(object, path, value, customizer) 3 function setWith(e, t, i, n) 15 customizer = typeof customizer == 'function' ? customi . .. zer : undefined; . 16 return object == null ? object : baseSet(object, path, 4 return n = "function" == typeof n ? n : void 0, null = .. value, customizer); . = e ? e : baseSet(e, t, i, n); 17 5 18 6 19 module.exports = setWith; 7 module.exports = setWith; Figure 6: One example of the vulnerable sinks inlodash@4.17.15before (left) and after (right) transformation. In “Before transformation”, some parts of the comments were omitted for brevity and replaced with “...”. identify the relevant data flows and construct a working exploit. Figure 5(c) examines this relationship by plotting exploit outcomes against package token count. The plot shows that successful ex- ploits are common for small and medium-sized packages, while the proportion of non-exploited cases increases as package size grows. 5.4.2 A case study of missed generalization. To better understand the generalization capabilities ofLLMVD.js, we investigate the sin- gle package that was successfully exploited on the original dataset but missed on the transformed dataset:lodash@4.17.15, which contains prototype-pollution vulnerabilities. In detecting the trans- formed version,LLMVD.jsdid not identify any potential vulnera- bilities and therefore stopped at the finder stage. Figure 6 shows a side-by-side comparison of one of the many vulnerable sinks in both the original and transformed code. The transformation in- cluded renaming variables and functions, removing comments, and altering the formatting. In the original package,LLMVD.jsreported eight vulnerabil- ity findings and successfully identified prototype pollution across multiple relevant files (e.g., _baseAssignValue.js, _baseSet.js, and _baseMerge.js). The agent’s reasoning benefited from semantic cues such as informative variable names and developer comments, which helped it interpret code intent and localize vulnerable behaviors. In contrast, on the transformed package,LLMVD.jsproduced zero findings and hit the recursion limit (54 iterations). The logs indicate that the agent spent most of its budget attempting to navigate and interpret the obfuscated codebase, becoming effectively lost among approximately 1,046 JS files without meaningful semantic hints. For example, it repeatedly revisited file-tree listings and performed pattern searches, but failed to form a coherent understanding of the code necessary to confirm vulnerabilities. This case study suggests that certain obfuscation and anonymiza- tion patterns can effectively blind LLM-based agents by removing the semantic cues they rely on for navigation and comprehension, leading to substantially degraded vulnerability detection and con- firmation. A systematic characterization of which transformations cause these failures, and how to mitigate them, is beyond the scope of this paper, and we leave detailed studies to future work. 5.4.3 Vulnerabilities Missed byLLMVD.jsbut Detected by Rule- based Tools. Most of the vulnerable packages thatLLMVD.jsover- looked but rule-based program analysis tools successfully exploited are due to manual validations. The LLM believes it has generated a valid PoC and exits, but the PoCs are manually rejected. We discuss this in Section 5.4.4. Ruling out this part, there are 8 pack- ages in all of SecBench.js, VulcaN and NodeMedic datasets that LLMVD.jsmissed but traditional tools successfully exploited. We manually inspected these packages and found that they mainly fall into two categories: (1) The tool identified the location of the vulnerability but concluded that it had sufficient sanitization or had already been patched, so it did not report it. (2) The tool loaded a large but irrelevant portion of the codebase into the context, which caused confusion and led to missing the vulnerability. Although these issues occur in only 4 out of the 776 evaluated packages, they highlight opportunities to further improve the currentLLMVD.js pipeline. 5.4.4 Invalid PoC Generations and False Positives. Table 7 sum- marizes why some PoCs that pass automated validation are still deemed invalid after manual inspection. Prototype pollution con- tributes the largest number of invalid cases (41 packages), and the dominant failure mode is FP4, where the PoC unrealistically passes ObjectorObject.prototypeas an input to the target API (34/41, 82.9%). For OS command injection, most invalid PoCs fall under FP1 (11/14, 78.6%), where the agent emulates missing environments or dependencies by redefining built-ins or stubbing key functionality, and a small fraction require external tools (FP5). For code injection, invalid cases are split between environment emulation (FP1: 5/8, 62.5%) and reliance on non-public/internal code paths (FP3: 3/8, 12 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning Table 7: Analysis of invalid exploit reasons by vulnerability type. For each vulnerability type, shows the total number of packages with invalid exploits and breakdown by reason with percentages. Invalid ReasonPath Traversal Command Injection Code Injection Prototype Pollution Total Packages (Invalid)614841 FP1: Emulated Environment-11 (78.6%)5 (62.5%)4 (9.8%) FP2: Strong Assumptions1 (16.7%)1 (7.1%)-1 (2.4%) FP3: Non-Public API5 (83.3%)1 (7.1%)3 (37.5%)2 (4.9%) FP4: Direct Object Use---34 (82.9%) FP5: External Tools Needed-1 (7.1%)-- 37.5%). For path traversal, the primary issue is FP3 (5/6, 83.3%), indicating that the agent often constructs PoCs that exercise inter- nal code paths, tests, or examples rather than public-facing APIs. Overall, these invalid generations concentrate in a few recurring, vulnerability-specific failure modes, which motivates future work on adding rule-based checks or LLM-based guardrails to discourage unrealistic assumptions and improve PoC validity. Overall, the evaluations do not suggest that LLMs universally outperform classical tools, but instead motivates rethinking the role of program analysis as a complementary technique, particularly in scenarios where formal guarantees or deep semantic reasoning are required. 6 Threats to Validity Even though we tried to design an evaluation to exclude the mem- orization effect of LLMs as much as possible, there are still some threats to validity that may affect the conclusions drawn from our experiments. First, the code transformations we applied to create the transformed dataset may not be sufficient to completely elim- inate the memorization effect, especially for larger models that may have seen similar code snippets during training. Future work could explore more sophisticated transformation techniques or use entirely synthetic datasets to further mitigate this threat. Second, the NodeMedic dataset has distribution bias compared with wild Node.jspackages since they only include those that NodeMedic- FINE [9] flagged as potentially vulnerable. Therefore, the perfor- mance of our framework on this dataset may not fully reflect its effectiveness in real-world scenarios. Third, our evaluation focuses on specific vulnerability types (i.e., path traversal, OS command injection, code injection, and prototype pollution), which may limit the generalizability of our findings to other types of vulnerabilities. Future work could extend the evaluation to a broader range of vulnerability types to assess the versatility of our framework. 7 Conclusion In this work, we demonstrate the strong capabilities of state-of- the-art LLMs in detecting and confirming vulnerabilities inNode.js packages using LLM-centric, tool-augmented reasoning without dedicated static/dynamic analysis engines for taint/path derivation. As LLMs continue to improve in reasoning and code understanding, the benefits of tightly integrating traditional program analysis tech- niques, as explored in prior work, may quickly diminish. We believe it is therefore promising to evaluate current LLM capabilities from lower-level perspectives and to carefully design program analysis tools that complement LLMs in ways that cannot be easily achieved through model scaling or architectural improvements alone. References [1] [n. d.]. Babel: The JavaScript Compiler. https://babeljs.io. [2] [n. d.]. Esprima: ECMAScript Parsing Infrastructure for Multipurpose Analysis. https://esprima.org/. [3] [n. d.]. Terser: JavaScript mangler and compressor toolkit. https://terser.org/. [4] Mir Masood Ali, Mohammad Ghasemisharif, Chris Kanich, and Jason Polakis. 2024. Rise of inspectron: Automated black-box auditing of cross-platform electron apps. In 33rd USENIX Security Symposium (USENIX Security 24). 775–792. [5]Thanassis Avgerinos, Sang Kil Cha, Alexandre Rebert, Edward J Schwartz, Mav- erick Woo, and David Brumley. 2014. Automatic exploit generation. Commun. ACM 57, 2 (2014), 74–84. [6]Masudul Hasan Masud Bhuiyan, Adithya Srinivas Parthasarathy, Nikos Vasilakis, Michael Pradel, and Cristian-Alexandru Staicu. 2023. SecBench. js: An executable security benchmark suite for server-side JavaScript. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1059–1070. [7]Tiago Brito, Mafalda Ferreira, Miguel Monteiro, Pedro Lopes, Miguel Barros, José Fragoso Santos, and Nuno Santos. 2023. Study of javascript static analysis tools for vulnerability detection in node. js packages. IEEE Transactions on Reliability 72, 4 (2023), 1324–1339. [8] Tiago Brito, Mafalda Ferreira, Miguel Monteiro, Pedro Lopes, Miguel Barros, José Fragoso Santos, and Nuno Santos. 2023. Study of javascript static analysis tools for vulnerability detection in node. js packages. IEEE Transactions on Reliability 72, 4 (2023), 1324–1339. [9]Darion Cassel, Nuno Sabino, Min-Chien Hsu, Ruben Martins, and Limin Jia. 2025. NODEMEDIC-FINE: Automatic Detection and Exploit Synthesis for Node. js Vulnerabilities. In Proceedings of the 2025 Network and Distributed System Security Symposium (NDSS’25). doi, Vol. 10. [10]Darion Cassel, Wai Tuck Wong, and Limin Jia. 2023. Nodemedic: End-to-end analysis of node. js vulnerabilities with provenance graphs. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 1101–1127. [11]Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [12] Brian Chess and Gary McGraw. 2004. Static analysis for security. IEEE security & privacy 2, 6 (2004), 76–79. [13]Edmund Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith. 2003. Counterexample-guided abstraction refinement for symbolic model checking. Journal of the ACM (JACM) 50, 5 (2003), 752–794. [14]Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337–340. [15]Alexandre Decan, Tom Mens, and Eleni Constantinou. 2018. On the impact of security vulnerabilities in the npm package dependency network. In Proceedings of the 15th international conference on mining software repositories. 181–191. [16]Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024.PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24). 847–864. [17]Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineer- ing: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53. [18]Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 13 Ronghao Ni, Mihai Christodorescu, and Limin Jia 1469–1481. [19]Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Ryan Tsang, Najmeh Nazari, Han Wang, Houman Homayoun, et al.2024. Large language models for code analysis: DoLLMsreally do their job?. In 33rd USENIX Security Symposium (USENIX Security 24). 829–846. [20]Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144 (2024). [21] Mafalda Ferreira, Miguel Monteiro, Tiago Brito, Miguel E Coimbra, Nuno Santos, Limin Jia, and José Fragoso Santos. 2024. Efficient static vulnerability analysis for javascript with multiversion dependency graphs. Proceedings of the ACM on Programming Languages 8, PLDI (2024), 417–441. [22] Tarek Gasmi, Ramzi Guesmi, Ines Belhadj, and Jihene Bennaceur. 2025. Bridging ai and software security: A comparative vulnerability assessment of llm agent deployment paradigms. arXiv preprint arXiv:2507.06323 (2025). [23]Zhiyong Guo, Mingqing Kang, VN Venkatakrishnan, Rigel Gjomemo, and Yinzhi Cao. 2024. ReactAppScan: Mining React Application Vulnerabilities via Compo- nent Graph. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 585–599. [24]Md Abdul Hannan, Ronghao Ni, Chi Zhang, Limin Jia, Ravi Mangal, and Co- rina S Pasareanu. 2025. On Selecting Few-Shot Examples for LLM-based Code Vulnerability Detection. arXiv preprint arXiv:2510.27675 (2025). [25]Julius Henke. 2025. AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents. arXiv preprint arXiv:2505.10321 (2025). [26]Hamed Jelodar, Samita Bai, Parisa Hamedi, Hesamodin Mohammadian, Roozbeh Razavi-Far, and Ali Ghorbani. 2025. Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineering. arXiv preprint arXiv:2504.07137 (2025). [27]Zihao Jin, Shuo Chen, Yang Chen, Haixin Duan, Jianjun Chen, and Jianping Wu. 2023. A Security Study about Electron Applications and a Programming Methodology to Tame DOM Functionalities.. In NDSS. [28] Mingqing Kang, Yichao Xu, Song Li, Rigel Gjomemo, Jianwei Hou, VN Venkatakr- ishnan, and Yinzhi Cao. 2023. Scaling javascript abstract interpretation to detect and exploit node. js taint-style vulnerability. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 1059–1076. [29]Hee Yeon Kim, Ji Hoon Kim, Ho Kyun Oh, Beom Jin Lee, Si Woo Mun, Jeong Hoon Shin, and Kyounggon Kim. 2022. DAPP: automatic detection and analysis of prototype pollution vulnerability in Node. js modules. International Journal of Information Security 21, 1 (2022), 1–23. [30]Raula Gaikovina Kula, Daniel M German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. 2018. Do developers update their library dependencies? An empirical study on the impact of security advisories on library migration. Empirical Software Engineering 23, 1 (2018), 384–417. [31] Carl E Landwehr, Alan R Bull, John P McDermott, and William S Choi. 1994. A taxonomy of computer program security flaws. ACM Computing Surveys (CSUR) 26, 3 (1994), 211–254. [32]Tan Khang Le, Saba Alimadadi, and Steven Y Ko. 2024. A study of vulnerabil- ity repair in javascript programs with large language models. In Companion Proceedings of the ACM Web Conference 2024. 666–669. [33]Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu. 2025. SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code. arXiv preprint arXiv:2506.05692 (2025). [34] Jie Lin and David Mohaisen. 2025. From large to mammoth: A comparative evaluation of large language models in vulnerability detection. In Proceedings of the 2025 Network and Distributed System Security Symposium (NDSS). [35]Filipe Marques, Mafalda Ferreira, André Nascimento, Miguel E Coimbra, Nuno Santos, Limin Jia, and José Fragoso Santos. 2025. Automated Exploit Generation for Node. js Packages. Proceedings of the ACM on Programming Languages 9, PLDI (2025), 1341–1366. [36]Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large language model guided protocol fuzzing. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS), Vol. 2024. [37]Yuzhou Nie, Hongwei Li, Chengquan Guo, Ruizhe Jiang, Zhun Wang, Bo Li, Dawn Song, and Wenbo Guo. 2025. VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection. arXiv preprint arXiv:2512.07533 (2025). [38]Yu Nong, Haoran Yang, Long Cheng, Hongxin Hu, and Haipeng Cai. 2025. APPATCH: Automated adaptive prompting large language models forReal- Worldsoftware vulnerability patching. In 34th USENIX Security Symposium (USENIX Security 25). 4481–4500. [39]Christoforos Ntantogian, Panagiotis Bountakas, Dimitris Antonaropoulos, Con- stantinos Patsakis, and Christos Xenakis. 2021. NodeXP: NOde. js server-side JavaScript injection vulnerability DEtection and eXPloitation. Journal of Infor- mation Security and Applications 58 (2021), 102752. [40]Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. 2020. Backstabber’s knife collection: A review of open source software supply chain attacks. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 23–43. [41]Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Commun. ACM 68, 2 (2025), 96–105. [42]Marco Pistoia, Satish Chandra, Stephen J Fink, and Eran Yahav. 2007. A survey of static analysis methods for identifying security vulnerabilities in software systems. IBM systems journal 46, 2 (2007), 265–288. [43]Zhuoyun Qian, Fangtian Zhong, Qin Hu, Yili Jiang, Jiaqi Huang, Mengfei Ren, and Jiguo Yu. 2025. Software Vulnerability Analysis Across Programming Language and Program Representation Landscapes: A Survey. arXiv preprint arXiv:2503.20244 (2025). [44]Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: A selective record-replay and dynamic analysis framework for JavaScript. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 488–498. [45]Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. 2025. Pentestagent: Incorporating llm agents to automated penetration testing. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security. 375–391. [46]Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang. 2025. Llms in software security: A survey of vulnerability detection techniques and insights. Comput. Surveys 58, 5 (2025), 1–35. [47]Deniz Simsek, Aryaz Eghbali, and Michael Pradel. 2025. PoCGen: Generating Proof-of-Concept Exploits for Vulnerabilities in Npm Packages. arXiv preprint arXiv:2506.04962 (2025). [48] Chao Wang, Ronny Ko, Yue Zhang, Yuqing Yang, and Zhiqiang Lin. 2023. Taint- mini: Detecting flow of sensitive data in mini-programs with static taint analysis. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 932–944. [49] Dawei Wang, Geng Zhou, Li Chen, Dan Li, and Yukai Miao. 2024. Prophetfuzz: Fully automated prediction and fuzzing of high-risk option combinations with only documentation via large language model. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 735–749. [50]Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al.2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024). [51]Yonghao Wu, Zheng Li, Jie M Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large language models in fault localisation. arXiv preprint arXiv:2308.15276 (2023). [52] HanXiang Xu, ShenAo Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and HaoYu Wang. 2024. Large language models for cyber se- curity: A systematic literature review. ACM Transactions on Software Engineering and Methodology (2024). [53]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [54] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations. [55]Zidong Zhang, Qinsheng Hou, Lingyun Ying, Wenrui Diao, Yacong Gu, Rui Li, Shanqing Guo, and Haixin Duan. 2024. Minicat: Understanding and detecting cross-page request forgery vulnerabilities in mini-programs. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 525–539. [56]Xiaogang Zhu, Wei Zhou, Qing-Long Han, Wanlun Ma, Sheng Wen, and Yang Xiang. 2025. When software security meets large language models: A survey. IEEE/CAA Journal of Automatica Sinica 12, 2 (2025), 317–334. [57]Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. 2024. Teams of llm agents can exploit zero-day vulnera- bilities. arXiv preprint arXiv:2406.01637 (2024). [58]Markus Zimmermann, Cristian-Alexandru Staicu, Cam Tenny, and Michael Pradel. 2019. Small world with high risks: A study of security threats in the npm ecosystem. In 28th USENIX Security symposium (USENIX security 19). 995–1010. 14 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning Appendix A Regular Expressions for Filtering Crawled npm Packages Here are the regular expressions we used to filter potentially vulner- able packages from the crawled npm packages for each vulnerability type: Code Injection. – (?:eval|Function) *\(: eval/Function usage. Command Injection. – child_process *\. *(?:exec|spawn) : child_process exec/spawn usage. – require\( *['\"]child_process['\"] *\): require (’child_process’). – (?:execSync|spawnSync) *\(: execSync/spawnSync usage. Path Traversal. – *\. *(?:join|resolve) *\([^)]* ( ⌋ ?:req *\. *(?:params|query|body|headers|u ⌋ rl|originalUrl)|process *\. *env) [^)]*\) : path.join/resolve fed by req.* or env. – (?:fs|node:fs) *\. *(?:readFile|readFileSy ⌋ nc|writeFile|writeFileSync|createReadStream|cr ⌋ eateWriteStream|readdir|readdirSync|rm|rmSync| ⌋ unlink|unlinkSync|open|openSync) *\([^)]* ⌋ q *\. *(?:params|query|body|headers|url|orig ⌋ inalUrl) : fs.* sink references req.*. bundle. Prototype Pollution. – *\. *assign *\([^)]* (?:req *\.\ ⌋ s*(?:body|query|params)|JSON *\. *parse *\( ⌋ |qs *\. *parse *\() [^)]*\) : Object.assign fed by req.* or JSON.parse. – = *\[^]*\.\.\.(?:req *\. *(?:body|query|p ⌋ arams)|JSON *\. *parse *\(|qs *\. *parse ⌋ *\()[^]*\: Spread merge with attacker-controlled object. – (?:set|assign|merge|extend|defaultsDeep|deep ⌋ Merge|deepExtend) *\([^)]* (?:req *\. *( ⌋ ?:body|query|params)|JSON *\. *parse *\(|q ⌋ s *\. *parse *\() : Generic deep merge helpers with tainted input. – _ *\. *(?:merge|mergeWith|defaultsDeep|set ⌋ |setWith|update|updateWith) *\(: lodash-style risky helpers. B Sampling Algorithms for Crawled npm Packages We characterize each package using four features: three struc- tural metricsjs_ts_loc(lines of JavaScript or TypeScript code), dependency_count(number of declared dependencies), and js_ts_files(number of JavaScript or TypeScript source files), together with a binary indicator of whether the package contains minified code (by simply checking whether.min.orbundle appears in the filenames of JavaScript or TypeScript files). Our goal is to sample a fixed number of packages per vulnerability (65 in our experiments) such that the distribution over the three structural metrics is approximately uniform, while the proportion of minified packages follows the distribution observed in the crawled dataset, subject to a minimum of one minified package whenever minified artifacts exist. We implement this goal using a stratified sampling procedure: •Metric Bucketing: For each vulnerability class푣and each structural metric푚 푗 ∈ js_ts_loc, dependency_count, js_ts_files, we discretize metric values into푏=5 buck- ets using empirical quantiles. For푘 ∈ 1, . . .,푏−1, we de- fine bucket cutoffs as휃 푗,푘 = quantile 푘/푏 (푚 푗 (푥) | 푥 ∈ 푣), and assign each package푥to a bucket퐵 푗 (푥)according to the interval in which푚 푗 (푥) falls. •Group Definition: Each package is assigned to a joint group 푆(푥)=(퐵 1 (푥),퐵 2 (푥),퐵 3 (푥),푀(푥)), where푀(푥) ∈ minified, plaindenotes the minification status. •Structural Allocation: Ignoring the minification indica- tor, we target an approximately uniform allocation across the structural bucket tuples(퐵 1 ,퐵 2 ,퐵 3 )by iterating these tuples in randomized round robin order and selecting pack- ages to spread samples evenly across different structural configurations. •Minification Allocation: Let퐶 min and퐶 plain denote the counts of minified and plain packages in the crawled dataset for vulnerability푣. We set sampling budgets푁 min and푁 plain proportional to퐶 min and퐶 plain , while enforcing푁 min ≥1 whenever퐶 min > 0. •Selection: Within each structural bucket tuple(퐵 1 ,퐵 2 ,퐵 3 ), we draw packages while respecting the remaining minified and plain budgets. If one category is not available within a bucket, we draw from the other category. If a quota cannot be met due to global exhaustion, we fill the remaining slots from the available category. •Post Check: For each metric푚 푗 and each bucket푡that ap- pears among the sampled packages, if the sample contains at least one plain package in(푚 푗 ,푡)but zero minified pack- ages in(푚 푗 ,푡), and the crawled dataset contains at least one minified package in(푚 푗 ,푡), then we replace one sam- pled plain package from(푚 푗 ,푡)with an unused minified package from(푚 푗 ,푡) if such a minified package exists. C Prompts Templates Here we include the detailed prompt templates used for each stage of the pipeline and for each vulnerability class. The placeholder vari- ables in the templates are dynamically filled during the execution of the pipeline. C.1 Stage 1: Finder Prompt C.1.1 System Prompt. You are an expert security researcher specializing in Node.js vulnerabilities.↩→ 15 Ronghao Ni, Mihai Christodorescu, and Limin Jia Your task is to find instances of <VULN_TYPE> vulnerabilities in the project at: <PROJECT_PATH> ↩→ ↩→ <VULN_DESCRIPTION> WORKFLOW: 1. Start by getting the file tree or listing files to understand the structure↩→ 2. Search for patterns related to <VULN_TYPE> 3. Read suspicious files to analyze the code 4. Identify exact locations (file + line number) of vulnerabilities↩→ 5. Determine which public APIs can reach these vulnerabilities↩→ 6. Call submit_findings(findings=[...]) with structured arguments (NO JSON STRINGS). You can call this multiple times as you discover items. ↩→ ↩→ 7. When you are completely done adding findings, call finish(summary="...optional...") to end the run. ↩→ ↩→ FINDINGS FORMAT (submit_findings arguments): - findings: [ "vuln_type": "<VULN_TYPE>", "file": "relative/path/to/file.js", "line": 42, "description": "Brief description", "evidence": "Code snippet showing the issue",↩→ "reachable_apis": ["api1", "api2"], "confidence": 0.85 ] Be thorough and precise. Focus on actionable evidence but err on the side of inclusion: if a spot looks plausibly exploitable yet you lack full confirmation, include it with a lower confidence score and clearly state any assumptions. When you have submitted all findings, call finish to end the run. ↩→ ↩→ ↩→ ↩→ ↩→ ↩→ C.1.2 User Prompt. Find all <VULN_TYPE> vulnerabilities in the project. Use tools to analyze the code, then submit your findings. ↩→ ↩→ C.2 Stage 2: Judge Prompt C.2.1 System Prompt. You are an expert security code reviewer specializing in Node.js vulnerabilities.↩→ Your task is to validate whether a reported <VULN_TYPE> vulnerability is actually exploitable. ↩→ ↩→ CURRENT DIRECTORY: <PROJECT_PATH> All file paths are relative to this directory. Call get_file_tree() first to see the structure if needed. ↩→ ↩→ ANALYSIS CHECKLIST: 1. Read the code at the reported location 2. Trace data flow to see if user input can reach the vulnerable sink↩→ 3. Check for any input validation or sanitization 4. Determine if the vulnerability is actually exploitable↩→ 5. Submit your verdict with detailed reasoning IMPORTANT - LIBRARY/PACKAGE ATTACK SURFACE: - When analyzing npm packages/libraries, EXPORTED functions (exports.*, module.exports) are the attack surface ↩→ ↩→ - If a vulnerable function is exported (even with no callers in the codebase), it IS reachable by external code ↩→ ↩→ - Focus on: Can user-controlled input reach the sink IF the exported function is called?↩→ - Do NOT search for external callers - the export itself makes it callable↩→ VERDICT FORMAT (submit_verdict arguments): - is_valid: true/false - reason: "Detailed explanation with evidence" - confidence: 0.0-1.0 Call submit_verdict(is_valid=true/false, reason="...", confidence=0.0-1.0). Do NOT wrap the arguments in JSON strings. ↩→ ↩→ Be conservative - only mark as valid if truly exploitable.↩→ C.2.2 User Prompt. Validate this vulnerability finding: Type: <VULN_TYPE> File: <FILE> Line: <LINE> Description: <DESCRIPTION> Evidence: <EVIDENCE> Reachable APIs: <REACHABLE_APIS> Confidence: <CONFIDENCE> Use tools to analyze the code and submit your verdict.↩→ C.3 Stage 3: Constraints Inferencer Prompt C.3.1 System Prompt. You are an expert security researcher analyzing how to exploit vulnerabilities.↩→ 16 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning Your task is to analyze a <VULN_TYPE> vulnerability and determine the constraints for exploitation.↩→ ANALYSIS OBJECTIVES: 1. Identify the entry point (URL, function, API) 2. Determine parameter names and how to provide them 3. Understand required format/structure of the payload↩→ 4. Identify any validation to bypass 5. Define success criteria for the exploit CONSTRAINTS FORMAT (submit_constraints arguments): - constraints: "Detailed natural language description of exploitation requirements" (REQUIRED) ↩→ ↩→ - entry_point: "How to reach the vulnerability" - parameters: ["param1", "param2"] - payload_format: "Required format for the payload" Be specific and actionable - the exploit agent will use this.↩→ C.3.2 User Prompt. Generate exploitation constraints for: Type: <VULN_TYPE> File: <FILE> Line: <LINE> Description: <DESCRIPTION> Evidence: <EVIDENCE> Analyze the code and submit detailed constraints. C.4 Stage 4: Exploiter Prompt C.4.1 System Prompt. You are an expert security researcher crafting exploits for <VULN_TYPE>.↩→ Your task is to create a working exploit based on the provided constraints.↩→ EXPLOIT GOAL: <EXPLOIT_INSTRUCTIONS_FROM_VULN_DEFINITION> Dependency setup: Install any missing dependencies in the package directory before running your exploit (use run_shell_command if you need npm/yarn). ↩→ ↩→ ↩→ EXPLOIT WORKFLOW: 1. Read the vulnerable code to understand it 2. Craft exploit code that imports the module: const mod = require('./index');↩→ 3. Execute the exploit using execute_javascript - Only if you truly need a background service (e.g., start a server), use start_persistent_process, inspect with check_persistent_process, and always call kill_persistent_process when finished. ↩→ ↩→ ↩→ ↩→ 4. Verify success (check file creation, output, etc.)↩→ 5. If import fails, check the exact filename and try again↩→ 6. Submit your result with submit_exploit_result(success=..., exploit_code="...", execution_output="...", explanation="...") ↩→ ↩→ ↩→ Be creative and persistent. Learn from failures. C.4.2 User Prompt. Create and execute an exploit for: Type: <VULN_TYPE> File: <FILE> Line: <LINE> Constraints: <CONSTRAINTS_TEXT> Create a working exploit and submit the result. C.5 Stage 4 Dynamic Goal Templates (Per Vulnerability Type) C.5.1 os_command_injection. Craft a payload that injects OS commands (e.g., using ; || && or backticks).↩→ The exploit should create the file /tmp/os_cmd_success to prove arbitrary command execution. ↩→ ↩→ Example:`; touch /tmp/os_cmd_success` or`$(touch /tmp/os_cmd_success)`↩→ C.5.2 code_injection. Craft a payload that injects JavaScript code into eval(), Function(), vm.runInNewContext(),↩→ or similar APIs. The exploit harness defines global.CTF = () => console.log('GLOBAL.CTF HIT'). ↩→ ↩→ Your malicious payload must call global.CTF() so that the marker appears in STDOUT, proving↩→ arbitrary code execution without relying on filesystem writes.↩→ C.5.3 path_traversal. Craft a payload using ../ sequences to escape the intended directory and read /tmp/path_traversal. ↩→ ↩→ The harness writes the sentinel string 'PATH_TRAVERSAL' into that file ahead of time.↩→ 17 Ronghao Ni, Mihai Christodorescu, and Limin Jia Use the vulnerability to read it and console.log the file contents so STDOUT contains the marker.↩→ C.5.4 prototype_pollution. Goal: trigger prototype pollution so that a newly created object inherits attacker-controlled data. ↩→ ↩→ Use the package's public API as intended (e.g., merge, copy, set, or path-based helpers).↩→ The attacker controls input values such as objects, keys, or paths.↩→ You may use payloads such as: - objects containing prototype-related keys - dotted paths (e.g.,'__proto__.polluted', 'constructor.prototype.polluted')↩→ - arrays of path segments if supported by the API Do NOT use unrealistic techniques such as Object.defineProperty, Object.setPrototypeOf, or ↩→ ↩→ direct writes to Object.prototype. IMPORTANT: - Do NOT try to manually check for success. - The JavaScript run tool automatically probes, and if pollution occurred it will emit↩→ 'PROTO_POLLUTION SUCCESS' on its own. - Do NOT print or hardcode the success token. C.6 Vulnerability Definitions C.6.1 os_command_injection. CWEs: 077, 078 OS Command Injection (CWE-78): The product constructs and executes OS commands↩→ using unsanitized input, allowing attackers to execute arbitrary system commands.↩→ Common sink APIs: exec, execSync, execFile, execFileSync, spawn, spawnSync,↩→ child_process module functions C.6.2 code_injection. CWEs: 094 Code Injection (CWE-94): The product dynamically generates or evaluates code↩→ using untrusted input, allowing attackers to inject and execute arbitrary JavaScript code.↩→ C.6.3 path_traversal. CWEs: 022, 035 Path Traversal (CWE-22): The product uses external input to construct a file or directory↩→ path but fails to properly sanitize special characters (e.g., ../), allowing access to↩→ unintended locations outside the intended directory.↩→ C.6.4 prototype_pollution. CWEs: 1321 Prototype Pollution (CWE-1321): attacker-controlled keys or paths are used in object writes or↩→ merge/copy operations, causing a shared prototype (often Object.prototype) to be modified and↩→ affecting subsequently created objects. Indicators include dynamic property access (obj[key]), merge/copy utilities, recursive assignment ↩→ ↩→ patterns, and path-based setters. The vulnerable code may not explicitly reference'__proto__' or ↩→ ↩→ 'constructor.prototype'. D Full File-Level Matching Results and Metric Definitions The metrics we used for file-level matching are formally defined as follows: let푑denote a dataset (SecBench.js, VulcaN, or their union) and 푠 ∈ Finder, Judge denote a stage. For each(푑,푠): •Let퐹 푑,Finder be the set of findings produced by the Finder stage, and let퐹 푑,Judge be the subset of findings deemed valid by the Judge stage. • Let 퐺 푑 be the set of ground-truth vulnerable files. •A finding푓 ∈ 퐹 푑,푠 is matched if there exists푔 ∈ 퐺 푑 under our file-level matching rule (same package-version pair, vulnerability type, and file). Otherwise, 푓 is unmatched. We report the following four metrics, which correspond directly to Table 6 and Table 8: Finder Findings Unmatched(푑) = |푓 ∈ 퐹 푑,Finder : 푓 unmatched| |퐹 푑,Finder | Judge Findings Unmatched(푑) = |푓 ∈ 퐹 푑,Judge : 푓 unmatched| |퐹 푑,Judge | Finder GT Unmatched(푑) = |푔 ∈ 퐺 푑 :푓 ∈ 퐹 푑,Finder matched to 푔| |퐺 푑 | Judge GT Unmatched(푑) = |푔 ∈ 퐺 푑 :푓 ∈ 퐹 푑,Judge matched to 푔| |퐺 푑 | The first two metrics measure the fraction of tool reports that are unmatched; the latter two measure the fraction of benchmark vulnerable files not covered by any report at each stage. As shown in Table 8, the detailed file-level matching results are broken down by benchmark (SecBench.js and VulcaN) and 18 Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning Table 8: Detailed file-level unmatched results by dataset and vulnerability type for Finder and Judge stages. “Findings Unmatched” reports unmatched-rate on tool findings, and “GT Unmatched” reports unmatched-rate on ground-truth files. DatasetVulnerability Finder Findings Unmatched Judge Findings Unmatched Finder GT Unmatched Judge GT Unmatched SecBench.js CWE-2227/221 (12.22%)25/218 (11.47%)2/156 (1.28%)2/156 (1.28%) SecBench.js CWE-471113/343 (32.94%)92/299 (30.77%)10/118 (8.47%)12/118 (10.17%) SecBench.js CWE-7821/201 (10.45%)18/194 (9.28%)3/78 (3.85%)3/78 (3.85%) SecBench.js CWE-9428/59 (47.46%)25/55 (45.45%)1/21 (4.76%)1/21 (4.76%) SecBench.js All189/824 (22.94%)160/766 (20.89%)16/373 (4.29%)18/373 (4.83%) VulcaNCWE-224/8 (50.00%)4/8 (50.00%)0/3 (0.00%)0/3 (0.00%) VulcaNCWE-47134/146 (23.29%)26/130 (20.00%)6/63 (9.52%)6/63 (9.52%) VulcaNCWE-7840/163 (24.54%)35/128 (27.34%)5/63 (7.94%)8/63 (12.70%) VulcaNCWE-9430/58 (51.72%)28/53 (52.83%)5/22 (22.73%)8/22 (36.36%) VulcaNAll108/375 (28.80%)93/319 (29.15%)16/151 (10.60%)22/151 (14.57%) AllCWE-2231/229 (13.54%)29/226 (12.83%)2/159 (1.26%)2/159 (1.26%) AllCWE-471147/489 (30.06%)118/429 (27.51%)16/181 (8.84%)18/181 (9.94%) AllCWE-7861/364 (16.76%)53/322 (16.46%)8/141 (5.67%)11/141 (7.80%) AllCWE-9458/117 (49.57%)53/108 (49.07%)6/43 (13.95%)9/43 (20.93%) AllAll297/1199 (24.77%)253/1085 (23.32%)32/524 (6.11%)40/524 (7.63%) vulnerability type, including both unmatched-finding rates and unmatched-ground-truth rates. 19