Paper deep dive
Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code
Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 9/11/2026, 3:56:28 AM
Summary
This paper introduces the Static-Pass Dynamic-Fail (SPDF) phenomenon, where code passes static security analysis but fails dynamic exploit verification. The authors propose a three-stage agentic pipeline combining Bandit/Semgrep static scanning, LLM-driven CWE reasoning, and Docker-based dynamic exploit verification. Evaluating 1,355 Python samples from SecurityEval, RedCode, and CyberNative datasets, they found that 14.53% of statically clean samples had confirmed or partially confirmed vulnerabilities at runtime. The study highlights that static analysis alone is insufficient for security assurance, particularly for LLM-generated code, and provides a reproducible pipeline and dataset for runtime vulnerability assessment.
Entities (12)
Relation Signals (14)
Bandit â isusedin â Static Analysis Stage
confidence 99% · We employ two complementary static-analysis tools: Bandit... and Semgrep.
Semgrep â isusedin â Static Analysis Stage
confidence 99% · We employ two complementary static-analysis tools: Bandit... and Semgrep.
SPDF â describes â Static-Pass Dynamic-Fail
confidence 98% · This paper introduces the Static-Pass Dynamic-Fail (SPDF) phenomenon... vulnerabilities that produce no findings under the evaluated composite Bandit-Semgrep static analysis gate but are subsequently demonstrated through runtime exploit verification.
Docker â enables â Dynamic Exploit Verification
confidence 98% · autonomous exploit verification in isolated Docker containers.
LLM â powers â CWE Vulnerability Detection Agent
confidence 98% · LLM-driven Common Weakness Enumeration (CWE) reasoning... Implemented using OpenAIâs gpt-5.4-nano model
RedCode â issourceof â Python Samples
confidence 97% · We evaluate 1,355 Python samples from SecurityEval, RedCode, and CyberNative datasets.
CyberNative â issourceof â Python Samples
confidence 97% · We evaluate 1,355 Python samples from SecurityEval, RedCode, and CyberNative datasets.
SecurityEval â issourceof â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Advances in large language models (LLMs) fuel the quest for scalable methods to assess the security of generated and security-sensitive software. Static analysis is widely adopted as a scalable, reproducible, and inexpensive security gate, but cannot directly observe runtime exploit behaviour. Vulnerabilities dependent on adversarial inputs, execution context, or exploit chaining may evade static checks while remaining exploitable in practice, yet passing static analysis is often treated as evidence of secure behaviour. This paper introduces the Static-Pass Dynamic-Fail (SPDF) phenomenon and a three-stage agentic pipeline combining static scanning, LLM-driven Common Weakness Enumeration (CWE) reasoning, and autonomous exploit verification in isolated Docker containers. We evaluate 1,355 Python samples from SecurityEval, RedCode, and CyberNative datasets. Of the 654 samples producing no findings under the composite Bandit-Semgrep gate, the LLM detection stage identified 394 candidate vulnerabilities across 235 files. Dynamic verification confirmed or partially confirmed exploitability in 95 files, yielding an inclusive pipeline rate of 14.53% (roughly 1 in 7 statically clean samples). This rate represents the proportion of Bandit-Semgrep-clean samples for which the pipeline identified a candidate vulnerability and obtained runtime evidence supporting exploitability. Outcomes varied by dataset: among candidate file--CWE pairs, confirmed exploitability was 33.7% for RedCode, 28.6% for CyberNative, and 5.4% for SecurityEval. Several frequently confirmed classes, including CWE-338 and CWE-916, were flagged by neither Bandit nor Semgrep. These findings indicate that static-analysis success and runtime security are hierarchical layers of software assurance rather than interchangeable measures, and have the potential to reshape how AI-generated and security-sensitive code is evaluated.
Tags
Links
- Source: https://arxiv.org/abs/2609.10762v1
- Canonical: https://arxiv.org/abs/2609.10762v1
Trouble viewing inline? Open PDF directly â
Full Text
57,881 characters extracted from source content.
Expand or collapse full text
Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code Jessica Pourleyli Toronto Metropolitan University Toronto, Canada jessica.pourleyli@torontomu.ca Maitreyee Das Urmi Toronto Metropolitan University Toronto, Canada maitreyee.urmi@torontomu.ca Glaucia Melo Toronto Metropolitan University Toronto, Canada glaucia@torontomu.ca AbstractâAdvances in large language models (LLMs) fuel the quest for scalable methods to assess the security of generated and security-sensitive software. Static analysis is widely adopted as a scalable, reproducible, and inexpensive security gate, but cannot directly observe runtime exploit behaviour. Vulnerabilities depen- dent on adversarial inputs, execution context, or exploit chaining may evade static checks while remaining exploitable in practice, yet passing static analysis is often treated as evidence of secure behaviour. This paper introduces the Static-Pass Dynamic-Fail (SPDF) phenomenon and a three-stage agentic pipeline combining static scanning, LLM-driven Common Weakness Enumeration (CWE) reasoning, and autonomous exploit verification in isolated Docker containers. We evaluate 1,355 Python samples from Secu- rityEval, RedCode, and CyberNative datasets. Of the 654 samples producing no findings under the composite BanditâSemgrep gate, the LLM detection stage identified 394 candidate vulnerabilities across 235 files. Dynamic verification confirmed or partially confirmed exploitability in 95 files, yielding an inclusive pipeline rate of 14.53% (roughly 1 in 7 statically clean samples). This rate represents the proportion of BanditâSemgrep-clean samples for which the pipeline identified a candidate vulnerability and obtained runtime evidence supporting exploitability. Outcomes varied by dataset: among candidate fileâCWE pairs, confirmed exploitability was 33.7% for RedCode, 28.6% for CyberNative, and 5.4% for SecurityEval. Several frequently confirmed classes, including CWE-338 and CWE-916, were flagged by neither Bandit nor Semgrep. These findings indicate that static-analysis success and runtime security are hierarchical layers of software assurance rather than interchangeable measures, and have the potential to reshape how AI-generated and security-sensitive code is evaluated. Index Termsâsoftware security, static analysis, dynamic analy- sis, exploit verification, agentic security evaluation, large language models I. INTRODUCTION The growing adoption of Large Language Models (LLMs) in software engineering has raised important questions regarding the security and trustworthiness of AI-generated code [1]. Modern coding assistants can generate functional programs, reason about implementation details, and autonomously interact with development tools [2], [3], yet vulnerabilities in generated code remain a persistent concern [4], [5]. As a result, a growing body of work has proposed security-focused benchmarks and Jessica Pourleyli and Maitreyee Das Urmi contributed equally and share first authorship. evaluation frameworks, including SecurityEval, AutoSafeCoder, RedCode, and VulnRepairEval, to assess the security of generated software [6]â[9]. Despite these advances, security evaluation pipelines con- tinue to rely heavily on static-analysis tools such as Bandit and CodeQL [6], [7]. Static analysis is scalable, reproducible, and inexpensive, but it reasons about code without executing it and therefore cannot directly observe runtime exploit behaviour [10]. Vulnerabilities that depend on adversarial inputs, execution context, or exploit chaining may therefore evade static analysis while remaining exploitable in practice. This limitation is particularly relevant in modern agentic workflows, where successful static checks are often treated as evidence of secure behaviour. In this paper, we investigate what we term the Static- Pass Dynamic-Fail (SPDF) phenomenon: vulnerabilities that produce no findings under the evaluated composite Bandit- Semgrep static analysis gate but are subsequently demonstrated through runtime exploit verification. More broadly, we view software-security evaluation as a hierarchy of evidence. Static analysis, vulnerability reasoning, and dynamic verification provide progressively stronger evidence about software security, but success at one level does not guarantee success at the next. To study this problem, we design a three-stage agentic security pipeline combining static scanning, LLM-driven CWE reasoning, and autonomous exploit generation within isolated Docker environments. The pipeline first filters programs using Bandit [11] and Semgrep [12]. Surviving statically clean samples are then analyzed by a CWE Vulnerability Detection Agent that retrieves and reasons over authoritative vulnerability knowledge. Finally, a Dynamic Exploit Verification Agent selectively evaluates candidate vulnerabilities through generated exploit harnesses executed in sandboxed Docker environments, providing higher-confidence exploitability assessments while avoiding the cost of exhaustive dynamic testing. To evaluate the prevalence of SPDF behaviour, we analyze 1,355 Python samples drawn from SecurityEval [6], RedCode [8], and the CyberNative Code Vulnerability Security DPO dataset [13]. These datasets contain security-sensitive bench- mark programs and LLM-generated code spanning multiple arXiv:2609.10762v1 [cs.CR] 9 Sep 2026 CWE categories, adversarial coding scenarios, and exploit- oriented tasks. We additionally quantify the practical cost of agentic exploit verification, since computational requirements influence the feasibility of deploying such techniques in real- world security evaluation workflows. The following research questions guide our study: RQ1: To what extent does code that produces no findings under the evaluated static-analysis gate remain dynamically exploitable under runtime testing? We aim to quantify the observed yield of the SPDF pipeline among samples producing no findings under the evaluated static-analysis gate and examine how often LLM-identified candidates can subsequently be demonstrated as exploitable at runtime. RQ2: What categories of vulnerabilities are most likely to evade static analysis while remaining exploitable during execution? We aim to identify which CWE categories are most frequently associated with SPDF behaviour and, therefore, most likely to require dynamic verification. RQ3: What is the cost of the agentic dynamic- exploitability pipeline in terms of latency and token usage? This question evaluates the practical feasibility of the proposed approach by quantifying its computational cost. The main contributions of this work are as follows: 1)We introduce and operationalize the Static-Pass Dynamic- Fail (SPDF) phenomenon and the SPDF Rate (SPDFR), providing an empirical characterization of its prevalence and the vulnerability categories most likely to evade static detection. 2) An empirical characterization of the computational cost of agentic exploit verification, including token consump- tion, latency, reasoning iterations, and exploit-generation overhead across 394 candidate vulnerabilities. 3)A curated collection of SPDF-positive samples identi- fied from existing security-oriented benchmarks, together with evaluation artifacts, exploit evidence, and exploit- verification code used during the verification process to support reproducible research on runtime vulnerability assessment for LLM-generated software. 4)A reproducible and publicly available multi-stage pipeline combining static analysis, LLM-driven CWE detection, and Docker-based exploit verification [14]. By focusing on the gap between static security guarantees and observed runtime exploitability, this work contributes toward a more execution-aware framework for evaluating the security of AI-generated and security-sensitive software systems. We emphasize that our corpus comprises security-sensitive bench- mark code and LLM-generated vulnerable samples, not code produced by a coding assistant operating within an interactive development workflow. Our results, therefore, characterize Static Application Security Testing (SAST) false-negative exposure on such corpora; extending them to assistant-in-the- loop pipelines is a hypothesis our findings motivate but do not establish. I. RELATED WORK A. Security of LLM-Generated Code LLM-based coding assistants can generate code that appears useful and may functionally satisfy program needs, yet still contain security weaknesses. Prior evaluations of GitHub Copi- lot and ChatGPT show that generated programs may contain vulnerabilities associated with known CWE categories, even when the generated code is syntactically valid or functionally plausible [4], [5]. These findings establish that functional code generation and secure code generation are not necessarily equivalent evaluation goals that should be treated equally during the code generation process. This distinction has become increasingly important as LLMs are adopted across software engineering tasks such as code generation, debugging, repair, and maintenance [1]. Security evaluation, therefore, requires assessing whether generated code is robust against vulnerability patterns, not only whether it satisfies the surface-level programming task. B. Security Benchmarks and Evaluation Frameworks for AI- Generated Code Security-focused benchmarks have been introduced to evalu- ate generated code beyond functional correctness. SecurityEval provides Python prompts mapped to CWE categories and demonstrates their use for evaluating code-generation models using both manual inspection and automated static analysis tools [6]. RedCode expands security evaluation to code agents by testing risky code execution across Python and Bash tasks, including sandboxed execution environments and evaluation metrics for unsafe behaviour [8]. Recent frameworks also show a shift toward execution- aware security evaluation. AutoSafeCoder combines LLM- based code generation with static analysis feedback and fuzzing-based dynamic testing in a multi-agent framework [7]. VulnRepairEval evaluates LLM-based vulnerability repair using functional proof-of-concept exploits and argues that exploit-based validation provides a stricter test of whether a vulnerability has actually been repaired [9]. Together, these works motivate security evaluation methods that go beyond static inspection alone, an area that our work complements and extends. C. Static and Dynamic Analysis Static analysis remains widely used for evaluating the security of generated code because it can be automated and applied to large numbers of code samples. Prior work has used tools such as Bandit and CodeQL to identify security weaknesses in generated code and benchmark datasets [6], [7]. However, empirical evaluations of SAST tools show that no single static-analysis tool detects all vulnerabilities, and substantial numbers of vulnerabilities may remain undetected even when multiple tools are combined [15]. Recent work improving Semgrep-based SAST pipelines further demonstrates that static-analysis effectiveness is heavily dependent on rule coverage and vulnerability-pattern representation, highlighting the potential for false negatives in automated security evaluation [15]. However, static analysis does not execute the program under concrete runtime conditions. It can identify many security- relevant patterns, but runtime exploitability may depend on adversarial inputs, execution context, or whether a vulnerable path can actually be triggered. This limitation motivates dy- namic testing approaches such as fuzzing, sandboxed execution, and exploit-based validation. Prior work has already used fuzzing and proof-of-concept exploits to strengthen security evaluation and aid vulnerability repairs [7], but the specific case of examining existing security-sensitive datasets for code that passes static analysis and later fails under dynamic exploit verification remains underexplored. D. Agentic Security Pipelines and Autonomous Software Eval- uation Modern LLM systems increasingly combine reasoning, tool use, and interaction with external environments. ReAct formalizes an approach in which language models interleave reasoning steps with actions that query tools or environments [16]. SWE-agent applies tool-mediated agent interaction to software engineering tasks, allowing agents to edit files, navigate repositories, and execute tests through an agent- computer interface [17]. Security-oriented systems have begun applying similar agen- tic patterns to code generation and evaluation. AutoSafeCoder uses multiple agents for code generation, static vulnerability analysis, and fuzzing-based testing [7]. RedCode evaluates code agents in scenarios involving risky code execution and generation, highlighting the need for safety evaluation in tool- using coding agents [8]. Our work builds on this direction by focusing on the gap between static validation and runtime exploitability in an agentic pipeline that combines static scanning, CWE-guided vulnerability reasoning for detection, and Docker-based exploit verification. I. METHODOLOGY We develop a three-stage security evaluation pipeline consist- ing of a composite Static Analysis Stage, a CWE Vulnerability Detection Agent, and a Dynamic Exploit Verification Agent. The latter two stages use LLM-based agents, while the initial stage deterministically orchestrates conventional static-analysis tools. The pipeline progressively filters, analyzes, and validates security weaknesses using both static and execution-based techniques. Samples successfully exploited after passing static analysis are classified as SPDF instances. A. Dataset Construction To evaluate SPDF across diverse software-security contexts, we construct a unified collection of 1,355 Python samples from three complementary security-oriented datasets: SecurityEval [6], RedCode [8], and the CyberNative Code Vulnerability Security DPO dataset [13]. SecurityEval provides manually curated vulnerability-oriented programming tasks, RedCode contributes adversarial execution scenarios, and CyberNative provides LLM-generated vulnerable code. Together, these datasets capture benchmark-driven, adversarial, and LLM- generated security contexts meant to provide complementary perspectives on software security and vulnerable code. Only Python samples were retained to ensure compatibility across Bandit, Semgrep, LLM-guided CWE analysis, and Docker- based execution. Each sample was normalized into a standalone Python file and assigned source-dataset metadata. Table I summarizes the final dataset collection used in this study. TABLE I DATASET COMPOSITION USED IN THE SPDF EVALUATION. DatasetSamplesRole in Study SecurityEval121Benchmark security evaluation RedCode810Adversarial execution scenarios CyberNative424LLM-generated vulnerable code Total1355Combined evaluation collection B. Pipeline Overview The pipeline consists of three stages, as shown in Figure 1: (1) BanditâSemgrep static analysis, (2) LLM-guided CWE vulnerability detection over statically clean files, and (3) Docker- based dynamic exploit verification of identified candidates. A sample qualifies as SPDF only when it passes the composite static gate, is assigned a candidate CWE by the detection agent, and subsequently exhibits runtime evidence of exploitability. The two agentic stages use separate configurations but share the same underlying model, a dependency considered when interpreting the results. C. Static Analysis Stage The Static Analysis Stage serves as the initial filtering stage of the proposed pipeline. Its objective is to establish the static- pass condition by identifying files containing vulnerabilities detectable by conventional static-analysis tools before any LLM- based reasoning or dynamic testing is performed. To accomplish this, we employ two complementary static- analysis tools: Bandit [11] and Semgrep [12]. Bandit focuses on Python-specific security weaknesses through a collection of security-oriented checks, while Semgrep provides rule-based detection of broader vulnerability patterns. Using both tools reduces reliance on a single detection strategy and establishes a stricter composite filtering stage. All static-analysis results were generated using Bandit v1.8.4.dev21 and Semgrep v1.155.0. Bandit was executed recur- sively across all Python files and configured to produce JSON output for downstream processing. Semgrep was executed using theautoruleset and JSON output. No additional confidence or severity thresholds were applied beyond those used by the tools themselves. Consequently, any file containing at least one finding reported by either Bandit or Semgrep was classified as statically flagged and removed from further analysis. Files producing no findings from either tool were classified as statically clean and advanced to the CWE Vulnerability Detection Agent. For each flagged file, the static-analysis stage records the originating tool, rule identifier, CWE mapping (when available), severity information, confidence information, and the associated source-code location. Findings from both tools are merged at the file level to produce a composite static-analysis result. A file is considered statically clean only when neither Bandit nor Semgrep reports any findings. Across the 1,355 Python samples analyzed, Bandit reported 922 findings, and Semgrep reported 816 findings. After merging detections across both tools and removing duplicate file-level reports, 701 unique files were flagged by at least one static- analysis tool. The remaining 654 files passed composite static analysis and formed the input to the second stage of the SPDF pipeline. D. CWE Vulnerability Detection Agent The CWE Vulnerability Detection Agent analyzes the statically clean samples produced by the Static Analysis Stage. Implemented using OpenAIâsgpt-5.4-nanomodel, the agent is prompted to act as a security analyst whose objective is to identify vulnerabilities that may have been missed by conven- tional static-analysis tools while grounding its assessments in authoritative vulnerability knowledge. All experiments used the OpenAI model identifiergpt-5.4-nanowith deterministic decoding (temperature=0). The complete system prompt, tool specifications, and agent configuration are provided in the replication package. The agent follows a ReAct-style reasoning process that combines source-code inspection with retrieval-augmented security analysis. For each sample, the agent examines program inputs, data flows, API usage, and security-sensitive operations to identify behaviours potentially associated with known vul- nerability classes. In this study, a candidate vulnerability refers to a fileâCWE pairing identified by the CWE Vulnerability Detection Agent based on security-relevant patterns observed in the source code and supported through comparison with external security knowledge and the corresponding MITRE CWE definition. These findings represent hypothesized vul- nerabilities that have not yet undergone dynamic exploit verification. When a candidateâs weakness is identified, the agent retrieves supporting information from external sources, including official MITRE CWE definitions, before assigning a vulnerability classification. The agent is explicitly instructed to validate suspected vulnerabilities against the corresponding CWE definition and supporting evidence before reporting them, thereby reducing unsupported or speculative findings. Three tools are available to the agent. Thesearch_web tool retrieves security information related to vulnerability patterns, CWEs, CVEs, and Python security weaknesses. Thefetch_cwe_detailtool retrieves the correspond- ing CWE entry from the MITRE repository. Finally, the report_findingstool records the agentâs final vulner- ability assessments. The system prompt explicitly prohibits reporting a CWE unless the corresponding MITRE CWE definition has been retrieved and examined. The agent produces structured vulnerability reports that include the predicted CWE identifier, vulnerability name, severity classification, and supporting rationale. By grounding assessments in established CWE definitions rather than relying solely on unconstrained model reasoning, the agent provides a more interpretable and security-focused analysis of statically clean code samples. Because LLM-based security judgments may vary across executions, the agent is executed independently five times for each sample. Each run is permitted a maximum of ten reasoning and tool-use iterations. Agent execution was bounded by interaction turns rather than a fixed token budget. No explicit per-instance token limit was imposed beyond the underlying modelâs context window, allowing token consumption to vary with analysis complexity. Final vulnerability predictions are consolidated using majority-vote aggregation at the fileâCWE level, where a vulnerability is retained only if it appears in a strict majority of runs (at least three of five executions). A five-run ensemble was selected as a practical balance between prediction stability and computational cost. This procedure reduces the impact of stochastic model behaviour and produces a more conservative and stable set of candidate vulnerabilities for subsequent dynamic verification. All other model settings were kept to their default parameters. E. Dynamic Exploit Verification Agent The Dynamic Exploit Verification Agent evaluates whether candidate vulnerabilities identified by the CWE Vulnerability Detection Agent can be triggered under concrete execution conditions, requiring observable runtime evidence before a vulnerability is considered CONFIRMED. The agent uses OpenAIâsgpt-5.4-nanowith temperature=0.3, automatic tool selection, and a maximum of 30 reasoning and tool-use iterations per execution. Agent execution was constrained by this turn budget rather than a fixed token budget; with no explicit token limit beyond the model context window. The non-zero temperature encourages exploration of diverse exploit strategies, while five-run majority voting mitigates variability across attempts. The agent follows a ReAct-style process combining retrieval- augmented vulnerability analysis with autonomous exploit generation. For each fileâCWE pair, the agent receives the target code, the predicted vulnerability classification, and supporting context from the previous stage. The agent then curates its own exploit-tests by retrieving information from the MITRE CWE database and the National Vulnerability Database (NVD) [18], leveraging vulnerability descriptions, attack patterns, and real-world examples associated with the target CWE. This grounding enables exploit generation to be informed by established vulnerability knowledge rather than relying solely on model reasoning. Six tools are available to the agent. Thefetch_cwe_info tool retrieves vulnerability definitions, examples, and support- ing information from the MITRE CWE repository, while fetch_nvd_cve_examplesretrieves real-world CVEs associated with the target CWE from the National Vulnerability Database (NVD). Thewrite_harnessandread_file tools enable the agent to generate and inspect exploit harnesses, run_in_dockerexecutes generated harnesses within an isolated runtime environment, andreport_verdictrecords the final exploitability assessment. The agent is required to retrieve both the corresponding MITRE CWE definition and NVD examples before exploit harness generation, grounding exploit construction in documented vulnerability behaviour rather than unconstrained model reasoning. Rather than executing a fixed set of tests, the agent dy- namically determines the number and type of exploit attempts required for a given vulnerability. Exploit generation is tailored to both the target CWE and the structure of the analyzed code. For example, command-injection vulnerabilities are evaluated across multiple payload variations and execution paths, path- traversal vulnerabilities are tested with alternative traversal strategies, and deserialization vulnerabilities are assessed with malicious serialized inputs. Additional exploit attempts may be generated when previous executions are inconclusive or when multiple attack surfaces are identified within the target program. This adaptive strategy allows the agent to explore vulnerability- specific attack vectors while avoiding a one-size-fits-all testing methodology. Generated exploit harnesses are executed within isolated Docker containers configured with disabled network access, a 256 MB memory limit, and configurable execution timeouts. This sandboxed environment enables the safe execution of potentially harmful payloads while providing a consistent, reproducible runtime. During execution, the pipeline records program output, execution status, resource utilization, and error conditions for subsequent analysis and to support final exploita- tion verdicts. The agent additionally records generated exploit harnesses, Docker execution outputs, exit codes, timeout events, and supporting evidence used to justify final exploitability decisions. A vulnerability is dynamically verified only when observable runtime behaviour consistent with the target CWE and NVD evidence is demonstrated. Criteria are vulnerability-specific: for example, command injection requires unintended command execution while path traversal requires unauthorized file access beyond intended boundaries. Runtime exceptions alone are not considered successful exploitation unless they constitute the target weakness. Following execution, the agent assigns one of four verdicts defined in Table I. A NOT TRIGGERED verdict indicates that testing did not produce sufficient runtime evidence of exploitation; it does not establish that the candidate is absent or unexploitable under other conditions. Each fileâCWE pair is evaluated indepen- dently across five executions, allowing multiple vulnerability hypotheses within the same file to be verified separately. A vulnerability is dynamically verified only when at least three of five runs returnCONFIRMED, reducing the impact of stochastic exploit generation. TABLE I DYNAMIC EXPLOIT VERIFICATION VERDICT DEFINITIONS. VerdictDefinition CONFIRMEDRuntime evidence demonstrates successful ex- ploitation of the target CWE. PARTIALTesting revealed behaviour consistent with the target CWE, but the observed evidence was insufficient to conclusively demonstrate full ex- ploitation. NOTTRIGGEREDMultiple exploit attempts fail to produce evidence of exploitation under the evaluated harnesses and execution conditions. INCONCLUSIVEThe vulnerability cannot be reliably evaluated under the available runtime conditions. Both LLM-based stages use the same underlying model (gpt-5.4-nano), although they operate with distinct prompts, tools, and objectives. Consequently, their outputs are not fully independent: model-specific reasoning tendencies may be shared across detection and verification, creating the possibility of correlated errors or self-confirmation. We account for this dependency when interpreting the resulting exploitability estimates and discuss its implications further in Section V and Section VI. F. SPDF Measurement To quantify the prevalence of the Static-Pass Dynamic-Fail (SPDF) phenomenon, we define the Static-Pass Dynamic-Fail Rate (SPDFR) as: SPDFR = |static-pass â© dynamic-fail| |static-pass| (1) where static-pass denotes unique files that produce no find- ings under the composite BanditâSemgrep static-analysis stage, and dynamic-fail denotes unique files for which the Dynamic Exploit Verification Agent subsequently confirms exploitability through runtime testing.SPDFR is reported at the file level using all samples that pass the evaluated BanditâSemgrep gate as the denominator. However, dynamic verification is applied only to fileâCWE candidates first identified by the CWE Vulnerability Detection Agent. Accordingly, SPDFR should be interpreted as the observed yield of the complete detection-and-verification pipeline among Bandit/Semgrep-clean samples, rather than as a direct estimate of the total prevalence of exploitable vulnerabilities in the entire static-pass set. Files not surfaced by the LLM detection stage are not dynamically tested and may therefore contain additional exploitable weaknesses that this pipeline does not observe. We include both conservative and inclusive definitions of the SPDFR, where conservative means only CONFIRMED verdicts are included in the calculation, while the inclusive definition incorporates both CONFIRMED and PARTIAL verdicts. In addition to SPDFR, we analyze the distribution of CWE categories among confirmed SPDF instances to identify vulnerability classes most likely to evade static detection. We also record latency, token consumption, reasoning iterations, and exploit-generation activity for the agentic stages to quantify the computational cost of the proposed pipeline. G. Statistical Analysis To assess whether dynamic exploit confirmation varies across benchmark datasets, we employ Pearsonâs Chi-Square Test of Independence on contingency tables constructed from CONFIRMED and non-CONFIRMED candidate fileâCWE pairs, as shown in Table I. Statistical significance is evaluated at α = 0.05. To examine whether exploit confirmation differs across vulnerability categories, we perform a permutation-based Chi-Square test using candidate fileâCWE pairs and their corresponding exploit-verification outcomes. This approach was selected because several CWE categories contain small sample sizes that violate the assumptions of standard asymptotic tests. TABLE I STATISTICAL ANALYSIS OF CONFIRMED EXPLOITABILITY AMONG CANDIDATE FILEâCWE PAIRS. FactorKey observationsSignificance Dataset RedCode106/315 (33.7%) CyberNative12/42 (28.6%) SecurityEval2/37 (5.4%)Ï 2 (2) = 12.55 , p = 0.0019 CWE (Top 5 Shown) CWE-40027/49 CWE-33815/15 CWE-33014/14 CWE-9169/14 CWE-227/35Ï 2 (22) = 138.06, p < 0.0001 â â Permutation-based p-value. Values are reported as CONFIRMED candidate fileâCWE pairs divided by total candidate fileâCWE pairs for the corresponding dataset or CWE category. H. Reproducibility and Artifact Release To support reproducibility and future research on the Static-Pass Dynamic-Fail (SPDF) phenomenon, we release the complete SPDF evaluation pipeline, generated exploit harnesses, and evaluation artifacts. We also release the subset of samples identified in this study as exhibiting SPDF behaviour, including exploit evidence, verification metadata, and dynamic analysis outcomes. These artifacts are intended to facilitate future research on runtime vulnerability verification, exploit- aware security evaluation, and the limitations of static-analysis- based security assessment. The complete system prompts, tool specifications, and agent implementations used by the CWE Vulnerability Detection Agent and Dynamic Exploit Verification Agent are included in the replication package [14]. IV. RESULTS This section presents the results of the SPDF evaluation pipeline. We first quantify the prevalence of vulnerabilities that survive static analysis yet remain exploitable at runtime. 1,355 Pyt hon Sam ples 654 St at ically Clean Fi les St at ic Scanner A gent CW E V ulner abi li t y D et ect ion A gent 235 Fi les (394 Candidat e CW Es) D ynam ic Exploi t at ion Ver i ficat ion A gent 120 / 3 9 4 CONFIRM ED D ynam ic Exploi t at ion R esult s 20 8 / 3 9 4 NOT TRIGGERED 20 / 3 9 4 PARTIAL 4 6 / 3 9 4 INCONCLUSIVE St at ic - Pass Dy namic - Fail Rate: Conser v at ive - 8 3 Files - 12.6 9 % Inc lusive - 9 5 Files - 14 .53 % Fig. 1. Summary statistics across stages of the dynamic exploit verification pipeline. Then, we examine how SPDF behaviour varies across bench- mark datasets and vulnerability categories, before assessing the computational cost of the proposed agentic verification process. Collectively, these analyses provide a multifaceted view of the relationship between static security assessment and demonstrated exploitability. A. RQ1: To What Extent Does Code That Passes Static Analysis Remain Dynamically Exploitable? Figure 1 presents the progression of samples through the proposed SPDF pipeline. Of the 1,355 Python samples analyzed, 701 unique files were flagged by either Bandit and/or Semgrep during static analysis. The remaining 654 samples passed the composite static-analysis stage and were therefore classified as statically clean and advanced to the second agent. The CWE Vulnerability Detection Agent subsequently iden- tified 394 candidate vulnerabilities across 235 unique statically clean files. These candidate filesâCWE pairs were evaluated for potential exploitation by the Dynamic Exploit Verification Agent, as shown in Table IV. TABLE IV DYNAMIC EXPLOIT VERIFICATION OUTCOMES. VERDICT COUNTS ENTAIL TESTED FILESâCWE INSTANCES THAT FALL INTO EACH CATEGORY. PERCENTAGES CALCULATED AS COUNT / CANDIDATE CWE FINDINGS OutcomeCountPercentage Files analyzed235â Candidate CWE findings394â CONFIRMED12030.5% PARTIAL205.1% NOTTRIGGERED20852.8% INCONCLUSIVE4611.7% Table V summarizes the resulting SPDFR metrics. Under the conservative definition, 120 confirmed fileâCWE pairs corresponded to 83 unique files, yielding an observed con- servative pipeline yield of 12.69% (83/654) relative to all Bandit/Semgrep-clean files. IncludingPARTIALoutcomes produced 140 fileâCWE pairs across 95 unique files, corre- sponding to an observed inclusive yield of 14.53% (95/654). These percentages characterize the output of the complete SPDF pipeline: dynamic verification was performed only on candidates surfaced by the CWE Detection Agent. They should therefore not be interpreted as direct estimates of the total prevalence of exploitable vulnerabilities among all 654 static-pass files. Because exploitability was assessed using a specific agent architecture, model configuration, and execution environment, these values should be interpreted as a model- dependent lower bound of demonstrated exploitability within the evaluated benchmark collection. The aggregate results were strongly influenced by dataset composition. RedCode accounted for 315 of the 394 candidate fileâCWE pairs (79.9%) and 106 of the 120CONFIRMED outcomes (88.3%). Among candidate fileâCWE pairs, confir- mation rates were 33.7% for RedCode (106/315), 28.6% for CyberNative (12/42), and 5.4% for SecurityEval (2/37). The association between dataset and confirmation outcome was statistically significant (Ï 2 (2) = 12.55, p = 0.0019). Across the dynamically evaluated candidates, the Dynamic Exploit Verification Agent produced 120CONFIRMEDexploita- tions and an additional 20PARTIALexploitations. Combined, 35.6% (140/394) of all candidate fileâCWE pairs exhibited at least some evidence of exploitability during dynamic verification. TABLE V STATIC-PASS DYNAMIC-FAIL RATES (SPDFR). MetricCount (Files)Rate Conservative SPDFR8312.69% Inclusive SPDFR9514.53% These findings provide direct evidence of the Static-Pass Dynamic-Fail phenomenon: more than one in eight samples that passed a composite BanditâSemgrep gate was subsequently shown to containCONFIRMEDvulnerabilities triggerable through execution-based testing. Static-analysis success should therefore not be read as evidence that software resists exploita- tion. Equally notable is the reduction between vulnerability suspicion and confirmation. Of the 394 candidate weaknesses across 235 files, 120 were confirmed and 20 partially confirmed, while 208 were not triggered under the generated harnesses and available execution conditions and 46 remained inconclusive. Importantly, a NOTTRIGGERED verdict indicates only that the agent did not obtain runtime evidence of exploitation during the performed tests; it does not establish that the candidate weakness is a false positive or is unexploitable under other harnesses or environments. Thus, fewer than half of the reasoning-identified weaknesses translated into demonstrated exploitability within our evaluation. Dynamic verification thus operates as both a discovery and a validation mechanism; we discuss the implications of this layered view in Section V. B. RQ2: What Categories of Vulnerabilities Are Most Likely to Evade Static Analysis While Remaining Exploitable During Execution? To examine whether exploit confirmation rates vary across benchmark sources, we performed a Pearson Chi-Square test of independence. As shown in Table I, the relationship between dataset source and confirmed exploitability was statistically significant (Ï 2 (2) = 12.55, p = 0.0019). 01020304050 Number of file--CWE evaluations CWE-434 CWE-285 CWE-798 CWE-200 CWE-401 CWE-770 CWE-916 CWE-330 CWE-338 CWE-532 CWE-250 CWE-94 CWE-73 CWE-22 CWE-400 CWE category 10 10 10 12 12 13 14 14 15 16 17 19 32 35 49 1 2 3 4 6 9 14 15 3 1 6 7 27 1 4 1 1 2 3 1 Top 15 candidate CWE findings and dynamic exploit verification outcomes Detected candidate CWEs Confirmed Partial Fig. 2. Top 15 CWE categories identified during dynamic verification. Among candidate fileâCWE pairs identified by the CWE Detection Agent, RedCode exhibited the highest exploit- confirmation rate (33.7%), followed by CyberNative (28.6%), while SecurityEval produced substantially fewer confirmed exploitations (5.4%). These results suggest that vulnerabilities surviving static analysis are not equally likely to remain exploitable across benchmark datasets. We next examined exploitability across vulnerability cat- egories. A permutation-based chi-square test revealed a sig- nificant association between CWE category and confirmed exploitability (Ï 2 (22) = 138.06,p < 0.0001), indicating that SPDF behaviour is strongly dependent on vulnerability class. Figure 2 compares candidate vulnerability findings against confirmed and partial exploitations across the 15 most fre- quently observed CWE categories. Several categories exhib- ited substantial reductions between candidate identification and confirmed exploitability, indicating that many suspected weaknesses could not be reproduced under dynamic testing. In contrast, CWE-400 (Uncontrolled Resource Consumption) produced both the largest number of candidate findings and the largest number of confirmed exploitations (27 cases). Other frequently confirmed categories included CWE-338 (Use of Cryptographically Weak Pseudo-Random Number Generator), CWE-330 (Use of Insufficiently Random Values), CWE-916 (Use of Password Hash With Insufficient Computational Effort), and CWE-22 (Path Traversal). The most frequently confirmed SPDF categories involve security properties difficult to evaluate through syntactic inspection alone. Resource-exhaustion weaknesses frequently manifested through excessive memory allocation, computational amplification, or non-terminating execution paths that only became apparent during execution. Similarly, randomness and cryptographic weaknesses often appeared functionally correct or unsuspicious to security patterns, while violating underlying security assumptions. Consequently, these vulnerabilities sur- vived static analysis despite remaining exploitable in practice. Confirmation rates also varied sharply across classes. CWE- 338 and CWE-330 reached 100% confirmation, with every candidate vulnerability identified by the CWE Detection Agent and then labeled CONFIRMED through dynamic verification. In contrast, several categories frequently identified during CWE reasoning failed to produce successful exploit demonstrations. Examples include CWE-94 (Code Injection), CWE-434 (Unre- stricted File Upload), and CWE-20 (Improper Input Validation), all of which produced candidate findings but no confirmed exploitations. This variation suggests that exploitability is highly dependent on vulnerability class and that the presence of a candidate weakness does not necessarily imply practical exploitability. An additional finding concerns vulnerability categories that were absent from the initial static analysis stage. As shown in Table VI, several of the most frequently confirmed SPDF categories produced no Bandit or Semgrep findings anywhere in the collection despite later being identified by the CWE Detection Agent and confirmed through dynamic verification. Most notably, CWE-338 resulted in 15 CONFIRMED exploits despite no static-analysis findings, while CWE-916 resulted in 9 CONFIRMED exploits despite being absent from both tools. Similar patterns were observed for CWE-73, CWE-770, and CWE-862. TABLE VI TOP-5 CONFIRMED CWES MISSED BY STATIC ANALYSIS CWEConfirmed Cases CWE-33815 CWE-9169 CWE-736 CWE-7706 CWE-8626 This indicates that the observed SPDF behaviour is not limited to isolated false negatives within cov- ered classes: entire cate- gories of exploitable weak- nesses were absent from the static-analysis output, yet confirmed at run- time. At the same time, some confirmed SPDF cat- egories were represented in the static-analysis out- put elsewhere in the collection. For example, CWE-400, CWE- 330, and CWE-22 appeared among Bandit findings but were nevertheless confirmed in statically clean samples that advanced through the pipeline. This suggests that even when a static tool provides coverage for a vulnerability class, individual instances may still evade detection depending on implementation details, program context, and runtime behaviour. C. RQ3: What Is the Computational Cost of the Agentic Dynamic-Exploitability Pipeline? The Dynamic Exploit Verification Agent evaluated 394 filesâCWE pairs and performed 1,304 Docker executions, corresponding to an average of approximately 3.3 exploit attempts per candidate vulnerability. Table VII reports aggregate cost. The CWE Detection Agent consumed roughly 24.6M tokens (10.5 hours) and the Dynamic Verification Agent 71.7M tokens (18.2 hours), for a pipeline total of approximately 96.3M tokens and 28.7 hours. TABLE VII AGGREGATE COMPUTATIONAL COST OF THE SPDF PIPELINE. StageTokensLatency (s) Static Analysis Stage077.3 CWE Detection Agent24.6M37833.7 Dynamic Verification Agent71.7M65383.1 Total96.3M103294.1 TABLE VIII PER-EXECUTION COST OF DYNAMIC EXPLOIT VERIFICATION ACROSS ALL FIVE VERIFICATION RUNS (1,970 TOTAL EXECUTIONS). MetricMeanMedianIQRMax Agent Turns8.3487â1030 Total Tokens36,38833,33128,763â42,911219,084 Latency (s)33.1925.1922.53â29.291231.08 While aggregate cost characterizes the resources required to conduct the full study, Table VIII reports the cost of individual dynamic-verification evaluations calculated across the 5 runs. Means consistently exceeded medians (Table VIII), indicating a small tail of difficult cases requiring substantially more reasoning and exploit generation; the most expensive evaluation consumed 219,084 tokens and over 20 minutes. These results highlight the costâdepth trade-off: dynamic exploit verification is substantially more expensive than static analysis and is therefore better suited to selective verification than exhaustive screening a point we return to in Section V. V. DISCUSSION The central finding of this study is that software security is better understood as a hierarchy of evidence than as a binary property. Static analysis, vulnerability reasoning, and exploit verification provide different levels of evidence, and success at one stage does not guarantee success at the next. Across the pipeline, 1,355 samples were reduced to 654 statically clean files, 394 candidate weaknesses, and 120 confirmed exploitations across 83 files, showing that exploitable behaviour can persist through increasingly stringent evaluation. The conservative SPDF rate of 12.69% (more than one in eight statically clean files) represents the observed yield of the full detection-and-verification pipeline and indicates that passing composite static analysis is better interpreted as evidence of reduced risk than as evidence of security. Because CyberNative contains LLM-generated code, the results also show that statically clean LLM-generated programs can remain exploitable. Whether this extends to code produced by assistants in interactive development workflows, where context, iteration, tool use, and human oversight differ, remains unresolved. A. Interpreting the SPDF Rate The aggregate SPDF rates should be interpreted in light of substantial dataset heterogeneity. RedCode contributed 315 of the 394 candidate fileâCWE pairs (80%) and 106 of the 120 CONFIRMED exploitations (88%), so the aggregate rate is overwhelmingly a property of a single benchmark. The significant association between dataset source and confirmed exploitability (Ï 2 (2) = 12.55,p = 0.0019) reinforces this: SPDF behaviour is not uniformly distributed across the corpus. Notably, SecurityEval, the only manually curated, generation- task benchmark of the three, produced the lowest confirmed rate (5.4%), whereas the adversarial and intentionally vulnerable corpora produced the highest (33.7% and 28.6%). The aggregate inclusive rate of roughly one in seven should therefore be read as an upper-leaning estimate driven by adversarial and vulnerability-seeded code, not as a direct prediction of how often statically clean code from realistic generation tasks will prove exploitable. A second consideration concerns the shared model backbone across pipeline stages. Both the CWE Detection Agent and the Dynamic Exploit Verification Agent are implemented withgpt-5.4-nano, so the verifier confirms hypotheses generated by a sibling configuration of the same model. This risks correlated reasoning errors and a degree of self- confirmation: weaknesses the detector is predisposed to surface may also be those the verifier is predisposed to demonstrate. The categories with perfect confirmation rates, CWE-338 and CWE-330, each confirmed in every candidate instance, are consistent with this concern, since their verification often reduces to demonstrating that a generator is non-cryptographic, a property both agents recognize from the same surface cues. Majority voting over five runs reduces stochastic noise but cannot break this correlation, because all runs draw on the same model; pairing a detector and verifier from different model families, or auditing a stratified subsample against human analysts, would establish how much of the signal is independent of the model that proposed it. Finally, the SPDF rate treats every confirmed exploitation as equivalent, yet the confirmed categories differ markedly in consequence. The three most frequently confirmed classes, CWE-400 (Uncontrolled Resource Consumption), CWE-338, and CWE-330, are dominated by denial-of-service and weak- randomness weaknesses, which are among the easiest to demonstrate at runtime and, in many deployment contexts, among the lower in direct impact. By contrast, CWE-94 (Code Injection) produced numerous candidate findings but no con- firmed exploitations. Because exploitability and consequence are conflated within a single rate, the headline figure may overstate the practical severity of the residual risk, motivating severity-weighted formulations that do not silently equate exploitable with consequential. B. Implications For security evaluation pipelines, the results caution against treating static analysis as the primary measure of risk in AI- generated code, as is common practice. Static analysis remains an efficient first-line filter, it eliminated 701 files here at negligible cost; however, it identifies vulnerable patterns rather than demonstrating consequences, so the strongest assessments combine it with execution-based verification rather than relying on either in isolation. For vulnerability detection, both exploitability and static coverage are unevenly distributed across weakness classes. Several frequently confirmed categories were absent from the static-analysis output entirely (Table VI), indicating that SPDF behaviour is not merely isolated false negatives within covered classes but, in some cases, whole categories falling outside rule-based coverage. These tend to involve semantic properties, randomness quality, cryptographic strength, authorization logic, and resource consumption, that resist syntactic pattern matching, suggesting that reasoning- and execution-based analysis can complement rule-based tooling. For agentic security systems, dynamic verification served as both discovery and validation: it confirmed 120 weaknesses that survived earlier stages, while 208 candidate weaknesses were not triggered under the generated harnesses and available execu- tion conditions. These results demonstrate how dynamic verifi- cation can distinguish reasoning-based vulnerability hypotheses from vulnerabilities for which concrete runtime evidence can be obtained, without treating unsuccessful exploitation as evidence that the candidate weakness is necessarily absent. This depth is expensive, roughly 96.3 million tokens and 29 hours, so exploit verification is best deployed selectively within a layered strategy in which static analysis narrows the search space and reasoning prioritizes candidates. The broader implication is that exploitability deserves treatment as a first-class security outcome rather than an implicit byproduct of detection. VI. THREATS TO VALIDITY A. Construct Validity The primary construct, runtime exploitability, is an oper- ationalization rather than a perfect measure of real-world security impact. Because the sandbox disabled network access and provided no external services, privileged contexts, or vulnerability-specific dependencies, and because successful verification depends on the agent generating an effective exploit harness, someNOT_TRIGGEREDverdicts and the 46INCONCLUSIVEoutcomes may reflect environmental limits rather than the absence of a vulnerability; the reported SPDF rates should therefore be read as a lower bound on demonstrated exploitability. Two further limitations apply: the CWE Detection Agent relies on LLM-based reasoning and may produce both false positives and false negatives, and all verdicts are generated automatically rather than validated by human security analysts, so individual outcomes may diverge from expert judgment. B. Internal Validity Several implementation choices may influence the observed rates. Both agents usegpt-5.4-nano; a different model could change detection, recognition, and exploit-generation behaviour, and, because the same model both proposes and confirms weaknesses, a portion of the confirmation signal may be self-reinforcing, as discussed in SectionV-A. In addition, the Docker sandboxâs execution and resource limits, while improving safety and reproducibility, may suppress exploits that require system privileges or external conditions. Finally, majority voting over five runs reduces but does not eliminate stochastic variability; alternative run counts or aggregation strategies may yield slightly different estimates. C. External Validity The study examines only Python code from SecurityEval, RedCode, and CyberNative. Vulnerability patterns, static- analysis effectiveness, and exploitability may differ for pro- duction systems and for languages such as C/C++, Java, and Rust, so the reported rates should not be read as universal static-analysis failure rates. The static-pass set also depends on Bandit v1.8.4.dev21 and Semgrep v1.155.0 under theauto ruleset with no additional filtering; alternative versions, rulesets, or commercial scanners would shift it. Because these corpora emphasize security-relevant tasks, SPDF prevalence in general- purpose development may differ. VII. CONCLUSION AND FUTURE WORK This work examined whether code producing no findings under a composite BanditâSemgrep static-analysis gate can nevertheless exhibit dynamically demonstrable vulnerabilities. Through the Static-Pass Dynamic-Fail (SPDF) framework, we found that static-analysis outcomes under this evaluated configuration and demonstrated runtime exploitability capture distinct forms of security evidence. More than one in eight samples producing no findings under the evaluated static- analysis gate were subsequently confirmed exploitable under our conservative dynamic-verification criterion, and several CWE categories absent from the BanditâSemgrep findings were nonetheless confirmed at runtime. Because dynamic verification was restricted to LLM-identified candidates, these values characterize the yield of the evaluated pipeline rather than the total prevalence of exploitable vulnerabilities among all static-pass samples. These results support treating software security assessment as an accumulation of evidence across complementary forms of analysis rather than relying on any single evaluated technique in isolation. Alongside these findings, we release an agentic exploitability-evaluation framework, a dataset of dynamically evaluated vulnerability candidates, and a reproducible methodology for relating static-analysis outcomes to runtime exploitability. Two directions follow most directly. First, pairing a detector and verifier from different model families would test how much of the confirmed-exploitability signal is independent of the model that proposed it. Second, weighting confirmed instances by severity would distinguish exploitable weaknesses from consequential ones, refining the SPDF rate as a measure of practical risk. ACKNOWLEDGEMENTS The authors acknowledge support from the Faculty of Science Undergraduate Research Opportunities program at Toronto Metropolitian University which made this work pos- sible. OpenAI ChatGPT and Grammarly were used to assist with language editing and manuscript refinement. All technical content, experimental design, analysis, and final manuscript decisions were reviewed and validated by the authors. REFERENCES [1]A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, âLarge language models for software engineering: Survey and open problems,â in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023, p. 31â53. [2]J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., âGpt-4 technical report,â arXiv preprint arXiv:2303.08774, 2023. [3]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, âSwe-bench: Can language models resolve real-world github issues?â in International Conference on Learning Representations, vol. 2024, 2024, p. 54 107â54 157. [4] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, âAsleep at the keyboard? assessing the security of github copilotâs code contributions,â Commun. ACM, vol. 68, no. 2, p. 96â105, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3610721 [5] R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, âHow secure is code generated by chatgpt?â arXiv preprint arXiv:2304.09655, 2023. [6]M. L. Siddiq and J. C. S. Santos, âSecurityeval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques,â in Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security, ser. MSR4P&S 2022.Association for Computing Machinery, 2022, p. 29â33. [Online]. Available: https://doi.org/10.1145/3549035.3561184 [7]A. Nunez, N. T. Islam, S. K. Jha, and P. Najafirad, âAutosafecoder: A multi-agent framework for securing llm code generation through static analysis and fuzz testing,â arXiv preprint arXiv:2409.10737, 2024. [8]C. Guo, X. Liu, C. Xie, A. Zhou, Y. Zeng, Z. Lin, D. Song, and B. Li, âRedcode: risky code execution and generation benchmark for code agents,â in Proceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS â24. Curran Associates Inc., 2024. [9]W. Wang, W. Ma, Q. Hu, Y. Zhang, J. Sun, B. Wu, Y. Liu, G. Xu, and L. Jiang, âVulnrepaireval: An exploit-based evaluation framework for assessing large language model vulnerability repair capabilities,â arXiv preprint arXiv:2509.03331, 2025. [10]B. Chess and G. McGraw, âStatic analysis for security,â IEEE Security & Privacy, vol. 2, no. 6, p. 76â79, 2004. [11]PyCQA, âBandit documentation,â https://bandit.readthedocs.io/en/latest/, 2026. [12]Semgrep, Inc., âSemgrep documentation,â https://semgrep.dev/docs/, 2026. [13]CyberNative, âCode vulnerability security dpo dataset,â https://huggingf ace.co/datasets/CyberNative/CodeVulnerabilitySecurityDPO, 2024. [14]Pourleyli et al., âBeyond static guarantees: Measuring the static-pass dynamic-fail gap in security-sensitive and llm-generated python code,â Jun. 2026. [Online]. Available: https://doi.org/10.5281/zenodo.20761891 [15]G. Bennett, T. Hall, E. Winter, and S. Counsell, âSemgrep*: Improving the limited performance of static application security testing (sast) tools,â in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE â24. Association for Computing Machinery, 2024, p. 614â623. [Online]. Available: https://doi.org/10.1145/3661167.3661262 [16]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, âReact: Synergizing reasoning and acting in language models,â in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [Online]. Available: https://openreview.net/forum?id=WEvluYUL-X [17]J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, âSwe-agent: agent-computer interfaces enable automated software engineering,â in Proceedings of the 38th International Confer- ence on Neural Information Processing Systems, ser. NIPS â24. Curran Associates Inc., 2024. [18] National Institute of Standards and Technology (NIST), âNvd api documentation,â https://nvd.nist.gov/developers, 2026.