Paper deep dive
The History Is the Detector: Executing CVE Patch History, End-to-End
Qiushi Wu, Kevin Eykholt, Youngja Park, Xiaokui Shu, Dhilung Kirat, Douglas Lee Schales, Ian Molloy
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/8/2026, 2:16:13 AM
Summary
The paper introduces BUGSTONE-E2E, a framework that automates the transformation of CVE patch history into executable detection rules and validates findings through a staged pipeline. It mines 1,033 detection rules from 2,710 verified fixing commits across 56 CWE families. The system uses a funnel-shaped approach: Tree-sitter for anchor enumeration, lightweight heuristics for filtering, LLM-based agents for semantic inspection, and runtime validation (including ASan and differential testing) for confirmation and patch generation. Applied to 14 programs, it produced runtime evidence for 644 findings.
Entities (7)
Relation Signals (5)
BUGSTONE-E2E â mines â CVE
confidence 98% ¡ Using 19,325 high-severity CVEs from 2022 to 2026, BUGSTONE-E2E identifies 2,710 fixing commits
BUGSTONE-E2E â generates â CWE
confidence 95% ¡ constructs 1,033 detection rules across 56 CWE families
BUGSTONE-E2E â uses â Tree-sitter
confidence 95% ¡ BUGSTONE-E2E first enumerates call sites matching rule anchors using Tree-sitter
BUGSTONE-E2E â uses â LLM
confidence 95% ¡ Next, LLM-based agents inspect the remaining candidates guided by the rule.
BUGSTONE-E2E â validatesusing â ASan
confidence 90% ¡ produced runtime evidence for 644 findings, including Address-Sanitizer (ASan [63]) violations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Public vulnerability databases collect rich information about known software flaws, including their weakness types, affected components, and related patches. Fixing commits provide the exact code changes that removed these flaws. While these records capture why the original code was unsafe, they are documented mainly for human inspection rather than automated reuse. Consequently, the same unsafe conditions may still exist elsewhere in code without a known advisory, leaving much of this detection knowledge unused. We present BUGSTONE-E2E, a framework that transforms vulnerability history into executable detection rules and validates their findings. First, BUGSTONE-E2E mines reusable rules from verified fixing commits, capturing scan anchors, fix semantics, and CVE provenance and organizing them by CWE and language. Second, detection follows a funnel-shaped pipeline: early stages process a large pool of candidates using lightweight analysis, while later stages apply increasingly capable and expensive models to a shrinking set of targets. Specifically, BUGSTONE-E2E first enumerates call sites matching rule anchors using Tree-sitter, then removes benign sites using lightweight heuristics without LLM calls. Next, LLM-based agents inspect the remaining candidates guided by the rule. Following this inspection, the system re-triages surviving candidates and builds runtime verifications, then generates scope-checked patches validated via two-sided differential tests. Using 19,325 high-severity CVEs from 2022 to 2026, BUGSTONE-E2E identifies 2,710 fixing commits and constructs 1,033 detection rules across 56 CWE families, packaged into 172 skills. When applied across 14 programs, it produced runtime evidence for 644 findings. These results demonstrate that CVE history can be turned into an executable workflow, transforming past vulnerabilities into reproducible detection and repair.
Tags
Links
- Source: https://arxiv.org/abs/2609.05335v1
- Canonical: https://arxiv.org/abs/2609.05335v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
97,444 characters extracted from source content.
Expand or collapse full text
The History Is the Detector: Executing CVE Patch History, End-to-End Qiushi WuKevin EykholtYoungja ParkXiaokui Shu Dhilung KiratDouglas Lee SchalesIan Molloy IBM Research Abstract Public vulnerability databases collect rich information about known software flaws, including their weakness types, af- fected components, and related patches. Additionally, fixing commits provide the exact code changes that removed these flaws. While these records capture why the original code was unsafe, they are documented mainly for human inspection rather than automated reuse. Consequently, the same unsafe conditions may still exist elsewhere in code with no known advisory, leaving much of this detection knowledge unused. We presentBUGSTONE-E2E, a framework that automati- cally transforms vulnerability history into executable detec- tion rules and validates their findings. First,BUGSTONE-E2E mines reusable rules from verified fixing commits, capturing scan anchors, fix semantics, and CVE provenance and orga- nize them by CWE and language. Second, detection follows a funnel-shaped pipeline; early stages process a large pool of candidates using lightweight analysis, while later stages apply increasingly capable and expensive models to a shrinking set of targets. Specifically,BUGSTONE-E2Efirst enumerates call sites matching rule anchors using Tree-sitter, then removes benign sites using lightweight heuristics without LLM calls. Next, LLM-based agents inspect the remaining candidates guided by the rule. Following this inspection, the system re- triages surviving candidates and builds runtime verifications, and finally generates scope-checked patches that are validated via two-sided differential tests. Using 19,325 high-severity CVEs from 2022 to 2026,BUGSTONE-E2Eidentifies 2,710 fixing commits and constructs 1,033 detection rules across 56 CWE families, packaged into 172 skills. When applied across 14 programs, it produced runtime evidence for 644 findings. These results demonstrate that CVE history can be turned into an executable workflow, transforming past vulnerabilities to reproducible detection and repair. 1 Introduction Modern vulnerability response leaves a detailed public record. Specifically, a CVE entry describes the affected behavior and severity, a CWE label identifies the underlying weakness class, and a fixing commit usually shows the code-level changes that mitigated the risk [17,50]. For a human reviewer, these records explain what went wrong. However, they are of little use to automated scanners because the lessons embedded in a patch do not translate into detectors capable of searching the next codebase. The missed opportunity. Many vulnerability patterns re- cur across projects and languages. For example, a command- injection fix may reveal that attacker-controlled data reaches a shell sink without proper quoting. Although API names and code structure may differ, the underlying security condition often remains the same. Prior work detects recurring vul- nerabilities using known vulnerable code or patch-enhanced signatures [35,82], while recent systems combine LLMs with retrieved vulnerability knowledge or program-analysis con- text [18,38,44]. VulGenie further extracts API security rules from Java security patches [14]. However, what remains miss- ing is a general pipeline that transforms CVE fixes across various weakness classes and languages into reusable, exe- cutable rules, applies them at scale, and validates and repairs the findings they produce. Key challenges. Turning these public vulnerability records into executable detectors raises three primary challenges. First, CVE records and patches vary widely in content, mak- ing it difficult to extract the precise components that can be reused for detection. The relevant security fixes may be buried in unrelated edits, poorly described, or tied to project-specific logic. A mining tool must therefore identify what made the original code unsafe and generalize that condition into an- chors that can match the same flaw elsewhere. Second, this knowledge must be represented as an executable rule with suf- ficient context to distinguish vulnerable patterns from benign cases. For instance, a simple match onsystem,pickle.loads, ormemcpyis too broad, whereas an overly specific signature risks missing the same vulnerability hidden behind a different wrapper or code structure. Third, these rules must scale effi- ciently to handle large codebases without incurring expensive 1 arXiv:2609.05335v1 [cs.CR] 4 Sep 2026 analysis on every candidate. Static analyzers such as Cod- eQL and Semgrep scale well once suitable queries or rules exist [28,62], but creating and maintaining such rules still requires manual efforts. From detection to verified remediation. Solving the three challenges above remains insufficient for end-to-end vulnera- bility analysis. While a rule violation or plausible taint path identifies a potential vulnerability, it does not prove that the behavior is reachable or exploitable at runtime. A complete workflow therefore requires dynamic validation that produces concrete runtime evidence. Furthermore, it must support reme- diation by generating a feasible patch from the vulnerability evidence and program context, and then verifying that the patch removes the demonstrated behavior. Our approach.BUGSTONE-E2Eaddresses this gap through a staged pipeline spanning four core phases (Phase AâD) that transitions seamlessly from reusable vulnerability knowledge to automated detection, runtime validation, and verified reme- diation. The system first mines known vulnerabilities from the cvelistV5dataset, verifies their fixing commits, and synthe- sizes reusable rules containingscan_apis, buggy and fixed code patterns, a verification contract, and related source CVEs. The rules are grouped by CWE and language into modular bug detection skills. For a target project, Phase A first indexes the codebase with Tree-sitter [69] and enumerates all call sites match rule anchors without using an LLM. Phase A then removes clearly benign candidates via deterministic checks. The remaining candidates are analyzed independently in Phase B, where an LLM-based agent uses the rule-specific verification contract to inspect taint flows, callers, guards, and sanitizers, finally returning a structured verdict accompanied by concrete code evidence. This funnel reduces the number of candidates before applying more expensive analysis and allows Phase B to run in parallel with cheaper models. However, static evidence alone is insufficient for full vul- nerability confirmation. After Phase B filters out most can- didates, Phase C re-triages the small set of survivors and attempts to reproduce each one in an isolated, concrete execu- tion environment. A finding gains definite runtime validation only when an expected runtime markerâsuch as a sanitizer report, crash, or controlled-sink signalâis observed, and only the subset with a working live proof-of-concept (PoC) is rated asCONFIRMED_EXPLOITABLE, separating non-exploitable flaws from those that could not be tested. Finally, Phase D generates a minimal, scope-checked patch and verifies it against the runtime evidence produced in Phase C. The exploit must fail on the patched program. It then reverts the patch and confirms that the exploit succeeds again, preventing a broken build or environment from being accepted as a fix. Exploit- and test-based validation is increas- ingly used in recent literature to evaluate vulnerability repair and patch correctness [9,26,75]. However, unlike these exist- ing settings,BUGSTONE-E2Ebegins with newly discovered findings that have no predefined PoC or validation environ- ment. It must therefore construct the runtime evidence by itself and independently verify each critical artifact before accepting a patch. Evidence. In this work,BUGSTONE-E2Esurveys 19,325 high-severity CVEs from 2022 to 2026, enriches 5,902 unique open-source CVEs, and verifies 2,710 fixing commits. From these commits, it constructs 1,033 rules across 56 CWE families, packaged into 172 productionbug detection skills. Across scans of 14 real-world projects, Phase B reports 2,933 deduplicated findings. Finally, Phase C successfully obtains runtime evidence for 644 of these findings, including Address- Sanitizer (ASan [63]) violations and crashes. This paper makes the following contributions: ⢠Vulnerability Rule Mining from CVE history: We design a general pipeline that surveys high-severity CVEs, verifies fixing commits, extracts each patchâs vulnerable sink and fix semantics, and rejects cases that do not generalize. â˘Cost-ordered Detection Rule Execution: Our CWE and language-specific rules carry scan anchors, buggy and fixed patterns, a verification contract, and source-CVE prove- nance, all packaged into modular deployable skills. This ar- chitecture leverages deterministic indexing and lightweight filters to absorb scale, allowing LLM-based agents to apply expensive semantic judgment only where a rule points. â˘Oracle-driven End-to-End Workflow: Our system exe- cutes a batched re-triage and an environment-first exploit stage that separates ânot exploitableâ from âcould not be testedâ, and judges patches using a two-sided differential testing under a machine-checked output contract. We report empirical evidence for every pipeline phase and state what each result does not establish. ⢠Deterministic Decompositionfor Scalability:In BUGSTONE-E2E, anchor enumeration and the application of a frozen filter are deterministic. Thus, we can split a very large candidate pool into bounded, independent batches grouped by rule and source file. Since the batches share no mutable execution state, the expensive LLM-based agentic stage can parallelize freelyâeven though candidates within a single batch are not statistically independentâresulting in high scalability. 2 Background and Problem 2.1 Motivation Vulnerability patterns recur across projects. Known vul- nerability patterns often reappear in other code locations or projects. Early systems, such as ReDeBug and VUDDY, demonstrated this problem by finding unpatched clones of known vulnerable codebases at large scale [31,35]. Later work generalized beyond exact clones by deriving vulnera- bility and patch signatures that identify recurring vulnerabil- 2 Table 1: Representative high-recurrence patterns from our CVE background study, spanning ten of the twelve languages BUGSTONE-E2Ecovers. #CVEs counts distinct CVEs whose patches support the pattern, and the same weakness class recurs across languages. CWELang.#CVEsRepr. API CWE-79PHP27 echo CWE-502PHP16 unserialize CWE-78Python13 os.system CWE-79JS9 innerHTML CWE-639Go7 Decode CWE-79TypeScript7 src CWE-22Java5 getNextEntry CWE-122C4 memcpy CWE-122C++4 memcpy CWE-22C#4 Path.Combine CWE-78Ruby2 system ities despite code changes [30,77,82]. More recently, Wu et al. [81] showed that identical security-relevant usage pat- terns often recur many times within a single software project. Starting with 135 rules, their study flagged 22,568 potential violations in the Linux kernel, where a subsequent manual review of 400 cases confirmed 246 true vulnerabilities. Cross-language recurrence. Prior evidence has focused largely on code reuse and recurring patterns within a partic- ular language or software ecosystem. To examine whether the same property holds more broadly, we study public CVE fixing commits across weakness classes and programming languages. Table 1 reports representative high-recurrence pat- terns for different languages. We observe that many patterns are supported by multiple independent CVEs, rather than a single project-specific fix. We also find that the same weak- ness class can recur across languages through similar security- relevant operations. For example, cross-site scripting (CWE- 79) appears across PHP, JavaScript, and TypeScript, while command injection (CWE-78) affects Python and Ruby. CWE-78-Py-R1, our running example, is supported by thir- teen CVEs from different Python projects. The common pat- tern is that externally derived data reaches a shell API without sufficient quoting or sanitization. In one instance from Pad- dleOCR, a path selected throughQFileDialogis used to de- rive an output directory and then concatenated directly into an os.systemcommand. Although the surrounding application logic differs across projects, the security-relevant condition is the same. This recurrence motivates extracting such condi- tions from CVE fixes and reusing them to guide detection in previously unseen code. 2.2Challenges in Precise and Scalable Vulner- ability Detection Traditional static analysis. Static analyzers such as Cod- eQL [28] and Semgrep [62] can efficiently scan large code- bases once suitable queries or rules are available. However, encoding a new vulnerability pattern still requires analyst ef- fort, which limits how quickly these tools can absorb the broad and evolving set of patterns recorded in CVE history. Their findings also often require additional context to distinguish true vulnerabilities from benign uses, and false positives re- main a major barrier to developer adoption [33]. Recent work therefore combines static analysis with semantic reasoning to improve the triage of candidate findings [44]. Static analysis provides scalable candidate discovery, but extending its cover- age and resolving ambiguous findings still require substantial manual or semantic analysis. Dynamic analysis and fuzzing. Dynamic techniques provide stronger evidence because they observe concrete program be- havior. Tools such as ASan [63] detect memory errors during execution, while fuzzers such as AFL++ [21] explore pro- gram behavior through generated inputs. Their effectiveness, however, depends on both reaching the vulnerable behavior and having an oracle that recognizes it. Memory errors of- ten produce crashes or sanitizer violations, but many other vulnerabilities do not. For command injection, path traversal, deserialization, and similar weaknesses, successful execution may appear normal unless the analysis encodes the relevant security condition. Dynamic validation is therefore valuable for confirming vulnerabilities, but it is difficult to use as the primary mechanism for broad vulnerability discovery. LLM and agent-based detection. Recent work uses LLMs to provide semantic context that is difficult to encode in fixed rules. VulRAG [18] retrieves vulnerability knowledge to guide analysis, IRIS [44] and LLift [40] combine LLM reason- ing with static-analysis results, and LLMxCPG [38] uses code property graphs to provide structured program context. More recent systems employ agentic workflows that can inspect code, reason about attack surfaces, and validate candidate vul- nerabilities [5,29,36,57,72,86]. These systems improve semantic reasoning but introduce a different scaling problem. Repository-wide exploration gives the agent a large search space and little guidance about where expensive reasoning is most useful. Deeper analysis of one region consumes bud- get that could otherwise cover additional code, and the result depends strongly on the capability of the underlying model. Moreover, an individual scan usually produces findings rather than reusable detection knowledge, so similar reasoning may need to be repeated on the next target. This motivates a differ- ent design: use persistent vulnerability knowledge to identify where to look, apply inexpensive analysis broadly, and reserve expensive semantic and runtime reasoning for a progressively smaller set of candidates. 3 2.3 Research Scope BUGSTONE-E2E targets vulnerabilities that follow recurring API-misuse patterns, where the security-relevant behavior is visible at or near a call site. Its scope does not include one-off design flaws, configuration errors, or vulnerabilities that require whole-program symbolic reasoning to identify. BUGSTONE-E2Eproduces candidate findings with structured static and runtime evidence, while human-in-the-loop disclo- sure decisions and patch acceptance remain separate pro- cesses. 3 System Overview Figure 1 summarizesBUGSTONE-E2Ein three parts. Part 1 converts public vulnerability history into reusable detection skills. Part 2 applies those skills to a target project and identifies candidates that remain plausible after static analy- sis. Part 3 raises the evidence requirement through runtime validation and verified remediation. The pipeline therefore moves from historical vulnerability knowledge to increas- ingly stronger evidence while applying expensive analysis to progressively fewer candidates. Part 1: From CVE History to Detection Skills. BUGSTONE-E2E treats CVE fixing commits as rich sources of reusable detection knowledge rather than project-specific repairs. It surveys 19,325 high-severity CVEs and verifies 2,710 fixing commits from open-source projects. From these commits,BUGSTONE-E2Eextracts reusable scan anchors, buggy and fixed patterns, verification contracts, and CVE provenance, while rejecting patterns that fail to generalize. It then consolidates related CWE categories to group rules by CWE family and language. The resulting knowledge base contains 1,033 production rules across 56 CWE families, packaged into 172 deployable bug detection skills. Part 2: Rule-Driven Detection. Given a target repository, Phase A enumerates rule-anchored call sites, and Phase A.1 removes false positives using lightweight filtering, leaving a reduced candidate set for semantic verification. However, this stage can still produce hundreds to thousands of candidates for medium-to-large projects. Phase B then decomposes this workload into independent tasks, assigning one agent to each candidate together with the corresponding rule. Instead of exploring or reasoning about the repository as a whole, each agent only needs to determine whether that specific candidate violates the rule and constitutes the reported bug pattern. This narrow task scope substantially lowers the required agent ca- pability, enabling effective verification using smaller, more cost-efficient models. Because these candidates are indepen- dent, Phase B can also process them with high parallelism, scaling semantic analysis to large candidate sets without re- quiring a shared state. Part 3: Validation and Remediation. The output of Part 2 is still a static claim, so Part 3 applies stronger evidence re- quirements only to the surviving candidates from Phase B. Phase C first re-triages these findings in batches and then attempts to construct a concrete execution environment for runtime validation. It records runtime evidence such as sink reachability, sanitizer violations, or crashes, while only find- ings that satisfy the stronger exploit-confirmation criterion are markedCONFIRMED_EXPLOITABLE. Failure to construct or exercise an environment remains distinct from evidence that a finding is not exploitable. This ordering keeps expensive runtime analysis focused on the small set of candidates that survive earlier stages. For findings with sufficient runtime evidence, Phase D gen- erates a minimal, scope-checked patch using the vulnerability evidence and program context. It then applies a two-sided dif- ferential test: the proof of concept must fail on the patched tree and succeed again after the patch is reverted. This prevents a broken build or invalid environment from being accepted as a successful repair. Finally,BUGSTONE-E2Efollows the project documentation to produce a vulnerability report in the maintainerâs required format, together with supporting evidence and a proposed patch. 4 Design BUGSTONE-E2Eis designed around three problems that arise when turning vulnerability history into an end-to-end detection workflow. First, public CVE records and fixing com- mits must be converted into reusable detection knowledge without overfitting to one patch or learning the fix rather than the vulnerability. Second, that knowledge must guide seman- tic analysis at repository scale without requiring expensive model reasoning over the entire codebase. Third, findings and generated patches must be supported by evidence that does not depend on the modelâs own judgment. These problems motivate the three research challenges below. The following subsections then describe howBUGSTONE-E2Eaddresses them. 4.1 Research Challenges RC1 (Turning vulnerability history into reusable detec- tion knowledge). A CVE record is not directly usable as a detection specification. Vulnerability descriptions are often brief, fixing commits may not be linked explicitly, and patches can mix the security fix with unrelated changes. Even when the correct fixing commit is known, the changed line does not necessarily identify the underlying vulnerability condition. For example, a patch may place a check, sanitizer, or guard away from the actual vulnerable operation. Directly trans- lating such diffs into rules can therefore produce detectors for the fix rather than for the vulnerable code. Additionally, generating one rule per CVE results in highly redundant, project-specific rules, whereas merging them aggressively can combine distinct root causes. The challenge is to isolate 4 Part 1: CVE Intelligence and Skill Development CVE MiningRule & Skill Synthesis Host Agent Part 2: Rule-Driven DetectionPart 3: Validation & Remediation Guest VM Survey CVEs 19,325 high-sev. Identify + Enrich OSS public repositories Clone + Date Filter commit window Light LLM Screen pre-filter patches Verify Fix Commit 2,710 confirmed Rule Synthesis extract + generalize Skill Synthesis 1,033 rules 172 skills tree-sitter Index Phase A Filter.py Phase A.1 LLM Agent Verify Phase B Triage + Exploit Phase C Patch + Verify Phase D Report + Package JSON + HTML optimize filter (1â2 rounds)re-run PoC skills + rules Figure 1: Overview of BUGSTONE-E2E. and validate the relevant fix, identify the security-relevant condition, and generalize it only when supported by multiple instances. RC2 (Scaling semantic analysis through task decompo- sition). Applying an extensive rule set to a large code- base can produce an enormous amount of semantic analysis. While rules provide explicit candidate locations, determining whether a candidate is truly vulnerable may require inspect- ing surrounding data flows, guards, sanitization, and project- specific conventions. Simply assigning one agent to each rule does not solve this problem. A single rule can match hundreds or thousands of sites, overwhelming the agent with a large context and a long sequence of rule-specific judgments. Such tasks place substantial demands on both model capability and context handling. Therefore, the challenge is to minimize the number of candidates that reach semantic analysis and to break down the remaining work into small, independent reasoning tasks. RC3 (Establishing evidence beyond model judgment). A modelâs conclusion that a candidate violates a rule is merely a static claim, not runtime evidence of an acutal vulnerability. Likewise, a patch generated by the same model cannot be considered correct simply because the model reports that it works. Both stages require evidence that can be checked inde- pendently of the model. For vulnerability validation, the exe- cution must distinguish observed runtime evidence, stronger exploit confirmation, unsuccessful tests, and cases where the target cannot be exercised. For remediation, verification must demonstrate that the triggering input no longer succeeds af- ter the patch, but the same input succeeds when the patch is reverted. The challenge is therefore to raise the burden of proof from structured static evidence, to runtime evidence, and finally to verified remediation while applying the more expensive checks only to the progressively smaller set of surviving findings, none of which is taken on the reporting agentâs word. 4.2 CVE-to-Skill Pipeline StageArtifactCount CVE surveyHigh-severity CVEs19,325 OSS enrichmentOpen-source CVEs5,902 Fix recoveryVerified fixing commits2,710 Case validationValidated CVE cases2,662 Rule synthesisPre-consolidation rules / CWEs1,757 / 266 Family consolidationProduction rules / CWE families1,033 / 56 Skill synthesisDeployable bug detection skills172 Table 2: CVE-to-skill pipeline and the number of artifacts retained at each stage. To address RC1,BUGSTONE-E2Eturns incomplete CVE records into deployable detection skills in three steps: it en- riches CVE metadata and recovers verified fixing commits, extracts reusable vulnerability conditions from those fixes, and consolidates related rules into CWE-familyâlanguage skills. Table 2 summarizes this progression. 5 CVE enrichment and fix recovery. CVE metadata is often insufficient to identify either the affected code or its fix. A record may contain only a short vulnerability description, omit the affected repository or fixing commit, or reference an issue, advisory, or third-party page rather than the relevant code change. CWE metadata can also be missing, ambiguous, or contain several weakness categories in one vulnerability. BUGSTONE-E2Etherefore enriches each record before rule extraction. Starting from 19,325 high-severity CVEs pub- lished between 2022 and 2026, it combines structured CVE fields, natural-language descriptions, referenced pages, and public repository information to resolve the affected open- source project and primary weakness category. When a reli- able fixing-commit reference is available,BUGSTONE-E2E verifies the cited commit directly. Otherwise, it uses the vul- nerability reporting and publication window to guide an agen- tic search over repository history. Candidate commits are checked against the CVE description, patch, and surrounding code to determine whether they actually fix the reported vul- nerability. This process yields 2,710 verified fixing commits, which form the evidence base for rule generation. Patch-guided rule extraction. A verified fixing commit is still not a detection rule. A patch may mix the security fix with unrelated changes, and the code added by the fix may be a guard or sanitizer rather than the operation that was originally unsafe. The goal is therefore to recover the underlying vulnerability condition rather than reproduce the textual shape of the patch. For each verified fix,BUGSTONE-E2Ejointly analyzes the CVE context, the vulnerable and fixed code, and the surround- ing functions. From this evidence, it identifies the vulnerable operation, candidate scan anchors, the condition that makes the operation unsafe, and the change that removes that con- dition. The resulting rule describes what should be detected in vulnerable code rather than what was introduced by the fix. A validation step checks this direction explicitly: scan anchors must refer to operations present before the fix and must not point to a sanitizer, guard, or replacement API added by the patch. Rules that cannot be supported by the available evidence or generalized beyond the individual patch are not promoted to the production rule set. Each accepted rule re- tains its source CVEs so that later findings remain traceable to the vulnerability evidence behind the rule. This process produces 1,757 pre-consolidation rules cover- ing 266 CWE categories. These numbers describe the mined rule set before related weakness categories and overlapping rules are consolidated. Rule consolidation and skill synthesis. CWE categories do not map one-to-one to distinct detection problems. Re- lated CWEs often describe different manifestations of the same underlying condition and consequently share scan an- chors and semantic checks. Keeping them independent would repeatedly enumerate and analyze the same code sites. BUGSTONE-E2Etherefore groups closely related CWE categories into broader CWE families and consolidates over- lapping rules within each family. The resulting production knowledge base contains 1,033 rules across 56 CWE fami- lies, reduced from 1,757 rules across 266 CWE categories. This consolidation removes redundant detection work while preserving rules for distinct vulnerability conditions. The consolidated rules are organized by CWE family and target language into 172 deployablebug detection skills. A skill is not a single rule; it is an execution template for one CWE-familyâlanguage combination and may contain multi- ple rules covering different APIs or variants of the same vul- nerability family. Its rule file stores rule-specific scan anchors and vulnerability conditions, while a skill file (SKILL.md) defines the semantic checks and evidence expected from Phase B. Each skill also contains a filtering script for inexpen- sive deterministic filtering. This filter may initially be empty and, when candidate volume justifies it, can be refined from Phase B feedback during large-scale scans as described in §4.3. The CVE-derived rules themselves remain unchanged. For example, the Python command-injection family is supported by multiple CVEs, where externally derived data reaches a shell-execution API without sufficient quoting or sanitization. Although the individual patches differ, they sup- port the same reusable condition. The corresponding skill therefore anchors on the relevant shell APIs and asks down- stream analysis whether attacker-controlled data can reach them without an adequate guard. 4.3 Layered Target Scanner Oncebug detection skillsare constructed, the remaining chal- lenge is to apply them to a large repository without turning ev- ery rule match into an expensive semantic or runtime analysis. BUGSTONE-E2Etherefore uses a layered scanner that raises both cost and burden of proof as the candidate set shrinks. Phase A performs broad build-free enumeration, Phase A.1 reduces redundant or repeatedly benign candidates, Phase B performs bounded semantic verification, Phase C establishes runtime evidence and exploitability, and Phase D verifies re- mediation. Build-freeenumerationandconservativeself- improvement (RC2).In Phase A,BUGSTONE-E2E parses source files with Tree-sitter [69] and constructs a shared index of call sites that is reused across all applicable skills. Each rule contributes explicit scan anchors, and Phase A enumerates matching call sites without invoking an LLM. For a small target, or for a rule that matches only a modest number of sites, these candidates can proceed directly to semantic verification. However, large repositories create a different problem. Common APIs such as memory allocation, deallocation, or string operations can appear thousands of times, making se- mantic inspection of every matching site unnecessarily expen- sive. When this occurs,BUGSTONE-E2Ecan enable an iter- 6 ative filter-synthesis loop (Phase A.1). A new bug detection skill may begin with an empty filter. Phase B then examines a bounded sample of candidates and identifies false positives. From these results, an agent summarizes recurring benign pat- terns that can be expressed using inexpensive local syntax or AST checks and updates the filter conservatively. The filter is then applied to subsequent candidates, whose Phase B results can in turn refine it further. Thus, semantic judgments from early candidates are distilled into cheap checks that reduce the workload for later candidates. The filter is deliberately asymmetric relative to the CVE- derived rule. It only removes patterns judged safe by deter- ministic checks and does not modify the ruleâs vulnerability condition or scan anchors. If a filter cannot determine that a candidate is benign, the candidate is retained. The refined filter is stored with the skill and can be reused in later scans. Phase A.1 also removes duplicate work that arises only af- ter rules are applied to a target. When overlapping rules within the same CWE family identify the same source location, the scanner keeps a representative candidate rather than sending multiple equivalent tasks to Phase B. Note that Phase A.1 reduces redundant work in a particular target, while rule con- solidation in §4.2 reduces redundancy in the knowledge base. Bounded semantic verification (RC2). The remaining can- didates require semantic reasoning because safety may de- pend on data flow, surrounding guards, sanitization, callers, or project-specific conventions. Assigning all candidates for one skill to a single agent would still create a large reasoning task: one rule may match hundreds or thousands of locations, forcing the agent to process extensive context and make many independent judgments in one session. BUGSTONE-E2Einstead decomposes verification into small, bounded tasks. Candidates are grouped only when they match the same rule in the same source file, allowing them to reuse rule instructions and source context without turning veri- fication back into a repository-scale task. The maximum num- ber of candidates assigned to one worker is configurable and can be adjusted to the capability of the Phase B model. Each worker receives the corresponding skill and candidate loca- tions and may inspect nearby code, callers, data flow, guards, and sanitizers as needed. It returns a verdict together with structured rule-specific evidence, such as the matched sink, relevant data flow, and guard or sanitizer analysis. Because these tasks are small and independent, they can be processed in parallel while placing substantially lower contextual and reasoning demands on each model invocation. Runtime validation and exploit confirmation (RC3). A Phase B finding provides structured static evidence, but it does not establish that the reported condition can be exercised at runtime. Phase C therefore moves each surviving finding into an isolated dynamic validation workflow. The first step constructs a reusable execution environment for the target. A LLM-based agent runs inside a dedicated VM or container, installs the required dependencies, builds the project, and enables sanitizers when applicable. Because this setup is shared by all later validation tasks for the same target, BUGSTONE-E2Eperforms it once and records a compact environment report describing how the target was built and exercised. Each Phase B finding is then validated independently by a Phase C agent using this prepared environment and its build report. The agent focuses on one candidate at a time, con- structs an input or execution path that reaches the reported op- eration, and looks for a vulnerability-specific runtime oracle, such as an ASan violation, crash, controlled sink reachabil- ity, or other observable security-relevant behavior. Dynamic validation is executed sequentially rather than concurrently to avoid interference between concurrent tests. After each candidate,BUGSTONE-E2Esaves the resulting evidence and restores the environment to its pre-test state before validating the next finding. Runtime evidence is recorded separately from stronger exploit confirmation, because observing the re- ported behavior does not, by itself, establish a working exploit. Thus, Phase C distinguishes among four categories: findings with runtime evidence, exploit-confirmed findings, findings that could not be proved exploitable, and cases that could not be tested due to an insufficient execution environment. Differentially verified remediation (RC3). For findings with a working proof of concept, Phase D attempts to produce a minimal, scope-checked patch using the vulnerability report and runtime evidence. Patch generation alone is insufficient because the model that writes a patch cannot certify its own fix. A one-sided test is also weak, because the proof of con- cept may stop working because the patch broke the build or disabled the relevant execution path rather than because it removed the vulnerability. BUGSTONE-E2Etherefore uses a two-sided differential test. It first applies the patch and reruns the same proof of concept, which must no longer reproduce the vulnerability. It then reverts the patch and runs the proof of concept again, which must reproduce the original behavior. Only patches that satisfy both directions are marked as verified. The final output packages the maintainer-format vulnerability report together with the proof of concept, runtime evidence, proposed patch, and verification artifacts. This completes the progression from structured static evidence, to runtime evidence and exploit confirmation, and finally to verified remediation. 5 Evaluation We evaluateBUGSTONE-E2Ealong the main design ques- tions introduced in §4. We first examine how effectively the CVE-to-skill pipeline converts public vulnerability history into deployable detection knowledge. We then evaluate the target scanner through controlled comparisons, reproducibil- ity experiments, and cost measurements. Real-world findings and runtime validation results are reported separately in §6. 7 5.1 Experimental Setup Hardware. All experiments run on a dual-socket server with two Intel Xeon Gold 6336Y processors, 96 hardware threads, and 2 TB of RAM. Phase A uses up to 96 threads for deterministic enumeration, while model-based stages are bounded by API rate limits and the concurrency configured for each experiment. Models and agent harnesses. Unless otherwise stated, Phase B usesgpt-5-mini. Controlled experiments addi- tionally evaluate GPT-5.5, GPT-5.6-sol, Claude Sonnet 4.6, Kimi-K2.5, and other models where noted.BUGSTONE-E2E supports multiple coding-agent harnesses, including Claude Code, Codex, and Pi. For the cross-harness study in §5.4, we compare against three recent agentic security harnesses: the OpenAI/Codex security-scan harness [57], Anthropicâs open-source defending code reference harness [5], and the Visa Vulnerability Agentic Harness [72]. Evaluation targets. We use different targets for different controlled experiments. wolfSSL is rolled back to a pre-fix revision for the cross-harness comparison in §5.4. Pillow is similarly rolled back and used for the reproducibility and model studies in §5.5. The larger deployment corpus used to measure end-to-end scanning is described separately in §6. This separation prevents findings accumulated across repeated production scans from being mixed with controlled experimental results. 5.2 CVE Knowledge Construction We first evaluate the CVE-to-skill pipeline of §4.2. Starting from 19,325 high-severity CVEs published between 2022 and 2026, the pipeline identifies 5,902 records associated with public open-source repositories and verifies 2,710 fix- ing commits. Patch-guided synthesis produces 1,757 pre- consolidation rules spanning 266 CWE categories. Family- level consolidation reduces this knowledge base to 1,033 production rules across 56 CWE families, organized into 172 language-specificbug detection skills. Table 2 summarizes the full progression. Recovering fixes beyond structured CVE metadata. Of the 19,325 high-severity CVEs surveyed,BUGSTONE-E2E enriches 6,448 records using structured metadata, natural- language descriptions, and external references. This process resolves 5,902 records to open-source projects with identifi- able GitHub repositories. Among these 5,902 records, 2,439 contain a direct commit URL, from whichBUGSTONE-E2E verifies 1,944 fixing commits. For records with a resolved repository but no usable commit pointer,BUGSTONE-E2E searches repository history using the CVE context and vul- nerability timeline, recovering 766 additional fixing commits. Together, the direct-reference and search paths yield 2,710 verified fixing commits. Thus, 766 of the final fixes (28.3%) come from repository-history recovery and would not be avail- able from direct commit references alone. The remaining 3,192 resolved records did not yield a veri- fied fixing commit, which illustrates why this recovery step is difficult. In some, repository search finds no commit that can be confidently linked to the vulnerability; in others, the relevant time window contains either no useful commit his- tory or too many plausible changes to verify reliably. Thus, agent-assisted enrichment expands the usable evidence base, but does not replace explicit patch references. From fixing commits to reusable rules. Verification further reduces the corpus because not every security patch provides reusable detection knowledge. Some patches contain mostly unrelated refactoring, lack a clear vulnerable operation, or describe conditions that are too project-specific to generalize. After validation, 2,662 CVE cases provide usable rule evi- dence, from whichBUGSTONE-E2Esynthesizes 1,757 rules across 266 CWE categories. Each accepted rule retains links to its source CVEs, preserving provenance through later con- solidation and scanning. The raw rule set is intentionally broader than the deployed knowledge base. Closely related CWE categories frequently encode overlapping detection conditions and anchors, which would otherwise cause the same target site to be scanned repeatedly. Family-level consolidation therefore merges re- lated CWE categories and redundant rules, reducing the store from 1,757 rules over 266 CWEs to 1,033 rules over 56 CWE families. These rules are packaged into 172 CWE-familyâ language skills. Coverage across languages and weakness families. The mined rules span both memory-safety and higher-level API- misuse patterns. Before consolidation, the largest language- specific rule sets include Python, PHP, Go, JavaScript, Java, TypeScript, C, and C++. The same weakness can also appear across languages. For example, path traversal, injection, de- serialization, and related families produce language-specific rules with different APIs but similar underlying security con- ditions. Conversely, C and C++ contribute more heavily to memory-safety families. This coverage follows the scope defined in §4.1. BUGSTONE-E2Etargets recurring vulnerability conditions that can be anchored at or near security-relevant API use. Vul- nerabilities that depend primarily on global protocol state, race conditions, or other API-independent logic may there- fore have no corresponding rule even when they appear in the CVE corpus. 5.3Scanner Efficiency and Task Decomposi- tion We next evaluate the two design choices behind RC2: reduc- ing the number of candidates that require semantic analysis and bounding the semantic work assigned to each Phase B 8 worker. Candidate reduction before semantic analysis. Across 15 targets, deterministic filtering reduces the aggregate Phase A candidate pool from approximately 745K to 293K, removing 60.6% of candidates before model-based verification. The benefit is largest for high-volume rules: on FreeBSD, for ex- ample,cwe122-cppalone produces 65.5K call sites, which filtering reduces to 42.0K. On smaller or lower-volume tar- gets the reduction can be modest, supporting the design of Phase A.1 as an optional cost-control layer rather than a pre- requisite for scanning. Bootstrapping filters from Phase B feedback. We further evaluate filter synthesis on 10 high-volume skills from four targets, using up to 100 Phase B-labeled candidates per skill (979 total). Of these candidates, 814 are labeled false positive. The first synthesis iteration produces conservative filters for seven skills and removes 261 candidates, corresponding to 32.1% of the observed false positives; for the remaining three skills, no deterministic pattern satisfying the safety criteria is found. Applied to the full candidate pool, these newly synthe- sized filters further reduce the workload from approximately 299K to 293K candidates. During synthesis, candidates labeledBUGGYorUNKNOWNare protected: all 165 such candidates in the synthesis samples survive the accepted filters. This is a construction-time safety check rather than an independent recall measurement, but it prevents the synthesis procedure from accepting filters that contradict available positive or uncertain evidence. Bounded semantic tasks. Filtering controls how many candi- dates reach semantic verification; task decomposition controls how much reasoning each worker performs. Phase B groups only small batches from the same rule and source file, avoid- ing repository-scale reasoning while retaining shared rule and source context. We evaluate the resulting model requirement separately in §5.5, where multiple models verify the identical set of 8,115 Pillow candidates. Together, these two mecha- nisms reduce both the volume and the scope of model-based analysis. 5.4 Cross-Harness Comparison on wolfSSL Rolled-back target and five configurations. We chose wolfSSL, an embeddable TLS/DTLS and cryptography li- brary, because it is heavily audited and is the target of An- thropicâs Mythos and Glasswing disclosure work, so its open findings give a ready basis for comparingBUGSTONE-E2E with independent harnesses. We ran all five configurations on it under their own default settings. The target is commit 0c4ca257a07a(wolfSSL 5.8.4), deliberately rolled back to a pre-fix state. Two areBUGSTONE-E2Eunder the Pi agent with gpt-5-mini and with Kimi-K2.5, and Phase A delivers exactly the same 6,099 candidates to Phase B in both. This subsection quotes that later Pi run throughout, whereas the wolfSSL row of §6 quotes an earlier registry run with a smaller skill set and 3,166 candidates. The three baselines are the OpenAI/Codex security-scan harness with GPT-5.5 [57], An- thropicâs open-source defending code reference harness with Claude Opus 4.8 [5], and the Visa Vulnerability Agentic Har- ness (VVAH) with Claude Opus 4.8 [72]. Expanded ground truth.Recall was originally mea- sured against nine disclosed cases from Anthropicâs Mythos coordinated-disclosure program 1 , and nine points cannot sep- arate a 40% detector from a 55% one. We therefore rebuilt the ground truth from the vendor release notes for 5.9.0, 5.9.1, and 5.9.2, the three releases published after the snapshot. Those notes name 68 CVE identifiers, of which 64 localize into the rolled-back tree by aligning each fixâs pre-image against the snapshot source, and the other four are excluded with a recorded reason. A harness finding counts as a hit when it names the same file and the same enclosing function as a localized fix hunk. We calibrated the automatic file-and- function matcher against the existing hand adjudication of the nine Mythos cases across the seven harness configurations recorded at calibration time, which gives 63 labeled decisions, and it agrees on 59 (93.7%). Of the four disagreements, two are false positives that credit a baseline with a case its author did not identify, so the recall gap below is conservative. Recall on 64 cases. Figure 2 breaks recall down by case set, and the expansion confirms the ordering the nine cases gave rather than changing it.BUGSTONE-E2Ewith gpt-5-mini leads at 33 of 64 (52%), the union of its two model configura- tions reaches 35 (55%), the Codex harness reaches 22 (34%), the Anthropic harness 15 (23%), and VVAH 3 (5%). Recall on the 55 later-fixed CVEs tracks recall on all 64 within a few points for every system, so the expansion did not select a favorable population. The expansion also corrects the earlier result in the baselinesâ favor. Specifically, the Anthropic har- ness and VVAH score 0 of the 9 Mythos cases yet match 15 and 3 of the 64 localized CVEs, almost all of them among the 55 later-fixed ones, so reporting only the nine Mythos cases understated both. The disclosed CVEs are a lower bound on true positives rather than the full set, since a harness can re- port a real bug that no vendor advisory yet covers. Each bar spans the systemâs total reported findings, and the trailing count restates that volume, which the lightest segment beyond the matched cases makes visible. The twoBUGSTONE-E2E models reported 103 and 70 findings, above every baseline, since the Codex harness reported 52 and neither the Anthropic harness nor VVAH exceeded 26. No harness subsumes another. 43 of the 64 cases are found by at least one system against 35 for the twoBUGSTONE-E2E configurations together, so the baselines contribute eight cases BUGSTONE-E2Emisses. These systems are better read as an ensemble than as competitors on a single ranking. 21 of the 64 cases are found by nobody. Four of those sit in hand-written 1 https://red.anthropic.com/2026/cvd/ 9 020406080100 reported findings (64 known CVEs localized) Bugstone gpt-5-mini Bugstone Kimi-K2.5 Codex GPT-5.5 Anthropic Opus 4.8 VVAH Opus 4.8 33 (52%) matched, 103 found 26 (41%) matched, 70 found 22 (34%) matched, 52 found 15 (23%) matched, 19 found 3 (5%) matched, 26 found Mythos GT (of 9)later-fixed CVEs (of 55)other findings Figure 2: Recall on the rolled-back wolfSSL 5.8.4 snapshot, over the 64 post-snapshot CVEs that localize into the snapshot tree. Each bar spans the systemâs total reported findings, split into the nine hand-adjudicated Mythos ground-truth cases matched, the 55 later-fixed CVEs matched, and the remaining findings that match no disclosed CVE. The two matched segments therefore give the recall count, and the full bar gives reported volume, since the disclosed CVEs are a lower bound on true positives rather than all of them.BUGSTONE-E2E rows count Phase B BUGGY output. assembly and a header that no harness here parses, so part of the residual miss set is a shared input-language limit rather than a reasoning failure. The one Mythos case no system reports is CVE-2026-5446, ARIA-GCM explicit-IV/nonce reuse, and that miss is instructive rather than incidental. Nonce reuse is a stateful cryptographic invariant spanning successive record encryptions, not the misuse of a dangerous sink at one call site. Therefore no rule in the CVE-derived rule base expresses it, which is the scope boundary already declared in §5.2. Caveats. Three limits bound this comparison. First, ground truth is vendor-disclosed CVEs only and the clone we mined ends at 2026-06-24, so silently fixed bugs and later fixes are absent. Second, the match rule is enclosing-function identity, which is coarser than root-cause identity. Third, this is a single C/TLS target chosen because ground truth exists for it, so the recall ordering above is not a general ranking of these harnesses. 5.5 Run-to-Run Reproducibility Ten identical runs. Because Phase B relies on an LLM agent, identical scans need not produce identical findings. We therefore rolled Pillow back to commitd56032047d11and scanned it ten times with the same runner andgpt-5-mini model, using a frozen 174-skill snapshot of the rule base (the current production library contains 172 skills). All runs enumerate the same 30,594 raw matches and remove the same RunRaw FP Unk. Final Excl. Tokens run1246 7,84261388731M run2245 7,857131439690M run3235 7,87281318696M run4240 7,87051314699M run5219 7,8751413310794M run6220 7,89051355679M run7196 7,91541321765M run8261 7,85041474718M run9233 7,872101285743M run10235 7,861191251716M Table 3: Ten independent runs on the same Pillow snapshot with an identical configuration. Raw:BUGGYjudgments before collapsing by location; Unk.:UNKNOWNverdicts. run1 and run5 additionally left 21 and 7 candidates unjudged, so their rows sum to fewer than 8,115. Final: distinctBUGGYlocations; Excl.: locations this run reports and no other run does. 21,584 with the deterministic Phase A.1 filter, leaving 9,010 candidates that dedup to an identical set of 8,115 delivered to Phase B in the same 1,345 batches. Any variation after this point therefore comes from semantic verification rather than candidate delivery. Despite identical input, individual runs report 125â147 find- ings (Table 3). Their union contains 243 distinct locations, while only 56 appear in all ten runs. A typical run recovers 55.3Thus, aggregate finding counts are relatively stable, but the identity of the reported findings is not. A stable core and a stochastic tail. The variation is highly structured rather than uniform. Of the 243 findings in the ten-run union, 56 appear in every run and 55 appear in only one; the remaining 132 fall between these extremes (Fig- ure 3b). At the candidate level, 93.1% of the 8,115 candi- dates receive the same verdict in all ten runs onceUNKNOWN abstentions are counted as changes; restricting to outright BUGGYâFALSE_POSITIVEflips leaves 509 contested candidates (6.3%). Either way, disagreement is concentrated in a rela- tively small set of borderline cases. Findings supported by multipleBUGGYjudgments within a run are also much more stable across runs: 73.0% are unanimous across all ten runs, compared with 15.8% for findings supported by a single judg- ment. This suggests that within-run corroboration provides a useful indicator of expected stability without requiring an- other scan. Reruns improve coverage with diminishing returns. Re- peating Phase B expands the finding set, but the marginal gain decreases quickly. Averaged over all subsets of the ten runs, three runs recover 75.8% of the ten-run union and five re- cover 85.7% (Figure 3a). The second run adds an expected 32 findings, the third adds 18, and later runs contribute progres- sively fewer. A Chao2 incidence-based extrapolation [13,16] over the runs-by-locations incidence matrix estimates approx- imately 302 Phase B-reportable locations under this harness, suggesting that even ten runs recover only about 80.4% of 10 135710152025 number of independent runs 100 150 200 250 300 distinct findings (union) Chao2 estimate Ě S=302 1 run: 134 (55% of 10-run union) 10 runs: 243 +32 +18 +13 +11 +9 +8 +7 +6 +5 (a) how much a rerun adds best / worst subset of runs expected union (all subsets) extrapolation 12345678910 reported by how many of the 10 runs 0 20 40 60 80 distinct findings stochastic tail: 55 found once reproducible core: 56 in every run (b) reproducible core vs. stochastic tail observed single-probability model (p=0.55) 020406080100120140 distinct findings from one run GPT-5.5 GPT-5.6-sol GPT-5.6-luna Gemini 3.1 Pro Sonnet 4.6 Qwen3-6-35B-A3B n=127; 1 run 78%, 10 runs 99% n=121; 1 run 78%, 10 runs 99% n=125; 1 run 72%, 10 runs 97% n=36; 1 run 41%, 10 runs 69% n=8; 1 run 90%, 10 runs 100% n=31; 1 run 94%, 10 runs 97% gpt-5-mini 1-run mean 134 (c) one-run findings and cost by model in all 10 gpt-5-mini runs (stable core) in some runs (>=1, not all) in no run (unique) 1 random gpt-5-mini run reaches (mean, min-max) 05001000 per-run API cost (USD) $900 $869 $71 $575 $330 free gpt-5-mini mean $161 12345710 number of runs 0.0 0.2 0.4 0.6 0.8 1.0 union / harness's own total 56 in all 10 (reproducible core) 0 in all 10 (no core) (d) rerun diversity by harness Bugstone (n=10, ratio 0.23) Codex (n=10, ratio 0.00) Figure 3: Reproducibility of Phase B on the same Pillow snapshot. (a) Expected finding coverage from repeatedgpt-5-mini runs. (b) Finding frequency across ten runs. (c) Findings and API cost for six alternative models on the same 8,115 candidates(d) Growth of the finding union across reruns of BUGSTONE-E2E and the OpenAI Codex security harness. them. This 302 is the population reachable by this rule set and candidate stage, not an estimate of the true number of vulnerabilities in Pillow. Repetition is therefore useful for ex- panding coverage, but no small number of reruns eliminates the stochastic tail. Where does the variation come from?. The variation is almost entirely due to verdict changes rather than worker or reporting failures. Across the 1,087 missing occurrences relative to the ten-run union, 1,061 (97.6%) areBUGGY-to- FALSE_POSITIVEflips, 26 are abstentions, and none result from worker failure. These flips are also correlated within a semantic task: the 509 contested candidates occur in only 153 batches, with strong within-batch correlation (Ď= 0.71) and essentially no correlation across batches (Ď=â0.001). Thus, a bounded batch behaves partly as one shared reasoning decision, rather than as several independent candidate classifi- cations. This observation motivates keeping Phase B batches small even when candidates share rule and file context. Reproducibility is not recall. More findings from repeated runs do not necessarily imply proportionally higher vulner- ability recall. The rolled-back Pillow snapshot contains 21 vulnerabilities disclosed and fixed after the selected revision, but only five reach Phase B as candidates. Across ten runs, Phase B reports four of those five at least once, while a typical run reports only one. The remaining sixteen are lost before Phase B: fourteen are never enumerated at all, eight of them because the deployed skill set has no rule for their CWE, and two more are enumerated but removed by the Phase A.1 filter. Repetition can therefore recover semantic-verification misses, but it cannot compensate for missing detection knowledge or candidate coverage. Rerunning a small model versus using a stronger model. We next run six additional models once each on the same 11 Model$/1M in/outTokensFindingsCost gpt-5-mini0.25 / 2.00723M134$161 gpt-5.6-luna0.20 / 1.20479M125$71 sonnet-4.63.00 / 15.0166M8$330 gemini-3.1-pro2.00 / 12.0311M36$575 gpt-5.6-sol5.00 / 30.0311M121$869 gpt-5.55.00 / 30.0309M127$900 qwen3-6-35b-a3bself-hosted346M31free Table 4: Per-run Phase B cost on the same Pillow snapshot and candidates. $/1M: listed input and output prices per million tokens. Cost is provider-billed for the whole run and includes retries and cached-input tokens priced well below the listed input rate, so it is not the product of the Tokens column and the in/out rates.gpt-5-miniFindings and Cost are the ten-run mean; the other rows are a single run. Prices are the provider list rates as of 2026-08. free: Qwen3-6-35B-A3B runs on a local A100, so it incurs no per-token API charge. Pillow snapshot and the same 8,115 Phase B candidates. A singlegpt-5-minirun already covers 78% of the findings reported by GPT-5.5, 78% of GPT-5.6-sol, and 72% of GPT- 5.6-luna, close to the 79% overlap between twogpt-5-mini runs. Across tengpt-5-miniruns, coverage of these stronger- model finding sets rises to 97â99%, leaving at most four findings unique to any one model (Figure 3c). Several frontier configurations are also substantially more expensive (Table 4). Onegpt-5-minirun averages approx- imately $161, compared with about $900 for GPT-5.5 and $869 for GPT-5.6-sol on the same candidate set, while report- ing a comparable or larger number of Phase B findings. For this bounded verification task, spending the same budget on several small model runs therefore provides greater Phase B finding coverage than replacing the small model with a sin- gle frontier-model run. This result supports the RC2 design choice: once semantic analysis is decomposed into narrow candidate-level tasks, Phase B does not require the strongest available model for every judgment. Harness dependence and limitations. The benefit of re- running is not universal to all agentic scanners. Ten runs of the OpenAI Codex security harness on the same Pil- low snapshot show substantially less recurring overlap than BUGSTONE-E2E(Figure 3d), indicating that rerun behav- ior depends on how a harness structures its reasoning tasks. We treat this comparison as a diversity observation rather than a like-for-like recall result because the harnesses expose findings at different stages. Finally, this study covers one target and one primary Phase B configuration, and the repeated runs stop before dy- namic validation. The additional findings in the stochastic tail therefore should not be interpreted as confirmed vulner- abilities. Instead, the experiment measures the stability and coverage behavior of the semantic-verification stage itself. 5.6 Remediation Prototype Patch quality is difficult to evaluate automatically: whether a repair is complete, regression-free, and acceptable to main- tainers often depends on project-specific tests and human review. We therefore evaluate Phase D only as a prototype for exploit-guided remediation rather than as a general patch correctness system. We apply Phase D to 23 findings across 3 targets for which Phase C produces a reproducible proof of concept. Only one of these targets, the openai-python SDK (11 of the 23 find- ings), is an independent third-party codebase; the other two are a small internal Flask exercise and a C++ vulnerability- hunting exercise. For each finding, Phase D generates a patch and applies the two-sided differential test from §4.3: the proof of concept must reproduce on the original program, stop re- producing after the patch, and reproduce again after the patch is reverted. Phase D produced a patch for all 23 findings, and all 23 pass this differential check. The resulting patches are compact, with a median of 8 changed lines (6 added, 2 removed), and 21 of the 23 touch a single file. These results provide machine-checkable evidence that the current prototype can turn a live finding into a compact candidate remediation that blocks the demonstrated behav- ior. The differential outcome is the workerâs own transcribed runtime output rather than an execution we re-ran, and this small sample does not measure general patch correctness or maintainer acceptance, which require substantially broader project-specific testing and review. 6 Real-World Bug Detection BUGSTONE-E2Escanned 14 third-party projects and recorded every finding in a deduplicating registry. §6 re- ports the outcome per target, from the Phase A candidate pool through the Phase C verification tiers. Each row is one project, and the finding counts are deduplicated across every configuration run on it, so a finding seen by two configura- tions is counted once. The registry omits Pillow, which is scanned only for the reproducibility study of §5.5. The scan portfolio by verification tier. The columns of §6 are strictly nested, so a reader can pick an evidentiary bar and read off the count. EveryCONFIRMED_EXPLOITABLE finding has a live proof-of-concept, and every live proof- of-concept is a runtime observation. Across the 14 targets, BUGSTONE-E2Ededuplicated 2,933 Phase B findings, of which Phase C adjudicated 2,125 and 644 carry runtime evi- dence, meaning a sanitizer report, a crash, or a controlled-sink signal at the reported location. Runtime evidence is the head- line bar we report, and the stricter live-proof-of-concept and CONFIRMED_EXPLOITABLEtiers in §6 narrow it further for read- ers who want them. Of the 2,125 findings Phase C adjudicated, it rejected 1,099 as not a bug or not exploitable and left 382 with a static-reachability argument but no runtime signal, so 12 TargetLang.DomainCand.Phase BPhase CRuntimeLiveConf. FreeBSDC/C++OS kernel76,8432,1891,4854594016 PyTorchPythonML framework38,605295195959527 OpenSSLC/C++Crypto lib.20,3572016552 WordPress ⥠PHP/JSCMS8,6361818000 NVIDIA OGKCGPU driver6,4492626880 ImageMagickC/C++Image tooling5,161162162442 wxo-clientsPythonEnterprise SDK3,907464641410 guavaJavaJava core lib.3,8432222000 opensshCSSH impl.3,44111000 wolfSSL â CEmbedded TLS3,166313116164 openai-pythonPythonAPI client8413434000 MCP serversTS/JSAgent servers7393636121210 codex-srcPython/RustCoding agent1,1504747220 claude-codePython/TSCoding agent58666211 Total2,9332,12564422462 Table 5: Scanned targets, ordered by candidate volume, with deduplicated finding counts from theBUGSTONE-E2Eregistry. Cand. is Phase A candidates after the Phase A.1 filter; Phase B distinctBUGGYlocations; Phase C findings reaching Phase C; Runtime a sanitizer, crash, or PoC signal at the sink; Live an exploitability ofLIVE_POC; and Conf. the strongestCONFIRMED_EXPLOITABLE tier. Each tier is strictly stronger than the one to its left. â wolfSSL is the cross-harness target of §5.4; this row quotes an earlier registry run with a smaller skill set (3,166 candidates) rather than the later rolled-back Pi run analyzed there. ⥠Five WordPress stored-XSS reports share one file and line and collapse to a single registry record, so the row is reported at its true 18. 644 reached the runtime bar. The gap between the Phase B and Phase C columns is a separate matter: it is backlog, not a negative result. Specifically, 808 findings have never been given a runtime at all, 704 of them in FreeBSD. Reporting the rejected set and the untested backlog separately is what keeps ânot exploitableâ distinct from âcould not be tested,â so the runtime-evidenced counts are lower bounds set by how much Phase C compute we spent, not by how many findings survive scrutiny. Phase A enumeration. Summed across the 14 targets, the Cand. column of §6 totals 173,724 post-filter candidates that reach Phase B, after the deterministic Phase A.1 filter has already removed benign sites without any model call. Each candidate then reaches Phase B, which follows the callers, weighs every guard on the taint path, and returns aBUGGY verdict only with code evidence. Phase B and Phase C findings. Runtime-evidence rates dif- fer sharply by target and by weakness class. On guava, all 22 Phase B deserialization candidates reach Phase C, but none yet carries runtime evidence, so guava contributes no findings at the runtime bar. The single openssh finding, a CWE-190 integer overflow insetenv.c:183, illustrates precision in a widely audited codebase, since 3,441 candidates narrow to exactly one Phase B finding, still unverified dynamically. Py- Torch concentrates on RCE-class deserialization bugs, with 195 findings reaching Phase C and 95 carrying a runtime observation. FreeBSD dominates raw volume, with 2,189 Phase B findings against a 76,843-candidate pool. Only 459 have a runtime signal so far, because the Phase C backlog there is the largest. False positives cluster in three shapes. Dominant false- positive sources fall into three categories. Benign wrappers are internal utility functions that call a dangerous API with a constant or already-validated argument, where the Phase A pattern matches but Phase B identifies the effective guard. Machine-generated code is the second category. For exam- ple, opensshâs libcrux_mlkem768 ML-KEM implementation containsâ635 syntactic buffer-write matches, and all were correctly dismissed because the generated code uses bounded index arithmetic throughout. Incomplete taint chains are the third category, in which the dangerous sink exists but user- controlled input does not reach it within the analyzed call depth. Phase B labels theseFALSE_POSITIVEwith a taint- chain explanation, so the structured evidence format enables principled labeling rather than a bare verdict. CWE families and languages. Runtime-evidenced findings concentrate in families with deep rule coverage. CWE-502 in- secure deserialization dominates PyTorch and guava, CWE-79 cross-site scripting dominates WordPress, and the C memory- safety families CWE-120, CWE-122, CWE-125, CWE-787, and CWE-416 account for the OpenSSL and FreeBSD find- ings. CWE-22 path traversal, CWE-77 and CWE-78 com- mand injection, and CWE-918 server-side request forgery carry runtime evidence in wxo-clients under multiple con- figurations. The runtime-evidenced findings therefore span injection and deserialization as well as memory-safety fami- lies, and they cover six languages from C to TypeScript. This spread matches the cross-language coverage of the rule base in §5.2. The scan model shifts volume, not runtime evidence. The scan model changes how many candidates Phase B marks BUGGY, but Phase C then filters that raw volume down to what a runtime can support. Under the controlled Pillow compar- ison of §5.5, larger models and gpt-5-mini report compara- 13 ble Phase B finding counts on the same candidate set, so a stronger scan model mainly reshapes which candidates are flagged rather than how many survive runtime checking. The extra candidates a noisier model raises are not free: each one still consumes Phase C compute, and the 808-finding Phase C backlog above is exactly the cost of that raw volume. Split- ting Phase B into a simple per-candidate judgment keeps each judgment cheap enough for a small model, while the dy- namic Phase C confirmation carries the load-bearing runtime evidence. ThereforeBUGSTONE-E2Edoes not depend on the strongest and most expensive model to produce runtime- evidenced findings. A harness that concentrates all reasoning in one large model has the opposite property. 7 Discussion 7.1 Limitations Detection scope and completeness.BUGSTONE-E2Eis designed for recurring vulnerability patterns that can be an- chored to security-relevant operations or API uses. It therefore does not target one-off design flaws, configuration errors, or global and stateful invariants without a suitable local anchor. CVE-derived rules may also remain incomplete. A rule can miss semantically equivalent APIs or conditions not repre- sented in the source fixes. In addition, conservative candidate filtering and bounded Phase B reasoning can introduce false negatives, particularly when a decision requires deeper in- terprocedural context. Because Phase B is model-driven, its verdicts are also stochastic; reruns can recover some semantic- analysis misses but cannot recover vulnerabilities outside the deployed rule and candidate coverage. Runtime validation. Runtime confirmation depends on the target admitting a reproducible build and test environ- ment. Complex dependencies, unavailable inputs, or platform- specific behavior can prevent Phase C from exercising an otherwise plausible finding. Such cases should therefore be interpreted as unverified rather than benign. Dynamic valida- tion is also necessarily more expensive than static verification, so large Phase-B finding sets may only be partially exercised. Remediation guarantees. Phase D remains a prototype. Its differential oracle establishes that a patch blocks the demon- strated proof of concept and that reverting the patch restores the behavior. It does not establish that the repair is complete, regression-free, or acceptable to maintainers. Evaluating these properties at scale requires project-specific testing and sub- stantial human review, so we treat remediation as an end- to-end capability rather than a measured patch correctness result. 7.2 Future Work Future work can extendBUGSTONE-E2Ealong these limita- tions. Broader CVE coverage and rule auditing can improve generality across CWE families, while richer interprocedural context can increase detection coverage beyond locally an- chored cases. Runtime validation can likewise benefit from more reusable build environments and broader confirmation of Phase B findings. Remediation requires substantially more project-specific and human-in-the-loop evaluation: future ver- sions of Phase D can incorporate native regression tests, as- sess whether patches address the complete root cause, and track maintainer feedback as more patches are submitted. We therefore view the current remediation stage as an evolving prototype rather than a complete automated repair system. 8 Related Work Static and query-based vulnerability analysis. Code prop- erty graphs, QL and CodeQL, Semgrep, and Infer enumer- ate candidates from human-authored queries or rules [6,7, 10,28,62,84]. Recent work attacks that authoring bottle- neck. Specifically, QLCoder synthesizes CodeQL queries from CVE metadata, and QRS generates queries agentically and validates findings by exploit synthesis [70,73]. Sem- Taint likewise extracts per-package taint specifications for CodeQL [27]. ThereforeBUGSTONE-E2Eis not the first sys- tem to turn CVE history into executable detectors. However, BUGSTONE-E2Eemits CWE- and language-level rules that group verified fixing commits, keepsource_cvesprovenance, and carry an agent-consumed verification contract. Vulnerability clone and patch-signature detection. ReDe- Bug, VUDDY, MVP, MOVERY, V1SCAN, and VMud reuse vulnerability history through code, component, or patch-line signatures [30,31,35,76,77,82]. MAVM replaces those sig- natures with agents that detect, confirm, repair, and validate recurring cases [88]. In contrast,BUGSTONE-E2Ematches CWE- and API-level rules, so a target need not clone the seed vulnerability, and confirmation requires an executed exploit. Patch analysis and rule inference. Empirical studies and datasets such as BigVul show that security patches encode security properties [19,39]. Prior work infers those prop- erties for missing checks, security impact, disordered error handling, and severity prioritization [48,78â80]. Closer to BUGSTONE-E2E, VulGenie derives Java API rules by attack- defense cross-analysis, while RuleForge and RulePilot con- vert CVE records and analyst annotations into validated de- tection rules [14,25,74]. In addition, AutoTrace turns one fixing commit into a trigger location, and GONDAR sup- plies CWE-specific sink knowledge to exploit agents [22,92]. However,BUGSTONE-E2Ediffers in the unit and scale of reuse, since it groups public CVE history into CWE-language rules packaged as deployable agent skills. 14 Learning-based vulnerability prediction. VulDeePecker, SySeVR, Devign, ReVeal, LineVul, DeepDFA, and FVD- DPM learn vulnerability signals from gadgets, graphs, or transformer encodings [12,23,42,43,64,66,89]. That line now extends into agent training. For example, VulAgentRL rewards only verdicts whose evidence checks against a code property graph, and Antares distills compact localization mod- els [41,71]. However, these systems predict over learned representations, whereasBUGSTONE-E2Ekeeps CVE prove- nance per rule and demands taint, guard, and caller rationale. LLMs for vulnerability detection. LLift, Vul-RAG, IRIS, LLMxCPG, and specialized reasoning models guide LLMs with static analysis, graphs, or retrieved vulnerability knowl- edge [18,38,40,44,54]. Benchmark studies measure that behavior across model sizes, prompts, repositories, and agent settings [45,55,68,85]. Repository-scale agents followed quickly. Specifically, TitanCA reports a deployed match, fil- ter, inspect, and adapt pipeline, LLMVD.js exploits taint bugs in Node.js packages, and DREA separates exploration from reasoning [53,67,87]. Evaluation moved with it, since Vul- nGym, SastBench, and RealVuln supply traces, triage dis- tributions, and scanner rankings [20,32,59]. Also, agen- tic filtering removes most SAST noise yet suppresses true positives, and a Vul-RAG replication reports an accuracy plateau [34,83]. ThereforeBUGSTONE-E2Elayers filtering into a cost-ordered cascade, in which Phase A.1 rejects deter- ministically and only survivors reach an agent. Agentic program repair and self-validating patches. Agen- ticRepair assembles repair context through subagents, while KeaRepair grounds patches in verified facts and mined vulnerabilityâpatch pairs [11,24]. However, acceptance cri- teria are fragile, since adversarial issue reports yield newly vulnerable patches and most Java CVE patches fail stricter or- acles [2,15]. Execution-based patch acceptance therefore has prior art. Specifically, VulnRepairEval requires the original exploit to fail, Vul4Py adds a functional oracle, and Veri- Port chains evidence across affected versions [9,26,75]. In contrast, Phase D patches findings thatBUGSTONE-E2Edis- covered itself, so acceptance needs a two-sided test in which the exploit fails under the patch and succeeds after reverting it. Exploit generation, dynamic confirmation, and agent de- terminism. Big Sleep, ATLANTIS, and the AIxCC sys- tematization show agents finding and repairing real bugs alongside fuzzing and symbolic execution [29,36,86]. K- Repro, SEC-bench Pro, and CVE-Bench measure proof-of- concept construction on kernel N-days, browser engines, and web CVEs [37,60,91]. However, agents still fail on service- based vulnerabilities whose environments they cannot stand up [47]. RECEIPT restores verdict trust through isolation and binding, and FalseCrashReducer validates caller context against infeasible crashes [4,49]. Repeated-run studies re- port that single-run scores overstate coverage, that inference backends shift results, and that injected static structure halves variance [46,58,90]. ThereforeBUGSTONE-E2Efixes its Phase A candidate set before any model runs and measures run-to-run reproducibility, so execution collapses the remain- ing spread. Security taxonomies and public vulnerability data. BUGSTONE-E2Ebuilds on public infrastructure, namely cvelistV5records, NVD metadata, the CWE taxonomy, and OSV advisories [17,50,51,56]. CVEfixes, More- Fixes, MegaVul, and VulZoo unify CVE-linked fixes, while ThreatKG and VulnScopper mine missing relations among security entities [1,3,8,52,61,65]. Therefore BUGSTONE-E2Ediffers by turning verified fixing commits into executable scanner inputs with provenance rather than into datasets. 9 Conclusion BUGSTONE-E2E turns vulnerability history into reusable de- tection knowledge and applies it through a staged workflow from scalable detection to runtime validation and candidate remediation. Its current knowledge base contains 1,033 rules consolidated into 56 CWE families and deployed through 172 skills. Across controlled experiments and real-world scans of 14 projects, these skills surface recurring vulnerability con- ditions beyond the projects from which they were originally derived, with 644 findings supported by runtime evidence. More broadly,BUGSTONE-E2Econcentrates increasingly expensive analysis on progressively fewer candidates while grounding stronger security claims in observable evidence rather than model judgment alone. Our results show that his- torical vulnerability fixes can serve not only as records of past failures, but also as reusable knowledge for finding and validating future ones. References [1]J. Akhoundali, S. R. Nouri, K. Rietveld, and O. Gady- atskaya. MoreFixes: A large-scale dataset of CVE fix commits mined through enhanced repository discovery. In International Conference on Predictive Models and Data Analytics in Software Engineering, pages 42â51, 2024. doi: 10.1145/3663533.3664036. [2]A. Al-Maamari. Why LLMs fail: A failure analysis and partial success measurement for automated security patch generation, 2026. URLhttps://arxiv.org/ abs/2603.10072. arXiv:2603.10072. [3] D. Alfasi, T. Shapira, and A. Bremler-Barr. VulnScop- per: Unveiling hidden links between unseen security entities. In International Workshop on Graph Neu- ral Networking, pages 33â40, 2024. doi: 10.1145/ 3694811.3697819. 15 [4]P. C. Amusuo, D. Liu, R. A. C. Mendez, J. Metzman, O. Chang, and J. C. Davis. FalseCrashReducer: Mit- igating false positive crashes in OSS-Fuzz-Gen using agentic AI, 2025. URLhttps://arxiv.org/abs/ 2510.02185. arXiv:2510.02185. [5] Anthropic.Defending code reference harness. https://github.com/anthropics/defending- code-reference-harness, 2026. Accessed 2026- 08-10. [6]P. Avgustinov, O. de Moor, M. P. Jones, and M. Schäfer. QL: Object-oriented queries on relational data. In Eu- ropean Conference on Object-Oriented Programming, pages 2:1â2:25, 2016. doi: 10.4230/LIPIcs.ECOOP. 2016.2. [7]G. Bennett, T. Hall, E. Winter, and S. Counsell. Sem- grep*: Improving the limited performance of static application security testing (SAST) tools. In Inter- national Conference on Evaluation and Assessment in Software Engineering, pages 614â623, 2024. doi: 10.1145/3661167.3661262. [8]G. Bhandari, A. Naseer, and L. Moonen. CVEfixes: Automated collection of vulnerabilities and their fixes from open-source software. In International Conference on Predictive Models and Data Analytics in Software Engineering, pages 30â39, 2021. doi: 10.1145/3475960. 3475985. [9] T. Bui, T. Zhang, F. Thung, Y. Xiong, P. Jiang, X. Zhou, and D. Lo. Vul4Py: Benchmarking automated vulner- ability repair in Python with paired exploit and func- tional oracles, 2026. URLhttps://arxiv.org/abs/ 2608.00692. arXiv:2608.00692. [10] C. Calcagno, D. Distefano, J. Dubreil, D. Gabi, P. Hooimeijer, M. Luca, P. OâHearn, I. Papakonstanti- nou, J. Purbrick, and D. Rodriguez. Moving fast with software verification. In NASA Formal Methods, pages 3â11, 2015. [11]S. Cao, H. Ma, L. Yu, K. Ding, X. Liu, T. Y. Zhuo, B. Wang, X. Lin, X. Sun, L. Wang, and D. Lo. Knowledge-enhanced agentic vulnerability re- pair, 2026. URLhttps://arxiv.org/abs/2607. 00820. arXiv:2607.00820. [12] S. Chakraborty, R. Krishna, Y. Ding, and B. Ray. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering, 48 (9):3280â3296, 2022. doi: 10.1109/TSE.2021.3087402. [13]A. Chao. Estimating the population size for capture- recapture data with unequal catchability. Biometrics, 43 (4):783â791, 1987. doi: 10.2307/2531532. [14]B. Chen, S. Liao, L. Zhang, C. Zhang, M. Payer, and Y. Zhang. Patch-guided vulnerability detection: Ex- tracting java API security rules via attack-defense cross- analysis. In USENIX Security Symposium, 2026. Pre- publication. [15] S. Chen, Y. He, S. Jana, and B. Ray. Red teaming pro- gram repair agents: When correct patches can hide vul- nerabilities, 2025. URLhttps://arxiv.org/abs/ 2509.25894. arXiv:2509.25894. [16] R. K. Colwell, A. Chao, N. J. Gotelli, S.-Y. Lin, C. X. Mao, R. L. Chazdon, and J. T. Longino. Models and estimators linking individual-based and sample-based rarefaction, extrapolation and comparison of assem- blages. Journal of Plant Ecology, 5(1):3â21, 2012. doi: 10.1093/jpe/rtr044. [17] CVE Program. CVE list V5.https://github.com/ CVEProject/cvelistV5, 2026. Official CVE List repository in CVE JSON 5 format. [18]X. Du, G. Zheng, K. Wang, Y. Zou, Y. Wang, W. Deng, J. Feng, M. Liu, B. Chen, X. Peng, T. Ma, and Y. Lou. Vul-RAG: Enhancing llm-based vulnerability detection via knowledge-level rag. ACM Transactions on Software Engineering and Methodology, 2026. doi: 10.1145/ 3797277. arXiv:2406.11147. [19]J. Fan, Y. Li, S. Wang, and T. N. Nguyen. A c/c++ code vulnerability dataset with code changes and cve summaries. In International Conference on Mining Software Repositories, 2020. [20]J. Feiglin and G. Dar. SastBench: A benchmark for testing agentic SAST triage, 2026. URLhttps:// arxiv.org/abs/2601.02941. arXiv:2601.02941. [21] A. Fioraldi, D. Maier, H. EiĂfeldt, and M. Heuse. AFL++: Combining incremental steps of fuzzing re- search. In USENIX Workshop on Offensive Technolo- gies, 2020. [22]F. Fleischer, C. Zhang, J. Jang, J. Cho, M. Xu, and T. Kim. Contextualizing sink knowledge for Java vulner- ability discovery, 2026. URLhttps://arxiv.org/ abs/2604.01645. arXiv:2604.01645. [23] M. Fu and C. Tantithamthavorn. LineVul: A transformer- based line-level vulnerability prediction. In Interna- tional Conference on Mining Software Repositories, 2022. [24] M. Fu, Q. Mei, P. Thongtanunam, and K. Tan- tithamthavorn. AgenticRepair: Multi-faceted program context engineering for agentic vulnerability repair, 2026. URLhttps://arxiv.org/abs/2607.29422. arXiv:2607.29422. 16 [25]A. Garg, S. Hager, J. Montiel, A. Tiwari, M. Gentile, Z. Reavis, D. Magnotti, and W. Fullen. RuleForge: Au- tomated generation and validation for web vulnerability detection at scale, 2026. URLhttps://arxiv.org/ abs/2604.01977. arXiv:2604.01977. [26] J. Ghebremichael, W. Jiang, M. Lysenko, B. B. Nielsen, W. Enck, and A. Kapravelos.VeriPort: Automated and verified patch backporting at scale, 2026. URLhttps://arxiv.org/abs/2606.22704. arXiv:2606.22704. [27]J. Ghebremichael, S. Vasan, S. Ullah, G. Tystahl, D. Adei, C. Kruegel, G. Vigna, W. Enck, and A. Kaprav- elos. Multi-agent taint specification extraction for vul- nerability detection, 2026. URLhttps://arxiv.org/ abs/2601.10865. arXiv:2601.10865. [28] GitHub. CodeQL: Semantic code analysis.https:// codeql.github.com/, 2026. [29] Google Project Zero and Google DeepMind Big Sleep Team. From Naptime to Big Sleep: Using large lan- guage models to catch vulnerabilities in real-world code. Google Project Zero blog, Nov. 2024. URLhttps:// googleprojectzero . blogspot . com / 2024 / 10 / from-naptime-to-big-sleep.html.Accessed 2026-08-04. [30]K. Huang, C. Lu, Y. Cao, B. Chen, and X. Peng. VMud: Detecting recurring vulnerabilities with multiple fixing functions via function selection and semantic equivalent statement matching. In ACM Conference on Computer and Communications Security, pages 3958â3972, 2024. doi: 10.1145/3658644.3690372. [31]J. Jang, A. Agrawal, and D. Brumley. ReDeBug: Finding unpatched code clones in entire OS distributions. In IEEE Symposium on Security and Privacy, pages 48â62, 2012. doi: 10.1109/SP.2012.13. [32] K. Ji, J. Liu, E. Hu, C. Gao, K. Lian, Y. Liu, L. Zhang, T. Dong, H. Chen, and W. Bin. VulnGym: Bench- marking coding agents for repository-level vulnerabil- ity detection, 2026. URLhttps://arxiv.org/abs/ 2608.02001. arXiv:2608.02001. [33]B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge. Why donât software developers use static analysis tools to find bugs? In International Conference on Software Engineering, pages 672â681, 2013. [34]S. Kaniewski, F. Schmidt, and T. Heer.Revisiting Vul-RAG: Reproducibility and replicability of RAG- based vulnerability detection with open-weight mod- els.In International Workshop on Artificial Intel- ligence and Cybersecurity for Critical and Cyber- Physical Systems (AI&CCPS), co-located with ARES, 2026. arXiv:2606.04739. [35]S. Kim, S. Woo, H. Lee, and H. Oh. VUDDY: A scalable approach for vulnerable code clone discovery. In IEEE Symposium on Security and Privacy, 2017. [36]T. Kim, H. Han, S. Park, D. R. Jeong, D. Kim, D. Kim, et al.ATLANTIS: AI-driven threat localization, analysis, and triage intelligence system, 2025. URL https://arxiv.org/abs/2509.14589. Team At- lanta, 1st place, DARPA AIxCC Final Competition; arXiv:2509.14589. [37]H. Lee, J. Liu, D. Kim, W. Xia, Z. Zhang, C. S. Xia, and L. Zhang. SEC-bench Pro: Can language models solve long-horizon software security tasks?, 2026. URLhttps://arxiv.org/abs/2605.26548. arXiv:2605.26548. [38]A. Lekssays, H. Mouhcine, K. Tran, T. Yu, and I. Khalil. LLMxCPG: Context-aware vulnerability detection through code property graph-guided large language models. In USENIX Security Symposium, pages 489â 507, 2025. [39] F. Li and V. Paxson. A large-scale empirical study of security patches. In ACM Conference on Computer and Communications Security, pages 2201â2215, 2017. [40]H. Li, Y. Hao, Y. Zhai, and Z. Qian. Enhancing static analysis for practical bug detection: An llm-integrated approach. Proceedings of the ACM on Programming Languages, 8(OOPSLA1):474â499, 2024. [41]Y. Li, T. Zhang, J. Liu, J. Jiang, Y. Yieh, Y. Yang, W. B. Leow, Y. Yin, Y. Huo, E. L. Ouh, L. K. Shar, and D. Lo. Graph is the verifier: Agentic reinforce- ment learning for interprocedural vulnerability detection, 2026. URLhttps://arxiv.org/abs/2607.26656. arXiv:2607.26656. [42]Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong. VulDeePecker: A deep learning-based system for vulnerability detection. In Network and Distributed System Security Symposium, 2018. doi: 10.14722/ndss.2018.23158. [43] Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, and Z. Chen. Sy- SeVR: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Depend- able and Secure Computing, 19(4):2244â2258, 2022. [44]Z. Li, S. Dutta, and M. Naik. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In In- ternational Conference on Learning Representations, 2025. 17 [45]J. Lin and D. Mohaisen. From large to mammoth: A comparative evaluation of large language models in vul- nerability detection. In Network and Distributed System Security Symposium, 2025. [46]Z. Lin, M. Zhou, Y. Yang, and L. Li. How much static structure do code agents need? A study of deterministic anchoring. In International Symposium on Software Testing and Analysis (ISSTA), 2026. arXiv:2606.26979. [47]B. Liu, Y. Zhao, G. Xu, and H. Wang. LLM agents for automated web vulnerability reproduction: Are we there yet?, 2025. URLhttps://arxiv.org/abs/ 2510.14700. arXiv:2510.14700. [48]K. Lu, A. Pakki, and Q. Wu. Detecting missing-check bugs via semantic- and context-aware criticalness and constraints inferences. In USENIX Security Symposium, pages 1769â1786, 2019. [49]M. Lyu, K. Shieh, Y. Hou, H. Wang, K. Sen, and D. Wagner. RECEIPT: Deterministic, reward-hacking- resistant verification for white-box agentic XSS dis- covery, 2026. URLhttps://arxiv.org/abs/2607. 18575. arXiv:2607.18575. [50]MITRE. Common Weakness Enumeration.https:// cwe.mitre.org/, 2026. [51] National Institute of Standards and Technology. Na- tional Vulnerability Database.https://nvd.nist. gov/, 2026. Standards-based vulnerability management data repository. [52]C. Ni, L. Shen, X. Yang, Y. Zhu, and S. Wang. MegaVul: A C/C++ vulnerability dataset with comprehensive code representations. In International Conference on Mining Software Repositories, pages 738â742, 2024. [53]R. Ni, M. Christodorescu, and L. Jia. Taint-style vulner- ability detection and confirmation for Node.js packages using LLM agent reasoning, 2026. URLhttps:// arxiv.org/abs/2604.20179. arXiv:2604.20179. [54] Y. Nie, H. Li, C. Guo, R. Jiang, Z. Wang, B. Li, D. Song, and W. Guo.VulnLLM-R: Specialized reasoning LLM with agent scaffold for vulnerability detection. arXiv:2512.07533, 2025. URLhttps://arxiv.org/ abs/2512.07533. [55]Y. Nie, Z. Wang, Y. Yang, R. Jiang, Y. Tang, X. Davies, Y. Gal, B. Li, W. Guo, and D. Song. SeCodePLT: A unified platform for evaluating the security of code GenAI.NeurIPS Datasets and Benchmarks Track, arXiv:2410.11096, 2025. URLhttps://arxiv.org/ abs/2410.11096. [56]Open Source Security Foundation. Open Source Vul- nerability Schema and Database.https://osv.dev/, 2026. Machine-readable open-source vulnerability ad- visory database. [57] OpenAI. Codex security plugin.https://openai. com/daybreak/codex-security-plugin/, 2026. Accessed 2026-08-10. See alsohttps://learn. chatgpt.com/docs/security/plugin. [58] D. Pape, J. Evertz, and L. SchĂśnherr. The silent hyper- parameter: Quantifying the impact of inference back- ends on LLM reproducibility, 2026. URLhttps:// arxiv.org/abs/2605.19537. arXiv:2605.19537. [59]J. Pellew and F. Raza. RealVuln: Benchmarking rule- based, general-purpose LLM, and security-specialized scanners on real-world code, 2026. URLhttps:// arxiv.org/abs/2604.13764. arXiv:2604.13764. [60] J. Pu, X. Li, Z. Liang, J. Cox, Y. Wu, K. Shehada, A. Sri- vastav, and Z. Qian. Patch-to-PoC: A systematic study of agentic LLM systems for Linux kernel N-day re- production, 2026. URLhttps://arxiv.org/abs/ 2602.07287. arXiv:2602.07287. [61]B. Ruan, J. Liu, W. Zhao, and Z. Liang. VulZoo: A comprehensive vulnerability intelligence dataset. In International Conference on Automated Software En- gineering, pages 2334â2337, 2024.doi: 10 . 1145 / 3691620.3695345. [62]Semgrep.Semgrep documentation.https:// semgrep.dev/docs/, 2026. [63]K. Serebryany, D. Bruening, A. Potapenko, and D. Vyukov. AddressSanitizer: A fast address sanity checker. In USENIX Annual Technical Conference, pages 309â318, 2012. [64]M. Shao and Y. Ding. FVD-DPM: Fine-grained vulner- ability detection via conditional diffusion probabilistic models. In USENIX Security Symposium, 2024. [65]Z. Shi, N. Matyunin, K. Graffi, and D. Starobinski. Un- covering CWE-CVE-CPE relations with threat knowl- edge graphs. ACM Transactions on Privacy and Secu- rity, 27(1):1â26, 2024. doi: 10.1145/3641819. [66]B. Steenhoek, H. Gao, and W. Le. Dataflow analysis- inspired deep learning for efficient vulnerability detec- tion. In International Conference on Software Engineer- ing, pages 1â13, 2024. doi: 10.1145/3597503.3623345. [67]M. Sun and G. Meng. DREA: Decoupled reasoning and exploration agents for repository-level vulnerability detection. In International Conference on Internetware (Internetware), 2026. arXiv:2607.13439. 18 [68]Y. Sun, D. Wu, Y. Xue, H. Liu, W. Ma, L. Zhang, Y. Liu, and Y. Li. LLM4Vuln: A unified evaluation framework for decoupling and enhancing LLMsâ vul- nerability reasoning. arXiv:2401.16185, 2024. URL https://arxiv.org/abs/2401.16185. [69]Tree-sitter Project. Tree-sitter: An incremental pars- ing system for programming tools.https://tree- sitter.github.io/tree-sitter/, 2026. [70]G. Tsigkourakos and C. Patsakis.QRS: A rule- synthesizing neuro-symbolic triad for autonomous vul- nerability discovery, 2026.URLhttps://arxiv. org/abs/2602.09774. arXiv:2602.09774. [71]S. Vijay, A. Priyanshu, D. Chapoteau, A. Gold- blatt, J. He, K. Majd, F. Burch, B. Saglam, T. Mat- sumoto, Z. Yang, and A. Karbasi.Antares: Foun- dation models for agentic vulnerability localization, 2026. URLhttps://arxiv.org/abs/2608.02407. arXiv:2608.02407. [72] Visa.VVAH: Visa vulnerability agentic har- ness.https :// github . com / visa / visa - vulnerability- agentic- harness, 2026.Ac- cessed 2026-08-10. [73] C. Wang, Z. Li, S. Dutta, and M. Naik. QLCoder: A query synthesizer for static analysis of security vulnera- bilities. In International Conference on Learning Rep- resentations, 2026. URLhttps://openreview.net/ forum?id=J91IKwJrqv. arXiv:2511.08462. [74] H. Wang, M. Xu, Y. Guo, W. Han, H. W. Lim, and J. S. Dong. RulePilot: An LLM-powered agent for security rule generation. In International Conference on Soft- ware Engineering (ICSE), 2026. arXiv:2511.12224. [75]W. Wang, W. Ma, Q. Hu, Y. Zhang, J. Sun, B. Wu, Y. Liu, G. Xu, and L. Jiang.VulnRepairEval: An exploit-based evaluation framework for assessing large language model vulnerability repair capabilities, 2025. URLhttps://arxiv.org/abs/2509.03331. arXiv:2509.03331. [76]S. Woo, H. Hong, E. Choi, and H. Lee. MOVERY: A precise approach for modified vulnerable code clone discovery from modified Open-Source software com- ponents. In USENIX Security Symposium, pages 3037â 3053, 2022. [77]S. Woo, E. Choi, H. Lee, and H. Oh. V1SCAN: Dis- covering 1-day vulnerabilities in reused C/C++ open- source software components using code classification techniques. In USENIX Security Symposium, pages 6541â6556, 2023. [78]Q. Wu, Y. He, S. McCamant, and K. Lu. Precisely characterizing security impact in a flood of patches via symbolic rule comparison. In Network and Distributed System Security Symposium, 2020. [79]Q. Wu, A. Pakki, N. Emamdoost, S. McCamant, and K. Lu. Understanding and detecting disordered error handling with precise function pairing. In USENIX Security Symposium, 2021. [80] Q. Wu, Y. Xiao, X. Liao, and K. Lu. OS-aware vulnera- bility prioritization via differential severity analysis. In USENIX Security Symposium, pages 395â412, 2022. [81]Q. Wu, Y. Xiao, D. Kirat, K. Eykholt, J. Jang, and D. L. Schales. One bug, hundreds behind: LLMs for large- scale bug discovery. arXiv:2510.14036, 2025. URL https://arxiv.org/abs/2510.14036. [82]Y. Xiao, B. Chen, C. Yu, Z. Xu, Z. Yuan, F. Li, B. Liu, Y. Liu, W. Huo, W. Zou, and W. Shi. MVP: Detect- ing vulnerabilities using patch-enhanced vulnerability signatures. In USENIX Security Symposium, pages 1165â 1182, 2020. [83] Y. Xiong and T. Zhang. Sifting the noise: A compara- tive study of LLM agents in vulnerability false positive filtering. In International Symposium on Software Test- ing and Analysis (ISSTA), 2026. doi: 10.1145/3832100. arXiv:2601.22952. [84] F. Yamaguchi, N. Golde, D. Arp, and K. Rieck. Model- ing and discovering vulnerabilities with code property graphs. In IEEE Symposium on Security and Privacy, 2014. [85]A. Yildiz, S. G. Teo, Y. Lou, Y. Feng, C. Wang, and D. M. Divakaran. Benchmarking LLMs and LLM-based agents in practical vulnerability detection for code repos- itories. In Annual Meeting of the Association for Com- putational Linguistics, pages 30848â30865, 2025. doi: 10.18653/v1/2025.acl-long.1490. [86]C. Zhang, Y. Park, F. Fleischer, Y.-F. Fu, J. Kim, D. Kim, Y. Kim, Q. Xu, A. Chin, Z. Sheng, H. Zhao, M. Peli- can, D. J. Musliner, J. Huang, J. Silliman, M. Mcdaniel, J. Casavant, I. Goldthwaite, N. Vidovich, M. Lehman, and T. Kim.SoK: DARPAâs AI cyber challenge (AIxCC): Competition design, architectures, and lessons learned. In USENIX Security Symposium (Security), 2026. arXiv:2602.07666. [87]T. Zhang, Y. Li, C. Yang, R. Widyasari, Y. Liu, N. T. Bui, P. T. Nguyen, Y. N. Tun, I. C. Irsan, H. H. Nguyen, H. Huang, J. Jiang, L. K. Shar, E. L. Ouh, D. Lo, H. J. Kang, Y. Yin, and W. B. Leow. TitanCA: Lessons from orchestrating LLM agents to discover 100+ CVEs. IEEE Security & Privacy, 2026. To appear; arXiv:2604.17860. 19 [88]Z. Zheng, J. Zhou, X. Hu, Y. Gao, and S. Pan. Multi- agent end-to-end vulnerability management for mitigat- ing recurring vulnerabilities, 2026. URLhttps:// arxiv.org/abs/2601.17762. arXiv:2601.17762. [89]Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu. Devign: Ef- fective vulnerability identification by learning compre- hensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems, 2019. [90] Y. Zhou, L. Y. Choi, J. Wen, and W. Ye.Accu- racy, stability, and repeated-run reliability of large lan- guage models on deterministic programming tasks, 2026. URLhttps://arxiv.org/abs/2606.00920. arXiv:2606.00920. [91]Y. Zhu, A. Kellermann, D. Bowman, P. Li, A. Gupta, A. Danda, R. Fang, C. Jensen, E. Ihli, J. Benn, J. Geron- imo, A. Dhir, S. Rao, K. Yu, T. Stone, and D. Kang. CVE-Bench: A benchmark for AI agentsâ ability to exploit real-world web application vulnerabilities, 2025. URLhttps://arxiv.org/abs/2503.17332. arXiv:2503.17332. [92]A. Zibaeirad, M. Vieira, and T. Zimmermann. Auto- Trace: From patches to triggers via agentic interproce- dural exploration, 2026. URLhttps://arxiv.org/ abs/2607.12058. arXiv:2607.12058. 20