Paper deep dive
Toward Scalable Automated Repository-Level Datasets for Software Vulnerability Detection
Amine Lbath
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/22/2026, 6:00:09 AM
Summary
This research proposes an automated benchmark generator that injects realistic, interprocedural vulnerabilities into real-world software repositories to create scalable, executable datasets for training and evaluating vulnerability detection agents. The approach utilizes a multi-agent system guided by CodeQL to synthesize proof-of-vulnerability (PoV) artifacts and introduces an adversarial co-evolution loop between injection and detection agents to enhance model robustness.
Entities (6)
Relation Signals (4)
Automated Benchmark Generator → generates → Proof-of-vulnerability (PoV)
confidence 95% · synthesizes reproducible proof-of-vulnerability (PoV) exploits
Amine Lbath → proposes → Automated Benchmark Generator
confidence 95% · This doctoral research proposes an automated benchmark generator that injects realistic vulnerabilities into real-world repositories
Automated Benchmark Generator → uses → CodeQL
confidence 95% · In particular, a CodeQL-guided analysis [1] is used to mine candidate locations
Adversarial co-evolution loop → involves → Vulnerability Injector
confidence 90% · consisting of two competing AI models: a Vulnerability Injector and a Vulnerability Detector
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Software vulnerabilities continue to grow in volume and remain difficult to detect in practice. Although learning-based vulnerability detection has progressed, existing benchmarks are largely function-centric and fail to capture realistic, executable, interprocedural settings. Recent repo-level security benchmarks demonstrate the importance of realistic environments, but their manual curation limits scale. This doctoral research proposes an automated benchmark generator that injects realistic vulnerabilities into real-world repositories and synthesizes reproducible proof-of-vulnerability (PoV) exploits, enabling precisely labeled datasets for training and evaluating repo-level vulnerability detection agents. We further investigate an adversarial co-evolution loop between injection and detection agents to improve robustness under realistic constraints.
Tags
Links
- Source: https://arxiv.org/abs/2603.17974v1
- Canonical: https://arxiv.org/abs/2603.17974v1
Trouble viewing inline? Open PDF directly →
Full Text
12,355 characters extracted from source content.
Expand or collapse full text
Toward Scalable Automated Repository-Level Datasets for Software Vulnerability Detection Amine Lbath Supervisor: Prof. Massih-Reza Amini first.last[at]nist.gov;first.last[at]univ-grenoble-alpes.fr Univeristy of Grenoble Grenoble, France Abstract Software vulnerabilities continue to grow in volume and remain difficult to detect in practice. Although learning-based vulnera- bility detection has progressed, existing benchmarks are largely function-centric and fail to capture realistic, executable, interproce- dural settings. Recent repo-level security benchmarks demonstrate the importance of realistic environments, but their manual curation limits scale. This doctoral research proposes an automated bench- mark generator that injects realistic vulnerabilities into real-world repositories and synthesizes reproducible proof-of-vulnerability (PoV) exploits, enabling precisely labeled datasets for training and evaluating repo-level vulnerability detection agents. We further investigate an adversarial co-evolution loop between injection and detection agents to improve robustness under realistic constraints. CCS Concepts • Security and privacy→Security services;• Software and its engineering→Software verification and validation;• Com- puting methodologies→ Multi-agent systems. Keywords software vulnerabilities, benchmarks, LLM agents, vulnerability detection, exploit synthesis ACM Reference Format: Amine Lbath and Supervisor: Prof. Massih-Reza Amini. 2026. Toward Scal- able Automated Repository-Level Datasets for Software Vulnerability De- tection. In Proceedings of ACM International Conference on the Foundations of Software Engineering (FSE 2026). ACM, New York, NY, USA, 2 pages. https://doi.org/X.X 1 Problem Statement and Motivation The number of publicly disclosed software vulnerabilities has in- creased steadily over the past decade, with recent years reporting tens of thousands of new CVEs annually [5]. At the same time, modern software systems have grown in size and complexity, in- creasingly relying on large, multi-module codebases and rapid de- velopment cycles. This trend is further amplified by the widespread Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. FSE 2026, Montreal, Canada © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X adoption of AI-assisted coding tools, which have been shown to generate insecure code patterns and to exacerbate existing security risks when used without rigorous analysis [8]. As a result, effective vulnerability detection must operate at the codebase level, where vul- nerabilities often emerge from interactions across functions, files, and configuration boundaries rather than from isolated functions. Why current benchmarks are the bottleneck. A large fraction of ML/LLM vulnerability detection research still reduces the task to function-level binary classification. Recent analyses show this for- mulation is often ill-defined: many functions cannot be labeled vulnerable/benign without their calling context, and high scores may be driven by spurious correlations rather than true reasoning about security [7]. Likewise, evaluations on traditional datasets can dramatically overestimate real-world effectiveness; more realistic settings reveal large performance drops [2, 3]. Repo-level datasets exist, but realism and scalability remain limited. Several recent efforts move beyond single functions by mining repository context from CVE/fix histories. ReposVul and VulEval provide interprocedural context and large-scale mining, but depend on proxy labels and evaluation setups that are not always executable end-to-end [9,10]. AI agent-focused benchmarks, such as Boun- tyBench and CVE-Bench, explicitly target realistic environments and security-agent tasks (detect/exploit/patch) with reproducible setups, but require substantial manual effort and remain limited in scale, which constrains their use for training [11,12]. Meanwhile, specialized benchmarks highlight that repo-level security tasks are difficult even for strong AI agents, motivating the need for larger, training-ready datasets [6, 11]. Gap. Existing benchmarks lack a scalable mechanism to generate repository-level, buildable, and executable vulnerability instances with precise labels and reproducible proof-of-vulnerability artifacts. Recent work on automated vulnerability injection at the function level, including our own (AVIATOR), demonstrates that controlled injection can produce realistic and high-quality security data [4]. This research builds on that insight by extending vulnerability injection to the repository scale and systematically pairing injected vulnerabilities with executable proofs, enabling realistic training and benchmarking of repo-level vulnerability detection agents. 2 Research Questions Recent LLM-based and agentic approaches demonstrate emerging capabilities for software vulnerability detection, yet their perfor- mance remains limited in realistic, repository-level settings. Several recent studies explicitly identify the lack of large-scale, executable, arXiv:2603.17974v1 [cs.SE] 18 Mar 2026 FSE 2026, July 05–09, 2026, Montreal, CanadaA. Lbath and accurately labeled codebase-level vulnerability datasets as a key obstacle to building stronger systems and to benchmarking them reliably [6,11]. To address this gap, this doctoral research is guided by the following questions: •RQ1 (Benchmark quality). Can AI agentic workflows inject realistic vulnerabilities into real repositories while preserving build/test validity and yielding reproducible PoVs? •RQ2 (Training utility). Does training on a large, codebase- level, dataset improve generalization and robustness of vulnera- bility detection models compared to training at function-level? •RQ3 (Adversarial co-evolution). Can an adversarial co-evolution between vulnerability injection and detection agents improve robustness in repository-level vulnerability detection? 3 Proposed Approach The core contribution is an automated benchmark generator driven by a workflow of AI agents. The design is human-expert inspired: it mirrors how security engineers (i) set up a target, (i) form a vulnerability hypothesis, (i) implement a minimal-but- plausible change, and (iv) validate exploitability with a PoV. Phase A: Target selection and executable harness. Given a repos- itory, the system automatically: (i) builds the project in a container- ized environment, (i) discovers or synthesizes a test harness (ex- isting tests, fuzz harnesses, or minimal driver programs), and (i) establishes invariants (tests passing, sanitizer baseline, API con- tracts) used to reject problematic injections. This step aims to ensure every benchmark item is runnable and evaluation is reproducible. Phase B: Vulnerability Injection with Multi-Agent Control. To identify realistic injection points at repository scale, we leverage structural and semantic signals extracted from the codebase. In par- ticular, a CodeQL-guided analysis [1] is used to mine candidate locations based on dataflow patterns, and security-relevant sinks and sources, providing a principled approximation of how human experts reason about vulnerability placement. A set of cooperating agents then performs the injection: •Planner agent selects a vulnerability class (e.g., CWE family) and candidate injection sites suggested by CodeQL queries (e.g., incomplete validation paths, risky sink reachability). •Implementer agent introduces minimal, cross-file changes that activate the vulnerability only along specific execution paths, favoring interprocedural flaws. •Reviewer agent enforces realism constraints (coding style, plausible developer intent, avoidance of synthetic signatures). • Verifier agent rebuilds the project and executes tests to ensure that only intended behaviors are affected. This combination of static query guidance and agentic editing en- ables scalable, human-inspired vulnerability injection while pre- serving repository integrity. Phase C: Proof-of-vulnerability synthesis and trace artifacts. For each injected vulnerability, an exploit-generation agent at- tempts to produce a PoV: •For memory safety: a triggering input with sanitizer/crash sig- nature and a minimal reproducer. •For logic/web-style flaws: an end-to-end request sequence and observable security violation. The system records executable traces (stack traces, crashing inputs, dependency slices, and minimal call paths) to support evaluation of localization and explanation quality. Each benchmark item includes: original code, vulnerable commit, container recipe, harness, PoV, and structured labels (CWE, affected components, trigger path). Phase D: Training and benchmarking repo-level detection agents. We leverage the generated corpus to train vulnerability detection agents that operate at repository scope: tool-augmented retrieval over the codebase, iterative localization, hypothesis re- finement, and validation (e.g., reproducing the PoV or generating a patch). By providing executable ground truth and cross-file con- text, the benchmark enables evaluation of detection systems in settings that more closely reflect real-world software engineering workflows. Phase E: Adversarial co-evolution loop. Finally, we propose a novel adversarial learning framework consisting of two compet- ing AI models: a Vulnerability Injector and a Vulnerability Detector. Through competitive co-evolution, these models will iteratively improve each other’s capabilities, with the injector learning to generate realistic code vulnerabilities while the detector becomes increasingly skilled at identifying them. 4 Evaluation Plan and Expected Contributions Benchmark validation will measure: (i) build/test pass rate af- ter injection, (i) PoV reproducibility, (i) realism via expert au- dit on a stratified sample, and (iv) diversity (CWE distribution, multi-file depth). Training utility will compare detectors trained with/without injected repo-level data, tested on external repo-level benchmarks where possible (e.g., BountyBench/CVE-Bench-style tasks) [11,12]. Contributions include: (1) a novel scalable repo- level vulnerability benchmark generator with PoVs, (2) a dataset to train repo-level vulnerability detection agents, and (3) an ad- versarial co-evolution methodology to continuously improve both injection and more importantly detection. References [1] Pavel Avgustinov et al. 2016. QL: Object-oriented Queries on Relational Data. [2]Partha Chakraborty et al.2024. Revisiting the Performance of Deep Learning- Based Vulnerability Detection on Realistic Datasets. [3]Yangruibo Ding et al.2024. Vulnerability Detection with Code Language Models: How Far Are We? [4]Amine Lbath et al.2025. AVIATOR: Automated Vulnerability Injection for Real- istic Security Benchmarks. [5]National Institute of Standards and Technology. 2026. National Vulnerability Database (NVD). https://w.nist.gov/itl/nvd. Accessed 2026-01-28. [6] Yujin Potter et al. 2025. Frontier AI’s Impact on the Cybersecurity Landscape. [7]Niklas Risse et al.2025. Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection. [8]Veracode. 2025. GenAI Code Security Report: Security Flaws in AI-Generated Code. Accessed 2026-01-28. [9]Xinchen Wang et al.2024. ReposVul: A Repository-Level High-Quality Vulnera- bility Dataset. [10]Xin-Cheng Wen et al.2024. VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection. [11]Andy K. Zhang et al.2025. BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems. [12]Yuxuan Zhu et al.2025. CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities.