Paper deep dive
Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda
Wei Lin, Tao Zhou, Zhaofei Xie, Changgui Hong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/24/2026, 6:11:58 AM
Summary
This paper presents an evidence-centered structured survey at the intersection of Software Engineering (SE) and Software Security (SS) regarding Large Language Models (LLMs). It synthesizes work up to May 2026, proposing a unified taxonomy and an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority. The authors argue that current evaluations are divided between SE tasks (functional completion) and SS tasks (vulnerability detection), and they identify validity threats such as weak test oracles and data leakage. The paper introduces a claim-evidence ladder (E1-E4) and a governance tiering system (G0-G3) for agent authority, concluding that model capability should be judged as an assurance case supported by task-appropriate evidence rather than single benchmark scores.
Entities (14)
Relation Signals (10)
Large Language Models → appliedto → Software Engineering
confidence 95% · LLMs are moving from code completion toward repository-scale agents that... participate in security-sensitive workflows.
Large Language Models → appliedto → Software Security
confidence 95% · ...software security evaluations centered on vulnerability detection, secure generation, or exploit-oriented validation.
SWE-bench → usedfor → Repository-level code generation
confidence 93% · SWE-bench [26] tasks systems with resolving real GitHub issues spanning entire codebases.
HumanEval → usedfor → Function-level code generation
confidence 93% · Early benchmarks such as HumanEval [8] and MBPP [2] evaluate function-level generation in Python.
Assurance Framework → proposedby → This survey
confidence 92% · we introduce an assurance framework that separates functional correctness, security, operational reliability...
Automated Program Repair → uses → LLMs
confidence 92% · LLM-based APR has become a prominent research direction...
Retrieval-Augmented Generation → improves → Repository-level code generation
confidence 91% · Retrieval-Augmented Generation (RAG) enhances generation quality... significantly improving repository-level code generation
CodeBERT → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are moving from code completion toward repository-scale agents that retrieve context, edit files, execute tools, and participate in security-sensitive workflows. The evidence for these systems, however, remains divided between software engineering evaluations centered on functional task completion and software security evaluations centered on vulnerability detection, secure generation, or exploit-oriented validation. This evidence-centered structured survey synthesizes representative work available through May 31, 2026 across software engineering tasks, software security tasks, adaptation mechanisms, artifact granularity, and evaluation design. In addition to a task taxonomy, we introduce an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority. The review shows that execution feedback and repository access can substantially improve engineering task completion, but do not by themselves establish security; conversely, static-analysis labels or vulnerability-classification scores rarely establish deployable correctness. We identify recurring validity threats--weak test oracles, duplicated and temporally leaked data, changing agent harnesses, proxy-only security checks, and under-reported budgets and human intervention--and derive a minimum reporting protocol for cross-study comparison. The resulting research agenda prioritizes jointly secure-and-functional benchmarks, repository-scale threat models, calibrated human oversight, longitudinal maintainability evidence, and reproducible agent evaluation. The central conclusion is that model capability should be judged as an assurance case supported by task-appropriate evidence, rather than by a single benchmark score.
Tags
Links
- Source: https://arxiv.org/abs/2608.21107v1
- Canonical: https://arxiv.org/abs/2608.21107v1
Trouble viewing inline? Open PDF directly →
Full Text
69,779 characters extracted from source content.
Expand or collapse full text
Large Language Models at the Intersection of Software Engineering and Software Security: An Evidence-Centered Structured Survey and Research Agenda Wei Lin Tao Zhou Zhaofei Xie Changgui Hong Affiliation: Nanjing Liancheng Intelligent Technology Group, Nanjing, China Affiliation: linwei, zhoutao, xiezhaofei, hongchanggui@chinaliancheng.com Abstract Large Language Models (LLMs) are moving from code completion toward repository-scale agents that retrieve context, edit files, execute tools, and participate in security-sensitive workflows. The evidence for these systems, however, remains divided between software engineering evaluations centered on functional task completion and software security evaluations centered on vulnerability detection, secure generation, or exploit-oriented validation. This evidence-centered structured survey synthesizes representative work available through May 31, 2026 across software engineering tasks, software security tasks, adaptation mechanisms, artifact granularity, and evaluation design. In addition to a task taxonomy, we introduce an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority. The review shows that execution feedback and repository access can substantially improve engineering task completion, but do not by themselves establish security; conversely, static-analysis labels or vulnerability-classification scores rarely establish deployable correctness. We identify recurring validity threats—weak test oracles, duplicated and temporally leaked data, changing agent harnesses, proxy-only security checks, and under-reported budgets and human intervention—and derive a minimum reporting protocol for cross-study comparison. The resulting research agenda prioritizes jointly secure-and-functional benchmarks, repository-scale threat models, calibrated human oversight, longitudinal maintainability evidence, and reproducible agent evaluation. The central conclusion is that model capability should be judged as an assurance case supported by task-appropriate evidence, rather than by a single benchmark score. Keywords: large language models, software engineering, software security, code generation, vulnerability detection, automated program repair, agentic systems 1 Introduction The emergence of Large Language Models has substantially changed software development research and practice. Models such as GPT-4, CodeLlama [37], StarCoder [31], and DeepSeek-Coder [19] can understand, generate, and reason about source code. Their applications span code generation, automated program repair (APR), testing, code review, documentation, and vulnerability analysis [22]. Practitioner evidence also suggests productivity benefits for selected development activities, although reported gains depend strongly on task, developer experience, codebase context, and evaluation design [21]. This progress has been accompanied by a proliferation of survey papers, summarized in Table 1. Hou et al. [22] provide a systematic mapping of LLM4SE, including some security-related tasks within the broader software engineering landscape. Guo et al. [20] focus on benchmarks and solutions for LLM-empowered agentic systems, while Khan et al. [28] examine code-review benchmarks and evaluation practices. On the security axis, Xu et al. [46] review LLM applications across cybersecurity, Zhou et al. [51] focus on vulnerability detection and repair, Sheng et al. [39] synthesize vulnerability-detection techniques, and Basic and Giaretta [3] trace evidence from insecure generation to detection and remediation. These reviews establish substantial task-specific knowledge, but they leave open a cross-domain question: what evidence is sufficient when a development artifact must be functional, secure, reproducible, and produced by an agent with non-trivial authority? Our survey adopts an explicitly intersectional perspective. First, it treats software engineering and software security as co-equal analytical pillars rather than presenting security only as a subtask of software engineering. Second, it covers the emergence of agentic systems, including SWE-bench-oriented architectures [26], conversational development paradigms [17], and autonomous security workflows [46]. Third, it connects the two domains by examining how code-generation reliability affects security, how security analysis enters development workflows, and which evaluation criteria are required when functionality and security must be satisfied simultaneously. We do not claim exhaustive coverage of either field; instead, the goal is to provide a structured bridge between them. Table 1: Comparison with representative surveys. ∘ = covered as a sub-topic; ∙ = primary focus; – = not emphasized. Survey Year SE Security Agentic Benchmarks Intersection Hou et al. [22] 2024 ∙ ∘ – ∘ ∘ Xu et al. [46] 2025 – ∙ ∘ ∘ ∘ Zhou et al. [51] 2024 – ∙ – ∘ ∘ Guo et al. [20] 2025 ∙ – ∙ ∙ – Gao et al. [15] 2025 ∙ ∘ – – ∘ Basic and Giaretta [3] 2026 ∘ ∙ ∘ ∘ ∘ This survey 2026 ∙ ∙ ∙ ∙ ∙ This survey makes four contributions: (i) a unified taxonomy that relates engineering and security tasks to artifact granularity, context requirements, adaptation mechanisms, and agent authority; (i) an assurance-centered framework that distinguishes functional, security, operational, reproducibility, and governance evidence instead of treating task success as a scalar score; (i) a structured comparison of benchmarks and metrics, including a claim–evidence ladder and a minimum reporting protocol for model and agent evaluations; and (iv) a testable research agenda connecting secure generation, repository-scale analysis, human oversight, maintainability, and standardized cross-task evaluation. 2 Review Methodology 2.1 Scope and Research Questions This article is an evidence-centered structured narrative and tertiary survey rather than a completed protocol-driven systematic literature review. The search cutoff is May 31, 2026. We prioritize foundational model and benchmark papers, recent systematic reviews, empirical evaluations, and work that directly connects development with software security. The unit of synthesis is not the paper alone, but the claim supported by a particular artifact, dataset, execution environment, metric, and threat model. Because the underlying studies use different model versions, prompts, tools, retry budgets, repository snapshots, and benchmark variants, reported scores are treated as study-specific observations rather than as inputs to a pooled meta-analysis. This scope is deliberately transparent: the review provides a traceable conceptual synthesis and an extensible evidence package, but does not claim exhaustive recall of the literature. The review is guided by four research questions: RQ1: Which software engineering and software security tasks are addressed by LLM-based systems, and what artifacts and context do they require? RQ2: How do prompting, fine-tuning, retrieval, and agentic tool use change the capabilities and failure modes observed across these tasks? RQ3: Which benchmarks and metrics provide credible evidence, and which validity threats prevent direct cross-study comparison? RQ4: Where do engineering and security requirements intersect, and what evaluation and governance mechanisms are needed for integrated agents? 2.2 Search, Eligibility, and Coding Sources were identified through backward chaining from recent domain surveys and targeted searches for task names, adaptation strategies, models, and benchmark names in arXiv, ACM, and IEEE indexes. The protocol defines four concept blocks: language or code models; software engineering; software security; and adaptation or agent mechanisms. A source was retained when it introduced a consequential model, dataset, benchmark, or method; reported empirical evidence relevant to one of the research questions; synthesized a clearly bounded subfield; or exposed a validity threat with direct implications for evaluation. Opinion-only articles, duplicate versions, sources without enough methodological detail to calibrate their claims, and general cybersecurity work without a software artifact were excluded from the core synthesis. The manuscript’s seed corpus is generated automatically from citation keys in the source file and deduplicated by DOI, arXiv identifier, and normalized title. Coding fields include publication status, task family, artifact granularity, programming language, model or system, adaptation strategy, context source, tools, benchmark, split, metric, principal result, limitations, and the SE–security intersection. A separate quality rubric scores research-question clarity, dataset provenance, split validity, configuration reporting, baseline quality, metric validity, uncertainty analysis, leakage analysis, reproducibility, external validity, and security validation. These artifacts support audit and extension; reviewer judgments and screening agreement are intentionally left for real human coding rather than generated automatically. 2.3 Synthesis and Reproducibility The analysis combines task-centered narrative synthesis with structured comparison. Findings are not pooled statistically because the studies differ in models, data, prompts, tool access, and evaluation protocols. Instead, evidence is compared at the level of claims that a benchmark can support: function-level correctness, repository-level task completion, vulnerability classification, secure generation, or operational agent behavior. Positive results are interpreted together with negative findings, proxy limitations, and stated threats to validity. The source package includes a prospective review protocol, machine-readable seed corpus, search-log and extraction templates, a quality rubric, and scripts for merging, deduplication, and descriptive statistics. This package separates what has already been synthesized from what a future exhaustive update must still screen. 2.4 Claim Calibration We calibrate conclusions using four evidence levels. E1 (illustrative) consists of qualitative examples, demonstrations, or small curated tasks and supports feasibility claims only. E2 (controlled benchmark) adds repeatable datasets, baselines, and task-aligned metrics, supporting comparative claims within a fixed configuration. E3 (repository and execution) requires realistic project context, executable tests or exploits, environment capture, and regression checks, supporting system-level claims on the sampled projects. E4 (operational) additionally requires longitudinal use, human-effort or cost measurement, incident or maintenance outcomes, and governance controls. Evidence at a higher level does not automatically imply better model quality, but it supports a broader deployment claim. Throughout the survey, we therefore distinguish “performs on a benchmark” from “is reliable or secure in practice.” The remainder of this paper is organized as follows. Section 3 provides background on LLM architectures and adaptation strategies for code. Section 5 surveys LLM applications across core software engineering tasks. Section 6 examines LLM-driven approaches for software security. Section 7 compares representative evaluation benchmarks. Section 8 discusses persistent challenges and open problems, and Section 9 concludes with key findings and future directions. 3 Background and Preliminaries 3.1 LLM Architectures for Code The application of LLMs to code spans three dominant architectural paradigms. Encoder-only models, exemplified by CodeBERT [13], employ bidirectional attention to produce contextual representations suitable for understanding tasks such as vulnerability detection, clone detection, and code search. Encoder-decoder architectures, such as CodeT5 [43], combine bidirectional encoding with autoregressive decoding for tasks including summarization, translation, and repair. Decoder-only models support open-ended generation and tool-mediated interaction. Codex provided an early demonstration [8], while CodeLlama [37], StarCoder [31], and DeepSeek-Coder [19] established widely used open model families. Architectural labels alone do not determine performance: training data, context length, adaptation, and execution feedback are often equally consequential. 3.2 LLM Adaptation Strategies for Code Four principal adaptation strategies have emerged. Zero-shot and few-shot prompting leverages the model’s pre-trained knowledge without parameter updates, using natural language instructions or input-output examples to guide code generation [8]. Fine-tuning adapts models to specific code domains through full parameter updates or parameter-efficient methods such as LoRA and QLoRA, which are particularly important for domain-specific languages [47] and security-sensitive tasks [39]. Retrieval-Augmented Generation (RAG) enhances generation quality by retrieving relevant code snippets, documentation, or issue reports from repositories, significantly improving repository-level code generation [41]. Agent-based approaches decompose complex software tasks into subtasks handled by specialized LLM agents with tool-use capabilities, enabling end-to-end automation of the software development lifecycle [5]. 3.3 Task Taxonomy We organize the LLM4SE+Security landscape along three intersecting dimensions: software engineering tasks – code generation/synthesis, automated program repair, software testing, code review, and documentation/maintainability; software security tasks – vulnerability detection, vulnerability repair, malware analysis, and security testing/fuzzing; and adaptation strategies – the four approaches described above. Figure 1 illustrates this taxonomy as a hierarchical tree spanning all three dimensions. Figure 2 situates key developments along a historical timeline from CodeBERT (2020) through the emergence of agentic SE systems (2024–2026). LLM4SE & SecuritySoftware EngineeringSoftware SecurityAdaptation StrategiesCode Gen & SynthesisAutomated RepairTesting & FuzzingCode ReviewMaintainabilityVuln. DetectionVuln. RepairMalware AnalysisSecurity TestingFine-tuningPromptingRAGAgent-based Figure 1: Unified taxonomy of LLM4SE and LLM4Security. The hierarchy separates engineering tasks, security tasks, and adaptation strategies; colors distinguish node roles and adaptation families. CodeBERT2020CodeT52021Codex2021GPT-3.5/42022-23StarCoderCodeLlama2023DeepSeek-Coder 2024AgenticSE 2024-26Encoder-only Encoder-Decoder Decoder-only Agentic Figure 2: Timeline of key LLM developments for code (2020–2026). Colors denote architectural paradigms. 4 An Assurance-Centered Framework Existing evaluations often compress a system into one number even though an LLM-based development workflow is a compound intervention. A model receives a context selected by a retriever or agent, acts through a harness, observes tool feedback, and is judged by tests or analyzers executed in a particular environment. We represent an evaluation configuration as ℰ=⟨A,C,M,T,V,B,H⟩,E= A,C,M,T,V,B,H , (1) where A is the software artifact and task, C the available context, M the model and adaptation, T the tools and permissions, V the validators, B the inference and retry budget, and H the human intervention. Two scores are directly comparable only when the differences among these components are either controlled or explicitly analyzed. This representation explains why a model-only label is insufficient for an agent result and why benchmark leaderboards can change when the harness or environment changes. 4.1 Joint Functional and Security Success For an output y on task i, let Fi(y)F_i(y) denote satisfaction of functional tests and Si(y)S_i(y) denote satisfaction of threat-model-aligned security checks. The primary intersectional outcome is J@k=1N∑i=1N[∃y∈Yi(k):Fi(y)∧Si(y)],J@k= 1N _i=1^NI\! [∃ y∈ Y_i^(k):F_i(y) S_i(y) ], (2) where Yi(k)Y_i^(k) is the set of up to k generated candidates. Reporting F and S separately remains necessary because the same joint score can arise from different failure profiles. The conjunction is nevertheless important: averaging functional and security scores can conceal a catastrophic failure in either dimension. For repair and agent tasks, the conjunction should be extended with regression freedom, environment reproducibility, and policy compliance. This formulation also separates security from mere absence of analyzer warnings. A security predicate may be supported by dynamic proof-of-vulnerability tests, exploit reproduction, expert review, property checking, or a documented combination of analyzers. The validator must match the claimed threat model. Recent repository-level benchmarks move in this direction by pairing developer tests with dynamic security checks or by evaluating proof-of-concept generation and patching in executable environments [38, 30, 42]. 4.2 Evidence Ladder Table 2 connects evidence design to the strongest defensible claim. The levels are cumulative in realism but not interchangeable: a large controlled benchmark may provide more precise comparative evidence than a small operational deployment, while the deployment provides stronger external-validity evidence. A mature assurance case should therefore combine complementary levels rather than replace one score with another. Table 2: Claim–evidence ladder for LLM-based software engineering and security systems. Level Evidence setting Minimum evidence Defensible claim Typical failure E1 Demonstration or curated examples Prompt, outputs, manual inspection, and explicit examples Feasibility on illustrated cases Cherry-picking and no denominator E2 Controlled task benchmark Frozen data and split, baselines, repeated runs, task-aligned metric, configuration disclosure Comparative performance under the stated benchmark Leakage, weak tests, and metric mismatch E3 Repository and executable environment Versioned repository, build and tests, dynamic or threat-model-aligned checks, regression evidence, tool and budget logs System effectiveness on sampled projects and environments Hidden harness effects and irreproducible infrastructure E4 Operational or longitudinal setting Real users and workflow, time and cost, maintenance or incident outcomes, access controls, audit trail, and failure escalation Utility, risk, and sustainability in the observed deployment context Selection bias, organizational confounding, and rare-event uncertainty 4.3 Authority and Governance Agent authority is a separate evaluation dimension. We distinguish four practical tiers: G0G_0, read-only explanation or retrieval; G1G_1, patch proposal without execution; G2G_2, sandboxed editing and tool execution; and G3G_3, actions that can affect shared repositories, dependencies, credentials, or deployment. Evidence sufficient for G0G_0 is not sufficient for G3G_3. As authority increases, evaluation should add least-privilege tool scopes, network and secret isolation, action logging, approval gates, rollback, and incident handling. This tiering avoids treating all “agents” as equivalent and makes governance requirements proportional to potential impact. 5 LLMs for Software Engineering 5.1 Code Generation and Synthesis Code generation represents one of the most extensively studied LLM4SE applications. Early benchmarks such as HumanEval [8] and MBPP [2] evaluate function-level generation in Python. Reported performance has improved substantially, but HumanEval+ [34], which augments the original tasks with many additional tests, demonstrates that solutions passing limited test suites may still fail on edge cases. This distinction between benchmark success and robust functional correctness is central to evaluating generated code. Repository-level code generation introduces significantly greater complexity. SWE-bench [26] tasks systems with resolving real GitHub issues spanning entire codebases. The original study reported a 1.96% resolution rate for its strongest baseline, whereas a recent secondary review reports a result of 78.4% for a later agentic configuration [5]. These endpoints illustrate rapid progress, but they should not be interpreted as a controlled longitudinal experiment: benchmark variants, model versions, scaffolding, tool access, and verification procedures changed over time. Table 3 therefore presents the results as illustrative source-reported observations rather than a unified leaderboard. Table 3: Illustrative SWE-bench results reported by primary and secondary sources. The rows are not directly comparable because evaluation configurations differ. Source/System Setting Resolved (%) Interpretation Jimenez et al. [26] Original benchmark baseline 1.96 Limited repository-level issue resolution Bhati et al. [5] Later agentic result reported in review 78.4 Stronger models, retrieval, tools, and iterative execution Retrieval-Augmented Generation (RAG) is a common mechanism for moving from isolated function generation toward repository-aware synthesis. Tao et al. [41] categorize RAG-for-code approaches by generation strategy, retrieval modality, and integration stage. Multilingual code generation remains uneven: Python dominates evaluation, while less represented languages and language-specific toolchains receive substantially less evidence [25]. Domain-specific generation, including hardware description languages such as Verilog, further illustrates that syntax, verification infrastructure, and domain constraints materially shape evaluation [47]. The emergence of “vibe coding” – a paradigm where developers provide high-level intent and iteratively refine LLM outputs through natural language feedback – represents a shift toward speculative, conversational code generation. Ge et al. [17] synthesize this emerging literature and propose a taxonomy spanning unconstrained iteration to context-enhanced development workflows. 5.2 Automated Program Repair LLM-based APR has become a prominent research direction alongside template-based, search-based, and constraint-based repair. Unlike methods restricted to predefined transformations, generative models can propose diverse patches, but this flexibility also increases the space of plausible yet incorrect repairs [1]. A typical pipeline combines fault localization, context construction, patch generation, and validation through compilation, tests, or additional program analyses. Recent work has explored reinforcement learning and iterative repair, where rewards or feedback encode compilation, test outcomes, and patch properties [35]. Such feedback can improve a particular pipeline, but gains depend on test quality and retry budgets. Key benchmarks include Defects4J [27], BugsInPy [44], and QuixBugs [33]. Persistent risks include regression introduction, test-suite overfitting, and semantically vacuous patches that suppress symptoms without satisfying the intended specification. 5.3 Software Testing and Fuzzing LLMs have been applied to test generation and to the automation of testing activities. Execution feedback can help models repair imports, assertions, and setup code, but generated tests may reproduce implementation behavior rather than validate the intended specification. TestGenEval evaluates real-world test authoring, completion, and coverage improvement across Python repositories [23]; project-level benchmarks further expose compilation and cascading-context failures that function-level tasks do not capture. Metamorphic testing, a technique for detecting software faults by verifying relations across program executions, has entered a bidirectional relationship with LLMs. Zheng et al. [50] distinguish metamorphic testing applied to LLMs themselves from LLMs applied to generating metamorphic relations and test oracles. Fuzzing has been extended by LLM-guided input generation. Rather than relying solely on mutation-based strategies, LLM-guided fuzzers can generate semantically meaningful inputs for structured formats and protocols. Yu et al. [49] review human-in-the-loop fuzzing and identify interaction modes including monitoring, strategy steering, and collaborative exploration. Wienczkowski et al. [45] describe a “structural-adaptive fragmentation” in which program analysis, DevSecOps, and LLM-based testing remain insufficiently integrated. 5.4 Code Review and Documentation Code review automation has expanded from static analysis and pattern matching to generative review. Khan et al. [28] document systems that produce comments, estimate change quality, or propose refinements. CodeReviewer provides data and tasks for change-quality estimation, review-comment generation, and code refinement [32]. CRUXEval instead probes input and output prediction for short programs [18]; it is useful for execution reasoning but should not be treated as a direct code-review benchmark. Code summarization and documentation generation are common LLM applications. Models can produce function-level docstrings, file-level overviews, and architectural descriptions, but linguistic fluency is not evidence of factual consistency. Evaluation should therefore check claims against signatures, call sites, tests, and repository evolution, particularly when documentation describes API behavior or security assumptions. 5.5 Software Design and Maintainability Beyond the coding phase, LLMs are increasingly applied to requirements engineering, architecture design, and design-pattern recommendation. Yu et al. [48] identify a mismatch between academic emphasis on code-level tasks and industrial demand for requirements, architecture, and reliability support. Matias et al. [36] similarly identify potential maintainability benefits alongside risks from hallucinations, inconsistent context, and weak lifecycle evidence. Maintainability claims require different evidence from one-shot correctness. A patch that passes tests today may increase coupling, duplicate logic, weaken an abstraction boundary, or create documentation debt. Useful longitudinal outcomes include subsequent defect density, review and rework effort, code churn, dependency risk, architectural conformance, and the rate at which generated changes are reverted. Current evidence is much stronger for immediate task completion than for these lifecycle outcomes, so productivity and maintainability should not be used as synonyms. 5.6 Repository-Level Agents and Human Work Repository-level agents combine issue interpretation, search, planning, editing, execution, and revision. Their apparent capability is therefore a property of the whole configuration ℰE, not only the base model. Retrieval quality determines which code and documentation become visible; the harness determines editing and tool behavior; the environment determines which tests can run; and retry limits determine how much search is permitted. Evaluations should include component ablations and report failed trajectories, not only the final successful patch. Human involvement also changes the task. An agent that independently resolves an issue is not directly comparable to a copilot whose user selects files, diagnoses failures, or approves intermediate plans. Human effort should be measured through intervention count, active time, review burden, correction type, and acceptance or rollback decisions. This makes it possible to distinguish automation from productivity support and to identify cases where faster code production merely transfers effort to review, security triage, or maintenance. 6 LLMs for Software Security 6.1 Vulnerability Detection Detecting security vulnerabilities in source code is one of the most critical and challenging LLM4Security applications. Unlike general bug detection, vulnerability detection requires reasoning about subtle semantic properties – improper input validation, buffer overflows, injection flaws, and access control violations – that may span multiple functions or files [46]. For analysis, we group vulnerability-detection approaches into three overlapping families rather than strict chronological generations. Representation-learning approaches fine-tune encoder models such as CodeBERT [13] for function-level classification on datasets including Devign [52] and Big-Vul [12]. Structure-aware approaches incorporate control-flow, data-flow, or syntax representations. Generative reasoning approaches prompt decoder-only models to produce classifications, explanations, or candidate data-flow traces, sometimes with retrieval and tool support. The families differ in interpretability and context capacity, but all remain sensitive to dataset construction and repository context. Sheng et al. [39] survey LLM-based vulnerability detection techniques, covering model architectures, adaptation strategies, and evaluation protocols. Zhou et al. [51] organize vulnerability detection and repair approaches from direct prompting and fine-tuning to multi-step reasoning. Detection granularity ranges from function-level through slice-level to repository-level analysis. Cross-language and multimodal approaches integrate code with commits, issue reports, and security advisories. Representative resources include Big-Vul and DiverseVul [12, 10], Devign [52], ReVeal [7], and PrimeVul [9]. These datasets differ substantially in labeling quality, class balance, deduplication, and temporal splitting; consequently, scores across them should not be treated as interchangeable. Gao et al. [16] further show that strong benchmark results do not by themselves establish production readiness. 6.2 Vulnerability Repair and Patch Generation Automated vulnerability repair extends APR to the security domain with heightened correctness requirements: a faulty patch may fail to remove the vulnerability or may introduce a new attack surface. Context-aware repair therefore benefits from information about the vulnerability category, affected paths, and existing mitigations [51]. CVEfixes provides a corpus linking disclosed vulnerabilities to real fixes [4]; it can support repair research but is not itself a complete repair-evaluation protocol. SecurityEval instead evaluates generated code from a software-security perspective and should be classified under secure code generation rather than vulnerability repair [40]. Integration of vulnerability repair with general APR pipelines represents an emerging trend. Rather than treating security bugs as a separate category, unified repair frameworks aim to localize a weakness, generate a patch, preserve intended behavior, and validate exploit resistance within one pipeline. SEC-bench, for example, evaluates proof-of-concept generation and vulnerability patching in reproducible task environments [30]; repository-level secure-generation benchmarks similarly combine project context with functional and dynamic security validation [38, 42]. These designs better align the evidence with the claim, although their project, language, and vulnerability coverage remains bounded. 6.3 Malware Analysis and Threat Intelligence Beyond source code analysis, LLMs have been applied to binary-level and behavioral security tasks. Malware detection and classification leverage LLMs’ pattern recognition capabilities to identify malicious code patterns, while behavioral analysis uses LLMs to reason about execution traces and system call sequences. Xu et al. [46] document LLM applications in threat intelligence, incident response, and security log analysis, noting the emergence of autonomous security agents that orchestrate multi-step security workflows. Hybrid systems combining traditional program analysis with LLM reasoning represent a particularly promising direction. Static and dynamic analysis provide precise structural information about code and execution, while LLMs contribute semantic understanding and natural language reasoning. This synergy is especially powerful for tasks like vulnerability reachability analysis – determining whether a detected vulnerability is actually exploitable in a given deployment context. 6.4 Security Testing and Exploit Generation LLM-guided fuzzing augments conventional fuzzers with semantically informed seeds, harness components, or protocol-aware inputs. Wienczkowski et al. [45] organize adaptive security testing around structural analysis, DevSecOps integration, feedback-driven fuzzing, LLM-based test generation, and hybrid systems. They identify insufficient integration among program analysis, development pipelines, and LLM-based generation as a central obstacle. Claims of improved coverage must therefore separate the contribution of the LLM from the coverage guidance and execution infrastructure supplied by the underlying fuzzer. Exploit generation – constructing inputs that demonstrate practical exploitability – is inherently dual use. LLM-based security agents may assist with triage, incident analysis, and response orchestration, but evidence for fully autonomous deployment remains limited [46]. False positives can increase analyst workload, while false negatives can miss critical threats. Human approval, audit logs, scoped tool permissions, and reproducible execution evidence are therefore important design requirements for security-agent evaluations. 6.5 Cross-Domain Integration The intersection of software engineering and security is most visible at four points in the development workflow. First, generated code should be evaluated for both functional correctness and security properties rather than unit-test passage alone. Second, repair systems should verify that a patch removes the targeted weakness without breaking behavior or creating a new vulnerability. Third, repository-level agents should consume feedback from tests, static analyzers, dependency scanners, and policy checks within the same refinement loop. Fourth, agent authority should depend on risk: read-only analysis, patch proposal, test execution, and deployment are materially different capabilities. These integration points motivate evaluations that report secure-and-functional success, evidence provenance, tool permissions, and the amount of human intervention required. 7 Benchmarks and Evaluation 7.1 Code Generation Benchmarks The code generation landscape is served by a diverse ecosystem of benchmarks, summarized in Table 4. HumanEval [8] (164 hand-written Python problems) and MBPP [2] (∼ 1,000 crowd-sourced Python problems) remain common function-level evaluations, typically measured via pass@k. HumanEval+ [34] adds substantially more tests to expose solutions that pass weak suites. DS-1000 targets realistic data-science problems across major Python libraries [29], while BigCodeBench evaluates complex instructions and diverse library calls [53]. LiveCodeBench continuously incorporates newly released contest problems and also evaluates self-repair, execution, and output prediction [24]. MultiPL-E translates HumanEval and MBPP to 18 additional programming languages [6]. At repository scale, SWE-bench [26] evaluates issue resolution on real GitHub projects. These benchmarks measure different capabilities and should be selected according to the artifact and deployment claim under study. Table 4: Key benchmarks for LLM4SE and LLM4Security. Category Benchmark Primary Task Code Generation HumanEval/HumanEval+ Function-level (Python) MBPP Function-level (Python) SWE-bench Verified Repository-level DS-1000 Data science code BigCodeBench/LiveCodeBench Multi-library/Continuous Repair Defects4J Real-world Java bugs BugsInPy Real-world Python bugs QuixBugs Multilingual bug fixing Security Detection Big-Vul C/C++ vulnerability data Devign/ReVeal Function-level detection DiverseVul Diverse vulnerability types PrimeVul Deduplicated temporal evaluation Security Repair CVEfixes Vulnerability–fix corpus Secure Generation SecurityEval Prompt-level security Secure Generation SecRepoBench/RealSec-bench Repository-level joint evaluation Security Agents SEC-bench Executable PoC and patching tasks Security Testing OSS-Fuzz Real-world fuzzing targets Understanding CodeXGLUE/CodeSearchNet Multi-task/Search 7.2 Code Repair and Testing Benchmarks Automated program repair is commonly evaluated on Defects4J [27], BugsInPy [44], and QuixBugs [33]. These resources differ in language, project realism, fault complexity, and test adequacy, so a repair result on short algorithmic programs does not establish repository-scale effectiveness. TestGenEval evaluates test authoring and completion on real Python repositories [23]. For fuzzing research, OSS-Fuzz offers real open-source targets and operational bug histories [11]; nevertheless, coverage or crash counts remain sensitive to time budgets, harness quality, and the underlying fuzzer configuration. 7.3 Security Benchmarks Security evaluation relies on resources with different purposes. Big-Vul [12], Devign [52], ReVeal [7], PrimeVul [9], and DiverseVul [10] support vulnerability-detection research but differ in construction, labeling, balance, and split strategy. PrimeVul emphasizes deduplication, chronological splitting, and realistic class imbalance, illustrating why scores on older datasets may not transfer to deployment settings. CVEfixes links vulnerabilities to fixing commits and code changes [4]. SecurityEval contains 130 Python prompts covering 75 CWE types and evaluates generated code from a security perspective [40]. Newer resources increasingly test functionality and security together. SecRepoBench supplies repository context, developer-written tests, and dynamic proof-of-vulnerability inputs for secure code completion [38]; RealSec-bench targets secure generation in real Java repositories and explicitly uses a joint secure-and-functional outcome [42]; SEC-bench evaluates agent behavior on executable proof-of-concept and patching tasks [30]. These benchmarks narrow the construct-validity gap, but they should still not be collapsed into a single leaderboard because they evaluate different languages, artifacts, authority levels, and threat models. 7.4 Metric Limitations and Evaluation Gaps Despite the breadth of available benchmarks, several evaluation gaps persist. First, test-suite adequacy: pass@k estimates the probability that at least one of k generated programs passes the available unit tests [8]; it measures functional correctness only with respect to those tests and does not establish full specification compliance or security. Second, benchmark saturation: strong HumanEval scores reduce its discriminative power for newer systems. Third, data leakage: training-data contamination may allow models to reproduce memorized solutions. Fourth, security-specific validity: accuracy, recall, and false-positive rate do not capture triage cost, exploitability, or time to remediation. Static analyzers are valuable validators but imperfect ground truth; human-validated comparison has exposed substantial per-sample disagreement between analyzer reports and actual security labels [14]. Fifth, agentic evaluation: multi-step systems [20] combine retrieval, reasoning, editing, tool use, and execution, making component attribution and reproducibility difficult. Sixth, selection by repeated sampling: best-of-k results can improve while cost, latency, and the number of insecure discarded candidates also increase. 7.5 Minimum Reporting Protocol To make results interpretable and reproducible, we propose the reporting protocol in Table 5. It applies to both model-centric and agentic studies; fields may be marked not applicable, but should not be silently omitted. The protocol operationalizes the evaluation configuration in Section 4 and allows reviewers to identify whether a reported gain comes from the model, context, harness, tools, budget, validator, or human assistance. Table 5: Minimum reporting protocol for LLM-based software engineering and security evaluations. Dimension Required disclosure Reason for inclusion Task and artifact Task definition, artifact granularity, language, repository version, and intended deployment claim Prevents function-level evidence from being generalized to repositories or operations Model and adaptation Exact model and date/version, decoding parameters, prompts, examples, fine-tuning data, and retrieval configuration Separates model capability from prompt, data, and context effects Harness and tools Agent framework and commit, tools, permissions, network access, sandbox, memory, and stopping rules Makes agent scores attributable and exposes authority-related risk Budget and human input Samples, retries, tokens, wall time, monetary or compute cost, interventions, and approval decisions Prevents hidden search and human labor from being counted as autonomous capability Data and contamination Dataset version, deduplication, temporal and project split, overlap checks, and unavailable training-data caveats Calibrates memorization and transfer claims Validation Functional tests, coverage or mutation evidence, security oracle, threat model, regressions, and manual adjudication Aligns validators with correctness and security claims Uncertainty and failures Repeated runs, confidence intervals where meaningful, negative results, failure categories, and sensitivity analyses Avoids conclusions based only on a favorable run or aggregate score Reproducibility Code, prompts, logs, containers, dependencies, seeds, generated artifacts, and archival identifier Enables independent verification after services and repositories change Table 6 summarizes the dominant adaptation strategies and their applicability across the SE and security task landscape, highlighting how different tasks benefit from different LLM utilization paradigms. Table 6: LLM adaptation strategies and their applicability across SE and security tasks. ∙ = primary strategy; ∘ = commonly used; – = rarely applied. Task Prompting Fine-tuning RAG Agent-based Code Generation (function) ∙ ∘ ∘ – Code Generation (repo-level) ∘ – ∙ ∙ Automated Program Repair ∘ ∙ ∘ ∘ Test Generation ∙ ∘ ∘ ∘ Code Review ∙ ∘ ∘ – Vulnerability Detection ∘ ∙ ∙ – Vulnerability Repair ∘ ∙ ∘ ∘ Fuzzing ∙ – – ∙ 7.6 Cross-Task Synthesis Table 7 consolidates the evidence across task families. Three patterns answer the research questions. First, the required context expands from a prompt and function signature to repository state, build systems, issue history, and organizational policy; adaptation correspondingly shifts from prompting and fine-tuning toward retrieval and tool-mediated agents (RQ1–RQ2). Second, evaluation validity depends on the claimed unit of deployment: function tests cannot justify repository-level or security claims, and detector accuracy cannot substitute for exploitability or analyst-cost evidence (RQ3). Third, the SE–security intersection is not a separate terminal task but a set of constraints on generation, repair, testing, and agent authority (RQ4). A high-quality evaluation should therefore report not only whether a task completed, but also which context and tools were available, which functional and security checks passed, what regressions occurred, and how much human intervention was required. Table 7: Cross-task synthesis of artifacts, adaptation strategies, evidence, and principal validity threats. Task family Artifact and context Common adaptation Typical evidence Principal validity threat Function synthesis Specification, signature, local tests Prompting; fine-tuning pass@k and expanded unit tests Weak tests, contamination, and limited API realism Repository issue resolution Repository, issue, build and test environment Retrieval; planning; tool-using agents Resolved issues and regression tests Changing benchmark versions, retry budgets, and hidden infrastructure Program repair Fault location, buggy code, tests, execution feedback Fine-tuning; iterative prompting; agents Plausible and correct patches Test-suite overfitting and behavior regressions Test generation and fuzzing Code under test, harness, runtime feedback Prompting; execution-guided agents Coverage, mutation score, crashes, valid tests Oracle weakness and confounding from the underlying fuzzer Review and documentation Diff, repository context, conventions, APIs Prompting; retrieval; task-specific tuning Comment relevance and human judgment Subjective labels and fluent but unsupported explanations Vulnerability detection Function, slice, data flow, repository context Fine-tuning; structure-aware models; retrieval Precision, recall, F1, localization Duplicates, temporal leakage, unrealistic balance, and label noise Secure generation and repair Specification, CWE/CVE context, tests, analyzers Prompting; fine-tuning; constrained or iterative generation Functional tests plus security checks Proxy analyzers, incomplete threat models, and new vulnerabilities Security agents Multi-tool environment, permissions, logs, policies Retrieval; planning; tool use; human approval Task completion, evidence trace, analyst effort Unsafe actions, irreproducible environments, and unbounded authority 8 Challenges and Open Problems 8.1 Reliability, Hallucination, and Calibration Code hallucinations—outputs that are syntactically plausible but semantically incorrect, insecure, or inconsistent with the repository—remain a fundamental reliability challenge. Gao et al. [15] relate these failures to training data, autoregressive generation, and insufficient context. At repository scale, the failure surface expands to nonexistent symbols, stale APIs, misunderstood build assumptions, incompatible dependency changes, and patches that address the visible symptom while violating an implicit invariant. Execution feedback reduces some errors but creates a risk of validator overfitting. An agent may learn to satisfy the observed tests without implementing the intended behavior, or remove an analyzer warning without eliminating exploitability. Reliability therefore requires diverse evidence: specification-derived tests, regression suites, mutation analysis, static and dynamic checks, differential behavior, and targeted human review. Uncertainty should also be calibrated at the level at which action is taken. A confidence statement attached to a fluent explanation is less useful than calibrated probabilities for patch acceptance, vulnerability presence, or the need for escalation. 8.2 Dataset Realism, Leakage, and Benchmark Governance Dataset construction can dominate apparent model progress. Duplicate or near-duplicate functions inflate random-split performance; vulnerability-fixing commits contain unrelated edits; negative examples may be sampled from unrealistically clean code; and labels based on a commit message or analyzer can be noisy. PrimeVul demonstrates the importance of deduplication, chronological splitting, and realistic imbalance [9]. Similar discipline is needed for generative tasks because benchmark problems, tests, solutions, and repository histories may all be present in pretraining or retrieval corpora. A benchmark should consequently be governed as a versioned scientific instrument. Its release should identify provenance, licenses, filtering, label adjudication, known contamination channels, test coverage, supported claims, and change history. For agent benchmarks, the repository image, dependency registry state, container, network policy, and evaluator must also be frozen. Periodic refreshes can reduce memorization, but they should not silently replace an existing test set because longitudinal comparisons then lose meaning. A stable core, a sequestered evaluation set, and a documented refresh track provide a more defensible compromise. 8.3 Reproducibility, Cost, and Environmental Drift Proprietary model updates, nondeterministic sampling, disappearing APIs, mutable repositories, and evolving package registries make exact replication difficult. Agent evaluations add further variation through search indexes, tool versions, timeouts, concurrency, network access, and retry policies. A result should therefore include both a logical specification of the task and an executable environment whenever licensing permits. Containers alone are insufficient if external services, model snapshots, or package artifacts remain mutable. Cost is also part of effectiveness. A system that obtains a higher best-of-k success rate through many samples or long repair loops may be unsuitable for routine use. Relevant quantities include tokens, model calls, wall time, compute or monetary cost, tool executions, failed candidates, and human review. Reporting a Pareto frontier over quality, security, cost, and latency is more informative than ranking systems by task success alone. For security triage, analyst time spent dismissing false positives may dominate model inference cost. 8.4 Adversarial Robustness and Agent Governance Security systems face distribution shift and active adversaries. Semantics-preserving renaming, control-flow restructuring, dead-code insertion, misleading comments, poisoned retrieval documents, and tool-output manipulation can alter model judgments. Repository-level agents additionally process untrusted files and issue text that may contain prompt injections. Evaluations should distinguish robustness to natural code evolution from robustness to deliberate manipulation and should test both the model and the surrounding tool chain. Governance becomes essential when agents can execute commands or modify shared assets. The authority tiers in Section 4 imply progressively stronger controls: isolation and content provenance for read-only analysis; mandatory diff review for proposed patches; sandboxing and egress restrictions for tool execution; and explicit approval, rollback, and audit requirements for changes to production or security infrastructure. Safety refusal rates alone do not measure these system properties. The relevant question is whether unsafe actions are prevented, detected, contained, and recoverable under the stated threat model. 8.5 The Academic–Industry Evidence Gap Yu et al. [48] identify mismatches in task emphasis, scale realism, and integration. Academic work is concentrated on code-level tasks with automatically checkable outcomes, whereas industrial adoption must also address requirements, architecture, migration, reliability, compliance, ownership, and coordination. Million-line repositories, private dependencies, legacy build systems, organization-specific conventions, and regulated data cannot be represented fully by small public benchmarks. The gap is not solved simply by obtaining proprietary data. Industrial studies also require credible counterfactuals: comparison teams or periods, developer-experience stratification, workflow instrumentation, and measurement of downstream review, incidents, and maintenance. Matias et al. [36] emphasize that long-term maintainability remains uncertain. Longitudinal studies should follow generated changes beyond merge time and should report rejected and reverted outputs, not only accepted suggestions. 8.6 Threats to the Validity of This Survey Selection validity is limited because this is a structured narrative review built from a traceable seed corpus rather than a completed exhaustive search and dual-reviewer screening process. Relevant work may be absent, and highly visible English-language papers and preprints may be over-represented. The protocol and templates make this limitation auditable but do not remove it. Construct validity is threatened by inconsistent uses of terms such as correctness, agent, autonomy, vulnerability detection, and secure generation; the taxonomy and assurance framework reduce but cannot eliminate these differences. Conclusion validity is limited by heterogeneous tasks and configurations, which is why source-reported scores are not pooled. External validity remains constrained by the dominance of a small set of languages, open-source repositories, curated datasets, and short evaluation windows. Temporal validity is unusually fragile because models, APIs, scaffolds, and leaderboards change rapidly. The cutoff date and benchmark versions should therefore accompany any reuse of the conclusions. 8.7 A Testable Research Agenda Table 8 translates broad directions into falsifiable questions and minimum evidence. The agenda favors studies that isolate mechanisms, preserve failure evidence, and jointly evaluate engineering value and security risk. Table 8: Research agenda derived from the evidence gaps. Direction Testable research question Minimum credible evaluation Joint secure generation Does an intervention improve J@kJ@k rather than shifting failures between functionality and security? Paired functional and threat-model-aligned tests; separate F, S, and joint outcomes; fixed sampling budget; regression and failure analysis Repository context and agents Which gains come from retrieval, the model, the harness, tools, or additional retries? Versioned repositories and environments; controlled component ablations; equalized budgets; trajectory and tool logs; multiple projects Verifier ensembles and human feedback When do tests, analyzers, dynamic exploits, and expert review provide complementary rather than correlated evidence? Human-adjudicated subset; precision, recall, calibration, disagreement analysis, analyst time, and sensitivity to the chosen oracle Risk-bounded autonomy How does increasing authority change utility, unsafe-action probability, and oversight cost? Evaluation across G0G_0–G3G_3 tiers; adversarial tasks; least-privilege controls; approval and rollback logs; incident-oriented metrics Lifecycle maintainability Do LLM-authored changes reduce total engineering effort without increasing future defects or debt? Longitudinal repository study; matched controls; review and rework time; defects, reversions, churn, and architecture measures Continuous and multilingual evaluation Do methods transfer across time, languages, ecosystems, and newly disclosed weakness patterns? Temporal and project-held-out splits; multiple language/toolchain families; benchmark refresh protocol; contamination audit and uncertainty reporting 9 Conclusion This survey connected software engineering and software security through a shared taxonomy, an explicit evaluation configuration, and an assurance-centered view of evidence. The synthesis answers the four research questions as follows. For RQ1, the tasks range from function-level generation and classification to repository issue resolution, security patching, fuzzing, and tool-mediated analysis; the decisive difference is often the artifact and context available rather than the task label alone. For RQ2, prompting and fine-tuning remain important for bounded tasks, while retrieval, execution feedback, and agents become more consequential as repository context and iteration increase. These mechanisms change both capability and failure modes, and their effects cannot be attributed to the base model without controlled ablations. For RQ3, credible evidence must match the deployment claim. Unit tests support only test-relative functional claims; vulnerability labels support classification claims; and repository or operational claims require versioned environments, realistic context, regression checks, security oracles, budgets, and human-effort evidence. The proposed configuration ℰE, evidence ladder, and minimum reporting protocol make these requirements explicit. For RQ4, security is not a downstream task appended to development. It is a constraint on generation, repair, testing, review, and agent authority. Joint outcomes such as J@kJ@k should therefore accompany separate functional and security results. The literature supports cautious optimism rather than unconditional automation. Repository access and iterative tool use can improve task completion, yet they also introduce harness dependence, larger attack surfaces, higher cost, and reproducibility challenges. Static analyzers, tests, and LLM judges are useful components of an assurance case but are not individually sufficient ground truth. Long-term maintainability, multilingual transfer, adversarial robustness, and the organizational cost of oversight remain under-evidenced. The central research priority is consequently not another isolated leaderboard, but a cumulative evidence infrastructure: versioned benchmarks, executable environments, contamination-aware splits, joint functional and security validation, transparent agent permissions, complete trajectory and budget logs, and longitudinal human-centered outcomes. Progress should be claimed in proportion to the evidence. Under this standard, an LLM-based software system is trustworthy not because it produces a plausible patch or achieves a high scalar score, but because its behavior is supported by a reproducible, threat-model-aligned, and governance-aware assurance case. References [1] A. Anand, A. Gupta, N. Yadav, and S. Bajaj (2024) A comprehensive survey of AI-driven advancements and techniques in automated program repair and code generation. Note: arXiv preprint arXiv:2411.07586 Cited by: §5.2. [2] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton (2021) Program synthesis with large language models. Note: arXiv preprint arXiv:2108.07732 Cited by: §5.1, §7.1. [3] E. Basic and A. Giaretta (2026) From vulnerabilities to remediation: a systematic literature review of LLMs in code security. Note: arXiv preprint arXiv:2412.15004 Cited by: Table 1, §1. [4] G. P. Bhandari, A. Naseer, and L. Moonen (2021) CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE), New York, NY, USA, p. 30–39. Note: arXiv:2107.08760 External Links: Document Cited by: §6.2, §7.3. [5] H. Bhati (2026) Agentic AI in the software development lifecycle: architecture, empirical evidence, and the reshaping of software engineering. Note: arXiv preprint arXiv:2604.26275 Cited by: §3.2, §5.1, Table 3. [6] F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, et al. (2022) MultiPL-E: a scalable and extensible approach to benchmarking neural code generation. Note: arXiv preprint arXiv:2208.08227 Cited by: §7.1. [7] S. Chakraborty, R. Krishna, Y. Ding, and B. Ray (2022) Deep learning based vulnerability detection: are we there yet?. IEEE Transactions on Software Engineering 48 (9), p. 3280–3296. Note: arXiv:2009.07235 External Links: Document Cited by: §6.1, §7.3. [8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. Note: arXiv preprint arXiv:2107.03374 Cited by: §3.1, §3.2, §5.1, §7.1, §7.4. [9] Y. Ding et al. (2025) Vulnerability detection with code language models: how far are we?. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE), Los Alamitos, CA, USA, p. 1729–1741. Note: arXiv:2403.18624 External Links: Document Cited by: §6.1, §7.3, §8.2. [10] Y. Ding, C. Sun, S. Liu, and H. Gao (2023) DiverseVul: a new vulnerable source code dataset for deep learning based vulnerability detection. Note: arXiv preprint arXiv:2304.00409 Cited by: §6.1, §7.3. [11] Z. Y. Ding and C. Le Goues (2021) An empirical study of OSS-Fuzz bugs. Note: arXiv preprint arXiv:2103.11518 Cited by: §7.2. [12] J. Fan, Y. Li, S. Wang, and T. N. Nguyen (2021) Big-vul: a C/C++ vulnerability dataset with comprehensive code representations. Note: arXiv preprint arXiv:2105.14511 Cited by: §6.1, §6.1, §7.3. [13] Z. Feng et al. (2020) CodeBERT: a pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, p. 1536–1547. External Links: Document Cited by: §3.1, §6.1. [14] E. Firouzi and M. Ghafari (2026) Persistent human feedback, LLMs, and static analyzers for secure code generation and vulnerability detection. Note: arXiv preprint arXiv:2602.05868 Cited by: §7.4. [15] C. Gao, G. Fan, C. Y. Chong, S. Chen, C. Liu, D. Lo, Z. Zheng, and Q. Liao (2025) A systematic literature review of code hallucinations in LLMs: characterization, mitigation methods, challenges, and future directions for reliable AI. Note: arXiv preprint arXiv:2511.00776 Cited by: Table 1, §8.1. [16] Z. Gao, H. Wang, Y. Zhou, W. Zhu, and C. Zhang (2023) How far have we gone in vulnerability detection using large language models. Note: arXiv preprint arXiv:2311.12420 Cited by: §6.1. [17] Y. Ge, L. Mei, Z. Duan, T. Li, Y. Zheng, Y. Wang, L. Wang, J. Yao, T. Liu, Y. Cai, B. Bi, F. Guo, J. Guo, S. Liu, and X. Cheng (2025) A survey of vibe coding with large language models. Note: arXiv preprint arXiv:2510.12399 Cited by: §1, §5.1. [18] A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang (2024) CRUXEval: a benchmark for code reasoning, understanding and execution. Note: arXiv preprint arXiv:2401.03065 Cited by: §5.4. [19] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, et al. (2024) DeepSeek-Coder: when the large language model meets programming. Note: arXiv preprint arXiv:2401.14196 Cited by: §1, §3.1. [20] J. Guo, Y. Zhao, Y. Liu, L. Li, K. Wang, and D. Lo (2025) A comprehensive survey on benchmarks and solutions in software engineering of LLM-empowered agentic system. Note: arXiv preprint arXiv:2510.09721 Cited by: Table 1, §1, §7.4. [21] V. Gurgul, R. Gubela, and S. Lessmann (2026) The state of generative AI in software development: insights from literature and a developer survey. Note: arXiv preprint arXiv:2603.16975 Cited by: §1. [22] X. Hou et al. (2024) Large language models for software engineering: a systematic literature review. Note: ACM Transactions on Software Engineering and MethodologyarXiv:2308.10620 External Links: Document Cited by: Table 1, §1, §1. [23] K. Jain, G. Synnaeve, and B. Rozière (2025) TestGenEval: a real world unit test generation and test completion benchmark. Note: Proceedings of the 13th International Conference on Learning Representations (ICLR)arXiv:2410.00752 Cited by: §5.3, §7.2. [24] N. Jain, K. Han, A. Gu, W. Li, F. Yan, et al. (2024) LiveCodeBench: holistic and contamination free evaluation of large language models for code. Note: arXiv preprint arXiv:2403.07974 Cited by: §7.1. [25] C. Jiang et al. (2026) Large language models for multilingual code intelligence: a survey. Note: arXiv preprint arXiv:2604.25960 Cited by: §5.1. [26] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024) SWE-bench: can language models resolve real-world GitHub issues?. Note: Proceedings of the 12th International Conference on Learning Representations (ICLR)arXiv:2310.06770 Cited by: §1, §5.1, Table 3, §7.1. [27] R. Just, D. Jalali, and M. D. Ernst (2014) Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA), New York, NY, USA, p. 437–440. Cited by: §5.2, §7.2. [28] T. I. Khan et al. (2026) A survey of code review benchmarks and evaluation practices in Pre-LLM and LLM era. Note: arXiv preprint arXiv:2602.13377 Cited by: §1, §5.4. [29] Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. W. Yih, D. Fried, S. Wang, and T. Yu (2023) DS-1000: a natural and reliable benchmark for data science code generation. Note: Proceedings of the 40th International Conference on Machine Learning (ICML)arXiv:2211.11501 Cited by: §7.1. [30] H. Lee, Z. Zhang, H. Lu, and L. Zhang (2025) SEC-bench: automated benchmarking of LLM agents on real-world software security tasks. Note: arXiv preprint arXiv:2506.11791 Cited by: §4.1, §6.2, §7.3. [31] R. Li et al. (2023) StarCoder: may the source be with you!. Note: Transactions on Machine Learning ResearcharXiv:2305.06161 Cited by: §1, §3.1. [32] Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, et al. (2022) Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), New York, NY, USA, p. 1035–1047. External Links: Document Cited by: §5.4. [33] D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama (2017) QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, New York, NY, USA, p. 55–56. External Links: Document Cited by: §5.2, §7.2. [34] J. Liu, C. S. Xia, Y. Wang, and L. Zhang (2023) Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 21558–21572. Cited by: §5.1, §7.1. [35] M. R. Masud et al. (2026) Reward engineering for reinforcement learning in software tasks. Note: arXiv preprint arXiv:2601.19100 Cited by: §5.2. [36] B. C. Matias et al. (2026) A survey on large language model impact on software evolvability and maintainability. Note: arXiv preprint arXiv:2601.20879 Cited by: §5.5, §8.5. [37] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, et al. (2023) Code Llama: open foundation models for code. Note: arXiv preprint arXiv:2308.12950 Cited by: §1, §3.1. [38] C. Shen, C. Dilgren, P. Chiniya, L. Griffith, Y. Ding, and Y. Chen (2026) SecRepoBench: benchmarking code agents for secure code completion in real-world repositories. Note: arXiv preprint arXiv:2504.21205 Cited by: §4.1, §6.2, §7.3. [39] Z. Sheng, Z. Zhan, Z. Qin, S. He, and J. Huang (2025) LLMs in software security: a survey of vulnerability detection techniques and insights. Note: arXiv preprint arXiv:2502.07049 Cited by: §1, §3.2, §6.1. [40] M. L. Siddiq and J. C. S. Santos (2022) SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security, New York, NY, USA, p. 29–33. External Links: Document Cited by: §6.2, §7.3. [41] Y. Tao et al. (2025) Retrieval-augmented code generation: a survey with focus on repository-level approaches. Note: arXiv preprint arXiv:2510.04905 Cited by: §3.2, §5.1. [42] Y. Wang, Z. Zhang, C. Wang, X. Xu, M. Liu, Y. Wang, J. Chen, and Z. Zheng (2026) RealSec-bench: a benchmark for evaluating secure code generation in real-world repositories. Note: arXiv preprint arXiv:2601.22706 Cited by: §4.1, §6.2, §7.3. [43] Y. Wang et al. (2021) CodeT5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, p. 8696–8708. External Links: Document Cited by: §3.1. [44] R. Widyasari, S. Q. Sim, C. Lok, H. Qi, J. Phan, Q. Tay, et al. (2020) BugsInPy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), New York, NY, USA, p. 1556–1560. External Links: Document Cited by: §5.2, §7.2. [45] M. Wienczkowski et al. (2026) Adaptive and AI-augmented security testing: a systematic survey. Note: arXiv preprint arXiv:2604.27000 Cited by: §5.3, §6.4. [46] H. Xu et al. (2025) Large language models for cyber security: a systematic literature review. Note: ACM Transactions on Software Engineering and MethodologyarXiv:2405.04760 External Links: Document Cited by: Table 1, §1, §1, §6.1, §6.3, §6.4. [47] G. Yang D. Lo et al. (2025) Large language model for Verilog code generation: literature review and the road ahead. Note: arXiv preprint arXiv:2512.00020 Cited by: §3.2, §5.1. [48] H. Yu D. Lo et al. (2025) Aligning academia with industry: an empirical study of industrial needs and academic capabilities in AI-driven software engineering. Note: arXiv preprint arXiv:2512.15148 Cited by: §5.5, §8.5. [49] J. Yu et al. (2026) Human in the loop for fuzz testing: literature review and the road ahead. Note: arXiv preprint arXiv:2603.13411 Cited by: §5.3. [50] Z. Zheng et al. (2026) Bidirectional empowerment of metamorphic testing and large language models: a systematic survey. Note: arXiv preprint arXiv:2605.13898 Cited by: §5.3. [51] X. Zhou, T. Zhang, and D. Lo (2024) Large language model for vulnerability detection and repair: literature review and the road ahead. Note: arXiv preprint arXiv:2404.02525 Cited by: Table 1, §1, §6.1, §6.2. [52] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu (2019) Devign: effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Note: Advances in Neural Information Processing Systems (NeurIPS), Vol. 32 Cited by: §6.1, §6.1, §7.3. [53] T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, et al. (2025) BigCodeBench: benchmarking code generation with diverse function calls and complex instructions. Note: Proceedings of the 13th International Conference on Learning Representations (ICLR)arXiv:2406.15877 Cited by: §7.1.