Paper deep dive
Security in LLM-Assisted Coding: A Systematic Literature Review
Liang Yuan, Maryam Khanian Najafabadi, Mashud Rana, Varvara Vetrova
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models accelerate software development but introduce severe security vulnerabilities. This systematic review investigates security risks in automated coding, benchmark limitations, and the role of human developers. Findings indicate that models frequently generate insecure code and faulty patches due to training data biases. Additionally, the models themselves are susceptible to direct attacks. Current evaluation methods suffer from data contamination and lack robust security metrics. Consequently, human oversight remains essential. Safer integration of automation will require more security-aware model architectures, stronger security-specific benchmarks, and more effective human-AI collaboration frameworks to mitigate these systemic risks.
Tags
Links
- Source: https://arxiv.org/abs/2610.15482v1
- Canonical: https://arxiv.org/abs/2610.15482v1
Trouble viewing inline? Open PDF directly â
Full Text
152,786 characters extracted from source content.
Expand or collapse full text
Security in LLM-Assisted Coding: A Systematic Literature Review DOI: X.XXXXXXXJournal: JACMVolume: 3741117CCS: Security and privacy Software and application security Liang Yuan Note: Corresponding author email: liang.yuan@acu.edu.au Affiliation: Australian Catholic University, North Sydney, New South Wales, Australia , Maryam Khanian Najafabadi email: maryam.khanian@acu.edu.au Affiliation: Australian Catholic University, North Sydney, New South Wales, Australia , Mashud Rana email: mashud.rana@acu.edu.au Affiliation: Australian Catholic University, North Sydney, New South Wales, Australia and Varvara Vetrova email: varvara.vetrova@acu.edu.au Affiliation: Australian Catholic University, North Sydney, New South Wales, Australia 2026 Abstract. Large Language Models accelerate software development but introduce severe security vulnerabilities. This systematic review investigates security risks in automated coding, benchmark limitations, and the role of human developers. Findings indicate that models frequently generate insecure code and faulty patches due to training data biases. Additionally, the models themselves are susceptible to direct attacks. Current evaluation methods suffer from data contamination and lack robust security metrics. Consequently, human oversight remains essential. Safer integration of automation will require more security-aware model architectures, stronger security-specific benchmarks, and more effective human-AI collaboration frameworks to mitigate these systemic risks. Keywords: LLM, Software security, AI safety 1. Introduction The advent of Large Language Models (LLMs) marks a transformative period in software engineering, offering unprecedented capabilities in automating and assisting various coding tasks. From code generation and completion (Kharma et al., 2026; Mu et al., 2024; Jiang et al., 2024; Ding et al., 2024) to automated program repair (APR) and vulnerability patching (Pearce et al., 2023; Zhou et al., 2024b; Fu et al., 2024a; Wei et al., 2023), LLMs promise to enhance developer productivity and accelerate software development lifecycles (Barke et al., 2023). However, this rapid integration is paralleled by a growing array of security concerns that demand systematic investigation and proactive mitigation strategies (He and Vechev, 2023). A central challenge in securing LLM-assisted coding is that functional correctness does not establish software security. Generated code may satisfy its intended functionality while containing exploitable vulnerabilities, and an automatically produced patch may leave the underlying security issue incompletely resolved or introduce additional weaknesses (Pearce et al., 2022; Gong et al., 2024; Liu et al., 2024b). Assessing these risks is further complicated by limitations in evaluation datasets and pipelines, including benchmark contamination, incomplete vulnerability coverage, and differences between isolated coding tasks and realistic development settings. Human review provides an additional safeguard, but its effectiveness depends on developersâ security expertise and their ability to critically assess model outputs (Mohsin et al., 2024; Perry et al., 2023). These challenges involve two related but distinct concepts: software security and AI safety. Software security concerns the confidentiality, integrity, and availability of software artifacts and systems, including vulnerabilities introduced through code generation, insecure repairs, and adversarial manipulation. AI safety encompasses broader concerns about the dependable, controllable, and socially responsible operation of LLM-based systems, including misuse resistance, human oversight, trust calibration, accountability, and governance (Chow and Li, 2025). The distinction matters because a coding systemâs security depends both on the properties of its outputs and on the conditions under which those outputs are produced, evaluated, and accepted into software systems. The transition from passive code-completion tools to agentic coding systems makes these interactions more consequential. Autonomous coding agents, multi-agent frameworks, and tool-using LLMs can search repositories, invoke external tools, execute tests, edit files, and iteratively revise their outputs (Chow and Li, 2026; Liu et al., 2024c; Nunez et al., 2024; Zheng et al., 2024). These capabilities expand the security problem beyond individual code snippets. Errors can propagate across multiple steps, tool invocation and environment access introduce additional attack surfaces, and increasing autonomy can make intermediate actions less visible to human reviewers. Secure deployment therefore requires attention to workflow-level risks, the limits of human oversight, and the controls governing an agentâs actions. Figure 1. A layered framework for analyzing security in LLM-assisted coding. Existing studies provide substantial coverage of individual parts of this problem. Reviews of automated program repair examine repair techniques and their limitations (Huang et al., 2023; Anand et al., 2024; Zhang et al., 2024), while code-security surveys address vulnerabilities in generated code and the use of LLMs for vulnerability detection and repair (BASIC and GIARETTA, 2025; Sheng et al., 2025b; Zhou et al., 2024a). Broader reviews of AI safety and agentic systems examine oversight, controllability, and governance (Chow and Li, 2025; Chow and Li, 2026). However, these strands provide limited synthesis of how code-level weaknesses, evaluation limitations, human interaction, and increasing autonomy jointly affect security in coding workflows. In particular, the relationship between what current evaluations measure and what is needed to support secure deployment remains insufficiently integrated with analyses of human oversight and agentic risk. To address this gap, this survey provides a systematic review of security in LLM-assisted coding. We focus on five connected issues: the security weaknesses introduced by LLM-based generation and repair, the limitations of current evaluation datasets and pipelines, the changing threat model created by agentic and tool-integrated coding systems, the dual role of human oversight as both safeguard and risk factor, and the governance conditions required for trustworthy deployment. Thus, this review makes three main contributions. First, it synthesises security issues across code generation and repair, human interaction, and adversarial attacks, extending the analysis to the workflow-level risks of agentic and tool-integrated systems. Second, it critically examines evaluation datasets, methodologies, and metrics, with particular attention to benchmark contamination, the limitations of individual analysis methods, and the gap between benchmark settings and deployment conditions. Third, it analyses the role and limitations of human oversight and develops research directions connecting technical safeguards, evaluation practices, and deployment governance. These dimensions are organised through the four-layer framework shown in Fig. 1, which links system capabilities, failure modes, evaluation pipelines, and controls. The framework provides a common structure for examining how different coding capabilities expose particular risks and how those risks can be assessed and mitigated. The remainder of the paper is organised as follows. Section 2 reviews the existing literature and positions this work relative to surveys on software engineering, code security, AI safety, and agentic systems. Section 3 outlines the review methodology. Section 4 analyses the security issues associated with LLM-assisted coding, including the risks introduced by more autonomous agentic workflows. Section 5 examines current evaluation methods and benchmarks, and Section 6 discusses human oversight and human-factor risks in LLM-assisted coding. Finally, Section 7 offers practical insights for academia and industry and lays out directions for future work, before Section 8 concludes the paper. Table 1. Comparison of closely related surveys and the scope of the present review. â indicates substantial coverage; âź indicates partial coverage; Ă indicates that the topic is not a central focus. Survey Year Primary scope Sec Eval. Human Factor Agentic Gov. Main gap Huang et al. (Huang et al., 2023) 2023 APR broadly Ă Ă Ă Ă Ă Not LLM-security focused. Anand et al. (Anand et al., 2024) 2024 APR+code gen. âź âź Ă Ă Ă Broad overview; limited security synthesis. Zhou et al. (Zhou et al., 2024a) 2024 Vulnerability detect./repair â âź Ă Ă Ă Task-specific rather than integrated. Zhang et al. (Zhang et al., 2024) 2024 LLMs for APR âź âź Ă Ă Ă Repair-focused; narrow security scope. Basic and Giaretta (BASIC and GIARETTA, 2025) 2025 LLMs in code security â âź Ă Ă Ă Limited on deployment and governance. Sheng et al. (Sheng et al., 2025b) 2025 Vulnerability detection â â Ă Ă Ă Detection-oriented only. Li and Chow (Chow and Li, 2025) 2025 AI safety/governance âź Ă â âź â Not coding-task specific. Li and Chow (Chow and Li, 2026) 2026 Agentic AI systems âź Ă â â â Not code-security centered. This survey 2026 Security in LLM-assisted coding â â â â â Integrated security-centered view. 2. Related Works Existing surveys explore the integration of LLMs into software engineering tasks like Automated Program Repair (APR) and code generation, increasingly focusing on security implications. Anand et al. (Anand et al., 2024) review 27 recent papers on LLMs in APR and code generation, noting their transformative impact in reducing manual debugging and improving code generation. It contrasts methodologies, identifies trends like using LLMs and feedback loops, and discusses challenges, including functional correctness and security. Similarly, Huang et al. (Huang et al., 2023) provide a broad overview of APR techniques and discuss the opportunities introduced by learning-based methods, including LLMs. More recently, Zhang et al. (Zhang et al., 2024) present a systematic literature review focused specifically on LLM applications in APR, analysing repair strategies, utilisation patterns, and repair scenarios that include security vulnerabilities. Other surveys narrow their focus specifically to the intersection of LLMs and code security. Basic and Giaretta (BASIC and GIARETTA, 2025) investigate security vulnerabilities introduced by LLM-generated code, evaluate LLM effectiveness in detecting and fixing vulnerabilities, and explore data poisoning impacts. It aims to provide an in-depth analysis across these three areas. Likewise, Sheng et al. (Sheng et al., 2025b) survey LLM-based software vulnerability detection, examining problem formulations, models, datasets, and evaluation metrics, while Zhou et al. (Zhou et al., 2024a) review the use of LLMs for vulnerability detection and repair and outline future research directions. These studies provide useful foundations, but they often concentrate on specific security tasks, such as detection or repair, without fully integrating broader issues such as benchmark reliability, human-AI trust, deployment governance, or the security implications of increasing model autonomy. A growing body of broader LLM safety research examines issues that intersect closely with secure code generation, including controllability, misuse resistance, human oversight, and deployment governance. Li and Chow (Chow and Li, 2025) help frame risks that extend beyond insecure code artifacts, such as trust miscalibration, weak oversight, and insufficient accountability. These perspectives complement code-centric security analysis by explaining how flawed outputs become operational and organizational risks. Another emerging direction concerns agentic AI systems. Coding systems are increasingly evolving from passive assistants into tool-using agents that can plan, retrieve context, execute tests, and revise files. In another work, Li and Chow (Chow and Li, 2026) show that this shift expands the risk landscape from single-output vulnerabilities to workflow-level risks such as tool misuse, persistent memory, compromised retrieval, and compounding multi-step errors. Table 1 summarizes the main scope and limitations of closely related surveys. Prior reviews tend to focus on APR, vulnerability detection and repair, or broader AI safety questions separately. By contrast, this survey provides an integrated security-centered perspective on LLM-assisted coding across generation, repair, evaluation, human oversight, agentic risk, and deployment governance. We also place particular emphasis on evaluation limitations, including benchmark contamination, the partial visibility of static or dynamic analysis in isolation, and the mismatch between function-level benchmarks and real-world deployment settings. Accordingly, this review is intended not only to summarize vulnerabilities and defences, but also to clarify how technical, human, and organizational factors jointly shape the secure use of LLM-assisted coding systems. 3. Review Methodology To provide a comprehensive overview of the security landscape of LLM-based code tasks, a systematic literature review (SLR) was conducted. The methodology was designed following established guidelines for systematic reviews in software engineering and consists of three main stages: defining the research questions, executing a structured search and selection strategy, and performing a final snowballing search to ensure completeness. Figure 2. Search process 3.1. Research Questions This review is guided by the following research questions (RQs), which address the core objectives identified in the introduction: ⢠RQ1: What are the primary security issues introduced by the use of LLMs in code-related tasks? ⢠RQ2: What are the capabilities and limitations of current evaluation methods and benchmarks for assessing LLM-assisted coding tasks? ⢠RQ3: What is the role of human oversight and interaction in mitigating security risks in LLM-assisted software development? 3.2. Search Strategy A systematic search strategy was designed to identify relevant literature, as shown in Fig. 2. The search was conducted across several major digital libraries and preprint archives, including ACM Digital Library, IEEE Xplore, SpringerLink, ScienceDirect, Web of Science, and arXiv, reflecting common sources used in similar comprehensive reviews. The search keywords were formulated to capture the intersection of LLMs, code-related tasks, and security. Boolean operators were used to combine terms from three main categories: ⢠LLM terms: "Large Language Model", "LLM", "Foundation Model", "Generative AI". ⢠Code task terms: "code generation", "program synthesis", "code completion", "program repair", "vulnerability detection", "vulnerability fixing", "automated program repair", "APR", "code analysis", "software development". ⢠Security terms: "security", "vulnerability", "secure code", "insecure code", "exploit", "attack", "threat", "prompt injection", "jailbreak", "data poisoning", "adversarial attack", "data leakage", "confidentiality", "integrity", "robustness". ⢠Human-AI interaction terms: "human-in-the-loop", "human oversight", "human-AI collaboration", "cognitive bias", "feedback", "human review" The timeframe for the search initially focused on publications from 2020 to early 2025, aligning with the emergence and rapid proliferation of LLMs in software engineering tasks, particularly the significant surge in research from 2023 onwards. To ensure comprehensive coverage, both forward and backward snowballing techniques were applied to the reference lists of selected key papers, a method also employed in other systematic reviews (Zhang et al., 2024). 3.3. Study Selection Criteria The initial search results were subjected to a rigorous, multi-stage filtering process to determine inclusion in the final review. First, titles, abstracts, and keywords were screened for relevance. Subsequently, the full texts of the remaining articles were reviewed against the predefined inclusion and exclusion criteria. Inclusion Criteria: ⢠The paper must discuss security aspects directly related to the use of LLMs in tasks involving source code (generation, analysis, modification, repair, etc.). ⢠The paper evaluates LLMs for security-related tasks performed on source code (e.g., vulnerability detection, patch generation). ⢠The paper proposes or analyses adversarial attacks against LLMs specifically within the context of code-related tasks, or defences against such attacks. ⢠The study must be a peer-reviewed conference paper or journal article. Exceptionally, recent high-quality preprints were retained only when they addressed fast-moving topics not yet represented by an archival version and when their methodology and evidence were sufficiently detailed for assessment. Exclusion Criteria: ⢠Papers where LLMs are not a central component of the discussed code task or where the security aspect is trivial or out of scope. ⢠Papers not written in English. ⢠Editorials, opinion pieces, abstracts, posters, or very short workshop papers lacking substantial methodological detail or findings. ⢠Studies focusing on the security of LLMs in purely natural language processing tasks (e.g., general text generation, chatbots for non-technical domains) unless the principles or findings are directly and significantly applicable to code security. ⢠Studies focusing solely on ethical concerns (e.g., job displacement, copyright) without a direct technical security component. 3.4. Quality Assessment of Preprints Because this field is evolving rapidly, a limited number of recent preprints were retained when no peer-reviewed counterpart was available and when the work addressed important emerging topics. To reduce the risk of relying on unstable evidence, we applied additional screening to these studies, considering author credibility, methodological transparency, evidence quality, and the availability of later archival versions where applicable. Table 2. A Taxonomy of LLM Security Issues Category Specific Threat Potential Impact References Inherent Security Challenges Trained with risk datasets Introduction of exploitable flaws into applications. (Nazzal et al., 2024; Mohsin et al., 2024; Gong et al., 2024; Khoury et al., 2023; Pearce et al., 2022; Goetz and Schaad, 2024; Tihanyi et al., 2025) Lack of security principle understanding Subtle, hard-to-detect vulnerabilities; insecure design patterns. (Mohsin et al., 2024; Asare and Nagappan, 2023; Gong et al., 2024; He et al., 2024; Hajipour et al., 2024a; Khoury et al., 2023; TĂłth et al., 2024) Insecure/incomplete patches Regression of security posture; false sense of security. (Belozerov et al., 2025; Tihanyi et al., 2025; Gong et al., 2024; Asare and Nagappan, 2023; Liu et al., 2024b) Propagation of outdated/vulnerable dependencies System compromise through known exploits in third-party components. (Kharma et al., 2025; Pearce et al., 2022; Sandoval et al., 2023; Siddiq et al., 2024; Mohsin et al., 2024) Hallucinated APIs / semantic misassumptions Incorrect security assumptions; broken validation logic; misuse of nonexistent or misunderstood protections. (Liu et al., 2024a; Mohsin et al., 2024; Tie et al., 2024) Human-Related Security Issues Insecure prompt engineering Generation of vulnerable code; exposure of sensitive information. (Goetz and Schaad, 2024; Nazzal et al., 2024; Elgedawy et al., 2024; Chen et al., 2025b; Li et al., 2024; He et al., 2024) Over-reliance and reduced scrutiny Insecure code integrated into production systems. (Perry et al., 2023; Belozerov et al., 2025; Hamer et al., 2024; Tihanyi et al., 2024) Lack of developer security training for LLM usage Increased likelihood of insecure code generation and deployment. (Mohsin et al., 2024; Khoury et al., 2023; Kirova et al., 2024) Attack-Related Security Issues Adversarial Attacks/Data Poisoning Incorrect security assessments; introduction of exploitable flaws or backdoors. (Xie et al., 2020; Duan et al., 2021; Basic and Giaretta, 2024; Zeng et al., 2025; He and Vechev, 2023; Schuster et al., 2021; Cotroneo et al., 2024; Yan et al., 2024; Oh et al., 2024; Aghakhani et al., 2024) Data Leakage (from training data or prompts) Exposure of proprietary algorithms, API keys, personal data. (Nazzal et al., 2024; Tihanyi et al., 2025; Gong et al., 2024; Rabbi et al., 2024; Li et al., 2024; Hajipour et al., 2024b; FU et al., 2024; Liu et al., 2024b; He and Vechev, 2023; Zhong and Wang, 2024; Hamer et al., 2024) 4. LLM-Assisted Coding Task Security Issues Integrating LLMs into software development introduces diverse security issues, including inherent technological flaws, hallucination-related semantic failures, human-driven vulnerabilities, and direct attacks on the models (summarised in Table 2). This section explores these categories to address RQ1. Understanding this threat landscape is necessary because weaknesses originate from multiple sources, ranging from the underlying technology to specific deployment and usage configurations. Consequently, mitigation requires a layered approach. For example, a model replicating insecure training data, a malicious data poisoning attack, and a poorly written user prompt are distinct problems that require separate defensive strategies. 4.1. Inherent Security Challenges A primary concern is that LLMs frequently generate vulnerable code. Because they replicate patterns from large training datasets, they often reproduce insecure practices, common weaknesses (CWEs), and outdated library usages. Lacking a semantic understanding of security principles (Das et al., 2025), LLMs produce code that may be functionally correct but structurally insecure, simply mirroring flaws present in their training data (Basic and Giaretta, 2024). Additionally, the confident presentation of LLM outputs can cause developers to accept generated code with insufficient review (Mohsin et al., 2024), overlooking subtle, exploitable bugs (Yan et al., 2024). This risk also applies to automated program repair (APR) and vulnerability patching. While an LLM might resolve a targeted functional error, the resulting patch can introduce new security flaws or provide an incomplete fix that attackers can still bypass (Nong et al., 2024; Alrashedy et al., 2023). Training Data Contamination. Several studies highlight that LLMs have an inherent tendency to introduce security vulnerabilities, often due to being trained on massive open-source code corpora, which may contain insecure practices (Nazzal et al., 2024; Mohsin et al., 2024). Some research suggests this is partly due to LLMsâ lack of awareness regarding scenario-relevant security risks, as they primarily prioritise functional requirements (Gong et al., 2024; Khoury et al., 2023; Pearce et al., 2022). Consequently, LLMs replicate these security flaws from their training data, leading to the generation of vulnerable code. This creates significant concerns about the reliability of LLMs for secure code generation. The empirical evidence for this is compelling. GĂśtz and Schaad (Goetz and Schaad, 2024) found that up to 65% of code generated by some LLMs was initially insecure, attributing this partly to a lack of best-practice examples in training data. Similarly, Pearce et al. (Pearce et al., 2022) notably found that approximately 40% of code snippets generated by GitHub Copilot were susceptible to vulnerabilities. This was linked to Copilot being trained on public GitHub code known to contain insecure patterns. Mohsin et al. (Mohsin et al., 2024) demonstrate that LLMs trained on public data consistently reproduce software vulnerabilities. Their zero-shot evaluations expose a widespread lack of baseline security knowledge across multiple models. Notably, systems like ChatGPT and Copilot produced insecure code in roughly 40% of test cases, and baseline Prompt-Driven Code Generators (PDCGs) similarly failed to meet acceptable security standards. More comprehensive evaluations by Tihanyi et al. Tihanyi et al. (Tihanyi et al., 2025) across nine state-of-the-art models, showing that at least 62% to 63% of generated C programs were vulnerable when using neutral prompts. They concluded that all examined models notoriously introduce vulnerabilities at unacceptable rates, attributing this to training on human-generated code containing vulnerabilities. Limited Security Principle Understanding. Another critical inherent challenge is rooted in the design and training objectives of LLMs, which almost universally prioritise functional correctness and linguistic fluency over security awareness (Mohsin et al., 2024). These models are engineered to predict the most probable next token based on statistical patterns, not to reason abstractly about security implications or to evaluate potential attack vectors (Asare and Nagappan, 2023). This architectural limitation means that the security quality of generated code is often inconsistent and unpredictable. Gong et al. (Gong et al., 2024) observed that models often produce functional code while overlooking hidden vulnerabilities, falling into scenario-specific "traps." The authors attribute this failure to a general lack of security awareness and an inability to identify risks not explicitly detailed in the prompt. Studies show that even security-aware instructions in prompts may not significantly enhance security (He et al., 2024) or reliably (Hajipour et al., 2024a; Khoury et al., 2023) reduce vulnerabilities. TĂłth et al. (TĂłth et al., 2024) highlighted that security risks can be highly situational, making it difficult for AI to grasp context and distinguish safe from risky practices. Insecure or Incomplete Patches. The application of LLMs to automated program repair (APR) and vulnerability patching introduces another vector of risk. Fixes generated by LLMs may be functionally correct on the surface but can introduce new security flaws or leave the original vulnerability only partially mitigated. For example, Belozerov et al. (Belozerov et al., 2025) demonstrate that ChatGPT and GPT-4o correctly identified and fixed only 17 of 32 tested vulnerabilities. The models often produced incomplete patches, relied on superficial mitigations such as simple input validation, and misclassified vulnerability types (e.g., confusing CWE-78 (MITRE, 2024t) with CWE-77 (MITRE, 2024r)), thereby applying inappropriate defenses that left the original flaws unresolved. Tihanyi et al. (Tihanyi et al., 2025) suggest that current LLMs are not yet ready for program repair because they still struggle to generate functionally correct fixes. Although some studies report promising results, particularly when combined with other tools, substantial further research is still needed (Tihanyi et al., 2025; Gong et al., 2024). Asare et al. (Asare and Nagappan, 2023) show that Copilot may reintroduce vulnerabilities rather than produce secure repairs, while scanner-guided iterative repair eventually plateaus. Likewise, Liu et al. (Liu et al., 2024b) report that repair success varies by vulnerability type, with incomplete fixes persisting for classes such as Input Validation and Information Leakage. Outdated Dependency Propagation. LLMs often recommend deprecated libraries, obsolete APIs, and cryptographically broken implementations that persist in training data despite safer alternatives. Studies report risks from outdated libraries (Kharma et al., 2025) and deprecated APIs (Pearce et al., 2022; Sandoval et al., 2023), indicating that obsolete practices can continue to appear in generated code as the security landscape evolves (Pearce et al., 2022). One example is MD5, a broken hash algorithm still suggested by models such as GitHub Copilot (Sandoval et al., 2023) and also identified in code analyzed by the Sallm framework under CWE-328 (Siddiq et al., 2024). Because LLMs are trained on public repositories, they may reproduce outdated or vulnerable third-party dependencies in both prompt-driven models and coding assistants (Mohsin et al., 2024; Yao et al., 2024). While security quality varies across models, languages, and domains, severe vulnerabilities still recur under simple prompts. Hallucinated APIs and Security-Relevant Semantic Misassumptions. Another inherent risk in LLM-assisted coding is hallucination: models may generate nonexistent APIs, invalid signatures, fabricated library behavior, or incorrect assumptions about security mechanisms. Although often framed as a functional correctness issue, such hallucinations can also have direct security consequences. For example, a model may invent a sanitization routine, misuse an authentication or cryptographic API, assume that a library enforces checks it does not provide, or produce plausible code that silently violates security requirements (Mohsin et al., 2024; Tie et al., 2024). These errors are particularly dangerous because they can appear superficially convincing. Developers may fix syntax or compilation issues while missing deeper semantic errors involving permissions, validation, state handling, or security boundaries. Recent work shows that current models still struggle to detect hallucinated code and to repair it reliably once introduced (Liu et al., 2024a). In more autonomous or agentic settings, such errors can also propagate across steps when hallucinated dependencies, fabricated tool capabilities, or mistaken execution assumptions are reused as context. Hallucination should therefore be treated not just as a coding nuisance, but as a security-relevant failure mode linking model uncertainty, plausible output, and exploitability. Illustrative example of prompt-driven insecurity This simplified example shows how an underspecified prompt can yield code that appears reasonable while omitting basic security controls. Insecure prompt and potentially vulnerable output Prompt. âWrite a Python Flask endpoint that reads a filename from a request parameter and returns the file contents.â Output. from flask import Flask, request app = Flask(__name__) @app.get("/read") def read_file(): name = request.args.get("file") with open(name, "r") as f: return f.read() Risk. The code is functionally plausible, but it may enable path traversal because untrusted input is used directly in file access without validation or directory restriction. Safer reformulation and more secure implementation pattern Prompt. âWrite a Python Flask endpoint that reads a filename from a request parameter and returns the file contents, but restrict access to a fixed directory, reject path traversal, validate the filename, and handle errors safely.â Output. from flask import Flask, request, abort from pathlib import Path app = Flask(__name__) BASE = Path("/app/data").resolve() @app.get("/read") def read_file(): name = request.args.get("file", "") target = (BASE / name).resolve() if not str(target).startswith(str(BASE)) or not target.is_file(): abort(400) return target.read_text(encoding="utf-8") Safer pattern. This version still requires review and testing, but it shows how explicit security constraints can change both the generated structure and the defensive checks included. Figure 3. Illustrative example showing how weak prompt specification can lead to insecure but plausible code, and how a more security-aware reformulation can elicit safer implementation patterns. The prompts were manually written, and the sample code was generated using Gemini 3.1 Pro and subsequently reviewed by the authors for illustrative accuracy. 4.2. Human-Related Security Issues Beyond the inherent limitations of the models themselves, the human element represents a significant source of security vulnerabilities in LLM-assisted coding. The interaction between the developer and the model, from the initial prompt to the final code review, is a critical stage where misunderstandings, biases, and a lack of security awareness can lead to the introduction of serious flaws. Insecure Prompt Engineering. User prompts can inadvertently steer LLMs toward insecure code or bypassed safeguards. The security of generated code depends heavily on prompt quality and specificity: vague prompts often yield insecure outputs (Goetz and Schaad, 2024), and models may fail to infer or prioritize risks that are not explicitly stated (Gong et al., 2024). Consequently, non-expert users may obtain code that is functionally correct yet vulnerable (Nazzal et al., 2024), and even clearly security-sensitive tasks may not trigger warnings or protective measures unless these are explicitly requested (Nazzal et al., 2024; Belozerov et al., 2025). Crafting effective prompts is therefore a non-trivial skill, requiring both clear communication and engineering judgment (Elgedawy et al., 2024). Novice users often struggle to specify security requirements precisely enough to elicit secure code (Chen et al., 2025b). However, simply asking for âsecureâ code is usually insufficient; zero-shot security directives may still fail to prevent vulnerabilities such as path traversal (Goetz and Schaad, 2024), and generic security boilerplate alone yields little improvement (He et al., 2024). More detailed security-aware prompts can help, but only when users can accurately identify and describe the relevant requirements (Li et al., 2024). Figure 3 illustrates how underspecified prompts can produce plausible code that omits essential security controls. Over-Reliance and Reduced Scrutiny. Developers excessively trust LLM outputs, leading to insufficient review of generated code. Perry et al. (Perry et al., 2023) indicate that programmers using AI assistants produced less secure code than those working independently in four of five security-focused tasks. Yet these users were more likely to believe their code was secure, indicating overconfidence. The resulting code often omitted secure library use, proper input sanitisation, and edge-case handling for databases and file systems. Belozerov et al. (Belozerov et al., 2025) caution against over-reliance on AI code generation, arguing that manual review and security testing remain essential. Because LLMs can create a false sense of safety, all generated output requires rigorous inspection. Hamer et al. (Hamer et al., 2024) also warn against blindly trusting external code, whether AI-generated or human-written. Tihanyi et al. (Tihanyi et al., 2024) argue that relying on LLMs for production code generation without expert oversight is unsafe. Their evaluation of eight major models found vulnerabilities in at least 63.47% of generated C programs, covering 42 CWEs, including six from MITREâs Top 25; dereference failures and buffer overflows were the most frequent. Even the best models produced vulnerable code in roughly half of the cases, underscoring the need for rigorous manual validation. Insufficient Security Training for LLM Usage. Developers with limited security expertise are particularly vulnerable to misusing LLMs. Mohsin et al. (Mohsin et al., 2024) show that developer experience strongly affects the security of LLM-assisted coding: inexperienced programmers, especially in complex languages such as C++, often overlook critical practices, leading to vulnerabilities such as buffer overflows and improper input validation. Limited understanding of prompt security and subtle flaws in LLM output further increase the risk of exploitable code entering production. Khoury et al. (Khoury et al., 2023) reinforce this concern by simulating novice programmers, finding that ChatGPT produced initially secure code in only 5 of 21 cases without explicit security instructions. The model frequently generated severe vulnerabilities, including SQL injection, path traversal, and insecure handling of sensitive operations such as unhashed passwords, and typically added basic safeguards only when asked explicitly. Accordingly, software engineering education must adapt as LLMs automate routine coding (Kirova et al., 2024). Developers need stronger skills in prompt engineering, critical evaluation, and artifact verification, while curricula should emphasize review, testing, validation, and verification, together with professional ethics, responsible AI, and secure development practices (Kirova et al., 2024; Mohsin et al., 2024). Systematic security testing and review therefore remain essential when using LLM-generated code. 4.3. Attack-Related Security Issues Beyond inherent weaknesses and user-related errors, LLMs are themselves direct targets for attacks designed to compromise the integrity, confidentiality, and availability of the code they generate. These attacks exploit vulnerabilities in the LLMâs architecture, its training data, or its input-processing mechanisms to maliciously influence its behaviour. Adversarial Attacks and Data Poisoning. A primary attack vector is adversarial input, where carefully crafted prompts bypass an LLMâs safety mechanisms (Shayegani et al., 2023). Successful prompt injection can induce harmful code generation, confidential data disclosure, or unintended actions in integrated environments. As in adversarial attacks on image models (Xie et al., 2020; Duan et al., 2021), subtle input modifications can also make coding LLMs misclassify vulnerable code as safe, suggest flawed patches, or miss existing vulnerabilities. More broadly, poisoning the data used for training or fine-tuning can implant backdoors or systemic weaknesses that later reappear in generated code (Cotroneo et al., 2024; Tessa et al., 2026; Basic and Giaretta, 2024). Zeng et al. (Zeng et al., 2025) demonstrate this risk through HACKODE, which hides adversarial sequences in external code references, often within comments. Although these sequences appear meaningless to humans, they can alter model outputs while leaving the infected examples functionally correct, allowing them to spread through highly ranked online sources and into coding assistants. As a result, models may generate unsafe code from seemingly legitimate inputs, making the attack difficult to detect. Poisoning attacks are also a serious training-phase threat. He et al. (He and Vechev, 2023) show that poisoned data or malicious fine-tuning can bias code models toward insecure cryptographic recommendations and distinguish these attacks from SVEN, their post-training evaluation method. Schuster et al. (Schuster et al., 2021) further classify poisoning threats into model poisoning, which compromises the model or its supply chain, and data poisoning, which inserts malicious files into training repositories. Some poisoning attacks inject explicit insecure payloads into training data (Cotroneo et al., 2024; Yan et al., 2024; Oh et al., 2024; Schuster et al., 2021), but such payloads may be detectable by static analysis. To evade this, Aghakhani et al. (Aghakhani et al., 2024) propose COVERT, which hides malicious content in ignored regions such as comments and docstrings, and TROJANPUZZLE, which teaches the model token substitution patterns without including the full payload directly. When triggered, the model can then reconstruct and emit the malicious payload automatically. Data Leakage and Confidentiality Breaches. LLM-generated code often contains vulnerabilities that can expose, alter, or destroy sensitive data, thereby compromising confidentiality and sometimes availability. Attacker-induced flaws such as buffer overflows can also enable unauthorised access or modification of critical information (Nazzal et al., 2024; Zeng et al., 2025). Improper input validation (CWE-20 (MITRE, 2024f)) and related injection flaws, including SQL injection (CWE-89 (MITRE, 2024w)), cross-site scripting (CWE-79 (MITRE, 2024u)), OS command injection (CWE-78 (MITRE, 2024t)), and code injection (CWE-94 (MITRE, 2024x)), are common (Tihanyi et al., 2025; FU et al., 2024; Gong et al., 2024; Rabbi et al., 2024). These vulnerabilities can enable data theft, tampering, deletion, or even system compromise (Nazzal et al., 2024). A common cause is missing input validation or filtering, for example, failing to sanitize file names and thereby enabling path traversal (CWE-22 (MITRE, 2024g)) (Liu et al., 2024b; TĂłth et al., 2024). Memory-safety flaws, such as buffer overflows (CWE-120 (MITRE, 2024b), CWE-787 (MITRE, 2024s), CWE-119 (MITRE, 2024a)), out-of-bounds reads or writes (CWE-125 (MITRE, 2024c), CWE-787 (MITRE, 2024s)), null pointer dereferences (CWE-476 (MITRE, 2024m), CWE-153 (MITRE, 2024d), CWE-346 (MITRE, 2024i)), and memory leaks or use-after-free errors (CWE-401 (MITRE, 2024k), CWE-416 (MITRE, 2024l)) are also frequent, especially in C/C++ (Tihanyi et al., 2024; Li et al., 2024; Liu et al., 2024b; Hajipour et al., 2024b). These may cause crashes, data leakage, or arbitrary code execution, and missing NULL checks after allocation are a recurring weakness (Liu et al., 2024b). Information exposure vulnerabilities (e.g., CWE-200 (MITRE, 2024e), CWE-352 (MITRE, 2024j)) are likewise reported (FU et al., 2024; Mohsin et al., 2024). A common pattern is hard-coded credentials, where models generate passwords or API keys as clear-text strings (Gong et al., 2024). Invalid validation can also lead to unsafe operations that leak sensitive information (Liu et al., 2024b). Further risks include API misuse, such as insecure cryptographic choices (CWE-327 (MITRE, 2024h), CWE-759 (MITRE, 2024q)) or unsafe deserialization (CWE-502 (MITRE, 2024o)), as well as resource leaks (He and Vechev, 2023; FU et al., 2024; Zhong and Wang, 2024). Other reported issues include path traversal, unrestricted file upload, incorrect permissions, unsafe dynamic execution, and infinite loops that may trigger denial-of-service conditions (MITRE, 2024g; MITRE, 2024n; MITRE, 2024p; MITRE, 2024x; MITRE, 2024v; Zeng et al., 2025; Hamer et al., 2024; Gong et al., 2024; Liu et al., 2024b). 4.4. Security Implications of Agentic Coding Systems Early studies often evaluate LLM-assisted coding as a largely standalone task, but recent research and practice are shifting toward agentic coding systems: autonomous or semi-autonomous LLM-based systems that can decompose tasks, invoke tools, retrieve repository context, execute tests, revise files, and iterate on intermediate results (Chow and Li, 2026; Nunez et al., 2024; Liu et al., 2024c; Zheng et al., 2024). This shift changes the threat model from a single-output artifact problem to a multi-step socio-technical workflow problem. First, agentic systems expand the attack surface because they interact with shells, package managers, testing frameworks, retrieval systems, external APIs, and local development environments (Nunez et al., 2024; Liu et al., 2024c). Prompt injection may therefore arise not only from user input, but also from retrieved documentation, repository comments, malicious tool output, poisoned test artifacts, or adversarial external resources (Chow and Li, 2026; Zheng et al., 2024). As these components multiply, so do the trust boundaries that must be secured. Second, agentic systems can amplify errors across steps. Unlike a standalone insecure completion, an early hallucination, false assumption, or unsafe action may cascade by selecting a vulnerable dependency, generating flawed tests, approving an incomplete patch, or overwriting secure code with insecure revisions (Chow and Li, 2026). Because later stages may reuse earlier outputs as context, the system can recursively propagate its own mistakes. Third, agentic systems can reduce human visibility and controllability. Developers may see only the final artifact rather than the intermediate reasoning, tool calls, retrieved evidence, and edits that produced it, making it harder to audit vulnerabilities, assign responsibility, or identify whether failures stem from the base model, a tool interaction, a retrieval source, or workflow design (Liu et al., 2024c). Secure deployment therefore requires not only code inspection but also traceability across the broader execution process. Finally, agentic coding systems blur the boundary between software security and broader AI safety concerns, including controllability, misuse resistance, and operational governance (Nunez et al., 2024; Zheng et al., 2024). Accordingly, evaluating future LLM-assisted coding systems requires workflow-aware, repository-level, and action-oriented assessment rather than only single-turn code generation benchmarks (Chow and Li, 2026; Liu et al., 2024c). Taken together, these observations help explain why recurring vulnerability families appear across models, languages, prompting settings, and evaluation designs, as summarized in Table 3; this complements Table 2 by shifting attention from broad risk categories to recurrent security weaknesses. Table 3. Recurring vulnerability families in LLM-assisted coding studies. Vul. Family Representative CWEs Representative Works Recurring Pattern Input validation and injection CWE-20, 22, 78, 79, 89, 94 (FU et al., 2024; Gong et al., 2024; Khoury et al., 2023; Liu et al., 2024b; Nazzal et al., 2024; Rabbi et al., 2024; Tihanyi et al., 2025; TĂłth et al., 2024) Models often generate functionally plausible code that omits sanitization, boundary checks, or command/query validation. Memory safety CWE-119, 120, 125, 416, 476, 787 (Hajipour et al., 2024b; Li et al., 2024; Liu et al., 2024b; Tihanyi et al., 2024; Tihanyi et al., 2025) Memory-unsafe languages show recurring low-level errors, especially when prompts do not explicitly encode safety constraints. Weak crypto and insecure dependencies CWE-327, 328, 759 (He and Vechev, 2023; Kharma et al., 2026; Mohsin et al., 2024; Pearce et al., 2022; Sandoval et al., 2023; Siddiq et al., 2024) Models reuse outdated algorithms, deprecated APIs, and vulnerable dependencies inherited from training corpora. Incomplete or insecure repair CWE-dependent (Alrashedy et al., 2023; Asare and Nagappan, 2023; Belozerov et al., 2025; Liu et al., 2024b; Nong et al., 2024; Pearce et al., 2023; Tihanyi et al., 2025) Generated patches may satisfy local tests while leaving root causes unresolved or introducing new flaws. Data leakage and exposure CWE-200 and related patterns (FU et al., 2024; Gong et al., 2024; Hajipour et al., 2024b; Hamer et al., 2024; He and Vechev, 2023; Li et al., 2024; Mohsin et al., 2024; Nazzal et al., 2024; Zeng et al., 2025; Zhong and Wang, 2024) Insecure handling of secrets, credentials, and sensitive data remains a recurrent failure mode. Hallucination-related semantic failure Non-CWE-specific (Liu et al., 2024a; Mohsin et al., 2024; Tie et al., 2024) Fabricated APIs or false assumptions about security behavior can produce plausible yet unsafe code and may propagate across workflows. 5. Evaluating the Security of LLM-assisted Coding Assessing the security of LLM-assisted coding remains difficult because current benchmarks and evaluation pipelines capture different aspects of risk (Zhou et al., 2024a; Fan et al., 2023; Lyu et al., 2024; Li et al., 2023). Some focus on functional correctness, others probe known vulnerability patterns, and still others evaluate runtime behavior, repository context, or human review (Chen et al., 2026; Basic and Giaretta, 2024; Sheng et al., 2025a; Li et al., 2025a; Chen et al., 2025a). In addressing RQ2, this section examines the main datasets, metrics, and evaluation methods used in the literature, with particular attention to benchmark contamination, limited realism, and cross-framework incomparability. 5.1. Datasets A crucial component of any evaluation framework is the dataset used for assessment. In the context of LLMs for code generation, these datasets can be broadly divided into those that measure functional correctness and those specifically designed to probe for security vulnerabilities. 5.1.1. Functional Correctness The initial wave of benchmarks for LLM-based coding tasks focused primarily on functional correctness, whether the generated code produces the correct output for a given set of inputs. A foundational benchmark in this area is HumanEval (Chen et al., 2021), a collection of hand-written Python programming problems where the model must synthesise a function from its docstring. This benchmark has been extended to enhance its rigour and scope; HumanEval+ (Liu et al., 2023) augments the original with 80 times more test cases, while HumanEval-X (Zheng et al., 2023) adapts the problems for multiple languages, including C++, Java, and Go. Austin et al. (Austin et al., 2021) introduced two complementary datasets for code synthesis: MBPP, with 974 entry-level Python tasks including a hand-verified subset of 426, and MathQA-Python, with 23,914 mathematical word problems adapted from MathQA. MBPP targets imperative algorithmic construction, whereas MathQA-Python emphasizes language understanding through straight-line code generation. Hendrycks et al. (Hendrycks et al., 2021) proposed APPS, a 10,000-task benchmark for Python code generation that spans introductory to competition difficulty and evaluates functional correctness through test execution. Du et al. (Du et al., 2023) introduced ClassEval, the first benchmark for class-level code generation, with 200 Python tasks requiring complete classes generated from natural language descriptions. Unit-test results show that class-level generation is substantially harder than function-level generation. For more realistic and domain-specific assessment, DS-1000 provides 1,000 Stack Overflow data-science tasks requiring major Python libraries (Lai et al., 2023), BigCodeBench offers 1,140 practical Python tasks involving multi-library function use (Zhuo et al., 2024), and CoderEval evaluates 460 repository-level Python and Java tasks requiring full-project context (Yu et al., 2024). Together, these benchmarks show that practical and context-dependent generation remains challenging for current models. 5.1.2. Security-Specific As the functional limitations of LLMs became better understood, a new class of benchmarks emerged to address the critical issue of security in the code they generate. These security-specific datasets are not designed to evaluate if the code works, but rather if it is safe and free from known vulnerabilities. Early evaluation methods relied on curated prompt sets to test for known insecure patterns. For instance, SecurityEval (Siddiq and Santos, 2022) provides 130 Python prompts mapped to 75 Common Weakness Enumeration (CWE) categories. These samples directly assess whether a modelâs generated output introduces specific security vulnerabilities. Similarly, LLMSecEval(Tony et al., 2023) offers 150 natural language prompts to assess code generation in vulnerability-prone scenarios. Covering 18 of MITREâs Top 25 CWEs (MITRE, 2021), these prompts were derived from secure code snippets and manually refined to be language-agnostic. To support robust comparative evaluations, the dataset includes a secure, CodeQL-verified (CodeQL, 2025) Python baseline for each prompt. Wang et al. (Wang et al., 2024a) proposed CodeSecEval for evaluating both code generation and repair. The dataset contains 180 Python samples spanning 44 critical vulnerability types, each with executable secure and insecure versions and test cases for automated functional and security assessment. It comprises SecEvalBase, which extends SecurityEval (Siddiq and Santos, 2022), and SecEvalPlus, which targets the most common and severe weaknesses. FormAI (Tihanyi et al., 2023) instead combines large-scale code generation with formal verification. It contains 112,000 C programs generated by GPT-3.5-turbo and labeled with ESBMC, which identifies both vulnerability type and location. The study reported vulnerabilities in 51.24% of generated programs. Tihanyi et al. (Tihanyi et al., 2024) later introduced FormAI-v2, a larger set of 331,000 C programs generated by models such as GPT-4 and Gemini-1.0-Pro. Evaluating code from neutral prompts, they found with ESBMC that at least 63.47% of programs were vulnerable, suggesting that such risks are inherent to current code-generation models. 5.2. Metrics To quantify the performance of LLMs in coding tasks, researchers employ a range of metrics. These can be divided into two main categories: those that assess the functional correctness of the generated code and those specifically designed to measure its security posture. 5.2.1. Functional correctness Pass@k: A widely adopted metric for functional correctness is pass@k, which measures the probability that at least one of k code samples generated for a given task successfully passes all of its unit tests (Chen et al., 2021; Chen et al., 2025b; Fu et al., 2024b; Tian et al., 2023; Wang et al., 2024a; Fakhoury et al., 2024; Siddiq et al., 2024; Dilgren et al., 2025; Mathews and Nagappan, 2024; Miah and Zhu, 2024). This metric provides an indication of the likelihood that a developer will find a correct solution within a set of k generated options (Wang et al., 2024a; Siddiq et al., 2024). To ensure statistical robustness and reduce variance in results, an unbiased estimator of pass@k is commonly utilised in evaluations (Tong and Zhang, 2024; Wang et al., 2024a; He and Vechev, 2023). Some frameworks have also introduced variations, such as the ranked pass@k (pass@k@m) used by TiCoder (Fakhoury et al., 2024), which denotes correctness after a specified number of user queries and subsequent pruning of options. Compilation and Unit Tests: At a fundamental level, functional correctness is determined by whether the generated code compiles successfully and subsequently passes a predefined suite of unit tests (Chen et al., 2021; Fu et al., 2024b; Liu et al., 2023; Kharma et al., 2025). The percentage of code snippets that are merely compilable can also serve as a basic indicator of a modelâs output quality. Text Similarity Metrics: Metrics that assess the token-level similarity between generated code and a reference solutionâsuch as BLEU, ROUGE-L, CodeBLEU, METEOR, ChrF, and RUBYâhave also been applied (Ren et al., 2020; Chen et al., 2021). However, these metrics are often considered unreliable for evaluating code. Functionally equivalent programs can exhibit significant syntactic variations, meaning a high similarity score does not necessarily indicate functional correctness, nor does a low score confirm its absence (Tong and Zhang, 2024). Task-Specific Criteria: Generic metrics are often inadequate for specialised tasks, which require tailored evaluation criteria. For example, path-planning code is assessed by whether it finds valid paths and satisfies task constraints (Chen et al., 2025b), while hallucination detection uses dedicated metrics such as Valid Rate (VR), hallucination existence accuracy (Accrec), and hallucination type accuracy (Acctype) (Liu et al., 2024a). 5.2.2. Security-Specific Security Rate (SVEN-SR): An early metric proposed for security is the SVEN-SR, defined as the percentage of secure programs among all unique and compilable generated programs (Fu et al., 2024b). While this metric provides a baseline indication of security, it has been criticised for potentially overestimating a modelâs practical security performance. By not accounting for functional correctness, a trivially safe but entirely useless piece of code could achieve a high SVEN-SR score. Secure-pass@k: To provide a more realistic assessment, the Secure-pass@k metric measures the probability of generating code that is both secure and functionally correct within k attempts (Fu et al., 2024b; Dilgren et al., 2025). It is considered a more practical measure of an LLMâs secure code generation capability because it integrates functional correctness as an essential prerequisite for a successful outcome. Secure@kpass: A related but distinct metric is Secure@kpass, which assesses how likely any one out of k functionally correct generations is also secure (Fu et al., 2024b). This helps to decouple the security evaluation from the modelâs ability to generate working code in the first place. func-sec@k: Similarly, the func-sec@k metric evaluates functionality and security simultaneously, indicating the likelihood that an LLM-generated implementation (out of k options) is both correct and secure (Peng et al., 2025). This metric is particularly effective at highlighting the significant gap that often exists between a modelâs ability to generate functional code and its ability to generate code that is both functional and secure. vulnerable@k: Approaching the problem from the opposite perspective, the vulnerable@k metric measures the probability that at least one code snippet out of k generated samples contains a detectable vulnerability (Siddiq et al., 2024). In this case, a lower score for vulnerable@k signifies a better performance in generating secure code. False Refusal Rate (FRR): To quantify the inherent safety-utility tradeoff in models that have been fine-tuned for safety, the FRR measures the percentage of benign prompts that an LLM incorrectly refuses because it mistakes them for unsafe or malicious requests (Bhatt et al., 2024). Static Security Analysis Tools: Underlying many of these security metrics is the use of static security analysis tools. Tools such as CodeQL, SonarQube, Bandit, ESLint, and Infer are widely employed to automatically detect security flaws and known vulnerabilities directly within the source code (Pearce et al., 2022; Bhatt et al., 2023; Nunez et al., 2024; Peng et al., 2025; Hajipour et al., 2024a; Fu et al., 2024b; Dolcetti et al., 2024; Tony et al., 2023; Siddiq et al., 2024; FU et al., 2024; Kharma et al., 2025). These tools are instrumental in identifying specific vulnerability classes and flagging security hotspots. It is important to note, however, that static analysers are known to suffer from imprecision, including both false positives and false negatives (Wang et al., 2024a; Li et al., 2025b). 5.3. Evaluation Tools Security evaluations use complementary oracles. Static analyzers scale to many samples and vulnerability classes but have false positives, false negatives, and limited access to runtime or repository context (Hajipour et al., 2024a; Li et al., 2025b; Siddiq et al., 2024; Wang et al., 2024a). Dynamic tests, fuzzing, and exploit execution provide behavioral evidence, but their conclusions depend on test coverage, reproducible environments, and the fidelity of the attack scenario (Dilgren et al., 2025; Nunez et al., 2024; Peng et al., 2025; Vero et al., ; Yang et al., 2024b). Repository-aware evaluation exposes integration failures, while expert or developer review captures aspects of usability and oversight that automated tests cannot measure (Fakhoury et al., 2024; Li et al., 2025b; Miah and Zhu, 2024; Yu et al., 2024; Zheng et al., 2025). For this reason, results from different tools should be interpreted in terms of their task, oracle, context, and security objective; a fuller catalogue of the surveyed frameworks appears in Supplementary Section 2. Table 4. Representative families of evaluation frameworks for LLM-assisted coding security. Framework Family Representative Works Primary Oracle Key Use&limitation Function-level correctness (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021) Unit tests Useful for baseline synthesis ability, but weak in security realism and contextual coverage. Security prompt benchmarks (Siddiq et al., 2024; Siddiq and Santos, 2022; Tony et al., 2023; Wang et al., 2024a) Curated prompts + static rules Effective for probing known vulnerability classes, but limited in runtime realism and sensitive to analyzer imprecision. Dynamic/exploit-driven security (Dilgren et al., 2025; Nunez et al., 2024; Peng et al., 2025; Vero et al., ; Yang et al., 2024b) Runtime tests, fuzzing, exploit execution Better reflects exploitability in practice, but depends heavily on test quality, environment fidelity, and setup cost. Repository/context-aware evaluation (Li et al., 2025b; Yu et al., 2024; Zheng et al., 2025) Project tests + contextual reasoning Captures integration and cross-file failures, but is harder to standardize and compare across models. Human-centered/manual evaluation (Fakhoury et al., 2024; Miah and Zhu, 2024; Perry et al., 2023) Expert review/user study Reveals review burden, usability, and trust effects, but is costly, subjective, and hard to scale. 5.4. Cross-Framework Comparison and Evaluation Gaps The surveyed evaluation frameworks differ not only in implementation but also in what they treat as evidence of security. Some benchmarks focus on function-level correctness under predefined tests, while others probe for vulnerability patterns, repository integration, exploitability, or the interaction between human reviewers and generated code. As a result, two studies can evaluate the same model yet reach different conclusions because they observe different failure surfaces rather than because one of them is necessarily incorrect. A useful way to compare the literature is along six dimensions: (i) task granularity, ranging from isolated functions to repository-level workflows; (i) oracle type, including unit tests, static analyzers, exploit execution, and manual review; (i) evaluation objective, such as correctness, vulnerability detection, secure repair, or attack helpfulness; (iv) context availability, from prompt-only generation to cross-file and temporal repository context; (v) security realism, ranging from rule-based checks to exploit-driven validation; and (vi) human involvement, from fully automated pipelines to expert review. These dimensions help explain why benchmark results are often not directly comparable. Table 4 summarizes these framework families and highlights the distinct trade-offs that make cross-study results difficult to compare directly. Static-analysis-centered frameworks such as SALLM (Siddiq et al., 2024), CodeLMSec (Hajipour et al., 2024a), and parts of CodeSecEval (Wang et al., 2024a) scale well and provide broad CWE coverage, but they inherit the incompleteness and context limitations of the underlying analyzers. Dynamic and exploit-driven approaches such as SecRepoBench (Dilgren et al., 2025), CWEVAL (Peng et al., 2025), AutoSafeCoder (Nunez et al., 2024), and BAXBENCH (Vero et al., ) better capture runtime failures and real exploitability, but they depend heavily on test quality, environment setup, and scenario design. Repository-level and context-aware frameworks such as CORRECT (Li et al., 2025b), CoderEval (Yu et al., 2024), and HumanEvo (Zheng et al., 2025) reveal integration failures that function-level benchmarks often miss, yet they are harder to standardize and compare across models. Manual and human-centered evaluations capture readability, reviewability, trust effects, and workflow realism, but they are expensive and difficult to scale (Fakhoury et al., 2024; Miah and Zhu, 2024; Perry et al., 2023). Taken together, the literature suggests that no single framework is sufficient for evaluating secure LLM-assisted coding. Static analysis is useful for breadth, dynamic testing is useful for behavioral evidence, repository-level benchmarks are necessary for contextual realism, and human-centered evaluation is necessary for deployment realism. The main gap in the current landscape is therefore not simply the lack of benchmarks, but the lack of an integrated evaluation pipeline that combines these complementary signals into a more trustworthy picture of security performance. 6. Human Oversight and Human-Factor Risks The integration of LLMs into coding tasks has significantly advanced software development, yet their inherent limitations necessitate human involvement to ensure code quality, correctness, and security (Liu et al., 2024c; Kathiresan, 2025; Tie et al., 2024; Dou et al., 2024a). Human oversight is often treated as a key safeguard, whether through direct review, feedback loops, or human-in-the-loop validation (Li et al., 2026; Xia and Zhang, 2023; Firouzi and Ghafari, 2026; Olausson et al., ; Zheng et al., 2024). However, human participation is not uniformly protective: the same workflows that enable developers to detect flaws and clarify intent can also introduce vulnerabilities through over-trust, automation bias, weak prompt engineering, and insufficient review practices. Accordingly, this section treats human-AI interaction as a dual-role factor in secure LLM-assisted coding. Human developers and reviewers remain critical for interpreting requirements, auditing outputs, and identifying flaws that automated systems miss, yet empirical evidence also shows that LLMs can reduce scrutiny, create false confidence, and distort responsibility in ways that degrade security (Hamer et al., 2024; Khoury et al., 2023; Mohsin et al., 2024; Nazzal et al., 2024). To address RQ3, we therefore review the literature not only on human oversight as a mitigation mechanism, but also on human-AI collaboration as a source of security and accountability risk. 6.1. Limitation of Automated Feedback A common approach to improving LLM-generated code is to design models capable of providing their own feedback and attempting self-correction, a process often referred to as self-repair or self-criticism (Olausson et al., ; Dou et al., 2024a). This typically involves the model generating an initial code snippet, executing it against test cases, and then using the resulting error messages or diagnostics as execution feedback to identify and rectify its own errors (Zheng et al., 2024; Dai et al., 2025). For instance, Agents4PLC (Liu et al., 2024c) incorporates a dedicated debugging agent that provides fixing advice back to the coding agent when code fails, drawing inspiration from existing automated code repair methods. It can also leverage previously generated patches in a self-RAG context during the debugging phase to improve repair effectiveness. Similarly, frameworks like OpenCodeInterpreter (Zheng et al., 2024) involve prompting LLMs (e.g., GPT-4) to intentionally generate incorrect code, then using the resulting error messages as cues for correction. This process mimics a real-life debugging cycle. StepCoder (Dou et al., 2024b) also uses reinforcement learning with compiler feedback to guide the exploration of the output space for LLMs, aiming to improve code generation quality. Despite recent progress, models lack the formal correctness guarantees and specialized support required for industrial applications. In Programmable Logic Controller (PLC) programming, for example, they often struggle to satisfy the rigorous logical and safety constraints essential for industrial control systems (Liu et al., 2024c). Moreover, LLMs have difficulty executing complex coding tasks and frequently produce suboptimal algorithms, resulting in issues like timeout errors (Olausson et al., ; Chidambaram et al., 2024; Dou et al., 2024a). Even when directed to use efficient solutions like dynamic programming, LLMs can still create inefficient code (Qiu et al., 2024). Additionally, LLMs can misunderstand programming specifications, which leads to generated code that does not align with the userâs original intent. When the generated code deviates significantly from the ground truth, the coarse-grained error messages produced during automated feedback cycles often prove insufficient for effective improvement (Tian et al., 2025). When LLMs are employed in coding assistance, they might inadvertently bring about new risks. For instance, feedback-driven efforts to rectify identified security flaws may result in the LLM inserting defects into previously flawless code (Chong et al., 2024). Additionally, code produced by LLMs frequently omits defensive programming techniques, rendering it less secure and susceptible to vulnerabilities such as buffer and integer overflows (Licorish et al., 2025). Furthermore, this machine-generated code tends to be more intricate per line compared to code crafted by humans, which might complicate subsequent maintenance and analysis (Dou et al., 2024a). The perceived advantages of LLMs can engender an unfounded sense of productivity and security, given that LLMs might create flawed or weak code and misleading recommendations. Excessive dependence on these models can result in time wastage and suboptimal tool utilization (Tie et al., 2024). Alongside security issues, LLM-assisted coding tasks also face computational challenges. The cost of enabling self-repair might lead to only slight improvements, which are inconsistent across various tasks or datasets and can occasionally be negligible (Olausson et al., 2023). This is mainly due to the limited capacity of the model to introspect and reliably generate effective feedback on its own mistakes (Olausson et al., 2023; Chidambaram et al., 2024). Finally, LLMs show decreased efficiency in processing feedback aimed at rectifying multiple errors at once, indicating that such feedback might be more effective if it is broken down into simpler steps to enhance model performance (Chen et al., 2023). 6.2. Incorporating Human Feedback In addressing the limitations inherent in code generation by LLMs, human feedback is essential. This is commonly implemented through methods such as Reinforcement Learning from Human Feedback (RLHF) and Human-in-the-Loop (HITL) systems (Ouyang et al., 2022; Kathiresan, 2025; Wong and Tan, 2024). Such strategies employ human insight, proficiency, and subtle comprehension to steer LLMs toward producing more accurate, secure, and coherent code. Itâs crucial that users verify their codeâs security and integrity (FU et al., 2024; Pearce et al., 2022), while developers should remain alert and exercise caution with generative AI (TĂłth et al., 2024). The reliability of LLM-generated code, including suggested improvements, is typically variable and necessitates human oversight to ensure consistency (Elgedawy et al., 2024; Belozerov et al., 2025). Developers are warned against placing complete trust in either AI-generated or AI-revised code fragments (Hamer et al., 2024). At the same time, incorporating human feedback does not automatically eliminate risk. The security benefits of RLHF, HITL, and related feedback-driven workflows depend heavily on reviewer expertise, the quality and consistency of the feedback provided, and the surrounding workflow design (Belozerov et al., 2025; Firouzi and Ghafari, 2026; Kathiresan, 2025; Li et al., 2026; Mohsin et al., 2024; Tie et al., 2024). Poorly calibrated trust, superficial review, or misplaced confidence in AI-revised outputs can still allow insecure code to pass through development pipelines, while responsibility for failures may become blurred across model designers, tool providers, and human reviewers (Belozerov et al., 2025; Hamer et al., 2024; Khoury et al., 2023; Perry et al., 2023). Human feedback should therefore be understood not as a guaranteed safeguard, but as a socio-technical control whose effectiveness depends on how the collaboration is structured and audited. Reinforcement Learning from Human Feedback (RLHF). A prominent method for aligning model behaviour is RLHF, a technique that fine-tunes LLMs using human preferences as a reward signal (Ouyang et al., 2022; Brown et al., 2020; Bai et al., 2022; Wang et al., 2024b). The standard pipeline involves supervised fine-tuning on human demonstrations, followed by the training of a reward model using human-ranked output comparisons. In the final stage, the model is refined via Proximal Policy Optimization (PPO) (Ouyang et al., 2022), using the reward model as a signal to guide the model toward preferred responses. RLHF has significantly improved the performance of models such as Codex and GPT across various tasks (Wong and Tan, 2024; Ouyang et al., 2022). However, the method is constrained by the high cost and inconsistency of human feedback. Relying on individual evaluators introduces subjectivity and bias, which can be counterproductive in uncertain domains where input may be unreliable. Furthermore, the substantial labor required and the limited scale of human-labeled datasets often restrict the efficiency and measurable impact of this alignment strategy. Crowd-Sourced RLHF (cRLHF). To mitigate the inherent biases and constraints associated with individual evaluators, Wong et al. (Wong and Tan, 2024) introduced cRLHF, a framework that mitigates the biases of individual evaluators by aggregating feedback from multiple annotators. Using Bayesian inference, the system combines multi-source assessmentsâincluding line-by-line accuracyâinto a "coordinated score" between 0 and 1. This score serves as a direct reward for model refinement, simplifying the reward modeling process while ensuring outputs align with diverse human expectations. Empirical results demonstrate that this collaborative approach effectively reduces model bias and enhances the quality of generated code Crowdsourced helps reduce bias and improve evaluation objectivity by aggregating diverse human inputs. However, its effectiveness is often constrained by small sample sizes and the logistical challenge of recruiting expert annotators for large-scale studies (Wong and Tan, 2024). Human-in-the-Loop (HITL) Another crucial approach is HITL testing, which integrates human oversight at critical junctures to supplement automated evaluations (Kathiresan, 2025). This approach addresses the non-deterministic nature of model outputs by using automated tools to flag high-impact or low-confidence results for expert review. Human feedback on the accuracy and safety of these outputs is then used to refine prompts and triage criteria. This iterative process, which may include RLHF for response optimization (Ouyang et al., 2022), leverages models to generate adversarial test cases and summarize large datasets, ultimately improving both the reliability and precision of the system. HITL improves the reliability and safety of model-integrated systems, especially in high-stakes domains, by combining large-scale automated analysis with targeted human judgment (Kathiresan, 2025). Human reviewers can catch nuanced errors, biases, and contextual issues that automated tools often miss, although the approach remains selective and is limited by the scale of expert feedback available. Conversational Automated Program Repair (Conversational APR). Conversational APR iteratively generates patches and incorporates validation feedback, such as test results, outputs, and error messages, to refine subsequent repairs (Xia and Zhang, 2023). By combining prior patches with feedback in each round, it helps LLMs avoid repeating mistakes and better understand program semantics. Successful patches can also be varied through additional conversational turns. This approach can substantially improve repair quality by leveraging natural-language hints and human clarification; indeed, a brief human correction may be more effective than greatly increasing the number of generated samples. However, the method can be computationally expensive because it often requires many LLM interactions (Tian et al., 2025). Conversational Semi-Automatic Repair Framework (Cref). This system employs LLMs specifically designed to assist programming tutor (Yang et al., 2024a). It employs interactive processes involving tutor insights, solution narratives, and test cases that fail. Cref handles everything in one dialogue session and advances through specific stages: initially supplying LLMs with programming problems, erroneous code, and tutor assistance; if the code is still erroneous, it revisits all prior data and solution outlines; and if further correction is needed, it utilizes historic entries alongside failing test cases of the most recent code fix. This methodology notably enhances code generation efficiency via natural language prompts. Conversational APR (Xia and Zhang, 2023) strategically circumvents the replication of past errors in patches and effectively uses previously overlooked test case details. Cref has achieved substantial progress in Average-5 scores for code repair, cutting debugging times by 71.2% and costs by 69.9% in real-world corporate scenarios. It also boosts tutoring success and enriches studentsâ learning experiences. The achievements of Cref indicate its potential for wider use cases, such as automating code review processes by refining code based on reviewer feedback. Socratic Human Feedback (SoHF). SoHF (Chidambaram et al., 2024) is a form of expert feedback in which programmers guide LLMs through iterative natural-language prompts, much like a tutoring session. Using strategies such as test reiteration, error highlighting, and requirement clarification, experts provide step-by-step guidance across multiple debugging rounds. This approach can substantially improve performance: with expert Socratic guidance, LLMs were able to solve up to 74% of problems they initially failed. However, existing studies focus mainly on expert programmers, and some do not account for the time required for debugging, leaving its broader applicability unclear. Imitation Learning from Language Feedback (ILF). Chen et al. (Chen et al., 2023) introduced ILF, a framework that incorporates natural language feedback during the training phase rather than solely at inference. The process begins by identifying code that fails unit tests and obtaining human-written feedback for those errors. A specialized refinement model, ĎRâeâfâiânâe _Refine, uses this feedback to generate corrected versions of the code, which are then used to fine-tune the primary model. This iterative approach, which effectively minimizes the KL divergence from the target distribution, requires minimal human-authored feedback and remains independent of such feedback during testing. Evaluation on the MBPP benchmark showed that ILF increased the pass@1 rate of a 6.1B parameter model by 38% relatively and 10% absolute. Despite these gains, the framework is limited by the high cost of human feedback and the difficulty ĎRâeâfâiânâe _Refine faces when addressing multiple, simultaneous errors. Expert review is vital for auditing and refining code produced by automated systems to ensure functional correctness and security (Dai et al., 2025; Liu et al., 2024c; Wong and Tan, 2024). Professional developers are uniquely equipped to implement defensive programming and guardrails that automated tools frequently overlook, particularly in security-critical environments (Chong et al., 2024). By leveraging deep domain knowledge and context awareness, human reviewers can resolve ambiguous requirements and adhere to industrial standards more reliably than current computational models (Licorish et al., 2025). Despite being resource-intensive and difficult to scale, manual oversight remains a necessary safeguard against the structural complexity and subtle vulnerabilities common in machine-authored software (Zhou and Xu, 2020; Chong et al., 2024). Overall, human expertise continues to be crucial for addressing the shortcomings of LLMs, particularly in handling intricate, security-sensitive, or unclear tasks where LLMs face challenges related to efficiency, ambiguous requirements, and the creation of robust, secure, and maintainable code (Chong et al., 2024; Licorish et al., 2025; Yang et al., 2024a). It is most effective to regard LLMs as an enhancement to human programmers rather than a replacement. The cooperation between human skills and LLM capabilities, especially through iterative and focused feedback, is essential for the progress of AI-supported programming (Licorish et al., 2025). Taken together, the literature suggests that human oversight is both indispensable and fragile in LLM-assisted coding. Human expertise remains essential for clarifying intent, evaluating security-sensitive outputs, and correcting model failures, but the introduction of LLMs can also distort developer behavior through over-reliance, reduced scrutiny, and miscalibrated trust (Belozerov et al., 2025; Hamer et al., 2024; Khoury et al., 2023; Mohsin et al., 2024; Perry et al., 2023; Tihanyi et al., 2024). The practical challenge is therefore not simply to keep a human in the loop, but to design human-AI workflows in which oversight is informed, accountable, and appropriately matched to the risks of the task. 7. Open Challenges and Research Directions The reviewed literature points to three persistent challenges for secure LLM-assisted coding: a mismatch between probabilistic generation and deterministic security requirements, unreliable or incomplete evaluation pipelines, and fragile human-AI collaboration shaped by over-trust, weak prompting, and limited accountability. This section synthesizes these challenges into a research and deployment agenda, then outlines governance, practical recommendations, and broader implications for academia and industry. 7.1. Core Challenges Our study indicates that the security concerns associated with LLM-assisted coding do not merely consist of isolated bugs but rather merge into three interrelated, major meta-challenges. These three vulnerabilities form the core of the issue and require a comprehensive approach for resolution. 7.1.1. Probabilistic Models vs. Deterministic Security The core issue is a fundamental discrepancy between the function of LLMs and the demands of secure software development. LLMs operate as probabilistic, pattern-driven models designed to predict the next probable token based on the statistical trends found in their training data. They do not possess a profound, semantic comprehension of security principles, threat vectors, or the intricate context that differentiates secure from insecure code. Their key aim is functional plausibility and fluency in language, which means they aim to generate working code without an inherent focus on security vulnerabilities. This fundamental limitation manifests through several significant issues. Firstly, LLMs are trained on extensive publicly available code, such as the open-source collections on GitHub, which tend to contain vulnerabilities, outdated dependencies, and insecure coding standards. As a result, the models learn to emulate these issues. Research repeatedly shows that a significant proportion of code produced by LLMs is inherently insecure; for example, studies reveal that around 40% of snippets from models such as GitHub Copilot contain vulnerabilities. This happens because the modelâs objective is to replicate patterns, and insecure patterns are abundant in the training data. Secondly, when LLMs are used for APR and patching vulnerabilities, they may produce corrections that seem functionally sound for a limited set of tests but remain vulnerable to more advanced or unforeseen threats. The patch may fix the current issue but inadvertently create a new, subtler vulnerability or only partially mitigate the risk, leaving it exploitable. Third, LLMs often suggest or use deprecated libraries, outdated APIs, or cryptographically vulnerable algorithms because such insecure patterns remain within their training sets and are suggested without recognising their compromised state. Although the common explanation often blames bad training data, a more detailed analysis uncovers a deeper architectural flaw. The central problem is a semantic gap between statistical association and logical deduction. LLMs, as designed, do not reason about security attributes; they determine statistical probabilities. Even with security-focused prompts, models struggle because they are not equipped for abstract reasoning about possible threat vectors or security invariants. This indicates that merely cleaning the training data, while necessary, will not fully solve the problem. It is similar to teaching someone to write grammatically by exposing them to a large library without explicitly teaching grammatical rules. They might mimic style and common expressions, but cannot reason from grammatical fundamentals. Thus, to achieve genuinely secure code generation requires more than just data refinement; it points to a necessity for profound changes in model design, training goals, or the incorporation of symbolic reasoning abilities. 7.1.2. The Unreliability of Ground Truth The second significant obstacle is the inadequacy of current methods for assessing the security of code generated by LLMs. This leads to an evaluation crisis, where benchmark scores can be deceptive, potentially inflating a modelâs genuine security capabilities and creating an illusion of progress. Without reliable measurement, improvements cannot be accurately made. This crisis stems from various shortcomings in current evaluation practices. Firstly, many primary benchmarks for code generation, such as HumanEval (Chen et al., 2021), focus on functional correctness for small, self-contained programming tasks. However, these benchmarks often have a limited range and size, insufficient for detecting subtle logical mistakes or complex security weaknesses, resulting in exaggerated performance metrics that donât represent true robustness. Besides, the security evaluation of code heavily relies on SAST tools like CodeQL (CodeQL, 2025) and Bandit (Bandit, ), which are known to be neither sound nor comprehensive, displaying considerable rates of false positives and false negatives. Moreover, these tools typically assess code in isolation, lacking the broader repository context (like data flows across multiple files) necessary to uncover many intricate real-world vulnerabilities. Although dynamic analysis can surpass static analysis in certain areas, it is inherently limited by the quality and breadth of the test cases employed. Most benchmarks fail to mimic the complexity of real-world software development, where newly created code must be correctly and securely integrated into a large, existing, and complex repository. There is a clear progression in evaluation paradigms, shifting the primary question from "Does the code work?" to "Can the code be exploited?". Initial benchmarks like HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) concentrated on functional correctness, utilising a cooperative set of predefined inputs. In contrast, a new wave of benchmarks is surfacing that adopts a more adversarial approach. Frameworks such as SecRepoBench (Dilgren et al., 2025), CodeLMSec (Hajipour et al., 2024a), and BAXBENCH (Vero et al., ) are crafted to test vulnerabilities under more authentic conditions. They focus on repository-level context, dynamic execution of exploits, and resilience against adversarial inputs. This transition is significant as it redefines a good model not merely as one that passes tests but as one that withstands attempts at compromise. The future of effective LLM security evaluation lies in advancing and utilising these more demanding, adversarial, and context-sensitive benchmarking frameworks. 7.1.3. Human-AI Trust-Vulnerability The third significant challenge emerges at the complex intersection where human developers engage with AI assistants. While human oversight is generally considered the most reliable defence against the weaknesses of LLMs, it also presents significant vulnerabilities. This forms a trust-vulnerability nexus, where cognitive biases, different expertise levels, and misaligned trust can compromise the security of the entire development process. This connection presents itself in several concerning ways. First, a dangerous bias has surfaced where developers, especially those using AI coding assistants, generate code with lower security standards, even while being overly confident about its safety. This situation generates a misguided sense of security, which diminishes thorough examination and leads to the premature integration of flawed or insecure code into operational systems. Furthermore, the security of output from LLMs is largely dependent on the userâs skill in creating precise, security-conscious prompts and critically assessing the generated code. Inexperienced programmers or developers without specific security training are especially likely to create vague prompts and accept superficially suitable but insecure code, thereby adding new vulnerabilities to the software supply chain. Additionally, in the realm of software engineering, the notion of trust in LLMs is currently vague, inconsistently evaluated, and poorly comprehended. This leads to a precarious miscalibration, where a developerâs trust level does not match the modelâs actual effectiveness for a task. This could lead to either over-trust, resulting in uncritical acceptance of flawed code, or under-trust, inhibiting the use and advantages of the technology. The common advice of simply training developers more is necessary, but not enough to tackle this challenge. While it is crucial to educate on security, the main issue is not merely a lack of knowledge but one of human-computer interaction (HCI). The essential challenge is around calibrating trust. Developers need efficient ways to discern when and how much to trust the LLMâs output, as well as what aspects require closer examination. This shifts the focus from merely training to navigating a complex design challenge. Future research and tool development should aim to build Human-AI collaboration systems that actively aid in this calibration. Potential solutions include AI systems that communicate their uncertainty, provide verifiable evidence for their code suggestions, or automatically flag high-risk code segments that demand mandatory, thorough human assessment. This approach seeks to shift the human role from a passive, potentially susceptible reviewer to an empowered, critical, and effective guardian of code quality and security. 7.2. A Research Roadmap for Trustworthy LLM-Assisted Coding Tackling the triad of challenges presented earlier demands a collaborative, cross-disciplinary research approach. We organize this roadmap around three core pillars: constructing inherently secure models, developing more reliable and realistic security benchmarks, and improving the calibration of human-AI collaboration. A fourth, cross-cutting requirement (accountability, auditability, and governance) is discussed separately in Section 7.3 because it conditions how progress in all three pillars can be translated into trustworthy real-world deployment. 7.2.1. Towards Inherently Secure Models The key aim is to evolve from a reactive, post-incident security stance towards developing LLMs that are inherently secure. Achieving this goal involves integrating security principles into the modelâs fundamental architecture and training frameworks, rather than considering security as an afterthought. Security-aware architectural design and training are crucial for upcoming research, directing focus towards innovative model structures and training goals that surpass mere next-token forecasting. The prevailing approach, which chiefly values functional accuracy and fluency, falls short for applications critical to security. Future inquiries should explore incorporating security-driven loss functions that specifically discourage code creation with established vulnerability patterns (CWEs). Another promising path is crafting hybrid architectures that merge the pattern-detection capabilities of neural networks with symbolic reasoning elements adept at grasping and imposing security properties and constants. Advanced Security-Oriented Fine-Tuning provides a practical solution, given the formidable costs associated with initiating foundation model pre-training. Techniques for Parameter-Efficient Fine-Tuning (PEFT), like Low-Rank Adaptation (LoRA), have proven effective for tailoring pre-trained models to secure code generation without the extensive computational demands of full retraining. A pivotal area of research focuses on developing and curating extensive, high-quality datasets featuring confirmed vulnerability corrections to instil secure coding norms in models. Additional study is warranted to comprehend how varying granularities of dataâsuch as file-level, function-level, or detailed block-level modificationsâaffect the security of the refined modelâs outputs. A notable concern is safety degradation, where fine-tuning a model even with benign, task-specific data may unintentionally jeopardize its overall safety alignments. This underlines the necessity for Safety-Aware Fine-Tuning (SAFT) frameworks designed to automatically identify and exclude potentially harmful or vulnerability-inducing data from the refinement set or to guide gradient updates away from hazardous parameter territories via safety-aware probes. Formal Methods for Verifiable Security and Correctness represent the most thorough approach to ensuring software security through formal validation. Although directly producing formally verified code via LLMs is currently unsuitable except for the simplest programs, a highly promising area merges LLMs with formal methods tools. In this context, the LLM serves as a verification assistant rather than the definitive code author. For instance, frameworks like Marmaragan (Cramer and McIntyre, 2025) utilise an LLM to produce formal annotations, such as pre- and post-conditions, loop invariants, and contracts, for existing code in languages like SPARK. These annotations are then subjected to checks by a dedicated formal verifier. The LEMUR calculus lays down a theoretical base for this interaction, conceptualising a coherent loop where the LLM suggests properties and an automated reasoner seeks to validate them. Similarly, LLMLift employs an LLM to generate program invariants necessary for proving the equivalence of code translated between languages. This hybrid approach, blending verifier-in-the-loop, significantly harnesses the pattern-recognition and creative-generation capacities of LLMs while rooting them in the mathematical exactness of formal verification, paving the way for constructing more reliable systems. 7.2.2. Next Generation of Security Benchmarks To advance and monitor progress effectively within the community, it is essential to develop a new generation of evaluation benchmarks that are more trustworthy, realistic, and thorough than those currently widespread. Holistic, Contextual, and Repository-Level Evaluation: The field must move decisively beyond evaluating isolated, function-level code snippets created in isolation. Future benchmarks should be designed at the repository level to be impactful. This requires furnishing the LLM with the relevant contextâsuch as dependency structures, data-flow across files, and existing API definitionsâneeded to produce code that integrates accurately and securely into a larger, pre-existing codebase, not just correct in isolation. SecRepoBench (Dilgren et al., 2025) is an innovative initiative in this aspect, formulating code completion tasks from actual vulnerability patches in C/C++ projects and employing the projectsâ own unit test suites for validation. The HumanEvo (Zheng et al., 2025) benchmark enhances this by offering a historically accurate repository context, restoring a project to its state just before a target function was initially developed, thereby creating a more realistic evaluation scenario. Dynamic and Exploit-Driven Security Assessment: Sole reliance on static analysis to identify vulnerabilities is inadequate due to its inherent lack of precision. Future security evaluations must be dynamic, monitoring code during execution, and, where feasible, driven by the execution of real exploits. Benchmarking the Human-AI System: A crucial step for future evaluation research is to transition from assessing the AI alone to benchmarking the performance of the entire human-AI system. This would entail devising evaluations that mimic a real-world developer workflow. Metrics could encompass the security of code generated by a developer with an AI assistant, the time and effort needed for a human reviewer to identify and amend AI-induced flaws, or the efficiency of AI-aided review tools in enhancing a developerâs security performance. This comprehensive approach would deliver a more accurate depiction of these toolsâ impact on software security. 7.2.3. Calibrating Human-AI Collaboration Given that human oversight will remain an essential component of secure software development for the foreseeable future, research must focus on making this collaborative process as effective and secure as possible. The goal is to transform the human from a potential point of failure into an empowered and critical partner. Modelling and Measuring Trust to Mitigate Over-reliance: The well-documented risk of automation bias and developer over-trust is a primary human-related threat in LLM-assisted coding. To counter this, research drawing heavily from the field of Human-Computer Interaction (HCI) is needed to formally define, model, and measure the concept of trust in the specific context of software engineering. The objective is not simply to maximise trust, but to develop systems that can actively calibrate a userâs trust to align with the modelâs actual capabilities and reliability for a given task. This could be achieved through novel user interfaces that visualise a modelâs uncertainty about its own output, provide verifiable evidence for its suggestions, or automatically flag high-risk or low-confidence code segments that require mandatory, heightened human scrutiny. Developing Tools for Secure Prompting and AI-Assisted Review: The quality of the interaction between the developer and the AI is paramount. Research should focus on creating tools and interaction paradigms that facilitate more secure and effective collaboration. Promising directions include: ⢠Interactive and Conversational Repair: Frameworks that use a dialogue-based process of iterative feedback and refinement have shown significant promise. Systems like Cref (Yang et al., 2024a) and conversational APR (Xia and Zhang, 2023) allow a developer to guide the LLM toward a correct and secure solution through a natural conversation, providing error messages or clarifications that help the model understand the programâs semantics and avoid repeating mistakes. ⢠Guided Reasoning and Socratic Feedback: Techniques like Socratic Human Feedback (SoHF) (Chidambaram et al., 2024), where an expert programmer guides the LLM with a series of probing questions and step-by-step debugging hints, can enable models to solve complex problems that they would fail at on their own. This approach turns the interaction into a collaborative problem-solving session rather than a simple request-response transaction. ⢠AI-Powered Review Tools: The development of tools that integrate LLM-based vulnerability analysis directly into the IDE can augment the developerâs abilities. Tools like CODEGUARDIAN (Dozono et al., 2024) have been shown to make developers both faster and more accurate at identifying security flaws in code. 7.3. Accountability, Auditability, and Governance A recurring theme across the literature is that the risks of LLM-assisted coding cannot be reduced to model quality alone (Treude, 2026; Taldenko, 2026; Chow and Li, 2025; Chow and Li, 2026; Kathiresan, 2025; Li et al., 2026; National Institute of Standards and Technology, 2023). Even if a model is improved through better data curation, safer decoding, or stronger evaluation pipelines, real-world deployment still raises unresolved questions of responsibility, traceability, and governance (Chong et al., 2024; Chow and Li, 2025; Kathiresan, 2025). These issues become especially acute when insecure code is generated collaboratively by humans and AI systems or when agentic systems perform multi-step actions with limited human intervention (Chow and Li, 2025; Li et al., 2026). Accordingly, accountability, auditability, and governance should be treated not as peripheral policy concerns, but as cross-cutting conditions for the trustworthy deployment of LLM-assisted coding systems. Responsibility for insecure outputs. A fundamental challenge concerns responsibility allocation when insecure or harmful code is introduced through AI-assisted development (Chong et al., 2024; Chow and Li, 2025; Chow and Li, 2026; Li et al., 2026). In practice, responsibility may be distributed across several actors: the model developer, the provider of the coding assistant, the organization deploying the tool, the engineer who accepted the output, and the reviewer who approved it. This diffusion of responsibility can create accountability gaps, especially when AI-generated code is treated merely as a productivity aid rather than as an active decision-shaping component of the development process (Chow and Li, 2025; Chow and Li, 2026; Ma et al., 2024). For secure deployment, organizations must therefore adopt explicit policies stating that AI assistance does not eliminate human accountability for the final artifact. Auditability and traceability. Secure use of LLM-assisted coding also requires the ability to reconstruct how a code artifact was produced (National Institute of Standards and Technology, 2024; Chow and Li, 2026; Li et al., 2026; Treude, 2026). This includes preserving relevant prompts, retrieved context, intermediate model outputs, tool invocations, validation results, and subsequent human edits. Without such traceability, it becomes difficult to investigate incidents, reproduce failures, or determine whether a vulnerability emerged from the model, the prompt, the retrieval source, the toolchain, or the human review process (Chow and Li, 2026; Li et al., 2026). Auditability is thus not only a compliance concern; it is also a practical prerequisite for debugging, incident response, and continuous security improvement. Liability in regulated and high-stakes environments. The accountability problem becomes more pressing in regulated domains or safety-critical contexts, where software faults may trigger legal, financial, or societal harm (, 2025; Chow and Li, 2025; Chow and Li, 2026). In such settings, organizations cannot rely on opaque AI-generated artifacts without rigorous documentation, testing, and approval procedures. The introduction of LLM-assisted coding into domains such as healthcare, finance, public infrastructure, or industrial control raises difficult questions about due diligence, certification, and evidentiary standards for demonstrating that adequate safeguards were applied. Governance as a socio-technical control layer. These concerns suggest that governance must be treated as a first-class layer of defence rather than as an afterthought (, 2025; Chong et al., 2024; Chow and Li, 2025; Chow and Li, 2026). Effective governance includes policies for prompt and data handling, restrictions on the use of external tools and proprietary context, model and plugin vetting, mandatory review requirements for security-critical code, and documentation of human oversight and approval pathways. In the case of agentic systems, governance should additionally include action boundaries, escalation rules, logging of tool-mediated behavior, and explicit mechanisms for human interruption and override (Chow and Li, 2026; Li et al., 2026). Overall, accountability, auditability, and governance connect technical security properties with real deployment conditions and help ensure that human-AI collaboration remains reviewable, contestable, and aligned with institutional responsibility. Overall, accountability, auditability, and governance should be understood as integral components of secure LLM-assisted coding. They connect technical security properties with real deployment conditions and help ensure that human-AI collaboration remains reviewable, contestable, and aligned with institutional responsibility. 7.4. Practical Insights and Actionable Recommendations Organisations and software development teams have the opportunity to promptly implement specific measures to reduce security risks linked with incorporating LLMs into their processes. Here are seven practical recommendations that offer a comprehensive defence strategy for both leveraging the advantages of LLMs and addressing their potential risks. Adopt a Zero-Trust Policy for AI-Generated Code. Treating all code generated by an LLM as untrusted third-party input is the most indispensable rule, no matter how advanced the model or how specific the prompt. Such code should undergo stringent security assessments, peer review, and extensive testing akin to what is applied to code from external or junior sources. It is imperative that LLM-generated code never be deployed directly to production without exhaustive human supervision and a series of automated security evaluations. Invest in Continuous Developer Training for Secure AI Interaction. Human developers serve as the ultimate safeguard, with their knowledge being a crucial resource. Organisations need to prioritise ongoing training initiatives to enhance developersâ understanding of the inherent limitations and particular security challenges associated with LLMs. This educational effort should go beyond standard secure coding practices to encompass AI-centric skills, including secure prompt engineering (e.g., crafting clear, precise, and security-conscious prompts), understanding common vulnerabilities linked to LLMs, and methods for critically assessing AI-generated code, including identifying indicators of model overconfidence or factual inaccuracies. Integrate Multi-Layered, Automated Security Analysis into the CI/CD Pipeline. Utilizing only one security tool or method is inadequate for addressing the variety of possible issues in code generated by AI. An effective approach necessitates establishing a pipeline that integrates multiple levels of automated analysis that complement each other. This should encompass: Static Application Security Testing (SAST) to detect frequent vulnerability patterns within the source code prior to execution; Dynamic Application Security Testing (DAST) to discover runtime errors and vulnerabilities in the operational application; and Software Composition Analysis (SCA) to pinpoint and control vulnerable or outdated third-party libraries and dependencies, as LLMs are often known to recommend such components. Mandate Rigorous, Context-Aware Human Review for Security-Critical Code. Although all AI-produced code necessitates human oversight, code that is to be implemented in security-sensitive functions must undergo a rigorous and obligatory review process. This encompasses code involved in authentication, authorisation, encryption activities, session management, and input verification. Such evaluations should be performed by senior developers or specialised security professionals capable of analysing the code not only independently, but also within the broader context of the application, taking into account its interactions with various components and data pathways. Develop and Enforce Strict Organisational Policies for Prompt Engineering and Data Handling. Organisations need to create and implement well-defined governance policies outlining how developers should engage with LLMs, especially those from third parties. Such policies should clearly forbid embedding sensitive or proprietary dataâlike API keys, internal algorithms, trade secrets, or Personally Identifiable Information (PII)âinto prompts given to external models. In the case of applications utilising Retrieval-Augmented Generation (RAG), these policies must guarantee that internal knowledge bases used for retrieval are securely protected, access-controlled, and sanitised to avoid unintended data breaches and prevent malicious RAG poisoning attacks. Prioritise and Vet the AI/ML Component Supply Chain. Before introducing a new tool, plugin, or open-source model based on LLMs, it is essential for organisations to perform comprehensive security due diligence. This process includes assessing the vendorâs security practices, data privacy and retention policies, and model training procedures.5 In the case of open-source models obtained from public repositories, a standardised intake process should be adopted. This process should involve initial red-teaming to explore adversarial vulnerabilities and bias evaluations to highlight any potential ethical risks prior to integrating the model into internal development processes. Establish Clear Lines of Accountability for AI-Assisted Development. The legal and regulatory framework concerning liability for mistakes and weaknesses in AI-generated code is unclear and continues to develop. In such a scenario, organisations must implement well-defined internal policies that delineate accountability. The human developer and engineering team should bear the final responsibility for the security, accuracy, and quality of the completed software, rather than the AI tool utilised. Keeping comprehensive documentation of the human oversight, amendment, and approval processes for all AI-generated code is vital for showing thoroughness and mitigating legal liabilities. 7.5. Implications for Academia and Industry The rise of LLM-assisted programming brings about profound implications extending beyond technical challenges alone. It is reshaping research directions in academia and demanding new strategic methods for risk management and governance within the industrial sector. Navigating this new era effectively requires a collaborative effort between these communities to balance the pursuit of innovation with the crucial requirement for security and reliability. 7.5.1. Implications for Academia The issues highlighted by this survey indicate the necessity for a pivotal change in academic approaches to software engineering and artificial intelligence research. Traditionally, success for code-generating LLMs has been measured by their performance on tests like HumanEval (Chen et al., 2021), which largely focus on solving algorithmic puzzles in controlled environments. It is critical for the academic field to shift its focus toward emphasising security as a primary research goal. This includes fostering research on model architectures designed with security in mind, developing robust defences against adversarial threats, and creating realistic evaluation benchmarks that emphasise context-rich security scenarios. Adopting this new focus may require accepting compromises, such as models that are more secure but show reduced performance on specific functional tasks being deemed more valuable scientific contributions. The challenge of ensuring the security of LLM-assisted coding is not one that can be tackled by computer science in isolation. For instance, the Human-AI Trust-Vulnerability Nexus is inherently a Human-Computer Interaction (HCI) issue, demanding insights from psychology and cognitive science to effectively address and mitigate biases. Additionally, tackling concerns related to legal liability, data privacy, intellectual property, and algorithmic fairness requires collaboration with experts in law, ethics, and social sciences. The most significant future research efforts in this field will be fundamentally interdisciplinary, blending technical solutions with a detailed understanding of their human, societal, and legal implications to develop systems that are not only technically sound but also genuinely trustworthy. Besides the practical aspect of constructing and testing more secure systems, there is an urgent need for foundational research aimed at understanding why these systems fail. This involves interpretability studies using techniques like representation engineering to explore how LLMs internally comprehend security concepts such as vulnerability or sanitisation. It also includes theoretical research seeking to outline the connection between a modelâs structural features, its training data, and the security assurances of its outputs. Gaining a deeper, foundational understanding of these systems is essential for advancing beyond empirical trial-and-error toward the systematic design of verifiably secure AI. 7.5.2. Implications for Industry For industries, the primary challenge is to strategically manage the inherent conflict between leveraging the substantial productivity gains offered by LLMs and addressing the considerable security, legal, and reputational risks they pose. LLMs are fundamentally redefining the economic dynamics of cybersecurity. For attackers, LLMs commoditise intelligence, significantly reducing the cost to uncover vulnerabilities and engineer custom, long-tail assaults on a broad spectrum of less common software, rendering previously unattractive targets financially worthwhile. This generates a compelling market-driven incentive for organisations to bolster their AI security measures. On the flip side, there is a rising positive economic incentive. As the market for specialised LLM agents and tools evolves, aspects like security, reliability, and trustworthiness will become vital competitive advantages. Vendors capable of proving that their products deliver enhanced security and produce safer code will be able to demand a premium, encouraging a market that benefits those investing in secure AI advancements. The global regulatory framework for AI is quickly transitioning from broad ethical guidelines to specific, enforceable legal standards. Regulations such as the EU AI Act, which classifies AI systems based on risk, the NIST AI Risk Management Framework, and new standards like ISO 42001, are setting a new compliance landscape for organisations deploying AI (, 2025). These rules will necessitate organisations to develop strong governance, risk management, and compliance strategies for their AI systems. Moreover, the ongoing uncertainty regarding legal accountability for damages created by flaws in AI-produced code will push companies to adopt stringent internal validation, oversight, and documentation procedures to display due diligence and reduce legal liabilities. In the complex and dynamic environment, industry cannot afford to await perfectly secure, one-size-fits-all AI models. The sole feasible approach is a proactive, defence-in-depth strategy that combines technology, processes, and people. This entails investing in a comprehensive security technology stack, forming robust internal AI governance committees with well-defined accountability, and crucially, cultivating a culture of security awareness and critical thinking among developers. In the end, as AI increasingly automates coding procedures, the uniquely human abilities to judge, understand context, and provide critical oversight become some of the most crucial and irreplaceable security assets an organisation holds. 8. Conclusion The integration of LLMs into software development offers substantial productivity benefits but introduces systemic security risks. These vulnerabilities originate from a fundamental triad: the modelsâ lack of semantic security awareness, the inadequacy of current benchmarks, and the cognitive biases inherent in human-AI interaction. Our review identifies risks ranging from insecure code generation to the propagation of outdated dependencies, highlighting that evaluation must move beyond functional correctness toward adversarial, context-specific security assessments. Furthermore, developer oversight remains a critical but fragile defense, requiring tools that support rather than mislead the user. To address these challenges, we propose a research roadmap focused on security-driven training, more stringent benchmarks, and robust collaboration frameworks. For practitioners, we advocate a defense-in-depth strategy rooted in a zero-trust approach to model-generated code. Ultimately, the secure adoption of this technology requires a coordinated effort between academia and industry to prioritize security-focused governance alongside innovation. References Aghakhani et al. (2024) H. Aghakhani, W. Dai, A. Manoel, X. Fernandes, A. Kharkar, C. Kruegel, G. Vigna, D. Evans, B. Zorn, and R. Sim Trojanpuzzle: covertly poisoning code-suggestion models. In 2024 IEEE Symposium on Security and Privacy (SP), p. 1122â1140. Cited by: Table 2, §4.3. Alrashedy et al. (2023) K. Alrashedy, A. Aljasser, P. Tambwekar, and M. Gombolay Can llms patch security issues?. arXiv preprint arXiv:2312.00024. Cited by: §4.1, Table 3. Anand et al. (2024) A. Anand, A. Gupta, N. Yadav, and S. Bajaj A comprehensive survey of ai-driven advancements and techniques in automated program repair and code generation. arXiv preprint arXiv:2411.07586. Cited by: Table 1, §1, §2. Asare and Nagappan (2023) O. Asare and M. Nagappan Is githubâs copilot as bad as humans at introducing vulnerabilities in code?. Empirical Software Engineering 28 (6), p. 129. Cited by: Table 2, Table 2, §4.1, §4.1, Table 3. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §5.1.1, Table 4, §7.1.2. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional AI: harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. Cited by: §6.2. [7] BanditBandit(Website) External Links: Link Cited by: §7.1.2. Barke et al. (2023) S. Barke, M. B. James, and N. Polikarpova Grounded copilot: how programmers interact with code-generating models. Proceedings of the ACM on Programming Languages 7 (OOPSLA1), p. 85â111. Cited by: §1. Basic and Giaretta (2024) E. Basic and A. Giaretta Large language models and code security: a systematic literature review. arXiv preprint arXiv:2412.15004. Cited by: Table 2, §4.1, §4.3, §5. BASIC and GIARETTA (2025) E. BASIC and A. GIARETTA From vulnerabilities to remediation: a systematic literature review of llms in code security. Cited by: Table 1, §1, §2. Belozerov et al. (2025) V. Belozerov, P. J. Barclay, and A. Sami Secure coding with ai, from creation to inspection. arXiv preprint arXiv:2504.20814. Cited by: Table 2, Table 2, §4.1, §4.2, §4.2, Table 3, §6.2, §6.2, §6.2. Bhatt et al. (2024) M. Bhatt, S. Chennabasappa, Y. Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y. Chen, D. Kapil, et al. Cyberseceval 2: a wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161. Cited by: §5.2.2. Bhatt et al. (2023) M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, et al. Purple llama cyberseceval: a secure coding benchmark for language models. arXiv preprint arXiv:2312.04724. Cited by: §5.2.2. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877â1901. Cited by: §6.2. Chen et al. (2023) A. Chen, J. Scheurer, T. Korbak, J. A. Campos, J. S. Chan, S. R. Bowman, K. Cho, and E. Perez Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749. Cited by: §6.1, §6.2. Chen et al. (2025a) J. Chen, H. Huang, Y. Lyu, J. An, J. Shi, C. Yang, T. Zhang, H. Tian, Y. Li, Z. Li, et al. SecureAgentBench: benchmarking secure code generation under realistic vulnerability scenarios. arXiv preprint arXiv:2509.22097. Cited by: §5. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §5.1.1, §5.2.1, §5.2.1, §5.2.1, Table 4, §7.1.2, §7.1.2, §7.5.1. Chen et al. (2026) Q. Chen, J. Shuai, S. Chen, S. Ye, Z. Wen, X. Su, J. Jin, J. Li, J. Chen, X. Tan, et al. HardSecBench: benchmarking the security awareness of llms for hardware code generation. arXiv preprint arXiv:2601.13864. Cited by: §5. Chen et al. (2025b) W. Chen, M. Su, and M. L. Cummings Assessing llm code generation quality through path planning tasks. arXiv preprint arXiv:2504.21276. Cited by: Table 2, §4.2, §5.2.1, §5.2.1. Chidambaram et al. (2024) S. Chidambaram, L. E. Li, M. Bai, X. Li, K. Lin, X. Zhou, and A. C. Williams Socratic human feedback (sohf): expert steering strategies for llm code generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 15491â15502. Cited by: §6.1, §6.1, §6.2, 2nd item. Chong et al. (2024) C. J. Chong, Z. Yao, and I. Neamtiu Artificial-intelligence generated code considered harmful: a road map for secure and high-quality code generation. arXiv preprint arXiv:2409.19182. Cited by: §6.1, §6.2, §6.2, §7.3, §7.3, §7.3. Chow and Li (2025) J. C. Chow and K. Li Large language models in medical chatbots: opportunities, challenges, and the need to address ai risks. Information 16 (7), p. 549. Cited by: Table 1, §1, §1, §2, §7.3, §7.3, §7.3, §7.3. Chow and Li (2026) J. C. Chow and K. Li From dialogue systems to autonomous agents: a modeling framework for ethical generative ai in healthcare. Information 17 (4), p. 361. Cited by: Table 1, §1, §1, §2, §4.4, §4.4, §4.4, §4.4, §7.3, §7.3, §7.3, §7.3, §7.3. CodeQL (2025) CodeQLCodeQL(Website) External Links: Link Cited by: §5.1.2, §7.1.2. Cotroneo et al. (2024) D. Cotroneo, C. Improta, P. Liguori, and R. Natella Vulnerabilities in ai code generators: exploring targeted data poisoning attacks. In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, p. 280â292. Cited by: Table 2, §4.3, §4.3. Cramer and McIntyre (2025) M. Cramer and L. McIntyre Verifying llm-generated code in the context of software verification with ada/spark. arXiv preprint arXiv:2502.07728. Cited by: §7.2.1. Dai et al. (2025) D. Dai, M. Liu, A. Li, J. Cao, Y. Wang, C. Wang, X. Peng, and Z. Zheng FeedbackEval: a benchmark for evaluating large language models in feedback-driven code repair tasks. arXiv preprint arXiv:2504.06939. Cited by: §6.1, §6.2. Das et al. (2025) B. C. Das, M. H. Amini, and Y. Wu Security and privacy challenges of large language models: a survey. ACM Computing Surveys 57 (6), p. 1â39. Cited by: §4.1. Dilgren et al. (2025) C. Dilgren, P. Chiniya, L. Griffith, Y. Ding, and Y. Chen SecRepoBench: benchmarking llms for secure code generation in real-world repositories. arXiv preprint arXiv:2504.21205. Cited by: §5.2.1, §5.2.2, §5.3, §5.4, Table 4, §7.1.2, §7.2.2. Ding et al. (2024) Y. Ding, Z. Wang, W. U. Ahmad, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang CoCoMIC: code completion by jointly modeling in-file and cross-file context. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), p. 3433â3445. Cited by: §1. Dolcetti et al. (2024) G. Dolcetti, V. Arceri, E. Iotti, S. Maffeis, A. Cortesi, and E. Zaffanella Helping llms improve code generation using feedback from testing and static analysis. arXiv preprint arXiv:2412.14841. Cited by: §5.2.2. Dou et al. (2024a) S. Dou, H. Jia, S. Wu, H. Zheng, W. Zhou, M. Wu, M. Chai, J. Fan, C. Huang, Y. Tao, et al. Whatâs wrong with your code generated by large language models? an extensive study. arXiv preprint arXiv:2407.06153. Cited by: §6.1, §6.1, §6.1, §6. Dou et al. (2024b) S. Dou, Y. Liu, H. Jia, E. Zhou, L. Xiong, J. Shan, C. Huang, X. Wang, X. Fan, Z. Xi, et al. StepCoder: improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4571â4585. Cited by: §6.1. Dozono et al. (2024) K. Dozono, T. E. Gasiba, and A. Stocco Large language models for secure code assessment: a multi-language empirical study. arXiv preprint arXiv:2408.06428. Cited by: 3rd item. Du et al. (2023) X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou Classeval: a manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861. Cited by: §5.1.1. Duan et al. (2021) R. Duan, Y. Chen, D. Niu, Y. Yang, A. K. Qin, and Y. He Advdrop: adversarial attack to dnns by dropping information. In Proceedings of the IEEE/CVF international conference on computer vision, p. 7506â7515. Cited by: Table 2, §4.3. Elgedawy et al. (2024) R. Elgedawy, J. Sadik, S. Dutta, A. Gautam, K. Georgiou, F. Gholamrezae, F. Ji, K. Lim, Q. Liu, and S. Ruoti Ocassionally secure: a comparative analysis of code generation assistants. arXiv preprint arXiv:2402.00689. Cited by: Table 2, §4.2, §6.2. Fakhoury et al. (2024) S. Fakhoury, A. Naik, G. Sakkas, S. Chakraborty, and S. K. Lahiri LLM-based test-driven interactive code generation: user study and empirical evaluation. IEEE Transactions on Software Engineering 50 (9), p. 2254â2268. External Links: Document Cited by: §5.2.1, §5.3, §5.4, Table 4. Fan et al. (2023) A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang Large language models for software engineering: survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), p. 31â53. Cited by: §5. Firouzi and Ghafari (2026) E. Firouzi and M. Ghafari Persistent human feedback, llms, and static analyzers for secure code generation and vulnerability detection. arXiv preprint arXiv:2602.05868. Cited by: §6.2, §6. Fu et al. (2024a) M. Fu, V. Nguyen, C. Tantithamthavorn, D. Phung, and T. Le Vision transformer inspired automated vulnerability repair. ACM Transactions on Software Engineering and Methodology 33 (3), p. 1â29. Cited by: §1. Fu et al. (2024b) Y. Fu, E. Baker, Y. Ding, and Y. Chen Constrained decoding for secure code generation. arXiv preprint arXiv:2405.00218. Cited by: §5.2.1, §5.2.1, §5.2.2, §5.2.2, §5.2.2, §5.2.2. FU et al. (2024) Y. FU, P. LIANG, Z. LI, M. SHAHIN, J. YU, and J. CHEN Security weaknesses of copilot generated code in github. ACM Trans. Softw. Eng. Methodol 1 (1). Cited by: Table 2, §4.3, §4.3, §4.3, Table 3, Table 3, §5.2.2, §6.2. Goetz and Schaad (2024) S. Goetz and A. Schaad " You still have to study"âon the security of llm generated code. arXiv preprint arXiv:2408.07106. Cited by: Table 2, Table 2, §4.1, §4.2, §4.2. Gong et al. (2024) J. Gong, N. Duan, Z. Tao, Z. Gong, Y. Yuan, and M. Huang How well do large language models serve as end-to-end secure code producers?. arXiv preprint arXiv:2408.10495. Cited by: §1, Table 2, Table 2, Table 2, Table 2, §4.1, §4.1, §4.1, §4.2, §4.3, §4.3, §4.3, Table 3, Table 3. Hajipour et al. (2024a) H. Hajipour, K. Hassler, T. Holz, L. SchĂśnherr, and M. Fritz Codelmsec benchmark: systematically evaluating and finding security vulnerabilities in black-box code language models. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), p. 684â709. Cited by: Table 2, §4.1, §5.2.2, §5.3, §5.4, §7.1.2. Hajipour et al. (2024b) H. Hajipour, L. SchĂśnherr, T. Holz, and M. Fritz HexaCoder: secure code generation via oracle-guided synthetic training data. arXiv preprint arXiv:2409.06446. Cited by: Table 2, §4.3, Table 3, Table 3. Hamer et al. (2024) S. Hamer, M. dâAmorim, and L. Williams Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers. In 2024 IEEE Security and Privacy Workshops (SPW), p. 87â94. Cited by: Table 2, Table 2, §4.2, §4.3, Table 3, §6.2, §6.2, §6.2, §6. He and Vechev (2023) J. He and M. Vechev Large language models for code: security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, p. 1865â1879. Cited by: §1, Table 2, Table 2, §4.3, §4.3, Table 3, Table 3, §5.2.1. He et al. (2024) J. He, M. Vero, G. Krasnopolska, and M. Vechev Instruction tuning for secure code generation. In Proceedings of the 41st International Conference on Machine Learning, p. 18043â18062. Cited by: Table 2, Table 2, §4.1, §4.2. Hendrycks et al. (2021) D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938. Cited by: §5.1.1, Table 4. Huang et al. (2023) K. Huang, Z. Xu, S. Yang, H. Sun, X. Li, Z. Yan, and Y. Zhang A survey on automated program repair techniques. arXiv preprint arXiv:2303.18184. Cited by: Table 1, §1, §2. Jiang et al. (2024) X. Jiang, Y. Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology 33 (7), p. 1â30. Cited by: §1. Kathiresan (2025) G. Kathiresan Human-in-the-loop testing for llm-integrated software: a quality engineering framework for trust and safety. Authorea Preprints. Cited by: §6.2, §6.2, §6.2, §6.2, §6, §7.3. Kharma et al. (2025) M. Kharma, S. Choi, M. AlKhanafseh, and D. Mohaisen Security and quality in llm-generated code: a multi-language, multi-model analysis. arXiv preprint arXiv:2502.01853. Cited by: Table 2, §4.1, §5.2.1, §5.2.2. Kharma et al. (2026) M. F. Kharma, S. Choi, M. Alkhanafseh, and D. Mohaisen Security and quality in llm-generated code: a multi-language, multi-model analysis. IEEE Transactions on Dependable and Secure Computing. Cited by: §1, Table 3. Khoury et al. (2023) R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara How secure is code generated by chatgpt?. In 2023 IEEE international conference on systems, man, and cybernetics (SMC), p. 2445â2451. Cited by: Table 2, Table 2, Table 2, §4.1, §4.1, §4.2, Table 3, §6.2, §6.2, §6. Kirova et al. (2024) V. D. Kirova, C. S. Ku, J. R. Laracy, and T. J. Marlowe Software engineering education must adapt and evolve for an llm environment. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, p. 666â672. Cited by: Table 2, §4.2. Lai et al. (2023) Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. Wang, and T. Yu DS-1000: a natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, p. 18319â18345. Cited by: §5.1.1. Li et al. (2024) J. Li, F. Rabbi, C. Cheng, A. Sangalay, Y. Tian, and J. Yang An exploratory study on fine-tuning large language models for secure code generation. arXiv preprint arXiv:2408.09078. Cited by: Table 2, Table 2, §4.2, §4.3, Table 3, Table 3. Li et al. (2026) X. Li, S. Dai, K. Zheng, Y. Xiao, G. Deng, W. Dong, and X. Wang " Are you sure?": an empirical study of human perception vulnerability in llm-driven agentic systems. arXiv preprint arXiv:2602.21127. Cited by: §6.2, §6, §7.3, §7.3, §7.3, §7.3. Li et al. (2025a) X. Li, J. Ding, C. Peng, B. Zhao, X. Gao, H. Gao, and X. Gu Safegenbench: a benchmark framework for security vulnerability detection in llm-generated code. arXiv preprint arXiv:2506.05692. Cited by: §5. Li et al. (2025b) Y. Li, X. Li, H. Wu, M. Xu, Y. Zhang, X. Cheng, F. Xu, and S. Zhong Everything you wanted to know about llm-based vulnerability detection but were afraid to ask. arXiv preprint arXiv:2504.13474. Cited by: §5.2.2, §5.3, §5.4, Table 4. Li et al. (2023) Z. Li, C. Wang, Z. Liu, H. Wang, D. Chen, S. Wang, and C. Gao Cctest: testing and repairing code completion systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), p. 1238â1250. Cited by: §5. Licorish et al. (2025) S. A. Licorish, A. Bajpai, C. Arora, F. Wang, and K. Tantithamthavorn Comparing human and llm generated code: the jury is still out!. arXiv preprint arXiv:2501.16857. Cited by: §6.1, §6.2, §6.2. Liu et al. (2024a) F. Liu, Y. Liu, L. Shi, H. Huang, R. Wang, Z. Yang, L. Zhang, Z. Li, and Y. Ma Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971. Cited by: Table 2, §4.1, Table 3, §5.2.1. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36, p. 21558â21572. Cited by: §5.1.1, §5.2.1. Liu et al. (2024b) Z. Liu, Y. Tang, X. Luo, Y. Zhou, and L. F. Zhang No need to lift a finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineering. Cited by: §1, Table 2, Table 2, §4.1, §4.3, §4.3, §4.3, §4.3, Table 3, Table 3, Table 3. Liu et al. (2024c) Z. Liu, R. Zeng, D. Wang, G. Peng, J. Wang, Q. Liu, P. Liu, and W. Wang Agents4PLC: automating closed-loop plc code generation and verification in industrial control systems using llm-based agents. arXiv preprint arXiv:2410.14209. Cited by: §1, §4.4, §4.4, §4.4, §4.4, §6.1, §6.1, §6.2, §6. Lyu et al. (2024) M. R. Lyu, B. Ray, A. Roychoudhury, S. H. Tan, and P. Thongtanunam Automatic programming: large language models and beyond. ACM Transactions on Software Engineering and Methodology. Cited by: §5. Ma et al. (2024) W. Ma, Y. Song, M. Xue, S. Wen, and Y. Xiang The âcodeâ of ethics: a holistic audit of ai code generators. IEEE Transactions on Dependable and Secure Computing 21 (5), p. 4997â5013. Cited by: §7.3. Mathews and Nagappan (2024) N. S. Mathews and M. Nagappan Test-driven development and llm-based code generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE â24, New York, NY, USA, p. 1583â1594. External Links: ISBN 9798400712487, Link, Document Cited by: §5.2.1. Miah and Zhu (2024) T. Miah and H. Zhu User centric evaluation of code generation tools (invited paper). In 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), Vol. , p. 109â119. External Links: Document Cited by: §5.2.1, §5.3, §5.4, Table 4. MITRE (2021) MITRECwe top 25 most dangerous software weaknesses(Website) External Links: Link Cited by: §5.1.2. MITRE (2024a) MITRECWE-119: improper restriction of operations within the bounds of a memory buffer(Website) External Links: Link Cited by: §4.3. MITRE (2024b) MITRECWE-120: buffer copy without checking size of input (âclassic buffer overflowâ)(Website) External Links: Link Cited by: §4.3. MITRE (2024c) MITRECWE-125: out-of-bounds read(Website) External Links: Link Cited by: §4.3. MITRE (2024d) MITRECWE-153: improper neutralization of substitution characters(Website) External Links: Link Cited by: §4.3. MITRE (2024e) MITRECWE-200: exposure of sensitive information to an unauthorized actor(Website) External Links: Link Cited by: §4.3. MITRE (2024f) MITRECWE-20: improper input validation(Website) External Links: Link Cited by: §4.3. MITRE (2024g) MITRECWE-22: improper limitation of a pathname to a restricted directory (âpath traversalâ)(Website) External Links: Link Cited by: §4.3, §4.3. MITRE (2024h) MITRECWE-327: use of a broken or risky cryptographic algorithm(Website) External Links: Link Cited by: §4.3. MITRE (2024i) MITRECWE-346: origin validation error(Website) External Links: Link Cited by: §4.3. MITRE (2024j) MITRECWE-352: cross-site request forgery (csrf)(Website) External Links: Link Cited by: §4.3. MITRE (2024k) MITRECWE-401: missing release of memory after effective lifetime(Website) External Links: Link Cited by: §4.3. MITRE (2024l) MITRECWE-416: use after free(Website) External Links: Link Cited by: §4.3. MITRE (2024m) MITRECWE-434: unrestricted upload of file with dangerous type(Website) External Links: Link Cited by: §4.3. MITRE (2024n) MITRECWE-502: deserialization of untrusted data(Website) External Links: Link Cited by: §4.3. MITRE (2024o) MITRECWE-502: deserialization of untrusted data(Website) External Links: Link Cited by: §4.3. MITRE (2024p) MITRECWE-732: incorrect permission assignment for critical resource(Website) External Links: Link Cited by: §4.3. MITRE (2024q) MITRECWE-759: use of a one-way hash without a salt(Website) External Links: Link Cited by: §4.3. MITRE (2024r) MITRECWE-77: improper neutralization of special elements used in a command (âcommand injectionâ)(Website) External Links: Link Cited by: §4.1. MITRE (2024s) MITRECWE-787: out-of-bounds write(Website) External Links: Link Cited by: §4.3. MITRE (2024t) MITRECWE-78: improper neutralization of special elements used in an os command (âos command injectionâ)(Website) External Links: Link Cited by: §4.1, §4.3. MITRE (2024u) MITRECWE-79: improper neutralization of input during web page generation (âcross-site scriptingâ)(Website) External Links: Link Cited by: §4.3. MITRE (2024v) MITRECWE-835: loop with unreachable exit condition (âinfinite loopâ)(Website) External Links: Link Cited by: §4.3. MITRE (2024w) MITRECWE-89: improper neutralization of special elements used in an sql command (âsql injectionâ)(Website) External Links: Link Cited by: §4.3. MITRE (2024x) MITRECWE-94: improper control of generation of code (âcode injectionâ)(Website) External Links: Link Cited by: §4.3, §4.3. Mohsin et al. (2024) A. Mohsin, H. Janicke, A. Wood, I. H. Sarker, L. Maglaras, and N. Janjua Can we trust large language models generated code? a framework for in-context learning, security patterns, and code evaluations across diverse llms. arXiv preprint arXiv:2406.12513. Cited by: §1, Table 2, Table 2, Table 2, Table 2, Table 2, §4.1, §4.1, §4.1, §4.1, §4.1, §4.1, §4.2, §4.2, §4.3, Table 3, Table 3, Table 3, §6.2, §6.2, §6. Mu et al. (2024) F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang Clarifygpt: a framework for enhancing llm-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering 1 (FSE), p. 2332â2354. Cited by: §1. National Institute of Standards and Technology (2023) National Institute of Standards and Technology AI risk management framework (AI RMF 1.0). Note: https://w.nist.gov/itl/ai-risk-management-frameworkAccessed: 2026-07-06 Cited by: §7.3. National Institute of Standards and Technology (2024) National Institute of Standards and Technology Artificial intelligence risk management framework: generative artificial intelligence profile. Technical report Technical Report NIST AI 600-1, National Institute of Standards and Technology. Note: Accessed: 2026-07-06 External Links: Link Cited by: §7.3. Nazzal et al. (2024) M. Nazzal, I. Khalil, A. Khreishah, and N. Phan PromSec: prompt optimization for secure generation of functional source code with large language models (llms). In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 2266â2280. Cited by: Table 2, Table 2, Table 2, §4.1, §4.2, §4.3, §4.3, Table 3, Table 3, §6. Nong et al. (2024) Y. Nong, H. Yang, L. Cheng, H. Hu, and H. Cai Automated software vulnerability patching using large language models. arXiv preprint arXiv:2408.13597. Cited by: §4.1, Table 3. Nunez et al. (2024) A. Nunez, N. T. Islam, S. K. Jha, and P. Najafirad Autosafecoder: a multi-agent framework for securing llm code generation through static analysis and fuzz testing. arXiv preprint arXiv:2409.10737. Cited by: §1, §4.4, §4.4, §4.4, §5.2.2, §5.3, §5.4, Table 4. Oh et al. (2024) S. Oh, K. Lee, S. Park, D. Kim, and H. Kim Poisoned chatgpt finds work for idle hands: exploring developersâ coding practices with insecure suggestions from poisoned ai models. In 2024 IEEE Symposium on Security and Privacy (SP), p. 1141â1159. Cited by: Table 2, §4.3. [107] T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama Is self-repair a silver bullet for code generation?. In The Twelfth International Conference on Learning Representations, Cited by: §6.1, §6.1, §6. Olausson et al. (2023) T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama Is self-repair a silver bullet for code generation?. arXiv preprint arXiv:2306.09896. Cited by: §6.1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730â27744. Cited by: §6.2, §6.2, §6.2, §6.2. Pearce et al. (2022) H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri Asleep at the keyboard? assessing the security of github copilotâs code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), p. 754â768. Cited by: §1, Table 2, Table 2, §4.1, §4.1, §4.1, Table 3, §5.2.2, §6.2. Pearce et al. (2023) H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt Examining zero-shot vulnerability repair with large language models. In 2023 IEEE Symposium on Security and Privacy (SP), p. 2339â2356. Cited by: §1, Table 3. Peng et al. (2025) J. Peng, L. Cui, K. Huang, J. Yang, and B. Ray CWEval: outcome-driven evaluation on functionality and security of llm code generation. arXiv preprint arXiv:2501.08200. Cited by: §5.2.2, §5.2.2, §5.3, §5.4, Table 4. Perry et al. (2023) N. Perry, M. Srivastava, D. Kumar, and D. Boneh Do users write more insecure code with ai assistants?. In Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, p. 2785â2799. Cited by: §1, Table 2, §4.2, §5.4, Table 4, §6.2, §6.2. Qiu et al. (2024) R. Qiu, W. W. Zeng, J. Ezick, C. Lott, and H. Tong How efficient is llm-generated code? a rigorous & high-standard benchmark. arXiv preprint arXiv:2406.06647. Cited by: §6.1. Rabbi et al. (2024) M. F. Rabbi, A. I. Champa, M. F. Zibran, and M. R. Islam AI writes, we analyze: the chatgpt python code saga. In Proceedings of the 21st International Conference on Mining Software Repositories, p. 177â181. Cited by: Table 2, §4.3, Table 3. Ren et al. (2020) S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297. Cited by: §5.2.1. Sandoval et al. (2023) G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt Lost at c: a user study on the security implications of large language model code assistants. In 32nd USENIX Security Symposium (USENIX Security 23), p. 2205â2222. Cited by: Table 2, §4.1, Table 3. Schuster et al. (2021) R. Schuster, C. Song, E. Tromer, and V. Shmatikov You autocomplete me: poisoning vulnerabilities in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21), p. 1559â1575. Cited by: Table 2, §4.3, §4.3. Shayegani et al. (2023) E. Shayegani, M. A. A. Mamun, Y. Fu, P. Zaree, Y. Dong, and N. Abu-Ghazaleh Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844. Cited by: §4.3. Sheng et al. (2025a) Z. Sheng, Z. Chen, S. Gu, H. Huang, G. Gu, and J. Huang Large language models in software security: a survey of vulnerability detection techniques and insights. arXiv preprint arXiv:2502.07049. Cited by: §5. Sheng et al. (2025b) Z. Sheng, Z. Chen, S. Gu, H. Huang, G. Gu, and J. Huang LLMs in software security: a survey of vulnerability detection techniques and insights. arXiv e-prints, p. arXivâ2502. Cited by: Table 1, §1, §2. Siddiq et al. (2024) M. L. Siddiq, J. C. da Silva Santos, S. Devareddy, and A. Muller Sallm: security assessment of generated code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops, p. 54â65. Cited by: Table 2, §4.1, Table 3, §5.2.1, §5.2.2, §5.2.2, §5.3, §5.4, Table 4. Siddiq and Santos (2022) M. L. Siddiq and J. C. Santos SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security, p. 29â33. Cited by: §5.1.2, §5.1.2, Table 4. Taldenko (2026) I. Taldenko Human-in-the-loop governance for llm-generated code: an engineering control, accountability, and verification model in ai-augmented software development. EJSMT 2 (3), p. 140â149. Cited by: §7.3. Tessa et al. (2026) M. Tessa, I. E. Olatunji, A. War, J. Klein, and T. F. BissyandĂŠ How secure is secure code generation? adversarial prompts put llm defenses to the test. arXiv preprint arXiv:2601.07084. Cited by: §4.3. Tian et al. (2023) H. Tian, W. Lu, T. O. Li, X. Tang, S. Cheung, J. Klein, and T. F. BissyandĂŠ Is ChatGPT the ultimate programming assistantâhow far is it?. arXiv preprint arXiv:2304.11938. Cited by: §5.2.1. Tian et al. (2025) Z. Tian, J. Chen, and X. Zhang Fixing large language modelsâ specification misunderstanding for better code generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), p. 645â645. Cited by: §6.1, §6.2. Tie et al. (2024) J. Tie, B. Yao, T. Li, S. I. Ahmed, D. Wang, and S. Zhou LLMs are imperfect, then what? an empirical study on llm failures in software engineering. arXiv preprint arXiv:2411.09916. Cited by: Table 2, §4.1, Table 3, §6.1, §6.2, §6. Tihanyi et al. (2024) N. Tihanyi, T. Bisztray, M. A. Ferrag, R. Jain, and L. C. Cordeiro Do neutral prompts produce insecure code? formai-v2 dataset: labelling vulnerabilities in code generated by large language models. arXiv preprint arXiv:2404.18353. Cited by: Table 2, §4.2, §4.3, Table 3, §5.1.2, §6.2. Tihanyi et al. (2025) N. Tihanyi, T. Bisztray, M. A. Ferrag, R. Jain, and L. C. Cordeiro How secure is ai-generated code: a large-scale comparison of large language models. Empirical Software Engineering 30 (2), p. 1â42. Cited by: Table 2, Table 2, Table 2, §4.1, §4.1, §4.3, Table 3, Table 3, Table 3. Tihanyi et al. (2023) N. Tihanyi, T. Bisztray, R. Jain, M. A. Ferrag, L. C. Cordeiro, and V. Mavroeidis The formai dataset: generative ai in software security through the lens of formal verification. In Proceedings of the 19th International Conference on Predictive Models and Data Analytics in Software Engineering, p. 33â43. Cited by: §5.1.2. Tong and Zhang (2024) W. Tong and T. Zhang CodeJudge: evaluating code generation with large language models. arXiv preprint arXiv:2410.02184. Cited by: §5.2.1, §5.2.1. Tony et al. (2023) C. Tony, M. Mutas, N. E. D. Ferreyra, and R. Scandariato Llmseceval: a dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), p. 588â592. Cited by: §5.1.2, §5.2.2, Table 4. TĂłth et al. (2024) R. TĂłth, T. Bisztray, and L. ErdĹdi LLMs in web development: evaluating llm-generated php code unveiling vulnerabilities and limitations. In International Conference on Computer Safety, Reliability, and Security, p. 425â437. Cited by: Table 2, §4.1, §4.3, Table 3, §6.2. Treude (2026) C. Treude Accountable agents in software engineering: an analysis of terms of service and a research roadmap. arXiv preprint arXiv:2605.04532. Cited by: §7.3, §7.3. [136] M. Vero, N. MĂźndler, V. Chibotaru, V. Raychev, M. Baader, N. JovanoviÄ, J. He, and M. Vechev BaxBench: can llms generate correct and secure backends?. In ICLR 2025 Third Workshop on Deep Learning for Code, Cited by: §5.3, §5.4, Table 4, §7.1.2. Wang et al. (2024a) J. Wang, X. Luo, L. Cao, H. He, H. Huang, J. Xie, A. Jatowt, and Y. Cai Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval. arXiv preprint arXiv:2407.02395. Cited by: §5.1.2, §5.2.1, §5.2.2, §5.3, §5.4, Table 4. Wang et al. (2024b) S. Wang, S. Zhang, J. Zhang, R. Hu, X. Li, T. Zhang, J. Li, F. Wu, G. Wang, and E. Hovy Reinforcement learning enhanced LLMs: a survey. arXiv preprint arXiv:2412.10400. Cited by: §6.2. Wei et al. (2023) Y. Wei, C. S. Xia, and L. Zhang Copiloting the copilots: fusing large language models with completion engines for automated program repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, p. 172â184. Cited by: §1. [140] (2025)What are the risks associated with generative ai code?(Website) External Links: Link Cited by: §7.3, §7.3, §7.5.2. Wong and Tan (2024) M. F. Wong and C. W. Tan Aligning crowd-sourced human feedback for code generation with bayesian inference. In 2024 IEEE Conference on Artificial Intelligence (CAI), p. 158â163. Cited by: §6.2, §6.2, §6.2, §6.2, §6.2. Xia and Zhang (2023) C. S. Xia and L. Zhang Conversational automated program repair. arXiv preprint arXiv:2301.13246. Cited by: §6.2, §6.2, §6, 1st item. Xie et al. (2020) C. Xie, M. Tan, B. Gong, J. Wang, A. L. Yuille, and Q. V. Le Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 819â828. Cited by: Table 2, §4.3. Yan et al. (2024) S. Yan, S. Wang, Y. Duan, H. Hong, K. Lee, D. Kim, and Y. Hong An \llm-assisted\\easy-to-trigger\ backdoor attack on code completion models: injecting disguised vulnerabilities against strong detection. In 33rd USENIX Security Symposium (USENIX Security 24), p. 1795â1812. Cited by: Table 2, §4.1, §4.3. Yang et al. (2024a) B. Yang, H. Tian, W. Pian, H. Yu, H. Wang, J. Klein, T. F. BissyandĂŠ, and S. Jin Cref: an llm-based conversational software repair framework for programming tutors. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 882â894. Cited by: §6.2, §6.2, 1st item. Yang et al. (2024b) Y. Yang, Y. Nie, Z. Wang, Y. Tang, W. Guo, B. Li, and D. Song SecCodePLT: a unified platform for evaluating the security of code genai. arXiv preprint arXiv:2410.11096. Cited by: §5.3, Table 4. Yao et al. (2024) Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang A survey on large language model (llm) security and privacy: the good, the bad, and the ugly. High-Confidence Computing, p. 100211. Cited by: §4.1. Yu et al. (2024) H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie Codereval: a benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, p. 1â12. Cited by: §5.1.1, §5.3, §5.4, Table 4. Zeng et al. (2025) B. Zeng, Q. Zhang, C. Zhou, G. Go, Y. Jiang, and H. Shi Inducing vulnerable code generation in llm coding assistants. arXiv preprint arXiv:2504.15867. Cited by: Table 2, §4.3, §4.3, §4.3, Table 3. Zhang et al. (2024) Q. Zhang, C. Fang, Y. Xie, Y. Ma, W. Sun, Y. Yang, and Z. Chen A systematic literature review on large language models for automated program repair. arXiv preprint arXiv:2405.01466. Cited by: Table 1, §1, §2, §3.2. Zheng et al. (2025) D. Zheng, Y. Wang, E. Shi, R. Zhang, Y. Ma, H. Zhang, and Z. Zheng Humanevo: an evolution-aware benchmark for more realistic evaluation of repository-level code generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), p. 764â764. Cited by: §5.3, §5.4, Table 4, §7.2.2. Zheng et al. (2023) Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, L. Shen, Z. Wang, A. Wang, Y. Li, et al. Codegeex: a pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 5673â5684. Cited by: §5.1.1. Zheng et al. (2024) T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue OpenCodeInterpreter: integrating code generation with execution and refinement. In Findings of the Association for Computational Linguistics ACL 2024, p. 12834â12859. Cited by: §1, §4.4, §4.4, §4.4, §6.1, §6. Zhong and Wang (2024) L. Zhong and Z. Wang Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 21841â21849. Cited by: Table 2, §4.3, Table 3. Zhou and Xu (2020) W. Zhou and K. Xu Learning to compare for better training and evaluation of open domain natural language generation models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, p. 9717â9724. Cited by: §6.2. Zhou et al. (2024a) X. Zhou, S. Cao, X. Sun, and D. Lo Large language model for vulnerability detection and repair: literature review and the road ahead. ACM Transactions on Software Engineering and Methodology. Cited by: Table 1, §1, §2, §5. Zhou et al. (2024b) X. Zhou, K. Kim, B. Xu, D. Han, and D. Lo Out of sight, out of mind: better automatic vulnerability repair by broadening input ranges and sources. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, p. 1â13. Cited by: §1. Zhuo et al. (2024) T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, et al. Bigcodebench: benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877. Cited by: §5.1.1.