Paper deep dive
MACGen: Toward Functionally Correct and Secure Code Generation via Multi-Agent Collaboration
Miseon Yu, Jaehoon Choi, Younghan Lee, Yunheung Paek
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/27/2026, 5:20:47 AM
Summary
The paper introduces MACGen, a multi-agent framework designed to generate code that is both functionally correct and secure. It addresses the limitations of existing methods by decomposing the generation process into four specialized agents: a Planner, a Security Advisor, a Code Generator, and a Reviewer. Unlike shared-dialogue approaches, MACGen uses artifact-only interfaces to enforce role specialization and reduce context bloat. The Security Advisor utilizes a standards-driven Knowledge Base (CERT, OWASP) to identify CWEs and generate task-specific guidelines. Evaluations on CWEval and BaxBench benchmarks demonstrate that MACGen significantly improves the joint success rate (F&S@1) compared to direct prompting and other baselines.
Entities (12)
Relation Signals (10)
MACGen â containsagent â Reviewer
confidence 95% ¡ MACGEN decomposes secure code generation into planning, security analysis, code synthesis, and refinement, each handled by a dedicated agent.
MACGen â containsagent â Security Advisor
confidence 95% ¡ MACGEN decomposes secure code generation into planning, security analysis, code synthesis, and refinement, each handled by a dedicated agent.
MACGen â containsagent â Code Generator
confidence 95% ¡ MACGEN decomposes secure code generation into planning, security analysis, code synthesis, and refinement, each handled by a dedicated agent.
MACGen â containsagent â Planner
confidence 95% ¡ MACGEN decomposes secure code generation into planning, security analysis, code synthesis, and refinement, each handled by a dedicated agent.
MACGen â evaluatedon â BaxBench
confidence 95% ¡ We conduct a comprehensive evaluation of MACGEN using two complementary benchmarks, CWEval (Peng et al., 2025) and BaxBench (Vero et al., 2025).
MACGen â evaluatedon â CWEval
confidence 95% ¡ We conduct a comprehensive evaluation of MACGEN using two complementary benchmarks, CWEval (Peng et al., 2025) and BaxBench (Vero et al., 2025).
Security Advisor â identifies â CWE
confidence 90% ¡ A security advisor identifies likely CWEs and synthesizes task-specific guidelines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Despite their strong ability to generate code, large language models often fail to produce secure code, as their outputs frequently contain security vulnerabilities. Secure code generation is inherently challenging because it requires solving a multi-objective problem: functional correctness and security. Existing approaches address this challenge by injecting external security knowledge or by using agentic feedback and iterative refinement. However, guideline retrieval often leaves the generator to translate generic advice into task-specific secure implementations, while shared-dialogue multi-agent feedback can blur role boundaries and suffer from context bloat. We present MACGen, a multi-agent framework that integrates planning, security analysis, code synthesis and refinement to jointly optimize security and functionality. A planner constructs a step-by-step plan to satisfy functional requirements. A security advisor identifies likely CWEs and synthesizes task-specific guidelines, a coder then generates code grounded in these artifacts, and a reviewer issues perspective-separated feedback. Rather than sharing full dialogue histories, each agent receives only structured artifacts from upstream stages, enforcing role specialization and reducing uncontrolled context growth. On CWEval and BaxBench, MACGen improves F&S@1 over direct prompting by 19.61 and 10.57 percentage points (pp) on average, respectively.
Tags
Links
- Source: https://arxiv.org/abs/2608.25457v1
- Canonical: https://arxiv.org/abs/2608.25457v1
Trouble viewing inline? Open PDF directly â
Full Text
88,331 characters extracted from source content.
Expand or collapse full text
MACGEN: Toward Functionally Correct and Secure Code Generation via Multi-Agent Collaboration Miseon Yu Seoul National University altjs543@snu.ac.kr Jaehoon Choi Seoul National University qkzkf1113@snu.ac.kr Younghan Lee Sungshin Womenâs University yhlee@sungshin.ac.kr Yunheung Paek Seoul National University ypaek@snu.ac.kr Abstract Despite their strong ability to generate code, large language models often fail to produce se- cure code, as their outputs frequently contain security vulnerabilities. Secure code generation is inherently challenging because it requires solving a multi-objective problem: functional correctness and security. Existing approaches address this challenge by injecting external se- curity knowledge or by using agentic feedback and iterative refinement. However, guideline retrieval often leaves the generator to translate generic advice into task-specific secure imple- mentations, while shared-dialogue multi-agent feedback can blur role boundaries and suffer from context bloat. We present MACGEN, a multi-agent framework that integrates planning, security analysis, code synthesis and refinement to jointly optimize security and functionality. A planner constructs a step-by-step plan to satisfy functional requirements. A security advisor identifies likely CWEs and synthesizes task- specific guidelines, a coder then generates code grounded in these artifacts, and a reviewer is- sues perspective-separated feedback. Rather than sharing full dialogue histories, each agent receives only structured artifacts from upstream stages, enforcing role specialization and reduc- ing uncontrolled context growth. On CWEval and BaxBench, MACGEN improves F&S@1 over direct prompting by 19.61 and 10.57 per- centage points (p) on average, respectively. 1 Introduction Large language models (LLMs) demonstrate strong code-generation capabilities and are rapidly be- ing adopted in practice (Dohmke, 2023; GitHub, Inc., 2024). Empirical studies, however, reveal that LLM-generated code often contains a signifi- cant number of vulnerabilities (Pearce et al., 2025; Khoury et al., 2023; Bhatt et al., 2023; Mou et al., Usingrealpathis a positive step, but lacks comprehensive path sanitization. ... bool extract_tar_to_path(... , const char *dest_path) ... snprintf(full_path, sizeof(full_path), "%s/%s", dest_path, entry); char *sanitized_path= realpath(full_path); 1. Generated code contains a logic error As the security agent said, realpathhelps, ... but additional security measures are needed. Functionality Agent Role interference 01 02 03 04 Result: Misses the logical error â NULL Code Generator Agent Security Agent sanitized_pathis lost 2. Shared dialogue causes role interference Figure 1: Example of role interference. The generated code callsrealpathon a non-existent destination path, causingsanitized_pathto becomeNULLrather than preserving the intended extraction path. After observ- ing the security criticâs feedback in the shared-dialogue setting, the functionality critic shifts toward security- oriented feedback rather than independently identifying this logic error. 2025), including many listed in the MITRE CWE 1 Top-25 (MITRE, 2024). To mitigate these vulnerabilities, recent work has framed secure code generation as the task of producing code that satisfies the requested func- tionality while avoiding implementation choices that introduce security weaknesses. Unlike a post- generation filtering problem, secure code genera- tion requires security decisions to be made within the implementation itself, where they can affect whether the program still satisfies the intended be- havior. Such dual-objective setting increases task complexity, as secure implementations require ad- ditional decisions such as input validation, safer 1 The Common Weakness Enumeration (CWE) is a public catalog of software weaknesses comprising over 900 types.https://cwe.mitre.org/news/archives/ news2025.html arXiv:2608.25457v1 [cs.CR] 26 Aug 2026 API selection, or permission checks. Moreover, the security risks relevant to a task are often un- derspecified in the prompt, requiring the model to infer applicable threats and translate them into con- crete, functionally consistent code choices. Empiri- cally, security-oriented prompting, which directly instructs the model to produce secure code, can reduce vulnerabilities but often compromises func- tional correctness (Tony et al., 2025a; Black et al., 2025; Dai et al., 2026). The expanded reasoning burden imposed by dual objectives is difficult to resolve reliably through prompting alone. One line of work mitigates this challenge by injecting external security knowledge into the gen- eration context, such as secure code examples or security guidelines (Zhang et al., 2024; Lin et al., 2025; Shi and Zhang, 2026). More recently, these approaches have been refined by selecting concise and task-relevant guidelines to better preserve func- tionality (Shi and Zhang, 2026). However, such guidelines are inherently generic and lack aware- ness of task-specific context such as variable names, function signatures, or control flow. Similarly, secure code examples are language-specific and costly to collect, limiting their practical coverage. As a result, even with retrieved knowledge in con- text, the generator must still determine how generic guidance applies to the specific task and translate it into secure, functional code. Thus, retrieval nar- rows the security-knowledge gap but leaves task- specific adaptation unresolved at generation time. Another line of work addresses this reasoning burden through agentic critique. Le et al. (2024) propose an internal-dialogue framework in which safety and helpfulness critic agents analyze gener- ated code from their respective perspectives. By assigning each concern to a distinct critic, this ap- proach makes functional and security-related eval- uation criteria explicit. However, shared-dialogue designs may blur these role boundaries when critics observe each otherâs intermediate reasoning, poten- tially leading to role interference in which critics produce overlapping feedback or gradually drift from their assigned perspective (Figure 1). Because role separation is enforced only through prompt in- structions rather than by the interface itself, such designs rely on accumulated dialogue to coordinate critiques. This increases token cost and latency while exposing agents to long-context degradation as the dialogue history grows (Liu et al., 2024a,b). These considerations suggest that effective se- cure code generation requires not just more knowl- edge or more feedback, but a clearer separation of what each stage needs to reason about. Each stage should receive only the information it needs and pass a compact artifact to the next, so that role boundaries are enforced by the interface rather than by prompts alone. We instantiate this prin- ciple as MACGEN, a multi-agent collaboration system for secure code generation. Inspired by the Secure Software Development Lifecycle (S- DLC) (Howard and Lipner, 2006), which empha- sizes incorporating security throughout develop- ment, MACGEN decomposes secure code genera- tion into planning, security analysis, synthesis, and refinement, each handled by a dedicated agent. Specifically, a planner first establishes a func- tional plan, analogous to clarifying requirements. A code generator then produces draft code from this plan and a security advisor then analyzes the task, plan, and draft code to identify threats and generate task-specific guidelines that cover relevant CWE risks, serving a role similar to incorporating secu- rity during design and bridging the gap between generic retrieved advice and the concrete coding task. Subsequently, a code generator implements secure code guided by both the plan and security constraints. Finally, a reviewer supports the veri- fication phase by issuing actionable, perspective- separated feedback routed back to the coder for targeted refinement. By resolving what to imple- ment and how to secure it before code synthesis, MACGEN helps the coder produce code that satis- fies both functional and security requirements from the outset. Unlike prior multi-agent approaches that rely on shared dialogue, our framework enforces role specialization efficiently through artifact-only interfaces. We conduct a comprehensive evaluation of MACGEN using two complementary benchmarks, CWEval (Peng et al., 2025) and BaxBench (Vero et al., 2025). On CWEval, across six LLMs and five programming languages, MACGEN improves F&S@1 over direct prompting by 19.61 percentage points (p) on average. On BaxBench, evaluated with GPT-4o and GPT-4o-mini across six backend languages, MACGEN improves F&S@1 by 9.95p and 11.18p, respectively. ⢠We introduce MACGEN, a multi-agent frame- work for functionally correct and secure code generation that decomposes generation into planning, security analysis, code synthesis, and refinement. â˘We show that artifact-only interfaces provide an effective coordination structure for special- ized agents, by reducing unnecessary context sharing between roles. â˘We evaluate MACGENon CWEval and BaxBench across multiple LLMs and languages, reporting improved joint functionality-security performance. 2 . 2 Related Works 2.1 Multi-Agents for Code Generation Multi-agent strategies have shown potential for im- proving code quality through structured task de- composition (Li et al., 2023; Hong et al., 2024; Qian et al., 2023; Islam et al., 2024, 2025). Espe- cially, Islam et al. (2025) introduces CODESIM, which significantly enhances code quality by lever- aging planning techniques inspired by human problem-solving processes. It generates a step-by- step implementation plan, which is then simulated using synthesized input/output samples to verify correctness. This plan is iteratively refined until the simulation succeeds. Despite these advances, such planning-based and human problem-solving approaches have yet to be fully utilized in the do- main of secure code generation. MACGEN extends this line of work by applying planning techniques to secure code generation. 2.2 Secure Code Generation Model adaptation improves security via security- centric prefix-/instruction-tuning, localized opti- mization, or internal feature control (He and Vechev, 2023; He et al., 2024; Li et al., 2024; Hasan et al., 2025; Huang et al., 2026; El Husseini et al., 2026). However, such approaches require access to model parameters or internals, limiting applica- bility when only inference-time interaction is avail- able. Prompting/Reflection can steer LLMs toward safer code at inference time (Nazzal et al., 2024; Tony et al., 2025a; Bruni et al., 2025); for example, Tony et al. (2025a) show that multi-turn Recursive Criticism and Improvement (RCI) significantly re- duces security weaknesses. Knowledge Injection augments the generation context with external se- curity knowledge, such as secure code examples or guidelines (Wang et al., 2025; Zhang et al., 2024; 2 All source code, evaluation logs, and prompts are available athttps://anonymous.4open.science/r/ macgen-secure-code-48C0 Tony et al., 2025b; Lin et al., 2025; Shi and Zhang, 2026). Agentic Critique uses specialized agents to review generated code for security (Nunez et al., 2024) or for both functional and security perspec- tives (Le et al., 2024). Collectively, these inference- time approaches treat security as added knowledge or post-generation feedback, rather than integrating it throughout the development process as SSDLC principles prescribe. 3MACGEN Design 3.1 Problem Definition Our goal is to generate source code that is syn- tactically valid, functionally correct, and secure. We approach this as a conditional generation task where a multi-agent system produces a code se- quenceYfrom a promptX. This inputXcan consist of a natural language instruction, a code prefix for completion, or a combination of both. 3.2 Overall Workflow MACGEN is a role-specialized multi-agent frame- work composed of four agents: a Planner, a Se- curity Advisor, a Code Generator, and a Reviewer. An overview of the framework is displayed in Fig 2. The process begins with the Planner establishing a functional plan, which the Code Generator uses to produce the draft code ( 1 ââ 2 âin Fig 2). The Security Advisor then analyzes the draft, either trig- gering an early exit or performing standards-driven reasoning via a security Knowledge Base (KB) to synthesize guidelines ( 3 â). Finally, the Code Gen- erator implements the final code, followed by itera- tive verification from the Reviewer ( 4 â). 3.3 Building a Standards-driven Security Knowledge Base To ensure the normative correctness of security reasoning, we construct a standards-grounded KB from official code security standards such as CERT and OWASP (CERT; OWASP). This pre- constructed KB serves as the foundation for the retrieval mechanism, enabling the Security Advi- sor to perform standards-grounded reasoning. Raw text parsed from heterogeneous sources (e.g., PDF, Markdown) often contains structural noise and formatting inconsistencies that hinder effective semantic retrieval. To address this, we leverage an LLM-based refinement process that normalizes each guideline into a compact, high- density schema consisting of four fields (WHAT, Generate Plan Code Generator Planner Func. Plan Task Prompt Generate Draft Code Draft Code Draft Code Code-based Guide Gen. Standard- Driven Guide Gen. Final Security Guide Generate Code Code Task Prompt Func. Plan Sec. Guide Revise Syntax Secure & Functional Code Task Prompt Func. Plan Yes Func. Plan Task Prompt Code Security Check Sec. Guide Task Prompt Code Func. Check Both Pass? Report Task Prompt 3 21 4 3-2 3-3 3-4 No(Early Exit) Security Risk Found? 3-1 Yes Security AdvisorReviewer Code Generator No (Need Refinement) Figure 2: Overview of MACGEN: The framework consists of four specialized agentsâPlanner, Security Advisor, Code Generator, and Reviewer. WHY, HOW, and EXAMPLE). The prompt used for this refinement is provided in Figure 8. To convert these structured guidelines into a searchable semantic space, each guideg i is en- coded into a vectorv i â R d using an embed- ding functionE(¡). During inference, the Security Advisor formulates a set of task-specific queries Q = q 1 , . . . , q m derived from the identified CWE groups. For each queryq j âQ, the retrieval process selects the top-Kmost relevant guidelines according to cosine similarity: G â (q j ) = arg max (K) g i âKB sim E(q j ), E(g i ) (1) wherearg max (K) denotes an operator that returns the set ofKelements with the highest similarity scores, andsim(¡,¡)represents cosine similarity. The final candidate setGis obtained by the union of retrieved guidelines across all queries inQ. 3.4 Role-specific Agents This subsection details the roles and internal logic of each agent. The complete prompts for all agents are provided in Appendix K. 3.4.1 Planner Inspired by prior work (Islam et al., 2025), the Plan- ner agent generates a concise, high-level functional plan for each task. This plan serves dual purposes. It guides the Code Generator toward a functionally correct implementation, and provides task-specific context that enables the Security Advisor to iden- tify relevant attack surfaces more precisely. 3.4.2 Security Advisor The Security Advisor follows a multi-stage pipeline designed for both inference efficiency and analyt- ical depth. The process begins with an early-exit Use and to generate concise, actionable security guidelines. Security Guide 1 Generate grouped CWE causeâeffectpairs. Output JSON with: ⢠group_name ⢠why (short rationale) ⢠red_flags ⢠causeâeffect(CWE IDs) ⢠likelihood (0..1) Prompt Func. Plan Task Prompt CWE Extraction (LLMInference) Security Advisor Agent Retrieve Relevant Security Guidelines (RAG) Standards-Driven Guide Generation (LLMInference) CWE Output RAG Output Prompt "group_name": "File Upload Handling", "why": "Unvalidated uploads can be exploited.", "red_flags": ["Bad file type", "Poor handling"], "cause_cwes": ["CWE-434", "CWE-22"], "effect_cwes": ["CWE-94", "CWE-552"], "likelihood": 0.8 CWE Output Query For RAG To mitigate this threat, use a random filename (e.g., UUID/GUID) RAG Output Figure 3: Detailed visualization of the standards-driven guideline generation process (Step 3-2 in Fig 2). triage, where the advisor inspects the task prompt and the initial draft code to identify potential attack surfaces under pre-defined categories. If no risks are detected (e.g., for simple algorithmic or purely functional tasks), the system executes an early exit to bypass unnecessary retrieval and minimize com- putational overhead. For tasks requiring deeper inspection, the ad- visor synthesizes task-specific guidelines through a three-step process. First, following the work- flow in Figure 3, it examines the task prompt and the functional plan to identify potential security risks, which are represented as corresponding CWE groups (e.g., insecure file path handling). These groups are then used to formulate targeted search queries for retrieving relevant security guidelines from the pre-constructed KB. Based on the re- trieved context, the advisor generates the first set of standards-driven security guidelines. Subsequently, the advisor performs a code-based guide generation by inspecting the draft code. This step is crucial for detecting insecure coding pat- terns typical of LLM outputs that may not be ap- parent from the high-level plan alone. Based on this code-level analysis, the second set of guide- lines is generated. Finally, the advisor concatenates both sets and validates them against the functional requirements to ensure no conflicts exist. This con- solidation guarantees that the security requirements are both comprehensive and compatible with the taskâs functionality. 3.4.3 Code Generator The Code Generator agent serves two distinct roles within the pipeline. First, it produces an initial draft code based on the task prompt and the functional plan, which is subsequently analyzed by the Secu- rity Advisor. Second, guided by the functional plan and the security guidelines, the agent generates the executable code, with both components acting as dual constraints on the output. After generation, the agent iteratively checks the generated code us- ing a syntax checker and performs self-refinement until the code is syntactically valid. 3.4.4 Reviewer The Reviewer agent conducts a structured code re- view through two perspective-separated checks. It verifies that the implementation accomplishes the intended functionality and that all security guide- lines have been correctly met. If it uncovers any functional errors (e.g., incorrect logic) or resid- ual security issues (e.g., missing input sanitization, use of risky functions), it provides targeted feed- back. This feedback is passed to the Code Gener- ator, which revises the implementation as needed. The iterative refinement cycle continues until the Reviewer approves the code as both functionally correct and secure, or the maximum number of iterations is reached. 4 Experimental Setup In this section, we describe the experimental setup, including benchmarks, models, baselines, imple- mentation details, and evaluation metrics. More details are provided in Appendix A. 4.1 Benchmarks We evaluate our approach using two security- oriented code generation benchmarks. First, we employ CWEval (Peng et al., 2025), an outcome- driven evaluation benchmark comprising 119 code completion tasks across five programming lan- guages (C, C++, Python, JavaScript, and Go). Sec- ond, we use BaxBench (Vero et al., 2025), a bench- mark for generating backend applications in realis- tic and diverse environments. It contains 392 tasks across 28 backend scenarios and 14 frameworks in six programming languages (Go, JavaScript, PHP, Python, Ruby, and Rust), spans both single- and multi-file application settings. 4.2 Baselines & LLM Models To evaluate the effectiveness of our framework, we compare it against five baselines: Direct Prompting where LLMs generate codeYdirectly from the task promptX, SECGUIDE (Tony et al., 2025b), CODEGUARDER (Lin et al., 2025) and RES- CUE (Shi and Zhang, 2026) which are RAG- based methods, and INDICT (Le et al., 2024), an internal-dialogue multi-agent framework with iterative refinement. We evaluate all using six LLMs: GPT-4o, GPT-4o-mini (Hurst et al., 2024), Gemini 2.5-Flash, Gemini 2.5-Flash-Lite (Co- manici et al., 2025), the open-source DeepSeek- R1-Distill-Llama-70B (Guo et al., 2025), distilled from Llama3.3-70B-Instruct (AI@Meta, 2024), and Qwen3-8B (Yang et al., 2025). 4.3 Implementation Details We use the same set of hyperparameters for MAC- GEN across all experiments unless otherwise speci- fied. For the security analysis, the Security Advisor infers a number of potential CWE groups per task, with up toC = 3for CWEval and up toC = 2 for BaxBench. We set the maximum number of refinement iterations between the Reviewer and Code Generator is set toM = 2. For all base- lines, we adopt the hyperparameters reported in their original papers (Tony et al., 2025b; Lin et al., 2025; Shi and Zhang, 2026; Le et al., 2024). All LLMs are decoded with a temperature of 0.0 for reproducibility. 4.4 Evaluation Metrics We evaluate the generated code in terms of func- tional correctness and security. For two bench- marks, both aspects are automatically evaluated using built-in test oracles. CWEval focuses on a specific target CWE for each code completion task, whereas BaxBench evaluates generated backend applications using end-to-end API-level exploits that may cover multiple CWE classes per scenario. Following the metric definition introduced by Peng ModelMethod Func@1 (%)Sec@1 (%)F&S@1 (%)#NC ValueââValueââValueââ GPT-4o Direct80.6753.9150.424 SECGUIDE33.61-47.0653.68-0.2329.42-21.0124 CODEGUARDER71.43-9.2465.49+11.5754.62+4.206 INDICT62.18-18.4975.44+21.5356.30+5.885 RESCUE78.15-2.5275.00+21.0967.23+16.817 MACGEN79.83-0.8479.31+25.4070.59+20.173 GPT-4o-mini Direct76.4750.0046.227 SECGUIDE31.93-44.5445.56-4.4423.53-22.6929 CODEGUARDER67.23-9.2466.02+16.0252.94+6.7216 INDICT56.30-20.1766.67+16.6748.74+2.5217 RESCUE70.59-5.8865.09+15.0951.26+5.0413 MACGEN74.79-1.6876.72+26.7266.39+20.173 Gemini 2.5 Flash Direct84.8754.8751.266 SECGUIDE50.42-34.4566.67+11.8044.54-6.7223 CODEGUARDER73.95-10.9270.54+15.6762.18+10.927 INDICT78.99-5.8874.11+19.2464.71+13.457 RESCUE74.79-10.0873.15+18.2863.87+12.6111 MACGEN77.31-7.5678.07+23.2070.59+19.335 Gemini 2.5 Flash-Lite Direct73.9549.0942.869 SECGUIDE31.09-42.8670.00+20.9129.41-13.4539 CODEGUARDER55.46-18.4962.37+13.2743.70+0.8426 INDICT65.55-8.4073.08+23.9958.82+15.9715 RESCUE74.79+0.8470.91+21.8261.34+18.499 MACGEN72.27-1.6875.45+26.3665.55+22.699 DeepSeek-R1 -Distill (70B) Direct65.5544.5534.4518 SECGUIDE15.13-50.4254.55+9.9912.61-21.8575 CODEGUARDER51.26-14.2967.78+23.2241.18+6.7229 INDICT55.46-10.0851.49+6.9335.29+0.8418 RESCUE57.14-8.4069.23+24.6848.74+14.2928 MACGEN63.03-2.5270.30+25.7452.10+17.6518 Qwen3-8B Direct42.8650.7726.0554 SECGUIDE3.36-39.5030.00-20.772.52-23.53109 CODEGUARDER33.61-9.2461.90+11.1429.41+3.3656 INDICT50.42+7.5650.54-0.2330.25+4.2026 RESCUE43.70+0.8463.29+12.5236.97+10.9240 MACGEN53.78+10.9266.29+15.5243.70+17.6530 Table 1: Evaluation of CWEval (119 tasks, 5 languages).âindicates the absolute percentage-point difference from Direct, and #NC counts non-compilable cases (lower is better). et al. (2025); Vero et al. (2025), we report Pass@1 as the primary evaluation metric in three vari- ants: Func@1, Sec@1, and F&S@1. Func@1 and F&S@1 are computed over all generations, mea- suring functional correctness and joint functional- security success, respectively, while Sec@1 is com- puted only over compilable generations. 5 Experimental Results This section evaluates MACGENâs ability to gen- erate secure and functional code, focusing on the following research questions: â˘RQ1. How effectively does MACGEN perform across various LLMs? (Section 5.1) â˘RQ2. How does MACGEN perform across dif- ferent programming languages? (Section 5.2) â˘RQ3. How cost-efficient is MACGEN in prac- tice? (Section 5.3) â˘RQ4. How do artifact-only interfaces impact the overall performance compared to a shared- context variant? (Section 5.4) 5.1 Performance on Secure and Functionally Correct Code Generation Table 1 shows that MACGEN achieves the best overall performance on CWEval across all six ModelMethodFunc@1 (%)F&S@1 (%) GPT-4o Direct45.6621.17 SECGUIDE10.978.16 CODEGUARDER38.0123.47 INDICT49.7430.36 RESCUE40.5625.00 MACGEN49.7431.12 GPT-4o-mini Direct27.6410.76 SECGUIDE8.907.40 CODEGUARDER28.3215.56 INDICT32.1418.37 RESCUE27.3016.58 MACGEN29.5921.94 GPT-5.1 Direct45.1535.71 SECGUIDExx.x.x CODEGUARDERxx.x.x INDICTxx.x.x RESCUE51.7942.35 MACGEN62.5051.28 Claude Sonnet 5 Direct58.1642.35 SECGUIDExx.x.x CODEGUARDERxx.x.x INDICTxx.x.x RESCUE65.0551.28 MACGEN79.3461.22 Table 2: Evaluation of BaxBench (392 tasks, 6 lan- guages). LLMs, reaching up to 70.59% F&S@1. While all baselines exhibit a functionalityâsecurity trade- off, MACGEN attains the smallest Func@1 drop (â0.56%p on average) while achieving the largest Sec@1 gain (+23.82%p). In contrast, the strongest multi-agent baseline INDICT reduces Func@1 by 12.6%p on average. Remarkably, on Qwen3-8B, MACGEN improves Func@1 by 11%p while si- multaneously achieving a 17.65%p gain in F&S@1, demonstrating that MACGEN delivers consistent gains across both large and small LLMs. Table 2 further evaluates the methods on BaxBench, which requires generating runnable backend applications. MACGEN achieves the highest F&S@1 on both GPT-4o and GPT-4o- mini. On GPT-4o, MACGEN improves F&S@1 from 21.17% to 31.12% over Direct while match- ing the best Func@1 score; notably, MACGEN achieves comparable F&S@1 to INDICT (31.12% vs. 30.36%) with substantially fewer tokens (see Section 5.3). On GPT-4o-mini, MACGEN im- proves F&S@1 by 3.57 percentage points over INDICT. These results suggest that MACGEN remains effective in the more complex backend- generation setting, where generated applications are evaluated using end-to-end API functionality tests and expert-written exploits. 5.2 Performance Across Different Programming Languages To evaluate cross-lingual robustness, we analyze F&S@1 scores on CWEval, averaged over GPT-4o and GPT-4o-mini. As shown in Figure 4, MAC- GEN achieves the best performance across all five languages. Especially, MACGEN shows large im- provements on C++ and Go, where retrieval-based baselines exhibit more limited performance. This trend highlights that MACGENâs efficacy relies on more than mere knowledge injection. By utilizing the Security Advisor to translate generic security concepts into concrete, task-specific constraints, it alleviates the Code Generatorâs burden of interpret- ing raw, language-variable guidelines. Additional analysis of language-specific behaviors is provided in Appendix H. C++GoPythonJavaScript 0 10 20 30 40 50 60 70 80 F&S@1 (%) 52 36 39 52 59 34 24 16 26 28 63 45 39 62 50 60 40 47 52 59 65 5050 62 65 69 64 71 72 65 DirectSecGuideCodeGuarderINDICTRESCUEMACGen Figure 4: Comparison of F&S@1 results across pro- gramming languages using GPT-4o and GPT-4o-mini. 5.3 Token Cost Table 3 reports the total token usage and esti- mated Application Programming Interface (API) cost 3 measured over 25 Python tasks from CWEval. While achieving superior performance, MACGEN maintains a cost profile comparable to prior meth- ods. Specifically, it incurs only a marginal $0.09 increase over SECGUIDE, while reducing costs by $3.77 compared to INDICT. Notably, MACGEN consumes only 16% of the tokens required by IN- DICT. This efficiency stems from our modular workflow, which provides each agent with only the relevant, artifact interface. Method Input Tokens Cached Tokens Output Tokens Total Tokens Total API Cost ($) Avg. End-to-End Latency (s) SecGuide88,9580.0059,442148,400$0.8227.41 CodeGuarder99,9170.0017,256117,173$0.428.80 INDICT1,211,92639,936159,892 1,411,754$4.68213.44 RESCUE28,0860.0015,55543,641$0.237.08 MACGEN155,28515,10450,527220,916$0.9128.34 Table 3: Token usage, API cost, and latency for 25 Python tasks from CWEval, using GPT-4o pricing. 5.4 Effect of Artifact-Only Coordination To assess whether artifact-only interfaces mitigate role interference and context accumulation, we compare MACGEN against MACGEN-Shared, a variant in which each agent receives the full accu- mulated context of all upstream agents including intermediate reasoning and scratchpads in addi- tion to its own role specification. As shown in Figure 5, MACGEN consistently outperforms the variant across all benchmarks and models. On CW- Eval, GPT-4o-mini improves by +20.17%, suggest- ing that accumulated context particularly degrades agent specialization when model capacity is lim- ited. On BaxBench, GPT-4o improves by +8.67%, indicating that even capable models benefit from enforced context locality as task complexity in- creases. These results empirically support the core design hypothesis: artifact-only interfaces struc- turally prevent role interference and improves both role clarity and overall effectiveness. Detailed re- sults are provided in Appendix I. 6 Ablation Study 6.1 Impact of Different Agents Table 4 shows that the full MACGEN configuration achieves the highest F&S@1 score, with each agent contributing distinct benefits. Starting from the configuration without specialized agents, adding the Security Advisor yields the largest single-agent 3 As of May 2026, GPT-4o is priced at $2.50 per 1M input tokens, $1.25 per 1M cached input tokens, and $10.00 per 1M output tokens. GPT-4oGPT-4o-mini 0 20 40 60 80 F&S@1 (%) 68.91 46.22 70.59 66.39 +1.68 +20.17 CWEval GPT-4oGPT-4o-mini 0 10 20 30 40 F&S@1 (%) 22.45 19.64 31.12 21.94 +8.67 +2.30 BaxBench MACGen-SharedMACGen Figure 5: F&S@1 comparison between MACGEN and MACGEN-Shared on CWEval and BaxBench. gain in F&S@1. This gain, however, comes with a functionality trade-off. The Reviewer provides strong standalone value as well, and partially off- sets the Security Advisorâs functionality drop when combined with it, while preserving high Sec@1. Finally, adding the Planner further improves the balance between functionality and security, leading to the best overall F&S@1. These results suggest that MACGENâs agents are complementary. PlannerSecurity Adv.ReviewerFunc@1Sec@1F&S@1â â80.6753.9150.42-20.17 ââ 84.8757.7654.62-15.97 ââ80.6771.1762.18-8.40 âââ84.8771.5565.55-5.04 âââ73.9577.1266.39-4.20 ââ74.7977.7866.39-4.20 ââ78.9977.5968.07-2.52 â79.8379.3170.59 Table 4: Ablation study of agent components on CWE- val with GPT-4o, whereâdenotes the absolute differ- ence in F&S@1 from MACGEN. 6.2 Effectiveness of Security Advisor Components Table 5 analyzes the contribution of individual Se- curity Advisor components by selectively removing each stage. The w/o standards-driven gen. variant removes the RAG-based retrieval step ( 3 âin Fig. 3), generating guidelines from extracted CWEs alone without retrieved security standards. The w/o code- based gen. variant eliminates draft-code analysis (Step 3-3 in Fig. 2), while w/o validation bypasses the final consolidation stage (Step 3-4 in Fig. 2). Across both models, removing any single compo- nent leads to degraded F&S@1 performance. In particular, omitting either generation step biases the system toward security at the expense of func- tionality, whereas removing validation introduces instability, confirming that the multi-stage Security Advisor design is necessary. ModelMethodFunc@1 (%) Sec@1 (%) F&S@1 (%) GPT-4o w/o standards-driven gen.77.3170.3462.18 w/o code-based gen.68.9173.0456.30 w/o validation73.9572.0362.18 MACGEN79.8379.3170.59 GPT-4o-mini w/o standards-driven gen.72.2769.0357.14 w/o code-based gen.71.4367.8357.14 w/o validation63.0372.6554.62 MACGEN74.7976.7266.39 Table 5: Ablation study of the Security Advisor compo- nents on the CWEval. ModelHumanEvalâHumanEval+âEarly Exit GPT-4o92.1+3.187.2+0.0164/164 GPT-4o-mini88.4-0.681.1-3.7162/164 Gemini-2.5-Flash97.0+3.187.8-3.1124/164 Gemini-2.5-Flash-Lite94.5+0.087.2+1.8114/164 DeepSeek-R1-Distill (70B)90.9+0.084.8-0.6148/164 Qwen3-8B90.2+17.684.1+14.6152/164 Table 6: Pass@1 (%) on HumanEval(+) and Early Exit ratio. â indicates change from Direct. ModelMethod Func@k (%)Sec@k (%)F&S@k (%) k=1k=3k=1k=3k=1k=3 GPT-4o Direct80.6786.5552.3859.6649.8652.94 SECGUIDE33.3349.1459.9177.5928.1643.10 CODEGUARDER70.0378.9966.1177.3157.1468.07 INDICT64.1575.6368.9184.8756.0270.59 RESCUE76.7584.8773.2584.0369.1875.63 MACGEN77.8789.0776.7584.8766.9581.51 Table 7: Evaluation on CWEval with GPT-4o. We re- port Func@k, Sec@k, and F&S@k fork = 1, 3with temperature 0.5. 6.3 Performance on General Functional Tasks To ensure MACGEN does not degrade general coding capabilities, we evaluate it on HumanEval and HumanEval+ (Chen, 2021; Liu et al., 2023). As shown in Table 6, MACGEN maintains per- formance comparable to Direct across all LLMs, with some models showing improvements. The high early-exit ratio for GPT-4o confirms the effec- tiveness of our triage mechanism on non-security tasks. 6.4Impact of Different Number of Samplings To evaluate performance stability under stochastic sampling, we report results fork = 3(temperature 0.5) on CWEval. As shown in Table 7, MACGEN outperforms all baselines across all metrics atk=3. This suggests that MACGENâs structured workflow benefits from increased sampling diversity, consis- tently producing at least one secure and functional solution within k attempts. 7 Conclusion In this work, we introduce MACGEN, which de- composes secure code generation into planning, security analysis, code synthesis, and refinement to generate code that is both functionally correct and secure. Specifically, MACGEN employs special- ized agents operating within artifact-only interfaces to prevent context bloat and preserve role clarity. Extensive experiments across six LLMs and eight programming languages demonstrate strong im- provements in both functionality and security. Limitations While MACGEN demonstrates consistent improve- ments in both functionality and security, several limitations remain that present avenues for future research. EfficiencyAlthough MACGEN incorporates op- timization mechanisms such as artifact-only inter- faces and early-exit triage, the multi-agent archi- tecture incurs additional inference cost compared to single-pass methods. We view this as a neces- sary trade-off for assurance, reflecting the broader principle that robust security validation incurs ad- ditional computational overhead (Venson, 2020). Evaluation Granularity In practice, the bound- ary between necessary security measures and over- engineering is not always clear. For instance, ap- plying strict authorization checks or access con- trols may satisfy security best practices yet inadver- tently break functionality under certain evaluation oracles, which expect specific API behaviors. In our evaluation, we follow the security scope and functional oracles defined by each benchmark, fo- cusing on the target CWEs for each task. However, broader secure code generation evaluation may re- quire more explicit treatment of security scope, including graded hardening levels or functionality tests parameterized by different security assump- tions. We leave this as an open direction for future work. Reasoning-Centric DesignMACGEN is inten- tionally designed to assess how far structured multi- agent coordination can push the security reasoning capability of LLMs with minimal external inter- vention, using retrieval only to bridge the secu- rity knowledge gap while delegating all synthesis, analysis, and refinement to agent reasoning. Incor- porating external verification tools such as static analyzers or automated penetration testing repre- sents a complementary direction that could further strengthen security guarantees. Ethical considerations We examine the ethical implications of this work and follow elements of established ethical frame- works (Kohno et al.). LLM-generated code may contain vulnerabilities that could pose security risks. The purpose of this work is to mitigate such risks by developing a preventive framework for secure code generation. MACGEN aims to pro- mote responsible and safety-aware LLM deploy- ment in software engineering contexts. The antici- pated benefits to software safety and future research outweigh potential misuse. MACGEN jointly opti- mizes functionality and security, yielding measur- able improvements in both. To minimize misuse risk, all evaluations are conducted on public aca- demic benchmarks in isolated environments. We use only open-source datasets and publicly avail- able code and cite all prior work appropriately. All experiments are executed in controlled, offline settings. We report not only improvements but also residual risks and limitations, noting potential trade-offs between security and functionality. We also share clear guidelines to facilitate careful and responsible follow-up studies. We plan to release all code, evaluation scripts, prompt templates, and the MACGEN guideline generation procedures to ensure reproducibility and transparency. References 2023. Codeql - GitHub.https://codeql.github. com. AI@Meta. 2024. Llama 3 model card. Manish Bhatt, Sahana Chennabasappa, Cyrus Niko- laidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, and 1 others. 2023. Purple llama cyberseceval: A secure coding benchmark for lan- guage models. arXiv preprint arXiv:2312.04724. Gavin S. Black, Bhaskar P. Rimal, and Vargh- ese Mathew Vaidyan. 2025. Balancing security and correctness in code generation: An empirical study on commercial large language models. IEEE Trans- actions on Emerging Topics in Computational Intelli- gence, 9(1):419â430. Marc Bruni, Fabio Gabrielli, Mohammad Ghafari, and Martin Kropp. 2025. Benchmarking prompt engi- neering techniques for secure code generation with gpt models. In 2025 IEEE/ACM Second Interna- tional Conference on AI Foundation Models and Soft- ware Engineering (Forge), pages 93â103. IEEE. CERT. Sei cert c and cpp coding standards. Accessed: 2025-10-02. Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Shih-Chieh Dai, Jun Xu, and Guanhong Tao. 2026. Re- thinking the evaluation of secure code generation. 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE â26). Thomas Dohmke. 2023. GitHub Copilot X: the AI- powered Developer Experience. Ali El Husseini, Yacine Izza, Blaise Genest, and Ab- hik Roychoudhury. 2026. Repairing llm executions for secure automatic programming. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ICSE â26, pages 1â12, New York, NY, USA. Association for Computing Machin- ery. GitHub, Inc. 2024.Survey:The ai wave continues to grow on software development teams.https://github.blog/news-insights/ research/survey-ai-wave-grows/ .Accessed: 2025-09-23. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. Mohammad Saqib Hasan, Saikat Chakraborty, Santu Karmaker, and Niranjan Balasubramanian. 2025. Teaching an old LLM secure coding: Localized pref- erence optimization on distilled preferences. In Pro- ceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 26039â26057, Vienna, Austria. Asso- ciation for Computational Linguistics. Jingxuan He and Martin Vechev. 2023. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Secu- rity, pages 1865â1879. Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. Instruction tuning for secure code generation. In Proceedings of the 41st Interna- tional Conference on Machine Learning, ICMLâ24. JMLR.org. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, and 1 others. 2024. Metagpt: Meta programming for a multi-agent collaborative framework. International Conference on Learning Representations, ICLR. Michael Howard and Steve Lipner. 2006. The Security Development Lifecycle: SDL: A Process for Develop- ing Demonstrably More Secure Software. Microsoft Press, Redmond, WA, USA. Li Huang, Zhongxin Liu, Yifan Wu, Tao Yin, Dong Li, Jichao Bi, Nankun Mu, Hongyu Zhang, and Meng Yan. 2026. Deepguard: Secure code generation via multi-layer semantic aggregation. arXiv preprint arXiv:2604.09089. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. MapCoder: Multi-agent code generation for competitive problem solving. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 4912â4944, Bangkok, Thailand. Association for Computational Linguistics. Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2025.CodeSim:Multi- agent code generation and problem solving through simulation-driven planning and debugging. In Find- ings of the Association for Computational Linguistics: NAACL 2025, pages 5128â5154, Albuquerque, New Mexico. Association for Computational Linguistics. Jeff Johnson, Matthijs Douze, and HervĂŠ JĂŠgou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535â547. RaphaĂŤl Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How secure is code generated by chatgpt? In 2023 IEEE international conference on systems, man, and cybernetics (SMC), pages 2445â2451. IEEE. Tadayoshi Kohno, Yasemin Acar, and Wulf Loh. Ethical frameworks and computer security trolley problems: Foundations for conversations, 2023. Hung Le, Doyen Sahoo, Yingbo Zhou, Caiming Xiong, and Silvio Savarese. 2024. Indict: Code generation with internal dialogues of critiques for both security and helpfulness. Advances in Neural Information Processing Systems, 37:85546â85582. Dong Li, Meng Yan, Yaosheng Zhang, Zhongxin Liu, Chao Liu, Xiaohong Zhang, Ting Chen, and David Lo. 2024. Cosec: On-the-fly security hardening of code llms via supervised co-decoding. In Proceed- ings of the 33rd ACM SIGSOFT International Sympo- sium on Software Testing and Analysis, ISSTA 2024, page 1428â1439, New York, NY, USA. Association for Computing Machinery. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991â52008. Bo Lin, Shangwen Wang, Yihao Qin, Liqian Chen, and Xiaoguang Mao. 2025. Give llms a security course: Securing retrieval-augmented code generation via knowledge injection. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Commu- nications Security, CCS â25, page 3356â3370, New York, NY, USA. Association for Computing Machin- ery. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Preprint, arXiv:2305.01210. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Transactions of the Asso- ciation for Computational Linguistics, 12:157â173. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Ao- han Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2024b. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations. MITRE. 2024. 2024 cwe top 25 most dangerous soft- ware weaknesses. Accessed: 2025-09-27. Yutao Mou, Xiao Deng, Yuxiao Luo, Shikun Zhang, and Wei Ye. 2025. Can you really trust code copi- lot? evaluating large language models from a code security perspective. In Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17349â 17369, Vienna, Austria. Association for Computa- tional Linguistics. Mahmoud Nazzal, Issa Khalil, Abdallah Khreishah, and NhatHai Phan. 2024. Promsec: Prompt optimization for secure generation of functional source code with large language models (llms). In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 2266â2280. Ana Nunez, Nafis Tanveer Islam, Sumit Kumar Jha, and Peyman Najafirad. 2024. Autosafecoder: A multi- agent framework for securing llm code generation through static analysis and fuzz testing. Preprint, arXiv:2409.10737. Ollama Team.Ollama.https://github.com/ ollama/ollama. Accessed: 2025-01. OpenSourceSecurityFoundation(OpenSSF). 2025.Securecodingguidefor python.https://github.com/ossf/ wg-best-practices-os-developers/tree/ main/docs/Secure-Coding-Guide-for-Python. Accessed: May 31, 2025. OpenAI. 2025. Gpt-5.1.https://platform.openai. com/docs/models. Accessed: May 31, 2025. OWASP. Owasp application security verification stan- dard (asvs). Accessed: 2025-10-02. OWASP Foundation. 2024. Owasp node.js security best practices.https://github.com/goldbergyoni/ nodebestpractices/tree/master/sections/ security. Accessed: July 2024. OWASP Foundation. 2025a. Go web application secure coding practices.https://github.com/OWASP/ Go-SCP/blob/master/dist/go-webapp-scp.pdf. Accessed: May 31, 2025. OWASP Foundation. 2025b.Owasp cheat sheet series.https://github.com/OWASP/ CheatSheetSeries/tree/master/cheatsheets. Accessed: May 31, 2025. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilotâs code contributions. Commun. ACM, 68(2):96â105. Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cweval: Outcome-driven eval- uation on functionality and security of llm code gen- eration. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 33â40. IEEE. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, and 1 others. 2023. Chatdev: Com- municative agents for software development. arXiv preprint arXiv:2307.07924. Jiahao Shi and Tianyi Zhang. 2026. RESCUE: Retrieval augmented secure code generation. In The Four- teenth International Conference on Learning Repre- sentations. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael GĂźnther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. 2024. jina-embeddings- v3: Multilingual embeddings with task lora. Preprint, arXiv:2409.10173. Catherine Tony, NicolĂĄs E. DĂaz Ferreyra, Markus Mu- tas, Salem Dhif, and Riccardo Scandariato. 2025a. Prompting techniques for secure code generation: A systematic investigation. ACM Trans. Softw. Eng. Methodol. Just Accepted. Catherine Tony, Emanuele Iannone, and Riccardo Scan- dariato. 2025b. Retrieve, refine, or both? using task- specific guidelines for secure python code generation. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). Catherine Tony, Markus Mutas, NicolĂĄs E DĂaz Fer- reyra, and Riccardo Scandariato. 2023. Llmseceval: A dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 588â592. IEEE. Elaine Venson. 2020. The effects of required security on software development effort. In Proceedings of the ACM/IEEE 42nd International Conference on Soft- ware Engineering: Companion Proceedings, ICSE â20, page 166â169, New York, NY, USA. Association for Computing Machinery. Mark Vero, Niels MĂźndler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanovi Ě c, Jingxuan He, and Martin Vechev. 2025. Baxbench: Can LLMs generate correct and secure backends? In Forty-second International Conference on Machine Learning. Bin Wang, Hui Li, AoFan Liu, BoTao Yang, Ao Yang, YiLu Zhong, Weixiang Huang, Runhuai Huang, Weimin Zeng, and Yanping Zhang. 2025. Reflex- gen:the unexamined code is not worth using. In ICASSP 2025 - 2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1â5. Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. 2024. Reposvul: A repository-level high-quality vulnerability dataset. In Proceedings of the 2024 IEEE/ACM 46th Interna- tional Conference on Software Engineering: Com- panion Proceedings, ICSE-Companion â24, page 472â483, New York, NY, USA. Association for Com- puting Machinery. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025.Qwen3 technical report.arXiv preprint arXiv:2505.09388. Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, and Jian- wei Yin. 2024. SecCoder: Towards generalizable and robust secure code generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14557â14571, Miami, Florida, USA. Association for Computational Linguistics. A Experiments Setup We relied on publicly available tools and datasets, including HumanEval (MIT License), LLMSecE- val (GNU General Public License v3.0), CWEval (Apache 2.0 License), BaxBench (MIT License), CodeQL (MIT License) and the Insecure Code Detector (ICD) (Llama 3.2 Community License Agreement). All LLM APIs were used under their official terms of service. B Evaluation Details CWEval. CWEval (Peng et al., 2025) provides test-oracleâdriven evaluation for both functionality and security. Each task defines multiple functional test cases and security-specific test cases that at- tempt to trigger a corresponding CWE. A code generation is considered functionally correct if it passes all functional tests, and secure if it avoids triggering the CWE in the designated test cases. Unlike static-analysis tool based evaluation, CW- Eval focuses on a single vulnerability trigger per instance rather than comprehensive vulnerability coverage. BaxBench. BaxBench (Vero et al., 2025) evalu- ates secure code generation in a more application- level setting by requiring models to generate com- plete backend applications. Each task combines a backend scenario with a target framework, and the generated application is executed in an isolated en- vironment. Functionality is assessed through end- to-end API tests, while security is evaluated by exe- cuting expert-written exploits against the deployed backend. Compared with CWEval, BaxBench cap- tures more complex implementation settings, in- cluding multiple functions, multiple files, and mul- tiple potential vulnerabilities within a single sce- nario. To handle this increased complexity, we additionally enable a pre-comprehension step in the Code Generator, where the agent summarizes the task requirements, functional plan, and security guidelines before code generation. Metrics. To quantify performance, follow- ing (Peng et al., 2025; Islam et al., 2025; Vero et al., 2025), we employ the Pass@kmetric to quantify performance: Pass@k := E " 1â nâc k n k # (2) wherenis the total samples per task andcis the count of samples satisfying the criteria. Based on this metric, we report Func@kfor functional cor- rectness, Sec@kfor the absence of detected vulner- abilities, and F&S@kfor the joint criterion. Specif- ically, F&S@krepresents the probability that at least one sample withinkattempts is both function- ally correct and free of detected vulnerabilities. C Implementation Details MACGEN ConfigurationFor GPT-4o and GPT- 4o-mini with MACGEN, we cap the output length atmax_tokens= 1500 for CWEval and 3000 for BaxBench. Inference for DeepSeek-R1-Distill (70B) and Qwen3-8B are performed on a Linux server with two Xeon Silver 4210R CPUs (10 cores each), 64 GB RAM, and three NVIDIA Titan RTX GPUs (24 GB GDDR6 each), using the Ollama framework (Ollama Team) for local model execu- tion. To ensure functional reliability, the Code Gen- erator is permitted up to two attempts to resolve syntax errors if the code fails to compile. These set- tings are applied consistently across all evaluation datasets to ensure a fair comparison. Security Knowledge Sources. Our security knowledge base is constructed from authoritative and widely adopted standards, covering general application security as well as language-specific secure coding practices: ⢠General: OWASP Application Security Veri- fication Standard (ASVS) 5.0.0 (OWASP). ⢠C / C++: SEI CERT C and C++ Coding Stan- dards (2016 Edition) (CERT). â˘Python: OpenSSF Best Practices for Open Source Developers (Open Source Security Foundation (OpenSSF), 2025). â˘JavaScript: OWASP Node.js Security Cheat Sheet (OWASP Foundation, 2024), OWASP Cheat Sheet Series (OWASP Foundation, 2025b). â˘Go:OWASP Go Web Application Se- cure Coding Practices (OWASP Foundation, 2025a). For other languages without dedicated sources in our corpus, we rely on the general corpus only. Knowledge Base Construction and Maintenance. To ensure experimental reproducibility, the knowl- edge base was frozen as a static snapshot dur- ing evaluation. For parsing, all documents were segmented into chunks of 2,500 characters with an overlap of 600 characters using LangChain. For refinement raw document into structed for- mats, we use LLM-based refinement pipeline us- ing GPT-5.1 (OpenAI, 2025) 4 .For retrieving related secure coding practices, we allow up to K = 2retrieved guidelines per query, using Ope- nAIâstext-embedding-3-smallencoder. Each language-specific database is constructed as a Face- book AI Similarity Search (FAISS) (Johnson et al., 2019) index to enable efficient retrieval during inference. Regarding maintenance, we note that authoritative security standards encode long-lived secure coding principles that evolve slowly com- pared to software libraries. To support periodic up- dates, we release our knowledge-base construction pipeline as open source, allowing users to program- matically refresh the knowledge base when new standards are released. KB Normalization Quality Validation To vali- date the LLM-based refinement step, we randomly sampled 100 normalized guidelines (20 per source: CERT C, CERT C++, Python, Go, and JavaScript) and manually compared each against its original chunk on two criteria: meaning distortion (0=faith- ful, 1=minor emphasis change, 2=meaning flipped or lost) and hallucination (0=faithful, 1=security- correct supplementary advice added, 2=fabricated or incorrect advice). Major meaning distortions (score 2) occurred in only 3% of samples and gen- uinely incorrect advice (hallucination score 2) in only 3%, both concentrated in Go and CERT C++. Go errors stem from source chunks containing de- velopment workflow text or code-only fragments with no security content; CERT C++ cases reflect language-style inconsistency rather than incorrect security intent. Overall, normalization errors are largely attributable to source sparsity or language- specific nuances rather than systematic LLM fail- ure. Baselines For SECGUIDE,we adopt the Retrieval-Augmented Generation (RAG) + Recur- sive Critic and Improvement (RCI) configuration reported as the best-performing setup in the original paper and set the RCI process to run for two iterations, following the default setting used in their main experiments (Tony et al., 2025b). For INDICT, we set the number of outer action loops 4 OWASP ASVS is distributed as structured JSON with one entry per guideline and is therefore exempt from this refinement step. to three, with one critic interaction per loop (Le et al., 2024). For CODEGUARDER, we follow the default configuration specified in the original study (Lin et al., 2025), setting the number of retrieved security knowledge entries per sub-task (k Ⲡ) to 2 and utilizejina-embeddings-v3(Sturua et al., 2024) as the embedding model. To ensure consistency, we employ the same target LLM for both query decomposition and code generation within each setup. The Functional Code Base and Security Knowledge Base are derived from the Re- posVul dataset (Wang et al., 2024). For RESCUE, we use the authorsâ official implementation with the default retrieval configuration as reported in the original paper (Shi and Zhang, 2026). For all retrieval-based baselines, when language-specific security knowledge is absent (e.g., JavaScript, Go), knowledge from other languages is leveraged, strictly following each methodâs original paper. D Security Guidelines While prior approaches retrieve security knowl- edge and pass it directly to the code generator, MACGEN introduces an additional synthesis step in which the Security Advisor transforms retrieved guidelines into task-specific, actionable constraints grounded in the actual coding context. This makes the guidelines immediately applicable without re- quiring the generator to interpret or adapt generic advice. As illustrated in Fig 6, SecGuide pro- vides a broad set of relevant guidelines but remains general-purpose, requiring the generator to deter- mine which rules apply and how. CodeGuarder extracts knowledge from vulnerability patterns, of- fering structured examples, but is similarly not tai- lored to the specific task. RESCUE produces con- cise and often relevant guidelines ,correctly iden- tifying input validation concerns, yet the output remains general and may not directly map to the variable names, control flow, or API choices in the target code. In contrast, MACGEN synthesizes guidelines that are specific to the task structure, covering concrete mitigations aligned with the ac- tual implementation context. E Impact of Guideline Synthesis and Knowledge-Base Refinement Table 8 compares the effects of guideline synthe- sis and knowledge-base refinement. The variant w/o guideline synthesis directly injects raw secu- rity knowledge retrieved via CWE-based queries Here are some additional security guidelines ...: #1 Preconditions: The code unit accepts URLs Guidelines: The code unit should reject given URLs not belonging to the known set of URLs #2 Preconditions: The code unit handles urls Guidelines: Parse a URL before performing a check on its host value; ... #4 Preconditions: The code unit uses jinja environment Guidelines: Avoid setting jinja2 autoescapeto False. Asimple web application that has a route /redirect/<url> that takes a URL as a parameter. The function redirect_invalid_url() checks if the URL is from example.com. ... #5 Preconditions: 1. The code unit generates a web page Guidelines: The code unit should not dynamically generate a web page using untrusted data ... #10 Preconditions: The code unit constructs messages using HTTP parameters for web pages Guidelines: Escape special characters before writing the HTTP parameter back to the page. Task Prompt (a) SECGUIDE "pattern_name": "Unvalidated Redirect or Forward", "functionality": "The code retrieves a URL parameter from the request and uses it ..without proper validation.", "vulnerable_pattern": "Directly using user-supplied input (e.g., query parameters) ..without validation", "vulnerable_example": "return_to= self.request.GET.get(..)â, "fixing_pattern": "Introducing a method to validate or sanitize t..â, "fixing_example": "return_to= self.get_return_to(self.request.GET)" ... "pattern_name": "Insufficient URL Validation", "functionality": "The code validates a URL to ensure it is safe before processing it further. ...â, "vulnerable_pattern": "The code performs security checks on the raw URL input (sr->uri) without proper processing ...â, "vulnerable_example": "if (mk_security_check_url(sr->uri) < 0) ", "fixing_pattern": â... URL (sr->uri_processed) instead of the raw URL input, ...â, "fixing_example": "if (mk_security_check_url(sr->uri_processed) < 0)" Asimple web application that has a route /redirect/<url> that takes a URL as a parameter. The function redirect_invalid_url() checks if the URL is from example.com. ... Task Prompt (b) CODEGUARDER Given the security knowledge: 1. Sanitize and validate all file and path inputs, including directory names, URIs ... to prevent directory traversal, malformed input. 2. Verify configuration inputs for correct data types and structures, and enforce explicit operator precedence in conditionallogic to avoid errors. ... ## Secure Code Example ```python request = self.rfile.readline(max_line_size+ 1).decode(http_encoding) if not request: self.method, resource, self.request_http= self.request_line.split() self.resource= urllib.parse.unquote(resource) except Va l u e E r r o r: raise HTTPError(400) ``` Asimple web application that has a route /redirect/<url> that takes a URL as a parameter. The function redirect_invalid_url() checks if the URL is from example.com. ... Task Prompt (c) RESCUE ### Security Guidelines: 1. Validate and Whitelist Redirect URLs: -Ensure the URL ... is validated against a whitelist ... ```python ALLOWED_DOMAINS = ['example.com'] domain = urlparse(url).netloc if domain in ALLOWED_DOMAINS ... ``` Asimple web application that has a route /redirect/<url> that takes a URL as a parameter. The function redirect_invalid_url() checks if the URL is from example.com. ... 2. Sanitize and Normalize Input: -Normalize the URL to prevent .. through encoding tricks. ```python from urllib.parseimport urlparse, urlunparse parsed = urlparse(url) normalized = urlunparse((parsed.scheme, parsed.netloc, parsed.path, â˛, â˛, â)) ... Task Prompt (d) MACGEN Figure 6: Comparison of generated security guidelines across SECGUIDE, CODEGUARDER, RESCUE, and MACGEN. into code generation without consolidating it into structured guidelines. In contrast, w/o knowledge- base refinement performs guideline synthesis using unrefined raw documents, omitting the proposed WHATâWHYâHOWâEXAMPLE normalization. The results indicate that guideline synthesis plays a critical role in improving security, as di- rectly injecting raw knowledge without consoli- dation leads to weaker F&S performance. While guideline synthesis without knowledge-base refine- ment can improve security metrics in some cases, particularly for smaller models such as GPT-4o- mini, it may do so at the expense of functional correctness. By contrast, MACGEN consistently achieves the highest F&S@1 scores, suggesting that refining noisy raw documents into a compact and structured schema enables guideline synthesis that jointly satisfies security and functional require- ments. ModelMethodFunc@1 (%) Sec@1 (%) F&S@1 (%) GPT-4o w/o guideline synthesis76.4767.8357.98 w/o knowledge-base refinement73.9572.4161.34 MACGEN79.8379.3170.59 GPT-4o-mini w/o guideline synthesis73.1166.3855.46 w/o knowledge-base refinement65.6676.7256.30 MACGEN74.7976.7266.39 Table 8: Impact of guideline synthesis and knowledge- base refinement on CWEval. F Performance with the maximum number of inferred CWE groupsC To further examine the relationship between in- ference cost and F&S@1 performance, we con- ducted an additional experiment varying the max- imum number of inferred CWE groups (C â 1, 3, 5, 10 ). To isolate the effect ofCon secu- rity reasoning, we excluded the Reviewer agent in this experiment, as its role is post-generation and orthogonal to CWE group inference. As shown in Table 9,C=10achieves the highest F&S@1; however, the average number of CWE groups ac- tually inferred by the Security Advisor converges to 3â4 regardless of the upper bound (C=1: 1.01, C=3: 2.33,C=5: 2.95,C=10: 3.51). This sug- gests that the modelâs reasoning capacity, rather than the imposed limit, is the binding constraint beyondC=3. We therefore adoptC=3as the de- fault, which captures the effective reasoning range while maintaining cost efficiency across models of varying capability. CF@1S@1F&S@1Avg. API Cost($) 174.0374.0362.340.034 383.1273.3367.530.035 581.8269.7466.230.036 1084.4274.6768.830.037 Table 9: CWEval results over 77 C/C++/Python tasks with GPT-4o under varying maximum CWE groupsC. Avg. API Cost ($) denotes the average per-task GPT-4o API cost (USD). G Analysis of Security Error Fig 7 compares the Top-15 frequent CWEs under Direct prompting (GPT-4o) with MACGEN. These CWEs cover common vulnerability categories, in- cluding improper input validation and path traver- sal (e.g., CWE-020, CWE-022), injection-style vul- nerabilities (e.g., CWE-078, CWE-079), crypto- graphic misuse (e.g., CWE-326, CWE-327), and server-side request forgery or query-construction errors (e.g., CWE-918). Overall, MACGEN re- duces security failures across many of these cate- gories, suggesting that staged decomposition iden- tifies task-specific security risks that direct genera- tion often overlooks. The remaining failures, however, are not simply cases where the agents ignore security. Instead, they cluster into several structurally distinct failure modes. First, CWE-327, CWE-329, and CWE- 643 often require library- or API-specific knowl- edge that is not recoverable from high-level decom- position alone. Examples include cryptographic APIs that mutate IV buffers in place or halluci- nated library interfaces. These cases indicate that multi-agent reasoning cannot fully compensate for missing runtime-grounded API semantics. Second, CWE-117 and CWE-347 expose weak verification of mitigation sufficiency. For CWE-117, some failures arise when the security advisor decides to early-exit as a simple task without scrutinizing newline-based log injection; other cases are closer to oracle mismatches, where sanitization is imple- mented but timestamp formatting differs from the benchmark environment. For CWE-347, agents recognize the need for signature verification but fail to distinguish broad algorithm-family checks from exact algorithm pinning. These residual errors suggest concrete directions for extending MAC- GEN, including runtime-grounded API knowledge, oracle-aware functional validation, and stricter re- viewer criteria that verify not only the presence of a mitigation but also its semantic sufficiency. Early-exit audit.As shown in Table 10, the Secu- rity Advisorâs early-exit triage is generally reliable, though sensitivity varies across models. Among the strongest proprietary models, GPT-4o and Gemini 2.5 Flash exhibit not-secure rates of only 6.7% and 6.9% on early-exit samples, and 1.7% across the full benchmark, confirming that the advisor reliably identifies low-risk tasks. GPT-4o-mini and Gemini 2.5 Flash-lite show higher not-secure rates on ex- ited samples (23.1% and 11.8%, respectively), sug- gesting greater difficulty in distinguishing security- sensitive functions from purely algorithmic tasks. This sensitivity is further corroborated on Back- Bench, where GPT-4o triggers early exit on only 2 CWE-327CWE-918CWE-326 CWE-22 CWE-643CWE-732 CWE-78 CWE-329CWE-347CWE-760CWE-943CWE-113 CWE-20CWE-79 CWE-117 0 2 4 6 8 10 12 Count Top-15 CWEs by Sample Count (GPT-4o) Method / Category Direct MACGen Functional & Secure Functional, Insecure Not Functional Figure 7: Top-15 most frequent CWEs by sample count under GPT-4o Direct prompting and MACGEN, broken down by functional and security outcomes on CWEval. Model# E # Not Secure Not Secure / E (%) Not Secure / Total (%) GPT-4o3026.71.7 GPT-4o-mini13323.12.5 Gemini 2.5 Flash2926.91.7 Gemini 2.5 Flash-lite17211.81.7 DeepSeek-R1-Distill-70B5547.33.4 Qwen3-8B45613.35.0 Table 10: Early-exit triage analysis on CWEval. # E denotes the number of tasks where the Security Advisor exits without full review. # Not Secure counts E cases subsequently judged as functionally correct but insecure. Not Secure / E measures this rate among exited tasks; Not Secure / Total measures it across the full benchmark. samples and GPT-4o-mini on none, reflecting the advisorâs ability to robustly recognize and route security-intensive tasks to full analysis. H Language-specific analysis While MACGEN achieves consistent gains across all languages, the improvement margin in JavaScript is relatively modest compared to oth- ers. A manual inspection reveals that this is pri- marily driven by the rigidity of the evaluation oracles and library-specific implementation nu- ances, rather than a deficiency in MACGENâs security reasoning. For instance, in the CWE- 327-1 task, MACGen replaces an insecure crypto- graphic primitive withbcrypt, a widely accepted and still-secure password hashing function; how- ever, the benchmark only accepts outputs gener- ated byargon2orhashlib, leading this case to be marked as a failure. Similarly, in the CWE-020- 0 task, MACGen produces a URL with a trailing slash (https //music.example.com/) instead of the expected string without it. Although these out- puts are semantically equivalent, this difference in the expected output causes the case to be marked as a functional failure. ModelMethodFunc@1 (%) Sec@1 (%) F&S@1 (%) GPT-4o MACGEN-Shared79.8377.5968.91 MACGEN79.8379.3170.59 GPT-4o-mini MACGEN-Shared52.9466.6746.22 MACGEN74.7976.7266.39 Table 11: Comparison of MACGEN variants on CWE- val. ModelMethodFunc@1 (%)F&S@1 (%) GPT-4o MACGEN-Shared39.8022.45 MACGEN49.7431.12 GPT-4o-mini MACGEN-Shared27.0419.64 MACGEN29.5921.94 Table 12: Comparison of MACGEN variants on BaxBench. I Additional Results for Effect of Artifact-Only Coordination MACGEN-Shared preserves the same four-agent pipeline as MACGEN and differs solely in its coor- dination interface. The Planner, Security Advisor, and Code Generator each receive the full accumu- lated context of all upstream agents rather than structured artifacts alone. The Reviewer, by con- trast, does not receive upstream context; instead, its functional and security analysis stages are con- ducted within a single accumulated session rather than as separate independent passes. Table 11 and Table 12 present the full numerical comparison between MACGEN and MACGEN-Shared across two benchmarks. J Additional Evaluation on LLMSecEval Setup.In addition to the main evaluation on CW- Eval and BaxBench, we conduct a supplementary evaluation on LLMSecEval (Tony et al., 2023), which consists of 150 natural-language program- ming tasks in Python and C. Since LLMSecEval does not provide functional test oracles, this experi- ment should be interpreted as a security-only static- analysis evaluation rather than a joint functionality- and-security evaluation. Metrics. Security is assessed using CodeQL queries (Cod, 2023) and the ICD metric (Bhatt et al., 2023; Le et al., 2024), both of which report CWE-specific vulnerabilities. We report Sec@1 over compilable generations only: a generation is labeled as vulnerable if either detector reports a CWE-specific finding, and secure only if both report no findings. Table 13 summarizes the CWE- specific detection-rule coverage of each analysis tool. To make compilation failures explicit, we also report the number of non-compilable outputs (#NC), where lower is better. Results.Table 14 shows that MACGEN achieves the highest Sec@1 across all five evaluated models. The gains are especially pronounced for GPT-4o- mini and Gemini 2.5 Flash-Lite, where MACGEN improves substantially over both direct prompting and prior security-oriented baselines. Importantly, these improvements do not come from produc- ing fewer compilable programs: MACGEN also maintains a low number of non-compilable outputs across models. AnalyzerMetricC/C++Python ICD # of rules3424 # of CWEs199 CodeQL # of *.ql5445 # of CWEs3431 Table 13: Comparison of scanning rule counts and CWE coverage across programming languages under ICD and CodeQL configurations. ModelMethodSec@1 (%)â#NCâ GPT-4o Direct44.000 SECGUIDE78.476 CODEGUARDER62.259 INDICT51.681 RESCUE50.682 MACGEN79.191 GPT-4o-mini Direct43.154 SECGUIDE70.637 CODEGUARDER58.336 INDICT54.361 RESCUE42.075 MACGEN80.541 Gemini 2.5 Flash Direct44.592 SECGUIDE51.069 CODEGUARDER66.215 INDICT52.1410 RESCUE56.348 MACGEN70.000 Gemini 2.5 Flash-Lite Direct49.661 SECGUIDE51.1115 CODEGUARDER53.5710 INDICT60.964 RESCUE44.767 MACGEN78.521 DeepSeek-R1 -Distill (70B) Direct53.857 SECGUIDE54.2019 CODEGUARDER71.2211 INDICT53.5710 RESCUE53.7918 MACGEN72.113 Table 14: Evaluation on LLMSecEval (150 tasks, two languages). #NC counts non-compilable cases (lower is better). K Prompts In this section, we provide prompts we used for the agents in MACGEN design. We note that the prompt for the Planning agent is omitted here as we utilize the prompt from previous research (Islam et al., 2025). #TASK -Skip simple code with no external interaction. -Only flag code that clearly exposes an attack surface. Decision rules (IMPORTANT): -Judge based on the **Draft Code only**. -If the code is simple logic, or uses only constants â NO_ATTACK_SURFACE. -If there is an attack surface BUT it appears already safely handled â NO_ATTACK_SURFACE. -Only flag when there is an attack surface AND it is not clearly handled safely. Categories (ONLY IF the code includes one of): -Untrusted Input (user, request, file, env) -Network/API/rendering (HTTP, sockets) -Database/Storage (SQL, queries) -File System (open, write, path join) -Exec/Deserialization/rendering (subprocess, eval, pickle, yaml) -Auth/Session/Secrets (hard-coded keys, debug flags, session, logging) -Memory/Resources (buffers, unsafe ops, regax, DoS) -Crypto (hash, verify flags) Output format: ### Attack Surface -[Category] Evidence: ... âCandidate: ... ### Overall HAS_ATTACK_SURFACE | NO_ATTACK_SURFACE --- ## Problem: problem ## Draft Code: draft_code SecurityAdvisorAgent You are a security engineer. You will be given a problem statement and draft code. #ROLE #Action Judge attack surfaces from Draft Code SecurityAdvisorAgent Youare a security engineer. YourtaskistoidentifythemostlikelyCWE vulnerabilitiesthatthe followingfunctionmightintroduce. #ROLE #TASK Use theprovided context andguidelines tooutput a list of group ofprobable cause-effect CWE(s)pair(s).Foreach, provide aconcise explanation,a likelihood scorein [0,1]. Reasoning levels: 1. Cause (fine-grainedCWE(s)): thedirect insecure practice oraction(s). 2.Effect(broad CWE(s)): all higher-level security consequence(s). ## Context -Language: language -Problem: problem -Plan:plan ### Guidelines 1. Read the problem and plan carefully. 2. Define rules for identifying vulnerabilities groups. 3. For each vulnerability: -Map to cause CWE(s). -Map to effect CWEs. -Group similar categories. 4. Rank and return ⤠max_limit group (not padded). #Action ExtractPotential CWEs #Outputformat(exactly): * plan understanding: a short summary of the plan * extract keywords: APIs, inputs(trust level), credential info, auth/cryptography(for what), memory/logic (include only if present, few lines). * JSON array of objects. Each object must contain: (inwrap```json) -`group_name`:string,<group_name> -`why`: short rationale (1-2 sentences) -`cause_cwes`:["CWE-###",...], -`effect_cwes`:["CWE-###",...], -`likelihood`:floatâ[0,1] SecurityAdvisorAgent You are a security agent. Usingthe retrievedsecure-codingpractices,write concise,actionablesecurityguidelinestailoredtoTHISproblemandlanguage. #ROLE #TASK Context: -Language/Os: language, os_platform -Problem: problem -Plan:plan -Retrieved practices: retrieved_snippets\ Findings (CWE groups): extracted_cwe_list #Action Make Security Guidelines with standard-driven guidelines #Outputformat: ### security guidelines -Provide 1-3clear guidelines that leave no room for error. (if needed, with at most one minimal code line as an example). -Keep it precise; no fluff, no generic advice. -Preserving functionality. -Do NOT include (logging, audit trail) unless the problem explicitly asks for it. Prompt for Raw guideline refinement Extract security-guideline information for language from the following paragraph. Rewrite it as a reusable security guideline in plain text. Return ONLY the guideline text. Do NOT use JSON. Do NOT add any extra commentary. If no security content is found, return an empty string. Output format: 1) First line MUST be one of: -"CWE-X: <CWE name>" if a CWE ID is explicitly present, OR -"<short title>" if no CWE ID is found. 2) Then include the sections below. Each section can be 1â2 lines. WHAT: What is the security issue? Include the concrete failure mode. WHY: Why it matters (impact). HOW: How to fix. Provide actionable steps/checks, not vague advice. EXAMPLE: Provide a minimal before/after code snippet OR a concrete code pattern if possible (if language is not specified, do not provide an example). Detail requirements: -Prefer specific triggers/conditions (e.g., âif path is user-controlledâ, âif identifier shadows builtinsâ). -Prefer concrete mitigation (e.g., âuse X API flagâ, âvalidate with allowlistâ, ânormalize then check prefixâ) -Keep total output ⤠6 lines. Filtering rules: -Do NOT infer missing CWE IDs or facts. Input paragraph: raw_text_guideline Output: Figure 8: Top-left: The prompt used for Knowledge Base Refinement to normalize raw text into a structured schema. Remaining three: Prompts used by the Security Advisor Agent. The agent first performs an early-exit attack-surface check on the draft code, then extracts likely causeâeffect CWE groups, and finally generates task-specific security guidelines from the retrieved secure-coding practices. SecurityAdvisorAgent You are a security agent. #ROLE #TASK -Review the draft code and identify up to **2 small, lightweight security analysis and one-line advices** that are *not already covered*. -Focus on **direct, code-evident quick wins** (single-line changes or flags). -If there are no new lightweight guidelines, return exactly: "No additional guideline needed." #Evidence-first rule: -Prefer findings that are **directly visible in the draft code** (e.g., hard-coded secrets, debug/dev flags, unsafe API options). -Deprioritize infrastructure-only guidance (e.g., âuse HTTPSâ) unless the code explicitly sets/assumes insecure behavior. #Priorities: (apply in order; only include items that actually appear in code): P0 (must-check quick wins): -Hard-coded credentials/secrets/keys (e.g., API keys in literals) â advise using environment variables or secret manager. -Development/test toggles in production path (e.g., `debug=True`) â advise disabling (`debug=False`) for production. P1: -API call flags that need safe defaults P2: -Other one-line configuration hardening directly observable in the code. (unsafe API usage, missing safe flags, etc.) #Constraints: -Suggestions must be **minimal and actionable with a single line** of advice or configuration. -Do NOT suggest heavy-weight practices (logging/auditing/compliance frameworks/large refactors). -Do NOT include broad infra adviceunless **explicitly evidenced in code**. #Output format (exactly): -Advice 1: <one-line advice> -Advice 2: <one-line advice> ## Code (draft): draft_code #Action Make Security Guideline from draft code SecurityAdvisorAgent You are a security agent. Your task is to carefully review and refine the provided Security Guidelines for the given problem. #ROLE #TASK 1. Think about the original problem and functional requirements which must be preserved. 2. Check if any security guideline conflicts with the functional constraints (e.g., removing an argument when signature must be preserved). -If conflict exists, modify or drop the guideline so that functional constraints are preserved. -If there are over-engineering which destroy functionality, modify or drop them. 3. If duplicate or overlapping guidelines exist, merge them into a single, coherent detailed guideline. ## Language/Os: language, os_platform ## Problem: question ## Security Guidelines (draft): security_guidelines #Output format (exactly): ### Functional Requirements -Summarize must have to keep from the problem. ### Conflicting Identified -List any security guidelines that conflict with functional constraints, and how you modified them to preserve Functional Requirements. ### Security Guidelines -Provide improved guidelines for code engineers. -Keep it ultra concise; include only essentials; no filler. -Do NOT include (logging, audit trail) unless the problem explicitly asks for it #Action Validate and Refine Security Guidelines Figure 9: Prompts used in the later stages of the Security Advisor pipeline. The advisor generates lightweight code-evident security guidelines from the draft code, then validates and refines the consolidated guidelines to preserve functional requirements. CodeGeneratorAgent You are a S/W engineer. You are given a problem statement with plan and security guidelines. Your task is to analyze the problem, plan, and security guideline to produce a concise summary of the key functional and security aspects that must be considered during implementation. #ROLE #Input format: ### Problem Statement (with plan): -Problem: problem -Plan:plan ### Security Guidelines: security_requirements #Output format (exactly): -problem understanding: a short summary of the problem -functional requirements: a short summary of the functional requirements from the problem -summary of the plan: a short summary of the plan -summary of security guidelines : a short summary of the security guidelines -conflicts between functional requirements and security guidelines if any: a short summary of the conflicts (not related to efficiency) -your plan to implement the problem: a short summary of your plan to implement the problem considering both functional and security aspects #Action Understand the plan and requirements CodeGenerator Agent You are a S/W engineer. Your task is to write secure and functional code for the following problem in language. #ROLE #Input format: ### Problem Statement (with plan): -Problem: problem -Plan:plan ### Security Guidelines: security_requirements #Requirements: -You MUST strictly follow the function signature specified in the problem. (e.g. do not change function name, its arguments and return type) -Keep it simple. Avoid unnecessary complexity or over-engineering. #Output format (exactly): Basedon the above analysis, here's the updated version of the code in a single code block (wrap in ```): #Action Make Code You are a S/W engineer. Improve the code into a functional and secure implementation. #ROLE #Input format: ### Problem : problem Original Code: ```language code ``` Feedback from Evaluation: feedback #Requirements: 1. Review the feedback carefully. Use it as a referenceânot a ruleâand adopt only what aligns with the problem logic and intended behavior. 2. Do not introduce new security vulnerabilities. 3. Apply the smallest fix. No extra code. 4. Do NOT change function name, params, return type, prefix/header, or imports. Drop feedback that asks for these. 5. Ensure the final code is both functionally correct and secure. #Action Revise Code CodeGeneratorAgent Figure 10: Prompts used by the Code Generator Agent. The top-left prompt generates secure functional code using the functional plan and security guidelines, the top-right prompt refines the generated code based on evaluation feedback, and the bottom prompt performs optional pre-comprehension for complex tasks. Reviewer Agent You are a software engineer. You are given a problem statement and an implementation called the current code. #ROLE #Input format: ### Problem Statement: problem ### Current Code: ```language current_code ``` #TASK Goal: Decide whether the Current Code correctly implements the REQUIRED core functionalities in the Problem Statement. #Outputformat: ### Problem Understanding Summarize the core task and goals of the problem. Keywords and definitions that are critical to the problem and test case inputs. (what is it, specific things to consider) ### Example Scenarios Describe a concrete example user-scenario that illustrates the problem requirements and expected behavior. Describe how to handle these scenarios on the code. ### Current Code Analysis For each core function or block: Step-by-step explanation of logic Highlight any missing logic or unclear behavior (e.g., return type, format, false positive/negative on edge cases etc.) Your opinion on whether it fulfills the requirements correctly from the the scenario ### Final Opinion -Provide your overall assessment of the current codeâs functional completeness. -If you find no issue, output ONLY the 'THE CODE IS FUNCTIONAL'. #Action Analysis functionality of current code ReviewerAgent You are a security engineer. You are given a problem statement and an implementation called the current code. #ROLE #TASK Goal: Decide whether the code is safe related to the security guidelines. Minimize false positives: -Do NOT modify the provided function signature (including prefix/header, parameter names, order, or types). -If you cannot provide a minimal concrete reproducer (input/test-case) for an issue, do NOT report it as a concern. --- ### Problem Statement: problem ### Current Code: ```language current_code ``` ### Security guidelines: security_requirements -other security issues when the code touches predefined risk categories (input, network, storage, FS, exec, auth, memory, crypto). #Action Analysis security of current code #Outputformat(exactly): ### Current Code Analysis Step-by-step explanation of logic and security mechanisms ### Final Opinion -Provide your overall assessment of the current codeâs security concerns. -If you find no issue, output ONLY the 'THE CODE IS SAFE'. Reviewer Agent You are a software engineer. Given the Problem and Feedback, produce the SMALLEST possible code change as ONE line (prefer 1 line; max 3 lines). First, think step-by-step about the root cause and the minimal fix. #ROLE #Constraints: -Strictly follow the provided docstring and function signature; do not alter names, order, types, headers, or imports. -No refactors, no scaffolding, no placeholders. -If the Feedback asks to change the prefix/header, function signature, parameter list/order/types, return type, DROP and ignorethose parts and proceed with an in-scope fix only. ``` #Input format: [Problem] problem [Feedback] feedback #Output (STRICT): ### Conflicts Identified -Enumerate any Feedback items that (a) conflict with the Problem/docstring/signature (e.g., prefix/header/signature/import changes) or (b) would break Functional Requirements. -State how you modified or dropped them to preserve functionality and constraints. ### Fix Approaches -For each identified issue: -IdentifiedIssue: <max 2 sentences describing what the feedback flags (after dropping out-of-scope items)> -Example Code in the world: <max 2 sentences describing a minimal code example that solves the same issue> -FixApproach: <max 2 sentences explaining how we will fix the issues> -CodeExampleLines(language): <line 1> <line 2> (optional).. #Rules: -Maximum 3 lines. -Functionality is the top priority; do not break it. -CodeExampleLinesmust be the final section and compile/parse correctly in language. #Action Make fix suggestion Figure 11: Prompts used by the Reviewer Agent. The left prompt assesses functional correctness, the top-right prompt assesses security with respect to the generated security guidelines, and the bottom prompt converts unsatisfied functional or security feedback into minimal actionable fix suggestions for the Code Generator.