Paper deep dive
LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software
Syed Md Mukit Rashid, Abdullah Al Ishtiaq, Kai Tu, Yilu Dong, Tianwei Wu, Ali Ranjbar, Tianchang Yang, Najrin Sultana, Shagufta Mehnaz, Syed Rafiul Hussain
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/15/2026, 1:45:52 AM
Summary
LogicEval is a systematic framework designed to evaluate automated repair techniques for logical vulnerabilities in software. It introduces LogicDS, a dataset of 86 real-world logical vulnerabilities with associated CVEs, and provides a pipeline for patch generation, compilation, testing, and reasoning-based evaluation to address the limitations of existing memory-safety-focused repair tools.
Entities (5)
Relation Signals (3)
LogicDS â contains â Logical Vulnerabilities
confidence 100% ¡ we created the first ever dataset, LogicDS, of 86 logical vulnerabilities
LogicEval â evaluates â Automated Vulnerability Repair
confidence 95% ¡ LogicEval, a systematic end-to-end framework that automatically generates and evaluates patches for logical vulnerabilities.
Large Language Models â usedfor â Automated Vulnerability Repair
confidence 90% ¡ Several LLM-based approaches have been proposed to localize vulnerabilities, generate repair patches, and assess patch correctness
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Logical vulnerabilities in software stem from flaws in program logic rather than memory safety, which can lead to critical security failures. Although existing automated program repair techniques primarily focus on repairing memory corruption vulnerabilities, they struggle with logical vulnerabilities because of their limited semantic understanding of the vulnerable code and its expected behavior. On the other hand, recent successes of large language models (LLMs) in understanding and repairing code are promising. However, no framework currently exists to analyze the capabilities and limitations of such techniques for logical vulnerabilities. This paper aims to systematically evaluate both traditional and LLM-based repair approaches for addressing real-world logical vulnerabilities. To facilitate our assessment, we created the first ever dataset, LogicDS, of 86 logical vulnerabilities with assigned CVEs reflecting tangible security impact. We also developed a systematic framework, LogicEval, to evaluate patches for logical vulnerabilities. Evaluations suggest that compilation and testing failures are primarily driven by prompt sensitivity, loss of code context, and difficulty in patch localization.
Tags
Links
- Source: https://arxiv.org/abs/2604.12994v1
- Canonical: https://arxiv.org/abs/2604.12994v1
Trouble viewing inline? Open PDF directly â
Full Text
112,224 characters extracted from source content.
Expand or collapse full text
LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software Syed Md Mukit Rashid, Abdullah Al Ishtiaq, Kai Tu, Yilu Dong, Tianwei Wu, Ali Ranjbar, Tianchang Yang, Najrin Sultana, Shagufta Mehnaz, Syed Rafiul Hussain The Pennsylvania State University szr5848, abdullah.ishtiaq, kjt5562, yiludong, tvw5452, aranjbar, tzy5088, nks5814, smehnaz, hussain1@psu.edu Abstract Logical vulnerabilities in software stem from flaws in program logic rather than memory safety, which can lead to critical security fail- ures. Although existing automated program- repair techniques primarily focus on repairing memory-corruption vulnerabilities, they strug- gle with logical vulnerabilities because of their limited semantic understanding of the vulner- able code and its expected behavior. On the other hand, recent successes of large language models (LLMs) in understanding and repairing code are promising. However, no framework currently exists to analyze the capabilities and limitations of such techniques for logical vul- nerabilities. This paper aims to systematically evaluate both traditional and LLM-based repair approaches for addressing real-world logical vulnerabilities. To facilitate our assessment, we created the first-ever dataset, LogicDS, of 86 logical vulnerabilities with assigned CVEs reflecting tangible security impact. We also developed a systematic framework, LogicEval, to evaluate patches for logical vulnerabilities. Evaluations suggest that compilation and test- ing failures are primarily driven by prompt sen- sitivity, loss of code context, and difficulty in patch localization. 1 1 Introduction Logical vulnerabilities in software pose significant risks because they stem from incorrect implemen- tation of the programâs logic, rather than violations of language safety measures (e.g., memory corrup- tions). These vulnerabilities can be exploited to induce critical security and privacy implications, including authentication bypass (Tu et al., 2024), sensitive data leakage (Ranjbar et al., 2025), or sys- tem operations disruptions (Dong et al., 2025), of- ten without triggering traditional security defenses (e.g., address sanitizers (Song et al., 2019)). Sev- eral works (Felmetsger et al., 2010; Deepa et al., 1 To appear in ACL 2026 Main Conference. 2018) have focused on identifying logical vulnera- bilities in complex software systems and evaluating LLM performance to identify them (Ullah et al., 2023). However, little attention has been given to automatically repairing them, leaving vulnerable software exposed to significant risk. Automatically repairing logical vulnerabilities presents unique challenges compared to fixing memory-corruption vulnerabilities. First, logical vulnerabilities do not follow consistent, reusable repair templates/patterns (Li et al., 2025). Generat- ing a correct patch often require deep semantic un- derstanding of the vulnerable code, its surrounding context, and the compromised functionality or prop- erty. They also do not necessarily lead to crashes or illegal memory access, thus conventional signals such as compilation logs, runtime logs, or memory sanitizers (Song et al., 2019) provide limited help for localization. Finally, generating effective test- cases to assess their patch correctness is difficult because these traditional tools cannot offer much guidance. Since manual vulnerability repair is often labor- intensive and time-consuming (Li and Paxson, 2017), prior works have proposed automated tech- niques to generate patches (Gao et al., 2021; Jiang et al., 2021; Le Goues et al., 2012). However, most of these approaches target memory safety viola- tions, making them ineffective or unreliable for logical vulnerabilities. Template-based (Lin et al., 2007; Novark et al., 2007; Cheng et al., 2019) and search-based (Le Goues et al., 2012; van Tonder and Le Goues, 2018) patch generation techniques rely on recognizable repair patterns that limit gen- eralization to logical vulnerabilities whose patches are diverse and context-specific. Deep learning- based methods (Chen et al., 2019; Jiang et al., 2021) seem a plausible option, but their effectiveness de- grades on previously unseen patterns. Recently, Large Language Models (LLMs) trained on vast training data have shown the ability arXiv:2604.12994v1 [cs.CR] 14 Apr 2026 to generate high-quality outputs across tasks such as text summarization (Jin et al., 2024; Urlana et al., 2024) and question answering (Saito et al., 2024; Yixing et al., 2024). Due to similarities between code and natural language, researchers have ex- plored LLMs for code generation (Ni et al., 2025; Liu et al., 2024a), analysis (Fang et al., 2024), and repair (Jin et al., 2023; Kulsum et al., 2024; Wang et al., 2024). LLMs can capture program syntax and semantics and reason about program behav- iors. Several LLM-based approaches have been proposed to localize vulnerabilities (Li et al., 2024), generate repair patches (Jin et al., 2023; Kulsum et al., 2024; Pearce et al., 2023), and assess patch correctness (Zhou et al., 2024), suggesting that LLMs may be promising for repairing logical vul- nerabilities. However, to our knowledge, there is no system- atic framework for analyzing the capabilities and limitations of automated vulnerability repair (AVR) for logical vulnerabilities. A targeted evaluation is critical for advancing AVR beyond memory-safety issues and toward the more subtle domain of log- ical vulnerabilities. To this end, we develop a systematic framework for evaluating repair tech- niques for logical vulnerabilities. We examine the performance of state-of-the-art non-learning-based, learning-based, and off-the-shelf LLM approaches, and study how prompting strategies, repair com- plexity, and auxiliary vulnerability information in- fluence repair outcomes. We also identify the key challenges LLM-based approaches must address to improve performance. Unfortunately,existingvulnerability datasets (Just et al., 2014; Gao et al., 2021; Fan et al., 2020; Jimenez et al., 2023) primarily focus on memory safety bugs and lack representa- tive samples of real-world logical vulnerabilities, limiting their usefulness for our purposes. To address this gap, we constructLogicDS, a curated dataset of 43 real-world logical vulnerabilities including vulnerable and fixed code, vulnerability descriptions, and, when available, behavioral specifications and developer-authored repair rationales. Building on this dataset, we designLogicEval, an end-to-end evaluation framework that takes lo- calized vulnerable code and auxiliary inputs (e.g., context, specifications, repair descriptions) and evaluates AVR techniques, including both tradi- tional tools and LLM-based methods, on their abil- ity to synthesize correct patches.LogicEvalsup- ports diverse prompt configurations and enforces structured patch outputs for seamless grafting into source code. It evaluates patches using compila- tion and testing pipelines and introduces automated reasoning to assess patch quality. Together, these capabilities enableLogicEvalto systematically characterize traditional and LLM-based AVR ap- proaches, as well as off-the-shelf LLMs, for patch generation and assessment on logical vulnerabili- ties. Through extensive empirical analysis, we iden- tify key strengths, limitations, and failure modes of LLMs and offer insights to guide future AVR development for logical vulnerabilities. 2 Logical Vulnerabilities A logical vulnerabilityθis a logical flaw in a pro- gramPthat leads the program to deviate from an expected behaviorEand compromises its security in terms of confidentiality, integrity, or availability. 1 typedef struct 2 ... 3++ #define CHACHA20_POLY1305_MAX_IVLEN 12 4 ... 5 6 static int chacha20_poly1305_ctrl(EVP_CIPHER_CTX ,â *ctx , int type , int arg) 7 ... 8 return 1; 9 10 case EVP_CTRL_AEAD_SET_IVLEN: 11â if (arg <= 0 || arg > CHACHA_CTR_SIZE) 12++ if (arg <= 0 || arg > CHACHA20_POLY1305_MAX_IVLEN) 13 return 0; 14 actx ->nonce_len = arg; 15 return 1; 16 ... Listing 1: Fix for CVE-2019-1543 (cve, 2019a). Lines prefixed with++denote additions, whereasâindicate deletions. Listing 1 shows a real-world logical vulnerability in OpenSSLâschacha20_poly1305_ctrl, which handles ChaCha20-Poly1305 control operations. RFC 7539 (rfc) requires a unique 96-bit (12-byte) nonce. However, OpenSSL allows variable-length nonces, pads those shorter than 12 bytes with lead- ing zeros, and even accepts lengths up to 16 bytes, using only the last 12 bytes and silently discard- ing the rest. This improper validation of thearg parameter risks nonce reuse and potentially enable severe cryptographic attacks (cve, 2019a). Expected behavior of logical vulnerabilities are often described by design specifications (e.g., RFCs) or implementation documentations. How- ever, they can stem from implicit knowledge as well (e.g., privilege escalation in CVE-2024-25420 (cve, 2024)). Logical vulnerabilities may compromise an implementationâs security assumptions even when the program is free from memory-safety issues. Logical vulnerabilities usually stem from incorrect or missing control flow or validation logic (e.g., skipping required state transitions, omitting essen- tial preconditions, placing checks in the wrong lo- cation, or accepting values outside the intended range). They do not exhibit consistent fix patterns. As such, resolving them requires a deep understand- ing of the program logic and its intended behavior. 3 Related Works AVR approaches. Automatic Vulnerability Re- pair (AVR) techniques aim to propose a patch that (i) eliminates an identified vulnerability and (i) preserves the original functionality after the fix. Existing AVR approaches can be broadly classified into non-learning-based, learning-based, and LLM- based approaches. Among non-learning-based AVR approaches, template-guided approaches (Shaw et al., 2014; Huang et al., 2019; Xing et al., 2024; Zhang et al., 2022) rely on patterns of vulner- ability properties or historical patches to generate patches, and thus usually excel where known fix templates can be reliably applied. In contrast, fix for logical vulnerabilities typically lack such com- mon patterns. Constraint-based approaches extract program constraints through static analysis (Oh, 2018; Chida and Terauchi, 2022), symbolic execu- tion (Shariffdeen et al., 2021), or dynamic analysis (Xuan et al., 2016; Agrawal and Horgan, 1990) against a set of test suites. However, typically each logical vulnerability has its own distinct expected behavior that does not necessarily map to program structure and often provides only limited exploit traces. Search-based approaches (Le Goues et al., 2012; Marginean et al., 2019; Jiang et al., 2018; Le et al., 2016) explore a hypothetical search space for repair through mutations in existing code or by ex- tracting similar code from other functions or source files within a project, however such mutations per- form poorly for real-world logical vulnerabilities. Deep learning-based AVR approaches (Chen et al., 2019; Jiang et al., 2021, 2023) treat program re- pair as a neural machine translation task. Although these approaches perform better than non-learning- based approaches, they require a dataset of vulnera- bilities and their fixes for training and attempt to ex- tract recurring repair patterns from source code. Re- cently, several LLM-based approaches have been proposed to generate repair patches (Jin et al., 2023; Kulsum et al., 2024; Pearce et al., 2023). LLMs ap- pear promising for repairing logical vulnerabilities Vulnerable Source Code Vulnerability Description Ground-Truth Fix Issue Report Patch Patch Location Identification Patch Generation Tool Suggested Patch Patch Grafting Judge LLM Patch Explanation Suggested Patch Reasoning Compilation & Testing Specification Patched Code Manual Judge Patch Generation Patch Evaluation Figure 1: Overview of LogicEval because they can understand security invariants, though they face limitations when evaluating com- plex constraints. However, to our knowledge, no prior research has systematically analyzed their ca- pabilities and limitations in this specific context. Further details regarding existing AVR techniques and their limitations are provided in Appendix A. AVR evaluation frameworks. Existing evalua- tions on LLMs for security vulnerabilities assess the security of generated code (Pearce et al., 2025), study LLM assistance in writing code (Sandoval et al., 2023), or evaluate LLMs for vulnerability identification and related security tasks (Deng et al., 2024; Ullah et al., 2023). Prior work also provides taxonomies and benchmarks for AVR approaches (Li et al., 2025), but does not address logical vul- nerabilities or the limitations of existing AVR ap- proaches for them. Most related to our work is that of Pearce et al. (Pearce et al., 2023), which evaluates the performance of LLMs towards repair- ing security vulnerabilities with zero-shot prompts. However, their evaluation framework does not take auxiliary vulnerability information as input, lacks automated metrics to identify reasonable patches, and uses address sanitizers and CodeQL to test patches, which are not suitable for evaluating logi- cal vulnerabilities. 4 LogicEval: Evaluation Framework To address limitations of existing evaluation ap- proaches, we developLogicEval, a systematic end-to-end framework that automatically generates and evaluates patches for logical vulnerabilities. We also define metrics to assess patch correctness and quality. Using the generated patches and evalu- ation results,LogicEvalenables analysis of differ- ent AVR approachesâ capabilities and limitations. Figure 1 outlines the LogicEval workflow. 4.1 Inputs for LogicEval We assume the following inputs to LogicEval: Vulnerable (S) and Fixed (F) Source Code. We provide code from the vulnerable implementation source codeSas input to the repair framework under evaluation. After patch generation, we as- sess patch quality by comparing generated patches against a ground-truth patch F . Vulnerability Description (D).We assume a natural text description of the vulnerability is available to clarify how it leads to deviations from expected behavior and violates security guarantees. Behavioral Specification (V S ). In some experimen- tal settings, we additionally provide natural-text be- havioral specifications (e.g., extracted from RFCs or formal implementation documentation). Since such specifications are not always available for all logical vulnerabilities, this input is optional and is only provided in certain experiments. Context (V ctx ). We define contextV ctx as local variable declarations at the start of the vulnerable function and global/static definitions at the top of the file/class. We optionally provideV ctx in some experiments, as it may help repair techniques gen- erate more plausible fixes. Compilation (C) and Testing (T) Scripts. We pro- vide a script to compile the full project with any suggested patch. If available, we also provide a test- ing scriptTthat runs existing testcases against the project, often including a proof-of-concept (PoC) exploit to check whether the vulnerability persists. 4.2 Patch Localization To generate a patch for a logical vulnerability, a developer must first locate the code segment that re- quires modification (i.e., patch localization). Most existing AVR approaches (Jiang et al., 2023; Kul- sum et al., 2024; Jiang et al., 2021; Lutellier et al., 2020) either assume perfect localization or rely on automated fault localization. Prior work on fault localization (Campos et al., 2012; Jones and Harrold, 2005; Abreu et al., 2007) mainly targets general bugs or memory-safety issues and is not de- signed for logical vulnerabilities, which are harder to localize, as automated techniques (e.g., address sanitizers) cannot identify code that triggers them. Moreover, no existing method achieves perfect ac- curacy, and localization errors frequently lead to incorrect or ineffective AVR patches (Jiang et al., 2018, 2023). Furthermore, both existing AVR tech- niques and off-the-shelf LLMs (§6.2) already strug- gle with logical vulnerabilities. To isolate patch- generation performance, we restrict our evaluation to single-hunk fixes within a single function and as- sume perfect patch localization by manually iden- tifying the precise vulnerable region. However, real-world patches for security- relevant logical vulnerabilities often span multiple locations. We observe that most such patches con- tain a core fix: a contiguous block of statements that implements the essential logic to eliminate the vulnerability, while other changes (e.g., vari- able declarations, helper functions, or supporting assignments) primarily support this core fix (ex- amples provided in Appendix G). Hence, we treat repairing the core fix as addressing a single-hunk vulnerability, motivated by (i) evidence that AVR performs better on single-hunk patches (Li et al., 2025) and (i) the intuition that multi-hunk vulnera- bilities can often be handled by generating patches independently at each location. We provide addi- tional examples and our identification procedure in the Appendix (§B). 4.3 Patch Generation for LogicEval LogicEval leverages the provided inputs and lo- calized vulnerability to generate patches using the AVR technique under evaluationM. For baseline approaches (Table 4), we provide vulnerable source codeVand the required localization (either the vul- nerable blockV b or functionV s ). When evaluating off-the-shelf LLMs, we construct prompts along the following dimensions using the inputs in §4.1: Portion of source code. We provide either the vulnerable blockV b or vulnerable functionV f ; in some experiments, we also include context V ctx . Auxiliary information. To assist LLM-based re- pair, we provide one or more of the vulnerability descriptionD, behavioral specificationS, and re- pair descriptionR. We vary these inputs across experiments §6.2.3 to measure how auxiliary infor- mation affects patch correctness and quality. LLM Configuration. We evaluate prompting strate- gies including zero-shot, few-shot, and chain-of- thought. We also configure temperature, and also explore task-oriented and role-oriented prompting styles. Patch grafting. While constructing prompts, we in- struct the LLM to place the suggested repair within tags and to output code that can directly replace the provided vulnerable function or block. After re- ceiving the output, we extract the tagged code and replace the annotated vulnerable block or function. Obtaining explanation for suggested patch. After each patch-generation, we ask the LLM in a follow- up prompt to provide a brief natural-language ex- planation of the suggested patch. The resulting explanationEis used for reasoning-based evalua- tion. 4.4 Patch Evaluation for LogicEval Evaluating patches for logical vulnerabilities poses unique challenges. Although automated patch vali- dation techniques exist (Tan et al., 2016; Xin and Reiss, 2017; Le-Cong et al., 2023), most rely on static code features or dynamic testing with pre- defined testcases or known exploits. Since logical vulnerabilities do not follow any pattern for iden- tification or repair, extracting features that capture their essence is difficult, thus limiting the applica- bility of existing automated patch evaluation tech- niques. Learning-based approaches have also been proposed (Tian et al., 2022, 2023), but the lack of large, high-quality datasets of logical vulner- ability patches constrains their effectiveness (Li et al., 2025). More recently, LLM-based valida- tion techniques (Zhou et al., 2024) have emerged, yet their reliability remains questionable due to dif- ficulty in handling complex constraints (Li et al., 2025; Liu et al., 2024b). Several works employ CodeQL queries (cod, 2021) for testing (Sandoval et al., 2023; Pearce et al., 2025, 2023), but craft- ing queries that capture logical vulnerabilities is non-trivial. Moreover, detecting and verifying logical vul- nerabilities often requires a holistic view of com- ponent interactions, precise tracking of data flows and dependencies, and rigorous enforcement of se- curity policies. These remain open challenges for automated validation of security patches (Li et al., 2025). Consequently, manual inspection, though labor-intensive, remains the most dependable way to confirm that a proposed patch addresses the un- derlying logical vulnerability without introducing side effects (Li et al., 2025; Pearce et al., 2025; San- doval et al., 2023). Based on these considerations, LogicEvalevaluates compilation and testing and reasoning to evaluate patches, as discussed below. 4.4.1 Compilation and Testing LogicEval first checks whether a suggested patch is compilable using the compilation scriptC. Be- cause we focus on the core fix (§4.2), we graft the suggested patchZonto the localized function F f or blockF b over the ground-truth fixed imple- mentationF, ensuring that auxiliary modifications needed for compilation and testing are present and we only focus on the core fix. If compilation suc- ceeds, the patch is compilable. If a testing script Tis available,LogicEvalruns it on compilable patches; a patch that passes all tests is plausible. This is the best automated outcome for any AVR framework. However, a plausible patch may still be incorrect (e.g., overfitting to provided tests). There- fore, we manually inspect all plausible patches to determine whether they are correct, i.e., manually verify whether they eliminate the vulnerability and introduce no unintended side effects. A patch that is not plausible cannot be correct. 4.4.2 Reasoning Aside from checking compilation and testing, an- alyzing the steps a patch performs to fix the vul- nerability is essential for logical vulnerabilities, especially to assess how reasonable the patch is as a suggestion to the developer. Thus,LogicEval also evaluates how closely a patch follows the steps needed to repair the vulnerability. Unfortunately, no technique currently exists to objectively and automatically evaluate suggested patches. To address this gap, we LLM-assisted rea- soning metrics similar to those used in prior work on vulnerability identification (Ullah et al., 2023). LLMs are known to understand program semantics well. As such, it can identify the similarity in ratio- nale among steps taken to repair a vulnerability in a suggested patch and a ground-truth patch. A high similarity would indicate that the candidate patchâs rational closely aligns with the ground-truth, which we use as a proxy for reasonable patches. Concretely,LogicEvaluses LLMs to compare the natural-language explanationEof each candi- date patch against a reference explanationE g of the ground-truth fix. We first prompt a âjudgeâ LLM in a zero-shot setting with the vulnerability descrip- tionD, vulnerable blockV b , and ground-truth fix blockF b to deriveE g . When the patch-generating LLM outputs a suggested fix, it also providesE (§4.3). We then compute reasoning metrics by com- paringEandE g using NLP-based measures such as cosine similarity (CS) (Ullah et al., 2023). We also use a judge LLM (Ullah et al., 2023) to pro- vide a binary verdict (J) on whetherEandE g are similar. We also evaluate these reasoning metrics in a separate experiment Section 6.4. 5 LogicDS : Logical Vulnerability Dataset Previous AVR approaches curated different bench- mark datasets for different programming languages such as Java (Just et al., 2014; Bui et al., 2022; Lin et al., 2017), C/C++ (Gao et al., 2021; Nikitopoulos et al., 2021; Fan et al., 2020) and Python (Jimenez et al., 2023). However, we observe that these datasets contain very few logical flaws, and most have no clear security impact (e.g., the changing- offset issue in the Time-3 Defects4J sample (tim)). We, therefore, focus on security-relevant logical vulnerabilities and buildLogicDS, a new dataset of 43 real-world cases. We identify vulnerabilities caused by deviations from expected behavior that undermine security guarantees by examining CVEs across 9 popular open-source implementations. For each sample, we use the CVE description asD, the fix commit as the fixed versionF, and its parent commit as the vulnerable versionV. We manually localize the core-fix vulnerable function and block (V f ,V b ), contextV ctx (§4.1), and the correspond- ing fixed function and block (F f ,F b ). We create a compilation scriptC, extract relevant specification textSwhen available, and collect testcases to build a test scriptTwhere applicable. All these steps were performed manually to ensure the correctness and reliability of the ground truth dataset. Con- structing each vulnerability data point tookâź10 person-hours, and the entire dataset requiredâź430 person-hours. To obtain a repair descriptionR, we use LLMs by providingD,V b , andF b , asking for a natural-language description of the repair steps, and then manually refining it. Synthetic Java Samples. Many state-of-the-art repair techniques (Jiang et al., 2018, 2023; Cam- pos et al., 2012) are tailored to Java constructs and adopt Java-based fault localization, driven by the popularity of Defects4J (tim) and Vul4J (Bui et al., 2022). This limits their direct applicability to other languages (e.g., C/C++). To facilitate log- ical vulnerability repair research and benchmark Java-focused approaches (Jiang et al., 2018, 2023; Kulsum et al., 2024), we also construct 43 syn- thetic Java samples derived from our real-world cases, each includingV,F, and testcases. Cura- tion details are provided in Appendix (§C). 6 Evaluations & Observations In this Section, we present the results and key find- ings from ourLogicEvalevaluation. We analyze baseline AVR approaches and also off-the-shelf LLM prompting by varying source code, auxil- iary information, and LLM configuration on both real-world and synthetic (LogicDS) vulnerabilities. Note that, for each claim presented in this Sec- tion, we also performed statistical significance tests for their justification, which are provided in Sec- tion F in the Appendix. The code forLogicEval, LogicDSdataset, and the generated patch sugges- tions are publicly available in (log, 2026). 6.1 Performance of Baseline Approaches This experiment evaluates the performance, gen- eral capabilities, and limitations of existing au- tomatic vulnerability repair (AVR) approaches. We use three state-of-the-art approaches to gener- ate patches for logical vulnerabilities inLogicDS: SimFix (Jiang et al., 2018), a non-learning-based method; KNOD (Jiang et al., 2023), a deep learning-based technique; and VRPilot (Kulsum et al., 2024), an LLM-based program repair ap- proach. Since SimFix and KNOD are Java-focused repair approaches, we only evaluate them on syn- thetic Java samples. On the other hand, we evaluate VRPilot on both real-world and synthetic samples ofLogicDS. Detailed adoption process of the base- lines are discussed in Appendix (§D). We present the compilation and testing results, and automated reasoning scores of the VRPilot ap- proach in both real-world and synthetic examples in Table 1 and 2, respectively. The results for Sim- Fix and KNOD in our synthetic Java examples are provided in Table 2. VRPilot, being an LLM-based approach, performs better compared to other base- line approaches. However, its CoT based approach has a lower reasoning score compared to zero-shot prompting techniques, which we discuss in §6.2. Both SimFix and KNOD mostly generate patches that are unreasonable and use incorrect logic for re- pair. E.g., patches from SimFix (Jiang et al., 2018) and from KNOD (Jiang et al., 2023) generate to- tally unreasonable logic. 6.2 Performance of Off-the-shelf LLMs This experiment aims to evaluate the performance of off-the-shelf LLMs to repair logical vulnerabil- ities usingLogicEval. Specifically, we evaluate the performance of LLMs along three dimensions when prompting for logical vulnerabilities: LLM configuration, source code, and auxiliary informa- tion. Examples for manual inspection are provided in Appendix (§G). LLMs considered and default configurations. We leverage three popular off-the-shelf LLMs: Meta Llama 3.1 (Grattafiori et al., 2024), Qwen 2.5 (Hui et al., 2024), and OpenAI o3-mini (OpenAI, 2025). For each experiment, each LLM is used to gen- Table 1: Compilation, testing, and reasoning score re- sults for real-world vulnerabilities of LogicDS. Llama-3.1Qwen-2.5OpenAI-o3-mini EPCT CSQ CSO JQ JOCT CSL CSO JL JOCT CSL CSQ JL JQ LLM Configuration P10.290.110.840.780.070.420.610.050.810.760.580.580.580.040.810.810.790.26 P20.340.080.820.780.090.330.670.050.820.750.510.51â P30.430.070.830.780.090.40.580.050.810.760.510.53â P40.370.070.830.750.090.530.610.050.80.720.490.510.640.050.810.820.770.47 P5 0.420.040.840.780.140.510.470.040.810.760.490.470.610.050.80.80.740.3 P60.430.00.840.760.190.420.580.050.80.740.560.60.580.00.480.480.370.14 P70.310.00.780.750.090.330.530.040.770.780.60.490.50.080.770.760.810.4 P8 0.640.050.710.690.020.070.720.060.680.660.090.070.640.050.640.630.230.05 Source Code P90.420.080.840.780.070.470.530.080.820.770.560.510.640.140.80.810.790.3 P100.440.080.810.780.00.370.590.10.810.770.610.590.680.150.780.780.730.37 P110.360.070.820.780.070.490.560.130.830.770.490.490.690.140.810.80.790.37 Auxilliary Information P120.610.050.70.650.00.120.720.050.650.620.120.160.610.050.660.660.210.07 P130.440.040.830.760.090.440.530.040.810.760.630.470.610.130.80.80.840.44 P14 0.360.00.830.790.160.560.420.00.820.760.770.650.530.090.810.80.810.47 P150.560.090.830.810.60.930.670.10.830.780.930.950.530.130.790.790.950.72 P160.550.00.830.810.451.00.550.00.820.790.970.940.420.050.780.761.00.65 Existing Technique P170.360.030.590.530.00.050.190.00.640.620.020.050.330.040.580.580.020.0 P18 0.250.00.650.610.00.050.190.00.630.60.050.050.280.070.650.640.070.02 P19 0.250.00.650.60.020.070.190.00.630.610.050.070.220.00.630.630.090.02 P20 0.220.00.720.660.070.190.190.00.660.640.280.190.280.00.710.710.420.05 P210.170.00.820.760.090.510.190.00.760.720.720.670.190.00.780.780.810.23 VRPilotâ0.810.090.650.660.020.06 âCâ and âTâ represent the % of generated patches successfully compiled and passed all tests, respectively. âCSâ represents average cosine similarity among suggested patches and ground-truth explanations, âJâ represents the % of patches a judging LLM determined similar to ground-truth fix. Suffixes âLâ, âQâ, and âOâ after each âCSâ and âJâ represent judging LLM Llama, Qwen, and OpenAI- o3-mini, respectively. âEâ represents the dimensions of experiments, and âPâ represents the prompt IDs. Table 2: Compilation, testing, and reasoning score re- sults for synthetic vulnerabilities of LogicDS. Llama-3.1Qwen-2.5OpenAI-o3-mini EPCT CSQ CSO JQ JOCT CSL CSO JL JOCT CSL CSQ JL JQ LLM Configuration P10.260.020.830.760.070.470.120.050.810.770.450.450.30.140.80.810.810.4 P20.260.050.820.760.120.450.10.020.790.750.480.48â P30.160.020.810.760.070.40.10.020.790.750.480.45â P40.190.050.830.760.070.470.170.020.780.730.50.50.330.140.810.810.70.26 P50.260.020.830.760.070.420.120.050.80.750.60.470.330.120.790.80.650.26 P60.210.050.820.740.120.370.20.070.780.750.510.390.370.160.520.520.30.12 P7 0.120.020.760.760.070.330.160.050.740.750.370.420.230.070.740.750.70.28 P8 0.510.020.690.670.00.050.490.00.620.60.020.070.530.070.620.620.190.05 Source Code P90.210.020.830.760.090.40.140.020.810.760.530.530.230.120.80.790.670.23 P100.370.090.780.760.090.370.330.160.790.750.440.510.470.230.770.770.790.3 P110.210.070.820.770.050.420.280.090.810.760.490.470.470.160.780.780.770.26 Auxilliary Information P120.560.020.680.620.00.020.470.00.590.560.050.050.470.070.620.620.120.0 P130.160.020.830.770.050.370.120.020.80.760.440.440.350.120.780.780.650.23 P140.070.020.80.790.140.530.120.020.790.760.740.650.120.050.80.790.810.26 P150.440.30.820.810.650.950.650.580.790.780.980.980.40.330.770.770.950.67 P160.450.260.810.810.710.90.610.550.780.790.971.00.350.260.760.760.940.58 Existing Technique P170.260.00.580.530.00.00.00.00.60.570.070.050.280.020.580.580.020.0 P180.120.00.630.610.00.070.00.00.610.580.020.020.120.00.580.590.090.02 P190.210.00.620.620.00.020.00.00.630.610.050.050.140.020.60.60.090.02 P200.050.00.70.660.090.30.00.00.660.620.230.30.070.020.690.690.420.19 P210.020.020.80.750.190.440.00.00.750.710.650.580.050.00.770.770.840.21 Baseline SimFixâ0.070.010.630.640.020.0 KNODâ0.480.030.640.660.020.0 VRPilotâ0.320.020.650.660.150.03 Column meanings are the same as in Table 1. erate and evaluate patches usingLogicEvalfor each vulnerability. Note that, our work aims not to benchmark all LLMs to repair logical vulnera- bilities, but rather to identify the capabilities and limitations of LLMs in general towards repairing logical vulnerabilities. In each experiment, we adjust one of the dimen- sions, and set the other dimensions to their defaults. We use a temperature of 0.2, role orientation, and zero-shot prompts as the default LLM configura- tion as they are the best-performing parameters (or at least equally performing compared to other choices) in their respective dimension as per our experiments described in detail below. Also, we use the vulnerable blockV b as the default provided vulnerable source code and the vulnerability in- formationDas the default provided auxiliary in- formation while varying configurations of other dimensions. We provide the prompt templates used in our experiments in Appendix (§E). For each LLM, we leverage the other two LLMs as judge LLMs, and obtain the cosine similarity scores among explanations, and also judging ver- dicts (§4.4). We display the average cosine similar- ity scores and percentage of patches whose expla- nation aligned with the ground-truth fix according to each judge-LLM. 6.2.1 Impact of Adjusting LLM configurations To assess the impact of LLM configurations on patch generation performance, we adjust the tem- perature, orientation, and prompting strategy of LLMs with the same prompt contents. We provide the vulnerable blockV b , the vulnerability descrip- tionD, and ask the LLM to suggest the patch using LogicEval. To assess the effect of adjusting LLM temperature, we run the patch with temperatures 0.2 (low), 0.5 (medium), and 0.9 (high) (prompts P1, P2, and P3, respectively). To assess the ef- fect of adjusting LLM orientation, we adopt a task- oriented prompt in P4. Finally, to assess the effect of adjusting prompt- ing strategy, we consider the following: (i) zero- shot prompt P5 havingV b andD, (i) few-shot prompt P6, where in addition toV b andD, we take a real-world sample fromLogicEval, and provide itsD,V b and ground-truthF b as an exam- ple fix, (i) chain-of-thought (CoT) based prompt P7, where in the first prompt we provideV b ,D and ask for stepsR suggested to repair the vulnera- bility; then, in the next prompt we provideV b and R suggested to obtain a patch, and (iv) CoT based prompt P8, similar to P7 except we do not include R suggested in the second prompt. We present the compilation and testing results, and results of the average reasoning scores for all the prompts in Table 1 and 2, for both real-world and synthetic logical vulnerability examples in LogicDS, respectively. We observe that the compi- lation, testing, and reasoning scores for P1, P2, and P3 are similar (we cannot adjust the temperature of openAI-o3-mini), indicating there is minimal effect of logical vulnerability repair performance based on adjusting temperature. Similarly, the scores for prompt P4 are also similar to P1, indicat- ing there is minimal effect of logical vulnerability repair performance based on orientation as well. When evaluating with different prompting strate- gies, we observe that usually zero-shot based prompt P5 obtains higher compilation percent- age and reasoning scores compared to CoT based prompts, especially in real-world examples. We observe that several compilation errors occur for CoT based prompt P7 because it tries to create extra undefined variables based on the reasoning suggestion obtained from its first prompt (as shown in Listing 9). The prompt P8 omits all auxiliary information and relies solely on the vulnerable source code (V b ). As a result, it generates more compilable patches but achieves significantly lower reasoning scores than both P5 and P7. This indicates auxiliary in- formation is required alongside the source code to achieve reasonable patches. Scores for few- shot based prompt P6 are usually marginally less than zero-shot prompts over the three LLMs; we manually did not observe any contrasting behavior pattern between the two strategies. 6.2.2 Impacts of Adjusting Source Code We evaluate the advantages and disadvantages of providing different portions of vulnerable source code, which is manually localized (§4.2). We ei- ther provideV b , where the core fix is required, or V f . ProvidingV b allows more fine-grained localiza- tion, whereasV f provides broader context. We first compare two prompting strategies: P9, which sup- plies only the vulnerable blockV b , and P10, which supplies the entire vulnerable function V f . Across both real-world and synthetic datasetsâand for all LLMs tested, P10 achieves marginally higher compilation and testing success rates, while P9 has marginally better reasoning scores. Manual inspection reveals that when only the vulnerable block is provided, LLMs sometimes cannot resolve the correct variable names and in- stead introduce placeholder variables, causing com- pilation failures 10. Supplying the full function context may mitigate such placeholder variable us- age. Conversely, when functions are very large, giving the entire function can make it harder for the LLMs to locate the exact block that needs to be modified, occasionally resulting in unreasonable patch suggestions (as in Listing 11), suggesting that providing only a vulnerable block without context sometimes leads LLMs to create place- holder variables. In contrast, when the entire vulnerable function is provided, LLMs sometimes fail to locate the exact block to modify to fix the vulnerability. We also consider another prompting strategy P11, which provides context (§4.1) and compare the performance with P9. We observe a marginal increase in compilation pass rate across all LLMs and sample types (both real-world and synthetic) except for Llama patches over real-world samples. In contrast, reasoning scores do not improve much for the same. This suggests that providing context somewhat facilitates the compilation process. 6.2.3 Impacts of Adjusting Auxiliary Information The goal of this experiment is to observe to what extent different auxiliary information facilitates re- pairing logical vulnerabilities with LLMs. Hence, we consider different combinations of auxiliary information in each of the five different prompts: prompt P12 having no auxiliary information (i.e., it only provides the vulnerable blockV b to LLMs and ask for a patch), prompt P13 provides only the vul- nerability informationD, prompt P14 additionally provides a specification text (if available, if not this defaults to P13), prompt P15 provides the repair stepsRalongside the vulnerability informationD, and prompt P16 only provides the repair stepsR, without the vulnerability information. The compilation, testing, and reasoning results are shown in Table 1 and 2 for real-world and syn- thetic samples, respectively. Here, first, we observe that prompt P12 has a much higher compilation rate, but much lower reasoning scores compared to other prompts. Upon manual inspection, we find that without any auxiliary information, LLM tends to treat the vulnerable source code as a possible memory vulnerability issue (as in this use-after- free example provided in Listing 13) and suggests patches to fix those vulnerabilities. These patches pass compilation, but doesnât meaningfully attempt to repair the logical vulnerability. This suggests that Auxiliary information about the vulnerabil- ity (e.g., vulnerability description, specification text, or repair steps) is essential for generating accurate patches for logical vulnerabilities. Furthermore, we observed that in some cases, since specification texts usually describe expected behavior in a different abstraction to the implemen- tation, LLMs tend to create undeclared variables adopted from the text (example provided in Table 14), which is a similar phenomenon we observed while evaluating prompt P7. Consequently, Prompt P13 has much higher reasoning scores compared to prompt P12, and prompt P14 has similar rea- soning scores but marginally less compilation rate compared to prompt P13. 6.2.4 Overall Observations Off-the-shelf LLMs perform significantly better when provided auxiliary information outperform- ing other baseline approaches. However, the overall performance leaves room for improvement, espe- cially on real-world environment where the LLMs generated correct patch for only 5 out of 43 sam- ples. However, the observations suggest that repair prompts should incorporate localization cues and more relevant context information related to the vulnerability. To reduce hallucinations, any text included in the prompt should be highly specific and aligned with the actual implementation. 6.3 Comparison with Other Evaluation Works Among existing evaluation frameworks for vulner- ability repairing, Pearce et al. (Pearce et al., 2023) propose a framework to evaluate LLM-generated patches for security vulnerability repair. In this ex- periment, we use their prompt templates (Table IV in (Pearce et al., 2023)) to generate patches for log- ical vulnerabilities (prompts P17 through P21) in Table 7. Note that, in these templates, we provide the source code according to the process prescribed in (Pearce et al., 2023). Except for P21, since the other four prompts (P17-P20) do not have informa- tion regarding the vulnerability, they often fail to generate reasonable patches, indicated by a lower reasoning score in Table 1 and 2. Prompt P21 uses the vulnerability description as the message, and consequently achieves similar reasoning scores as P1. This further proves that by failing to incorpo- rate auxiliary information, the prompt templates proposed by Pearce et al. are unable to generate more reasonable patches for logical vulnerabilities. 6.4 Effectiveness of Evaluation Metrics for Reasoning In this experiment, we assess how much our auto- matically calculated reasoning metrics align with human assessments of patch quality and, in turn, their reliability. We randomly selected 200 off- the-shelf LLM-generated patches from our off-the- shelf evaluation and manually annotated them as reasonable or unreasonable using the criteria in Section 4.4.2. Two security-knowledgeable an- notators independently labeled each patch as rea- sonable or unreasonable, blinded to all automated scores. Disagreements were rare (<5%) and re- MetricPrecisionRecallF1 ScoreAccuracy Compilation0.4240.60.4970.571 J-Qwen0.7730.6670.7160.8 J-OpenAI0.50.7210.590.701 J-LLaMA0.5470.8390.6620.677 CodeEmbed 70th (0.944)0.470.5080.4880.622 CodeEmbed 90th (0.981)0.6150.2620.3680.68 RougeL 70th (0.822)0.4520.40.4240.62 RougeL 90th (0.957)0.6820.2140.3260.69 CodeBLEU 70th (0.708)0.3520.3570.3550.545 CodeBLEU 90th (0.906)0.5790.1570.2470.665 Table 3: Precision, recall, F1 score, and accuracy across metrics. solved through discussion. We evaluate (i) compila- tion success, (i) the LLM-as-a-judge binary verdict (J), (i) cosine similarity between patch code em- beddings computed with Microsoft/unixcoder-base, (iv) ROUGE-L over patch, and (v) codeBLEU over patch. For each continuous metric, we treat patches above a percentile threshold as inferred reasonable, and report both the 70th and 90th percentile thresh- olds, including the corresponding threshold values. Table 3 reports precision, recall, F1 score, and ac- curacy for all metrics. We observe that compilation success is the weak- est indicator of patch reasonableness, with the low- est accuracy (0.571). In contrast, the LLM-as-a- judge metric using Qwen achieves the highest ac- curacy (0.8). While several metrics achieve higher accuracy at the 90th percentile threshold, this is largely driven by class imbalance, as the dataset is naturally skewed toward the unreasonable patch class (130/200). Conservative thresholds, therefore, tend to label most patches as unreasonable, which can inflate accuracy. Overall, these results suggest that LLM-as-a-judge is a reasonably reliable met- ric that aligns most closely with human judgments, although there remains room to improve automated reasoning evaluation. 7 Conclusion We introducedLogicEval, a systematic evalua- tion framework that leverages LLMs for patch validation. We also curatedLogicDS, a bench- mark dataset of logical vulnerabilities. Using LogicEval, we evaluated traditional AVR tools, LLM-based systems, and off-the-shelf LLMs across multiple dimensions and identified some key takeaways that would be helpful for future re- search. Limitations We discuss some of the limitations ofLogicEval, and our constructed dataset LogicDS below. Size of LogicDS. Our benchmark dataset,LogicDS, consists of 43 logical vulnerabilities drawn from real-world open-source projects. We focus on real vulnerabilities to reflect the conditions develop- ers actually face and to measure how well exist- ing techniques perform outside of synthetic set- tings. For our work, since we require provide full source code for evaluation, end-to-end validation of patches through compilation and, when avail- able executable-based testing, we are forced to limit ourselves to open-source projects to construct LogicDS. We curate samples by prioritizing (i) popular and widely used projects (e.g., OpenSSL, GnuTLS), (i) logical vulnerabilities with direct se- curity impact (e.g., privilege escalation, location tracking, DoS), and (i) publicly available commits for both the vulnerable version and its patch, along with build instructions and, if available, testing scripts. Constructing each sample requires substantial manual effort. For each selected project, we scan CVEs, identify issues that match our definition of logical vulnerabilities, confirm that a correspond- ing patch is publicly available, and then manually locate the core fix, as discussed in Section B. We additionally ensure the codebase compiles correctly (and run available tests when possible) so that each sample supports reliable compilation checking and practical evaluation. This high curation cost per sample make it challenging to scaleLogicDSto much larger sizes (e.g., thousands of samples). Assuming Perfect Localization and its Implica- tions. A real-world security vulnerability mitiga- tion workflow consists of four primary stages: (i) vulnerability detection, (i) root-cause analysis and localization, (i) patch generation, and (iv) valida- tion. Each stage introduces distinct technical chal- lenges, and extensive prior work exists across these phases, including automated detection (Ullah et al., 2023), localization (Yu et al., 2024), patch gener- ation (Li et al., 2025), and regression and security testing for validation (Zhou et al., 2024). However, none of these works produces the expected output (i.e., 100% sound and complete). As a result, all prior approaches focus on only one stage (e.g., lo- calization) of the vulnerability mitigation workflow and assume a perfect solution for prior stages (e.g., vulnerability detection). Consistent with prior work, this work aims to in- dependently evaluate the patch generation stage for logical vulnerabilities, isolating it from the effects of vulnerability detection and localization. Thus, we assume perfect vulnerability detection and lo- calization, which can be performed manually or semi-automatically by first applying existing auto- mated techniques and then refining them manually. Without this assumption, localization errors would propagate into the patch-generation stage, and fail- ures could stem from incorrect/partial localization rather than limitations of the patch generator itself, making it difficult to identify the true strengths and weaknesses of repair tools. This is why we adopted perfect localization rather than an end-to- end testing approach that also involves error-prone automated localization. Adoption of samples from existing repair dataset. Existing dataset curation efforts (Bhandari et al., 2021; Mei et al., 2024) collect CVEs and public commits based on CWE categories. However, sev- eral challenges limit their applicability for our pur- poses: (i) they do not explicitly target logical vul- nerabilities, which often do not map cleanly to specific CWEs, (i) they do not localize the patch in a commit, which often modifies several files at once where not all of the modifications are related to the vulnerability itself, (i) these datasets also typically do not directly provide test scripts, which are often provided in commits separate from the patch itself, and (iv) most of their collected sam- ples are memory-corruption vulnerabilities. Con- sequently, we adopted a manual curation approach to constructLogicDS. However, evaluation result of samples inLogicDSwas still enough to confirm the patterns observed in our evaluations as several patches demonstrated such behavior to confirm our observation. We plan to elaborateLogicDSin the future and also make the curation scripts and guide- lines publicly available for future elaboration and research. Adoption of older CVEs. A common concern for generating patches for known CVEs is that LLMs might simply reproduce the fixes they saw during training. However, studies (Jimenez et al., 2023) demonstrated that even if an LLM has encountered earlier versions of the code while training, it cannot memorize and regenerate those patches. Our patch validation analysis and manual observation also support this claim. Adopting ground-truth fix to reason patches. We measure reasoning by similarity to the ground-truth patch, which could undervalue correct fixes that implement the same logic differently. However, in our experiments, every plausible patch (i.e., a patch that passed both compilation and testing) scored above the80 th percentile in cosine similarity and was judged similar to the ground truth by the LLM, so we observed no such misclassifications. Variations in reasoning scores among LLMs. We observe that the reasoning scores differ when we use different judging LLMs. For example, Qwen as a judge provides lower reasoning scores across all prompts compared to Llama and OpenAI (Ta- bles 1 and 2). However, across different prompt approaches, the scores display the same trend, jus- tifying our assumptions based on the reasoning scores. Evaluating more baselines and LLMs. We pri- marily focused on repair baselines that can po- tentially produce patches that are plausible. Sev- eral categories of AVR approaches e.g., template- guided (Shaw et al., 2014; Huang et al., 2019), constraint-based (Chida and Terauchi, 2022; Xuan et al., 2016), and several search-based approaches (Marginean et al., 2019) are inapplicable to apply for repairing logical vulnerabilities primarily due to focusing on specific templates, struggling with path explosion when analyzing source code, or only being able to reuse existing source code. A detailed analysis of several categories of vulnerability re- pair approaches and their limitations in repairing logical vulnerabilities is presented Appendix A. Circularity Risk of Overlapping LLM Judges. In our automated reasoning metrics, we used LLMs as judges for many of the metrics (e.g., LLM as binary judge J). However, potentially overlapping LLMs for both patch generation and evaluation may introduce circularity. This can cause the same LLM to generate a patch and then evaluate its own patch suggestion, leading to bias (Panickssery et al., 2024; Wataoka et al., 2024). To avoid this, we ex- plicitly excluded any model from judging patches it generated. Concretely, a patch was evaluated only by Qwen3 and OpenAI models; a Qwen3- generated patch was judged only by Llama and OpenAI, and so on. Since patch generation and validation are performed independently, no LLM ever evaluates its own output in our experiments, mitigating the circularity concern. Manual efforts. We primarily require manual effort to (i) localize vulnerability and (i) evaluate patches for reasoning. To reduce the manual effort of in- specting all prompts, we use reasoning metrics to identify patches that score significantly higher un- der one prompt than the other. To uncover common failure patterns, we manually inspect a fixed subset of patches with low scores. Prioritizing patches according to their reasoning metrics allows us to significantly reduce the number of manually re- viewed patches. References https://github.com/rjust/defects4j/blob/ master/framework/projects/Time/patches/3. src.patch. [link]. Defects4J Time Patch 3.https://github.com/ rjust/defects4j/blob/master/framework/ projects/Time/patches/3.src.patch. Fixforcve-2014-0224.https:// github.com/openssl/openssl/commit/ bc8923b1ec9c467755cd86f7848c50e8812e441. National Vulnerability Database.https://nvd.nist. gov/. RFC 7539.https://datatracker.ietf.org/doc/ html/rfc7539. 2015.CVE-2015-1793.https://nvd.nist.gov/ vuln/detail/CVE-2015-1793. 2017.CVE-2017-3142.https://nvd.nist.gov/ vuln/detail/CVE-2017-3142. 2019a.https://nvd.nist.gov/vuln/detail/ CVE-2019-1543?cpeVersion=2.2. [link]. 2019b.https://github.com/ openssl/openssl/commit/ c62896c2c0cbd47ab01693d403e37fe5fe15aab8. [link]. 2019c.https://github.com/ openssl/openssl/commit/ e22257b1418438ebaf54df98af4e24f494d1809. [link]. 2019d.https://github.com/ openssl/openssl/commit/ a4f0b50eafb256b802f2724fc7f7580fb0fbabc. [link]. 2019e.https://github. com/openssl/openssl/blob/ c62896c2c0cbd47ab01693d403e37fe5fe15aab8/ crypto/evp/e_chacha20_poly1305.c. [link]. 2019f.https://github. com/openssl/openssl/blob/ e22257b1418438ebaf54df98af4e24f494d1809/ crypto/evp/e_chacha20_poly1305.c. [link]. 2019g. CVE-2019-1543.https://nvd.nist.gov/ vuln/detail/CVE-2019-1543. 2021. Codeql for research.https://securitylab. github.com/tools/codeql/. 2022.CVE-2022-1434.https://nvd.nist.gov/ vuln/detail/CVE-2022-1434. 2023a. [link]. 2023b. CVE-2023-48795.https://nvd.nist.gov/ vuln/detail/CVE-2023-48795. 2024. CVE-2024-25420.https://nvd.nist.gov/ vuln/detail/CVE-2024-25420. 2026. Logiceval repository.https://github.com/ SyNSec-den/LogicEval. Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007. On the accuracy of spectrum-based fault local- ization. In Testing: Academic and industrial confer- ence practice and research techniques-MUTATION (TAICPART-MUTATION 2007), pages 89â98. IEEE. Hiralal Agrawal and Joseph R Horgan. 1990. Dynamic program slicing. ACM SIGPlan Notices, 25(6):246â 256. Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. Cvefixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceed- ings of the 17th International Conference on Predic- tive Models and Data Analytics in Software Engineer- ing, pages 30â39. Quang-Cuong Bui, Riccardo Scandariato, and NicolĂĄs E DĂaz Ferreyra. 2022. Vul4j: a dataset of repro- ducible java vulnerabilities geared towards the study of program repair techniques. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 464â468. JosĂŠ Campos, AndrĂŠ Riboira, Alexandre Perez, and Rui Abreu. 2012. Gzoltar: an eclipse plug-in for testing and debugging. In Proceedings of the 27th IEEE/ACM international conference on automated software engineering, pages 378â381. Zimin Chen, Steve Kommrusch, Michele Tufano, Louis- NoĂŤl Pouchet, Denys Poshyvanyk, and Martin Mon- perrus. 2019. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Trans- actions on Software Engineering, 47(9):1943â1959. Xi Cheng, Min Zhou, Xiaoyu Song, Ming Gu, and Ji- aguang Sun. 2019. Automatic integer error repair by proper-type inference. IEEE Transactions on De- pendable and Secure Computing, 18(2):918â935. Nariyoshi Chida and Tachio Terauchi. 2022. Repair- ing dos vulnerability of real-world regexes.In 2022 IEEE Symposium on Security and Privacy (SP), pages 2060â2077. IEEE. Ganesan Deepa, P Santhi Thilagam, Amit Praseed, and Alwyn R Pais. 2018. Detlogic: A black-box ap- proach for detecting logic vulnerabilities in web ap- plications. Journal of Network and Computer Appli- cations, 109:89â109. Gelei Deng, Yi Liu, VĂctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024. PentestGPT: Evaluating and harnessing large lan- guage models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24), pages 847â864. Yilu Dong, Tianchang Yang, Abdullah Al Ishtiaq, Syed Md Mukit Rashid, Ali Ranjbar, Kai Tu, Tianwei Wu, Md Sultan Mahmud, and Syed Rafiul Hussain. 2025. Corecrisis: Threat-guided and context-aware iterative learning and fuzzing of 5g core networks. In 34th USENIX Security Symposium (USENIX Security 25). Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020. Ac/c++ code vulnerability dataset with code changes and cve summaries. In Proceedings of the 17th International Conference on Mining Software Repositories, pages 508â512. Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Ryan Tsang, Najmeh Nazari, Han Wang, Houman Homayoun, and 1 oth- ers. 2024. Large language models for code analysis: DoLLMsreally do their job? In 33rd USENIX Security Symposium (USENIX Security 24), pages 829â846. Viktoria Felmetsger, Ludovico Cavedon, Christopher Kruegel, and Giovanni Vigna. 2010. Toward auto- mated detection of logic vulnerabilities in web ap- plications. In 19th USENIX Security Symposium (USENIX Security 10). Xiang Gao, Bo Wang, Gregory J Duck, Ruyi Ji, Yingfei Xiong, and Abhik Roychoudhury. 2021. Beyond tests: Program vulnerability repair via crash con- straint extraction. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(2):1â 27. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Zhen Huang, David Lie, Gang Tan, and Trent Jaeger. 2019. Using safety properties to generate vulnera- bility patches. In 2019 IEEE symposium on security and privacy (SP), pages 539â554. IEEE. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186. Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018. Shaping program repair space with existing patches and similar code. In Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, pages 298â309. Nan Jiang, Thibaud Lutellier, Yiling Lou, Lin Tan, Dan Goldwasser, and Xiangyu Zhang. 2023. Knod: Do- main knowledge distilled tree decoder for automated program repair. In 2023 IEEE/ACM 45th Interna- tional Conference on Software Engineering (ICSE), pages 1251â1263. IEEE. Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd Interna- tional Conference on Software Engineering (ICSE), pages 1161â1173. IEEE. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language mod- els resolve real-world github issues? arXiv preprint arXiv:2310.06770. Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. 2024. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901. Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svy- atkovskiy. 2023. Inferfix: End-to-end program repair with llms. In Proceedings of the 31st ACM Joint Eu- ropean Software Engineering Conference and Sym- posium on the Foundations of Software Engineering, pages 1646â1656. James A Jones and Mary Jean Harrold. 2005. Em- pirical evaluation of the tarantula automatic fault- localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, pages 273â282. RenĂŠ Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Pro- ceedings of the 2014 international symposium on software testing and analysis, pages 437â440. Ummay Kulsum, Haotian Zhu, Bowen Xu, and Marcelo dâAmorim. 2024. A case study of llm for automated vulnerability repair: Assessing impact of reasoning and patch validation feedback. In Proceedings of the 1st ACM International Conference on AI-Powered Software, pages 103â111. Xuan Bach D Le, David Lo, and Claire Le Goues. 2016. History driven program repair. In 2016 IEEE 23rd international conference on software analysis, evolu- tion, and reengineering (SANER), volume 1, pages 213â224. IEEE. Thanh Le-Cong, Duc-Minh Luong, Xuan Bach D Le, David Lo, Nhat-Hoa Tran, Bui Quang-Huy, and Quyet-Thang Huynh. 2023. Invalidator: Automated patch correctness assessment via semantic and syn- tactic reasoning. IEEE Transactions on Software Engineering, 49(6):3411â3429. Claire Le Goues, Michael Dewey-Vogt, Stephanie For- rest, and Westley Weimer. 2012. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In 2012 34th international con- ference on software engineering (ICSE), pages 3â13. IEEE. Frank Li and Vern Paxson. 2017. A large-scale empiri- cal study of security patches. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 2201â2215. Yi Li, Shaohua Wang, and Tien N Nguyen. 2020. Dl- fix: Context-based code transformation learning for automated program repair. In Proceedings of the ACM/IEEE 42nd international conference on soft- ware engineering, pages 602â614. Ying Li, Gang Wang, Yuan Tian, and 1 others. 2025. Sok: Towards effective automated vulnerability re- pair. arXiv preprint arXiv:2501.18820. Yue Li, Xiao Li, Hao Wu, Yue Zhang, Xiuzhen Cheng, Sheng Zhong, and Fengyuan Xu. 2024. Attention is all you need for llm-based code vulnerability local- ization. arXiv preprint arXiv:2410.15288. Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. Quixbugs: A multi-lingual pro- gram repair benchmark set based on the quixey chal- lenge. In Proceedings Companion of the 2017 ACM SIGPLAN international conference on systems, pro- gramming, languages, and applications: software for humanity, pages 55â56. Zhiqiang Lin, Xuxian Jiang, Dongyan Xu, Bing Mao, and Li Xie. 2007. Autopag: towards automated soft- ware patch generation with source code root cause identification and repair. In Proceedings of the 2nd ACM symposium on Information, computer and com- munications security, pages 329â340. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024a. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36. Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, and Wenhai Wang. 2024b. Explor- ingChatGPTâscapabilities on vulnerability man- agement.In 33rd USENIX Security Symposium (USENIX Security 24), pages 811â828. Yu Liu, Sergey Mechtaev, Pavle Suboti Ě c, and Abhik Roychoudhury. 2023. Program repair guided by datalog-defined static analysis. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1216â1228. Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020. Coconut: combining context-aware neural translation models using ensemble for program repair. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, pages 101â114. Alexandru Marginean, Johannes Bader, Satish Chandra, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, and Andrew Scott. 2019. Sapfix: Automated end-to- end repair at scale. In 2019 IEEE/ACM 41st Inter- national Conference on Software Engineering: Soft- ware Engineering in Practice (ICSE-SEIP), pages 269â278. IEEE. Sergey Mechtaev, Jooyong Yi, and Abhik Roychoud- hury. 2016. Angelix: Scalable multiline program patch synthesis via symbolic analysis. In Proceed- ings of the 38th international conference on software engineering, pages 691â701. Xiang Mei, Pulkit Singh Singaria, Jordi Del Castillo, Haoran Xi, Tiffany Bao, Ruoyu Wang, Yan Shoshi- taishvili, Adam DoupĂŠ, Hammond Pearce, Brendan Dolan-Gavitt, and 1 others. 2024. Arvo: Atlas of reproducible vulnerabilities for open source software. arXiv preprint arXiv:2408.02153. Ziyi Ni, Yifan Li, Ning Yang, Dou Shen, Pin Lyu, and Daxiang Dong. 2025. Tree-of-code: A self-growing tree framework for end-to-end code generation and execution in complex tasks. In Findings of the As- sociation for Computational Linguistics: ACL 2025, pages 9804â9819. Georgios Nikitopoulos, Konstantina Dritsa, Panos Louridas, and Dimitris Mitropoulos. 2021. Crossvul: a cross-language vulnerability dataset with commit data. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing, pages 1565â1569. Gene Novark, Emery D Berger, and Benjamin G Zorn. 2007. Exterminator: automatically correcting mem- ory errors with high probability. In Proceedings of the 28th ACM SIGPLAN Conference on Program- ming Language Design and Implementation, pages 1â11. Hakjoo Oh. 2018. Memfix: Static analysis-based repair of memory deallocation errors for c. In FSE 2018: ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM. OpenAI. 2025. Openai o3 and o4-mini system card. Arjun Panickssery, Samuel R Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Advances in Neural Information Pro- cessing Systems, 37:68772â68802. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilotâs code contributions. Communications of the ACM, 68(2):96â105. Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Ex- amining zero-shot vulnerability repair with large lan- guage models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2339â2356. IEEE. Ali Ranjbar, Tianchang Yang, Kai Tu, Saaman Khalilol- lahi, and Syed Rafiul Hussain. 2025. Stateful Analy- sis and Fuzzing of Commercial Baseband Firmware . In 2025 IEEE Symposium on Security and Privacy (SP), pages 1120â1139, Los Alamitos, CA, USA. IEEE Computer Society. Kuniaki Saito, Kihyuk Sohn, Chen-Yu Lee, and Yoshitaka Ushiku. 2024. Unsupervised llm adap- tation for question answering.arXiv preprint arXiv:2402.12170. Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. 2023. Lost at c: A user study on the security impli- cations of large language model code assistants. In 32nd USENIX Security Symposium (USENIX Secu- rity 23), pages 2205â2222. Ridwan Shariffdeen, Yannic Noller, Lars Grunske, and Abhik Roychoudhury. 2021. Concolic program re- pair. In Proceedings of the 42nd ACM SIGPLAN In- ternational Conference on Programming Language Design and Implementation, pages 390â405. Alex Shaw, Dusten Doggett, and Munawar Hafiz. 2014. Automatically fixing c buffer overflows using pro- gram transformations. In 2014 44th Annual IEEE/I- FIP International Conference on Dependable Sys- tems and Networks, pages 124â135. IEEE. Dokyung Song, Julian Lettner, Prabhu Rajasekaran, Yeoul Na, Stijn Volckaert, Per Larsen, and Michael Franz. 2019.Sok: Sanitizing for security.In 2019 IEEE Symposium on Security and Privacy (SP), pages 1275â1295. IEEE. Shin Hwei Tan, Hiroaki Yoshida, Mukul R Prasad, and Abhik Roychoudhury. 2016. Anti-patterns in search- based program repair. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 727â 738. Haoye Tian, Kui Liu, Yinghua Li, Abdoul Kader KaborĂŠ, Anil Koyuncu, Andrew Habib, Li Li, Junhao Wen, Jacques Klein, and TegawendĂŠ F BissyandĂŠ. 2023. The best of both worlds: Combining learned embeddings with engineered features for accurate prediction of correct patches. ACM Transactions on Software Engineering and Methodology, 32(4):1â34. Haoye Tian, Xunzhu Tang, Andrew Habib, Shang- wen Wang, Kui Liu, Xin Xia, Jacques Klein, and TegawendĂŠ F BissyandĂŠ. 2022. Is this change the answer to that problem? correlating descriptions of bug and code changes for evaluating patch correct- ness. In Proceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineer- ing, pages 1â13. Kai Tu, Abdullah Al Ishtiaq, Syed Md Mukit Rashid, Yilu Dong, Weixuan Wang, Tianwei Wu, and Syed Rafiul Hussain. 2024. Logic gone astray: A security analysis framework for the control plane pro- tocols of 5g basebands. In 33rd USENIX Security Symposium (USENIX Security 24), pages 3063â3080. Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. 2023. Can large language models identify and reason about security vulnerabilities? not yet. arXiv preprint arXiv:2312.12575. Ashok Urlana, Pruthwik Mishra, Tathagato Roy, and Rahul Mishra. 2024.Controllable text summa- rization: unraveling challenges, approaches, and prospects-a survey. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1603â 1623. Rijnard van Tonder and Claire Le Goues. 2018. Static automated program repair for heap properties. In Proceedings of the 40th International Conference on Software Engineering, pages 151â162. Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. 2024. Inter- venor: Prompting the coding ability of large language models with the interactive chain of repair. In Find- ings of the Association for Computational Linguistics: ACL 2024, pages 2081â2107. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in llm-as-a-judge. arXiv preprint arXiv:2410.21819. Qi Xin and Steven P Reiss. 2017. Leveraging syntax- related code for automated program repair. In 2017 32nd IEEE/ACM International Conference on Auto- mated Software Engineering (ASE), pages 660â670. IEEE. Yunlong Xing, Shu Wang, Shiyu Sun, Xu He, Kun Sun, and Qi Li. 2024. WhatIFis not enough? fixing null pointer dereference with contextual check. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1367â1382. Zhengzi Xu, Yulong Zhang, Longri Zheng, Liangzhao Xia, Chenfu Bao, Zhi Wang, and Yang Liu. 2020. Au- tomatic hot patch generation for android kernels. In 29th USENIX Security Symposium (USENIX Security 20), pages 2397â2414. Jifeng Xuan, Matias Martinez, Favio Demarco, Maxime Clement, Sebastian Lamelas Marcote, Thomas Durieux, Daniel Le Berre, and Martin Monperrus. 2016. Nopol: Automatic repair of conditional state- ment bugs in java programs. IEEE Transactions on Software Engineering, 43(1):34â55. Peng Yixing, Quan Wang, Licheng Zhang, Yi Liu, and Zhendong Mao. 2024. Chain-of-question: A progres- sive question decomposition approach for complex knowledge base question answering. In Findings of the Association for Computational Linguistics ACL 2024, pages 4763â4776. Jinhong Yu, Yi Chen, Di Tang, Xiaozhong Liu, Xi- aoFeng Wang, Chen Wu, and Haixu Tang. 2024. Llm-enhanced software patch localization. arXiv preprint arXiv:2409.06816. Ying Zhang, Ya Xiao, Md Mahir Asef Kabir, Danfeng Yao, and Na Meng. 2022. Example-based vulnerabil- ity detection and repair in java code. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pages 190â201. Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Hung Huu Nguyen, Thanh Le-Cong, Junda He, Bach Le, and David Lo. 2024. Leveraging large lan- guage model for automatic patch correctness assess- ment. IEEE Transactions on Software Engineering, 50(11):2865â2883. Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021. A syntax-guided edit decoder for neural program repair. In Proceedings of the 29th ACM joint meeting on European software engineering conference and sym- posium on the foundations of software engineering, pages 341â353. Appendices A Characterizing AVR Approaches and Evaluation Frameworks for Logical Vulnerabilities A.1 Automatic Vulnerability Repair (AVR) Automatic Vulnerability Repair (AVR) techniques aim to automatically fix software vulnerabilities. Their primary objective is to propose a patch that (i) eliminates the identified vulnerability and (i) preserves the original functionality after the fix. In practice, vulnerability repair comprises three main stages: vulnerability localization, patch generation, and patch validation, which mirror the steps devel- opers naturally follow when repairing vulnerable code. In this work, evaluate the patch generation phase of logical vulnerabilities. We identify the primary challenges for automatically generating fixes for such vulnerabilities, design frameworks to system- atically evaluate the correctness of the generated fixes, and discuss the strengths and limitations of existing non-LLM and LLM approaches. Existing AVR approaches primarily focus on repairing memory corruption vulnerabilities and can be broadly classified into two categories: non- learning-based and learning-based approaches. A recent study by Li et al. (Li et al., 2025) provides a comprehensive analysis of various categories of AVR frameworks in addressing those vulnerabili- ties. In this section, we build upon previous studies CategoryOptimal Scenario (When it Works) Adaptable forLogical Vulnerability Repair Limitations When Applying for Logical Vulnerabilities Template guided When the vulnerability fix fol- lows a template/pattern NoLogical vulnerabilities does not follow any general fix pattern. Constraint based When the vulnerability can be fixed through enforcing con- straints easily extractable from source code Yes, but diffi- cult Constraint extraction for logical vulnerabilities struggle with path explosion in large codebases and often lacks the necessary specifi- cations or testcases to derive re- quired constraints. Search based When the vulnerability can be fixed by reusing or slightly adapting a similar code segment elsewhere in the source project Yes,but achieves poor performance Usually logical vulnerabilities cannot be fixed by reusing exist- ing code, as each one has distinct expected behavior and requires its own custom logic. Deep learning based When the vulnerability fix fol- lows a pattern learnable by deep learning models Yes,but achieves poor performance Usually logical vulnerabilities do not follow such recognizable pat- tern Table 4: Analysis of existing AVR techniques and limitations towards repairing logical vulnerabilities by examining each category of AVR approaches when applied to logical vulnerability repair, high- lighting the unique challenges and limitations they face in this context, which are summarized in Table 4. A.1.1 Non-Learning Based Approaches These non-machine learning based methods fall into three categories based on their repairing strategy:template-guided, search-based, and constraint-based approaches. Template-guided approaches. Template-guided ap- proaches rely on patterns of vulnerability properties (Shaw et al., 2014; Huang et al., 2019) or historical patches (Xing et al., 2024; Zhang et al., 2022) to generate patches. Due to their reliance on fixed patterns, these methods usually focus on vulnera- bilities exhibiting recurring structures (e.g., buffer overflow (Shaw et al., 2014), null pointer deref- erence (Xing et al., 2024), API misuses (Zhang et al., 2022)) where known fix templates can be reliably applied. In contrast, logical vulnerabilities typically involve highly context-specific behaviors and lack such common patterns, rendering these approaches inapplicable. Constraint-based approaches. These methods ex- tract program constraints that need to be satisfied to eliminate a vulnerability. These constraints are either extracted through (i) static analysis (Oh, 2018; Chida and Terauchi, 2022), (i) symbolic execution (Shariffdeen et al., 2021), or (i) dy- namic analysis through running against a set of test suites (Xuan et al., 2016; Agrawal and Horgan, 1990). (i) Among static analysisâbased techniques, Lee et al. (Oh, 2018) leverage typestate checking to detect memory vulnerabilities. Liu et al. (Liu et al., 2023) extract and modify Datalog facts to repair Java null-pointer exceptions, Python data leaks, and Solidity smart contract flaws, whereas Xu et al. (Xu et al., 2020) build a mathematical model from historical patches and use automated reasoning to generate hotpatches. However, these approaches rely on carefully defined properties to drive the static analysis and can only target specific vulnerability classes. Each logical vulnerability has its own distinct expected behavior, and formulating a corresponding property would require domain knowledge and significant manual effort compara- ble to writing the repair manually. (i) Symbolic executionâbased methods derive the constraints needed to satisfy given testcases (Shariffdeen et al., 2021; Mechtaev et al., 2016). Consequently, they rely on a comprehensive, vulnerability-specific test suite to extract precise constraints. However, real- world logical vulnerabilities often provide only a single or no exploit trace, which is insufficient to recover the necessary constraints (Li et al., 2025). (i) Dynamic testing-based approaches (Xuan et al., 2016; Agrawal and Horgan, 1990) also require a comprehensive test suite focusing on the vulnera- bility. Gao et al. (Gao et al., 2021) eliminate this requirement by using address sanitizers to extract crash-free constraints. However, logical vulnera- bilities rarely cause program crashes and cannot be captured using address sanitizers, rendering this approach inapplicable. Search-based approaches.Search-based ap- proaches explore a hypothetical search space for re- pair through mutations in existing code (Le Goues et al., 2012; Marginean et al., 2019) or by extract- ing similar code from other functions or source files within a project (Jiang et al., 2018; Le et al., 2016). However, their ability to repair a vulnera- bility depends on the availability of similar code in their search base that can either be directly placed or adopted with minimal changes to fix a vulnera- bility. In contrast, it is usually challenging to find similar code that can be used to fix a real-world logical vulnerability, as each logical vulnerability has a distinct expected behavior and thus distinct logic that can be used to fix it. As a result, although we can technically apply search-based mechanisms to repair logical vulnerabilities, their performance is poor, which we demonstrate in this paper using SimFix (Jiang et al., 2018) in §6.1. A.1.2 Learning Based Approaches Learning-based approaches pivot on machine learn- ing techniques and can be broadly divided into two categories: deep learning-based and large language model (LLM)-based approaches. Deep learning-based approaches. Deep learning- based approaches adopt deep learning models and treat the program repair problem as a neural ma- chine translation (NMT) problem, providing the faulty code as input and obtaining the fixed code as output. In this direction, SequenceR (Chen et al., 2019) is the first to adopt an encoder-decoder based supervised recurrent neural network (RNN) ma- chine translation model to generate patches. Later on, CURE (Jiang et al., 2021) improves on pre- vious techniques on NMT-based program repair through subword tokenization and a code-aware token search strategy to provide more accurate patches. A recent work, KNOD (Jiang et al., 2023), further increases the accuracy on Java program vul- nerabilities through leveraging a novel tree-based decoder and emitting an abstract syntax tree (AST) to repair programs. Although these approaches perform better than non-learning-based approaches (Lutellier et al., 2020; Jiang et al., 2023) they require a dataset of vulnerabilities and their fixes to train the model. Furthermore, while training, the model attempts to extract patterns in source code that can be used to generate patches for specific vulnerabilities. In- tuitively, these models work well when the under- lying vulnerability can be fixed using a language safety pattern (e.g., bounds checking for buffer overflow vulnerabilities). In contrast, logical vul- nerabilities do not follow any patterns. As we show in §6.1, the state-of-the-art repair framework KNOD with a deep learning approach perform poor in fixing logical vulnerabilities. A.1.3 Large Language Models for AVR Recently, Large Language Models (LLMs) trained on vast training data have shown the ability to gen- erate high-quality natural language outputs that are widely applicable across diverse areas such as text summarization (Jin et al., 2024) and question an- swering (Saito et al., 2024). Due to similarities between code and natural language, researchers have explored LLMs in programming-related tasks such as code generation (Liu et al., 2024a) and anal- ysis (Fang et al., 2024), and have been shown to be effective in understanding program syntax and semantics and analyzing program behaviors. Several LLM-based approaches have been re- cently proposed to generate repair patches (Jin et al., 2023; Kulsum et al., 2024; Pearce et al., 2023). However, their capabilities and limita- tions are unknown for logical vulnerabilities. Also, LLMs have the potential to understand security in- variants, even though they face limitations when computing complex constraints (Li et al., 2025). As such, LLMs and LLM-based approaches ap- pear promising for repairing logical vulnerabilities. However, to our knowledge, no prior research has systematically analyzed their capabilities and limi- tations in this specific context. A.2 Limitations of AVR Evaluation Frameworks Among existing evaluations on LLMs for security vulnerabilities, Pearce et al. (Pearce et al., 2025) assess the security of code generated by GitHub Copilot. Sandoval et al. (Sandoval et al., 2023) con- ducted a user study in which students used LLM assistance to write code, and the resulting programs were evaluated for security vulnerabilities. Deng et al. (Deng et al., 2024) propose and evaluate the use of LLMs for penetration testing. Ullah et al. (Ullah et al., 2023) automatically evaluates LLMs with various prompts to identify security vulnerability in source codes. However, none of them particu- larly addresses repairing vulnerabilities. Jimenez et al. (Jimenez et al., 2023) evaluate how LLMs generate fixes for GitHub issues; however, most of these issues are not labeled to indicate whether they address security vulnerabilities. Among evalu- ation works that focus on automatically repairing security vulnerabilities, Li et al. (Li et al., 2025) systematically study existing AVR approaches, and provide a taxonomy, discussing pros and cons of each category of approaches. They also benchmark several existing works based on existing datasets. However, they do not focus on logical vulnera- bilities and the limitations of using existing AVR approaches when applied to logical vulnerabilities. Most related to our work is that of Pearce et al. (Pearce et al., 2023), which evaluates the perfor- mance of LLMs towards repairing security vulner- abilities with zero-shot prompts. However, their evaluation framework have several limitations to- wards evaluating logical vulnerabilities: (i) Their evaluation approach do not take auxiliary infor- mation related to the vulnerability, such as vulner- ability description, specification, or repair steps, etc., as input, which facilitates generation of rea- sonable patches for logical vulnerabilities (§6.2.3), (i) They lack any automated metrics for evaluation to indicate reasoning / identify reasonable patches. They only manually evaluate some patches of the Extractfix (Gao et al., 2021) dataset and label them as identical, semantically equivalent, or reasonable, and (i) They use address sanitizers and CodeQL (cod, 2021) to test patches, which are not suitable for logical vulnerabilities. B Locating The Core Fix of a Patch We define the core fix as the portion of the patch that implements the decision logic needed to eliminate a vulnerability (e.g., the key authorization/validation/state-transition guard). As an example of a core fix, Listing 2 shows the fix for CVE-2014-0224 (hen), where we observe a macro definition at Line 3, assignment of the macro to a variable at lines 6 and 19, and a check based on the variable and macro at lines 11â17. However, the primary fix is to add the check at lines 11â17, which enforces a check based on the flag and en- 1 int ssl3_accept(SSL *s) 2 ... 3++ #define SSL3_FLAGS_CCS_OK 0x0080; 4 ... 5 case SSL3_ST_SR_FINISHED_B: 6++ s->s3->flags |= SSL3_FLAGS_CCS_OK; 7 ... 8 9 int ssl3_read_bytes(SSL *s, int type , unsigned ,â char *buf , int len , int peek) 10 ... 11++ if (!(s->s3->flags & SSL3_FLAGS_CCS_OK)) 12++ 13++ al = SSL_AD_UNEXPECTED_MESSAGE; 14++ SSLerr(SSL_F_SSL3_READ_BYTES, 15SSL_R_CCS_RECEIVED_EARLY); 16++ goto f_err; 17++ 18++ 19++ s->s3->flags &= SSL3_FLAGS_CCS_OK; 20 ... 21 Listing 2: Fix for CVE-2014-0224 (hen). sures that a ChangeCipherSpec message is received when expected. We refer to the if block at lines 11â17 as the core fix. In C/C++ and Java, the core fix typically corresponds to a compound statement enclosed in braces, with the function body as the top-level block. During identification and annotation, two security-knowledgeable human annotators selected the hunk(s) they believed contained this decision logic. Disagreements during annotation were rare (approximately 3 out of 43 cases) and were re- solved through discussion to reach a consensus. We manually identify and localize the core fix by observing the vulnerable and fix commits, the vul- nerability descriptionD, and, if available, the be- havioral specification S. During manual localization, we identify both the enclosing function and the enclosing block. The enclosing block provides fine-grained context for patch generation, while the enclosing function sup- plies the broader program context needed to under- stand the fix. To localize the enclosing function, we extract its full body and signature. To local- ize the enclosing block, we identify the smallest brace-enclosed region that contains the core fix. If the block appears within a conditional or loop, we include the associated control structure (e.g., the condition). In general, this block may range from a small basic block to the entire function body. When the fix logic is split across multiple hunks that are jointly required (e.g., a flag assigned in one place and enforced in another), in cases where possible, we expand the core-fix span to include all complementary hunks necessary to realize the intended security behavior, and treat that expanded region as the single-hunk core fix target for our evaluation. For example, in the fix presented in Listing 3 (representative of CVE-2023-0465, sim- plified for exposition), we place the entire block within the for loop as the core fix. 1- int i; 2+ int i, saw_err = 0; 3 4 // - - - start of hunk - - - 5- for (i = 1; i < N; i++) \ 6+ for (i = 0; i < N; i++) \ 7 item = chain[i]; 8+ if (item.has_invalid_policy) saw_err = 1; 9 FAIL_IF(item.has_invalid_policy); 10 \ 11 12+ if (!saw_err) return internal_error(); 13 // - - - end of hunk - - - 14 return OK; Listing 3: Simplified fix from CVE-2023-0465 In cases where two similar hunks in different code locations implement essentially the same de- cision logic for different modes/configurations, we select one of them as the representative fix, since both reflect the same underlying repair logic. Fi- nally, if the main logic resides in a newly added helper function, we treat the helper function body as part of the core-fix span (i.e., include it within the single localized region). We perform this localization on both the vulnera- ble source codeSand the fixed codeF, resulting in the corresponding localized blocks and functions: V b ,V f ,F b , andF f . The repair framework receives V b or V f as input, while F b and F f are used solely for evaluation. In some cases, such as the fix for CVE-2023- 5363 (cve, 2023a), the patch involves inserting logic between two basic blocks within a much larger compound statement. Including the entire enclosing block in such cases would dilute the rel- evance of the context. To preserve focus, if the enclosing block exceeds a token threshold (2048 in our case), we instead extract a focused span centered on the patch location, including the im- mediate predecessor and successor basic blocks surrounding the patch location (e.g., lines 227â250 in the vulnerable code for CVE-2023-5363). This threshold was intended to avoid context dilution while staying within a practical prompt token bud- get for some LLMs. This ensures precise and fo- cused block-level localization. C Curation of LogicDS For dataset curation, we first search CVEs of the selected targets from the years 2010 to 2024, and based on the CVE description and publicly avail- able fix, we determine whether the vulnerability is a logical vulnerability and has security implica- tions. While selecting vulnerability samples, we exclude memory-corruption vulnerabilities (e.g., buffer overflow) and vulnerabilities that are more related to safe programming practices rather than a specific logical issue (e.g., SQL injection). Then, we manually searched the internet to obtain the developer-provided fix commit for the open-source implementation, by examining commit logs and open source vulnerability websites (e.g., (bug; nvd)). A breakdown of the open-source projects used to construct LogicDS is given in Table 5. ProjectSample Count OpenSSL11 SrsRAN8 WolfSSL6 BIND4 GnuTLS3 MbedTLS3 OpenSSH2 DNSMasq1 Eclipse Mosquitto1 Hostapd1 PJSIP1 ProFTPD1 XMPP Openfire1 Table 5: Open-source projects considered while con- structing LogicDS An example of the different portions a sample is demonstrated in Table 6. C.1 Constructing Synthetic Java Samples. Manually creating such samples would require sig- nificant time and efforts. Thus, to alleviate the manual work, we develop an LLM-assisted semi- automated approach for synthetic sample creation, which will also help future research works in cre- ating synthetic Java examples from real-world log- ical vulnerabilities written in other programming languages. In this approach, we first manually create one synthetic sample based on a real-world vulnerabil- ity by replicating the vulnerable and fixed func- tionsV f &F f , with an intent to preserve original variable and function names where possible. We also write testcases that validate the created patch. Once we create one such example, we leverage LLMs to create subsequent examples, by adopting InputMandatoryExample VulnerableSource Code âCommit (cve, 2019b) which is the parent commit of the fix in (cve, 2019c) Fixed Source CodeâCommit (cve, 2019c) available in national vulnera- bility database (cve, 2019a) VulnerabilityDe- scription âCVE description for CVE-2019-1543 found in (cve, 2019a) Behavioral Specifica- tion âExtracted from RFC 7539 (rfc) Section 2.8 Context LinesâLines21-31and140-147in crypto/x509/x509_vfy.c in (cve, 2019b) Repair DescriptionâA text description generated by observing the fixed source code (cve, 2019c). Compilation ScriptsâCompilation script generated manually following im- plementation documents Testing ScriptsâTest found (cve, 2019d) by manually checking the fix commit (cve, 2019c) Faulty Function and block âLines 323-429 and 359-363 respectively in crypto/x509/x509_vfy.c in (cve, 2019e) Fixed Function and block âLines 500-607 and 537-541 respectively in crypto/x509/x509_vfy.c in (cve, 2019f) Table 6: Example input and vulnerability localization (CVE-2019-1543 (cve, 2019a)) for LogicEval. a few-shot technique. We do that by providing the manually created sample, and prompting it to gen- erate similar synthetic examples given a vulnerable real-world function and vulnerability description. In the subsequent prompt, we provide the gener- ated vulnerable Java function and the real-worldV b andF b . Finally, we prompt the LLM to generate testcases for validation. We also assist the LLM iteratively generate syntactically correct Java code by providing compilation logs in case of any er- rors. This few-shot technique assists the LLM in creating precise synthetic examples when provided with a real-world sample by clearly demonstrating the task at hand. Note that, this semi-automated technique only intended to alleviate the manual work. Although the generated samples needed to be manually verified after generation, this approach is significantly less time-consuming than creating synthetic samples from scratch. D Adopting Baselines for LogicEval Existing work on program repair can be broadly categorized into three main paradigms: (i) non learning-based approaches, (i) deep learning- based approaches, and (i) large language model (LLM)-based approaches. To evaluate the capabili- ties and limitations of each, we select representative state-of-the-art methods from each category. We aim to analyze their capabilities in repairing logi- cal vulnerabilities and derive insights that can be leveraged in future automated repair frameworks, focusing on them. Non Learning-Based Program Repair.We chose a state-of-the-art search-based method, Sim- Fix (Jiang et al., 2018), among non-learning-based approaches. Template-based methods focus on spe- cific classes of bugs and are unsuitable due to the diverse nature of logical vulnerabilities, whereas constraint-based methods struggle with path explo- sion in large projects and are often inapplicable due to the lack of necessary specifications or test cases. SimFix operates by mining AST-level modifica- tions from a corpus of buggy programs and their corresponding patches. During inference, it ana- lyzes the buggy project and attempts to replace faulty code snippets with similar code snippets from previously mined patches. Our experiments allow SimFix to generate up to 100 patches per bug. Also, since SimFix works with line-level localiza- tion, LogicEval provides a manually located line within the core fix most suitable as the fault loca- tion. Furthermore, we allow SimFix to collect can- didate patches from the largest project in Defects4J (Just et al., 2014), Closure, providing a sufficiently large search space for collecting candidate patches. Deep Learning-Based Program Repair. For deep learning-based repair, we evaluate KNOD (Jiang et al., 2023), a state-of-the-art deep learning-based framework, due to its superior performance com- pared to others (Jiang et al., 2021; Li et al., 2020; Zhu et al., 2021), on standard benchmarks (Just et al., 2014; Lin et al., 2017). KNOD uses a three- stage tree decoder to generate repaired Java AST patches, and incorporates domain-specific rules to enhance the accuracy of AST node predictions. KNOD defaults to an output token limit of 512; therefore, we only provide the vulnerable blockV b as input. LLM-Based Program Repair. We consider VR- Pilot (Kulsum et al., 2024) as our baseline for LLM-based repair methods because of its robust performance and widely adoption in recent pro- gram repair evaluations (Li et al., 2025). VRPilot leverages chain-of-thought (CoT) prompting first to elicit reasoning about a required fix and then generate candidate patches. It refines patch quality through iterative feedback using logs from compi- lation errors, functional test failures, and security tests until a plausible patch is identified or a pre- defined iteration budget is reached. Since we are testing for logical vulnerabilities, and through our experiments (§6.2.3) we observed that providing Dsignificantly improves performance, we addD in VRPilotâs initial reasoning prompt. In addition to baseline-specific adaptations, when evaluating a suggested patch from an adapted baseline for reasoning,LogicEvalprompts the judge LLM with the vulnerability descriptionD, the vulnerable blockV b , and the suggested patch to extract an explanation E for the patch. E Prompts Used for LLM-based Experiments The prompts used in our experiments are summa- rized in Table 7. As an example of a concrete prompt, we provide the template of prompt P7: ... Here is a portion of a code: <code> (Vulnerable source code block or function) </code> Here is a description of the specification telling what to do: <specification> (Natural-text specification) </specification> Here is a description of the vulnerability: <vulnerability> (Vulnerability description) </vulnerability> Provide a repair for the mentioned code snippet to fix the vulnerability. Provide repaired code exactly to replace the original buggy code mentioned between <code> and </code>. Provide repaired code between <repair> and </repair> tags. The code between <repair> and </repair> will be directly copied to replace the code between <code> and </code>. ... Here, we also test with prompts P15 and P16 with a repair description. From the results of the suggested patches of these prompts shown in Table 1 and 2 respectively, we observe that have much higher scores in all metrics, indicating LLMs can generate more reasonable patches if concrete re- pair steps can be provided. We do not focus on the reasoning scores for these two prompts, since the repair steps of ground-truth fix were already provided to the LLM under testing. F Statistical Significance Analysis of Primary Claims To evaluate whether the observed claims in our Experiments Section (§6) are statistically signif- icant, we performed paired significance tests for each claim. For binary outcomes, compilation suc- cess/failure (C), test-case success/failure (T), and the binary LLM-as-a-judge verdict (J): we used an exact McNemarâs test and report the discordant counts, p-value, and odds ratio (OR). For continu- ous outcomes, such as the LLM-as-a-judge cosine similarity score (CS) we used the Wilcoxon signed- rank test and report the p-value, Cliffâsδ(effect size), the mean paired difference, and the bootstrap confidence interval (CI). Our primary claims are as follows: 1. Adjusting temperature has minimal impact on repair performance (i.e., compilation, testing and reasoning scores for output patches from prompts P1âP3 are identical). We test this by comparing P1 vs. P2. 2. Orientation has minimal impact (i.e., output patches from prompt templates P1 and P4 yield similar results). 3. The zero-shot prompt template (P5) achieves IDDescription P1-P3Temperature set to 0.2, 0.5, and 0.9, respectively P4Task-oriented prompt P5Zero-shot prompt (same as P1) P6Few-shot prompt including an example real-world vulnerability description, and vulnerable and fixed blocks P7Chain-of-thought prompting, first prompt obtains a repair suggestionR suggestion , which is placed in the second prompt that asks for patch P8Same as P7, but R suggestion is not added to the second prompt P9Provides vulnerable block V b as input source code (same as P1) P10Provides entire vulnerable function V f as input source code P11Provides vulnerable block along with additional context lines V context P12Provides no auxiliary information, only the vulnerable block V b P13Provides the vulnerability description D (same as P1) P14Provides the vulnerability description D and the specification S P15Provides repair steps R and vulnerability description D P16Provides the repair steps R only, does not provide D P17Obtained from (Pearce et al., 2023), vulnerable code provided with no help (n.h.) P18Obtained from (Pearce et al., 2023), deletesV f and adds a comment âbugfix: fixed logical vulnerabilityâ (s.1). P19Obtained from (Pearce et al., 2023), deletesV f and adds comment âfixed logical vulnerabilityâ (s.2). P20Obtained from (Pearce et al., 2023), after a â// BUG: logical vulnerabilityâ comment,V b is shown in commented-out form, immediately followed by a â// FIXED:â section. (c.) P21Obtained from (Pearce et al., 2023), first include a â// BUG: logical vulnerabilityâ comment, then a â// MESSAGE:Dâ line. Next, showV b as commented-out lines, and finally introduce the corrected code under a â// FIXED VERSION:â header. (c.m.) Table 7: Prompt Descriptions for Different Prompts Tested With Off-the-shelf LLMs. For prompts P1 to P16, by default, the vulnerable block V b is provided as source code, higher compilation success and slightly better reasoning than the CoT prompt template (P7). 4.Including reasoning text in the patch- generation prompt improves patch quality. Specifically, P8 increases compilation suc- cess but lowers reasoning scores relative to P7, which includes vulnerability text. 5.Auxiliary information is essential for gener- ating reasonable patches for logical vulnera- bilities. Specifically, P12 (no auxiliary info) performs worse than P13 (vulnerability text) and P14 (specification info), even though P12 produces patches with a higher compilation success rate. 6. Adding context (P11) to a vulnerable source- code block (P9) improves compilation, while reasoning scores remain similar. 7. Providing only the vulnerable block (P9) yields more compilable patches and slightly better reasoning than providing the entire func- tion (P10). 8.Prompts from Pearce et al. (P17âP20) are less effective than our baseline prompt configura- tion (P5). We demonstrate this by comparing P17 vs. P5 and P20 vs. P5. For each claim, we report results aggregated across all evaluated LLMs for each prompt type. The statistical significance results are provided in Table 8. The significance tests largely support all of our claims, with two minor exceptions. For the binary LLM-as-a-judge reasoning metric (J), the differences are not statistically significant for the P5 vs. P7 comparison in claim 3 and for the P9 vs. P10 comparison in claim 6. In both cases, how- ever, the continuous reasoning metric (CS) remains statistically significant. In essence, the relative performance of P9 against P10 depends on the re- IDClaimC (McNemar)CS (Wilcoxon p, Cliffδ, Mean Diff., CI) J (McNemar) 1P1 and P2 show similar results 48/147 vs 46/147 (p=0.7266, OR=1.67) 0.7909 vs 0.7854;p=0.2107 (ns); δ=+0.143;â=+0.0055 [-0.0041, +0.0153] 116/300 vs 118/300 (p=0.8877, OR=0.92) 2P1 and P4 show similar results 81/224 vs 80/224 (p=1.0000, OR=1.08) 0.7888 vs 0.7860;p=0.4915 (ns); δ=+0.065;â=+0.0028 [-0.0073, +0.0130] 202/454 vs 204/454 (p=0.9050, OR=0.94) 3P5 shows better C, CS, J than P7 81/231 vs 67/231 (p=0.0385, OR=2.08) 0.7915 vs 0.7558;p⤠0.001(sig); δ=+0.506;â=+0.0357 [+0.0260, +0.0453] 217/510 vs 204/510 (p=0.2546, OR=1.27) 4P8 shows more C but less CS, J than P7 132/231 vs 67/231 (p⤠0.001, OR=5.33) 0.6509 vs 0.7558;p⤠0.001(sig);δ=- 0.859; â=-0.1049 [-0.1216, -0.0887] 39/510 vs 204/510 (p⤠0.001, OR=0.07) 5aP12 has higher C but lower CS, J than P13 128/231 vs 80/231 (p⤠0.001, OR=4.00) 0.6315 vs 0.7879;p⤠0.001(sig);δ=- 0.976; â=-0.1564 [-0.1725, -0.1408] 39/510 vs 216/510 (p⤠0.001, OR=0.03) 5bP12 has higher C but lower CS, J than P14 128/231 vs 57/231 (p⤠0.001, OR=8.10) 0.6315 vs 0.7948;p⤠0.001(sig);δ=- 0.976; â=-0.1633 [-0.1811, -0.1462] 39/510 vs 280/510 (p⤠0.001, OR=0.02) 6P11 has better C than P995/231 vs 79/231 (p=0.0166, OR=2.33) 0.7915 vs 0.7947;p=0.7474 (ns); δ=+0.082;â=-0.0033 [-0.0102, +0.0036] 215/510 vs 217/510 (p=0.9142, OR=0.95) 7P9 has lower C but higher CS, J than P10 77/225 vs 104/225 (p⤠0.001, OR=0.34) 0.7945 vs 0.7756;p⤠0.001(sig); δ=+0.398;â=+0.0189 [+0.0111, +0.0266] 212/498 vs 210/498 (p=0.9204, OR=1.04) 8aP17 has lower C, CS, J than P5 50/231 vs 81/231 (p⤠0.001, OR=0.40) 0.5794 vs 0.7915;p⤠0.001(sig);δ=- 0.976; â=-0.2121 [-0.2303, -0.1940] 12/510 vs 217/510 (p⤠0.001, OR=0.03) 8bP20 has lower C, CS, J than P5 27/231 vs 81/231 (p⤠0.001, OR=0.11) 0.6790 vs 0.7915;p⤠0.001(sig);δ=- 0.953; â=-0.1125 [-0.1288, -0.0971] 117/510 vs 217/510 (p⤠0.001, OR=0.29) Table 8: Paired significance test results for the primary claims, aggregated across all evaluated LLMs for each prompt type. quirements for vulnerability repair. Vulnerabilities that require broader, function-level context benefit from being provided in P10, whereas issues that are localized to a specific region are often better addressed with a block-level snippet in P9, which can facilitate repair while avoiding incorrect local- ization. G Example Codes Supporting Manual Inspection of Patches Table 9: Llama Fix for CVE-2015-1793 (cve, 2015) Before processing the certificate chain, we need to check if the certificate has aBasicConstraintsextension and if the cA flag is set. We can use the âX509_get_extension_flagsâ function tocheckiftheextensioniscritical and the âX509v3_get_ext_d2iâ function to parse the extension. Suggested Fix 1 // Check the Basic Constraints extension 2 X509_EXTENSION *ext = X509_get_ext(xtmp , ,â NID_basic_constraints); 3 if (ext!= NULL) 4 // Check if the extension is critical 5 if(X509_get_extension_flags(ext) & ,âEXFLAG_CRITICAL) 6 ... 7 if (bc!= NULL) 8 ... 9BASIC_CONSTRAINTS_free(bc); 10 ... Based on the prompt (highlighted in yellow), 2 undefined variables (highlighted in red) are generated. Table 10: Qwen Fix for CVE-2017-3142 (cve, 2017) Suggested Fix 1 if (msg ->tsigstatus != dns_rcode_noerror) 2 else if (tsig.error != dns_rcode_noerror) 3 else 4 // Extra check to ensure the requestor is ,â authorized 5 if (!is_requestor_authorized(msg)) 6 result =DNS_R_UNAUTHORIZED; 7 ... Ground Truth Fix 1 if (msg ->verified_sig && msg ->tsigstatus == ,â dns_rcode_noerror && tsig.error == ,â dns_rcode_noerror) 2 else if ((!msg ->verified_sig) || (msg -> ,â tsigstatus != dns_rcode_noerror)) 3 else 4 INSIST(tsig.error != dns_rcode_noerror); 5 result = DNS_R_TSIGERRORSET; 6 Undefined variables (highlighted in red) were created due to the lack of context. Table 11: Llama Fix for CVE-2015-1793 (cve, 2015) The whole function was input in the prompt, but the LLM failed to locate the vulnerable portion and add extra code after the vulnerable portion instead. The original vulnerable code has 319 lines, and the suggested patch has 336 lines. Suggested Fix 1 if (ok > 0) 2 X509_free(xtmp); 3 while (num > j) 4 xtmp = sk_X509_pop(ctx ->chain); 5 X509_free(xtmp); 6 num --; 7 ctx ->last_untrusted --; 8 9 retry = 1; 10 break; 11 12 ... 13 end: 14 ... 15// Fix: Check for Basic Constraints cA value 16if (ctx->chain != NULL) 17for (i = 0; i < sk_X509_num(ctx->chain); i++) 18x = sk_X509_value(ctx->chain, i); 19if (X509_check_ca(x)!= X509_CHECK_CA_OK) 20ctx->error = X509_V_ERR_INVALID_CA; 21ctx->current_cert = x; 22ctx->error_depth = i; 23ok = cb(0, ctx); 24if (!ok) 25goto end; 26 ... Ground Truth Fix 1 if (ok > 0) 2 X509_free(xtmp); 3 while (num > j) 4 xtmp = sk_X509_pop(ctx ->chain); 5 X509_free(xtmp); 6 num --; 7 8 ctx ->last_untrusted = sk_X509_num(ctx ->chain); 9 retry = 1; 10 break; 11 Table 12: OpenAI Fix for CVE-2022-1434 (cve, 2022) The fix adds a check after calling OSSL_PARAM_locate_const to verify that a valid pointer was returned rather than a NULL reference. Without this check, subsequent code that uses âpâ couldtrytodereferenceaNULL pointer if OSSL_PARAM_locate_const doesnât find the expected parameter, potentially leading to undefined behavior, a crash, or even exploitable security vulnerabilities. Suggested Fix 1 p = OSSL_PARAM_locate_const(params , ,â OSSL_CIPHER_PARAM_AEAD_TLS1_AAD); 2 if (p == NULL) 3 /* The parameter is required; if not found , ,â report an error and bail out */ 4 ERR_raise(ERR_LIB_CIPHER , ,â CIPHER_R_MISSING_AEAD_TLS1_AAD); 5 return 0; 6 Ground Truth Fix 1 p = OSSL_PARAM_locate_const(params , ,â OSSL_CIPHER_PARAM_AEAD_MAC_KEY); 2 ... Because no detailed vulnerability was specified in the prompt, the LLM assumed a NULL-pointer dereference risk when calling OSSL_PARAM_locate_const. Therefore, it adds a check immediately after this call to prevent any subsequent dereference of a NULL pointer. Table 13: Qwen Fix for CVE-2015-1793 (cve, 2015) Certainly!Theprovidedcodesnippetcontainsapotentialvulnerabilityrelated tomemorymanagement,specificallythehandlingofpointersafterfreeing memory. This can lead to undefined behavior, including crashes or security vulnerabilities such as use-after-free errors. Letâs break down the original code and then explain the repairs made. ... Suggested Fix 1 if (ok > 0) 2if (xtmp != NULL) 3X509_free(xtmp); 4xtmp = NULL; // Avoid dangling pointer 5 6 ... 7 Ground Truth Fix 1 if (ok > 0) 2 X509_free(xtmp); 3 ... 4 Because no detailed vulnerability was specified in the prompt, the LLM assumed that a use-after-free risk existed in the code and generated a patch to prevent any subsequent dereference of the dangling pointer. Table 14: OpenAI Fix for CVE-2023-48795 (cve, 2023b) ... The guess is considered wrong if the kex algorithm and/or the host key algorithm are guessed wrong (server and client have different preferred algo- rithms) or if any of the other algorithms cannot be agreed upon. Otherwise, the guess is considered to be right, and the optimisticallysentpacketMUST behandledasthefirstkeyexchangepacket.However,iftheguesswaswrong andapacketwasoptimisticallysentbyoneorbothparties,suchpackets MUSTbeignored, and the appropriate side MUST send the correct initial packet. ... Suggested Fix 1 if (first_kex_packet_follows || session -> ,â send_first_kex_follows) 2 ... 3 if (session ->first_kex_follows_guess_wrong) 4discard_first_kex_packet(session); 5 else 6 // Process the optimistically sent packet ,â normally 7process_first_kex_packet(session); 8 ... Provided the specification text in the prompt. However, since the code and the specification are at different abstraction levels, the LLM created functions that do not exist in the codebase, resulting in a patch that cannot pass compilation. Table 15: OpenAI Fix for CVE-2019-1543 (cve, 2019g) Suggested Fix 1 ... 2if (actx->aad != NULL || actx->len.aad != 0 || 3actx->len.text != 0) 4EVPerr(EVP_F_CHACHA20_POLY1305_CTRL, 5EVP_R_COPY_ERROR); 6return 0; 7 8 9 EVP_CIPHER_CTX *dst = (EVP_CIPHER_CTX *)ptr; 10 dst ->cipher_data = OPENSSL_memdup( 11 actx , sizeof (*actx) + Poly1305_ctx_size ()) ,â ; 12 if (dst ->cipher_data == NULL) 13 EVPerr(EVP_F_CHACHA20_POLY1305_CTRL , 14 EVP_R_COPY_ERROR); 15 return 0; 16 17 18 19 return 1; The generated patch has different logic from the ground-truth patch. The ground-truth is provided in 1. Table 16: KNOD (Jiang et al., 2023) Fix for CVE-2019- 1543 (cve, 2019g) (Synthetic) Suggested Fix 1 ... 2 case EVP_CTRL_AEAD_SET_IVLEN: 3if (actx == null ) 4return 0; 5 6 if (arg <= 0 || arg > CHACHA_CTR_SIZE) 7 return 0; 8 9 actx.nonce_len = arg; 10 return 1; 11 ... The generated patch has different logic from the ground-truth patch. The ground-truth for the real-world example is provided in 1. Table 17: SimFix (Jiang et al., 2018) Fix for CVE-2019- 1543 (cve, 2019g) (Synthetic) Suggested Fix 1 ... 2 case EVP_CTRL_AEAD_SET_IVLEN: 3 if (arg <= 0 || arg > CHACHA_CTR_SIZE) 4if (arg <= 0 || rhType > CHACHA_CTR_SIZE) 5return 0; 6 7 return 0; 8 9 actx.nonce_len = arg; 10 return 1; 11 ... The generated patch has different logic from the ground-truth patch. The ground-truth for the real-world example is provided in 1.