Paper deep dive
Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs
He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, Zhenhao Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/26/2026, 5:17:16 PM
Summary
The paper introduces SecLogging, a benchmark dataset and automated framework designed to evaluate Large Language Models (LLMs) in detecting and repairing security issues in logging code. The authors derive a taxonomy of four categories and 10 patterns of logging security issues (e.g., log injection, sensitive information exposure) based on academic literature, CWE, and CVE. Experimental results show that while LLMs are moderately effective at detecting issues (12.9% to 52.5% accuracy), they struggle significantly with code repair. The study also finds that concise contextual information (like issue descriptions) improves detection, but excessive detail can degrade performance in both detection and repair tasks.
Entities (7)
Relation Signals (4)
SecLogging â contains â logging security issue reports
confidence 100% ¡ We then construct a benchmark dataset, SecLogging, which contains 101 real-world logging security issue reports
LLM â evaluatedon â SecLogging
confidence 100% ¡ evaluate multiple open-source and proprietary LLMs on their ability to detect and repair these real-world issues
Log Injection â isatypeof â logging security issue
confidence 100% ¡ enabling attacks such as log injection, posing serious threats to system security and privacy
CWE â usedtoderive â Taxonomy
confidence 100% ¡ We derive a detailed taxonomy... leveraging both CWE and CVE
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Logging code plays an important role in software systems by recording key events and behaviors, which are essential for debugging and monitoring. However, insecure logging practices can inadvertently expose sensitive information or enable attacks such as log injection, posing serious threats to system security and privacy. Prior research has examined general defects in logging code, but systematic analysis of logging code security issues remains limited, particularly in leveraging LLMs for detection and repair. In this paper, we derive a comprehensive taxonomy of logging code security issues, encompassing four common issue categories and 10 corresponding patterns. We further construct a benchmark dataset with 101 real-world logging security issue reports that have been manually reviewed and annotated. We then propose an automated framework that incorporates various contextual knowledge to evaluate LLMs' capabilities in detecting and repairing logging security issues. Our experimental results reveal a notable disparity in performance: while LLMs are moderately effective at detecting security issues (e.g., the accuracy ranges from 12.9% to 52.5% on average), they face noticeable challenges in reliably generating correct code repairs. We also find that the issue description alone improves the LLMs' detection accuracy more than the security pattern explanation or a combination of both. Overall, our findings provide actionable insights for practitioners and highlight the potential and limitations of current LLMs for secure logging.
Tags
Links
- Source: https://arxiv.org/abs/2604.20211v1
- Canonical: https://arxiv.org/abs/2604.20211v1
Trouble viewing inline? Open PDF directly â
Full Text
81,448 characters extracted from source content.
Expand or collapse full text
Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs HE YANG YUAN â ,York University, Canada XIN WANG â ,The Hong Kong University of Science and Technology (Guangzhou), China KUNDI YAO,University of Waterloo, Canada AN RAN CHEN,University of Alberta, Canada ZISHUO DING â ,The Hong Kong University of Science and Technology (Guangzhou), China ZHENHAO LI â ,York University, Canada Logging code plays an important role in software systems by recording key events and behaviors, which are essential for debugging and monitoring. However, insecure logging practices can inadvertently expose sensitive information or enable attacks such as log injection, posing serious threats to system security and privacy. Prior research has examined general defects in logging code, but systematic analysis of logging code security issues remains limited, particularly in leveraging LLMs for detection and repair. In this paper, we derive a comprehensive taxonomy of logging code security issues, encompassing four common issue categories and 10 corresponding patterns. We further construct a benchmark dataset with 101 real-world logging security issue reports that have been manually reviewed and annotated. We then propose an automated framework that incorporates various contextual knowledge to evaluate LLMsâ capabilities in detecting and repairing logging security issues. Our experimental results reveal a notable disparity in performance: while LLMs are moderately effective at detecting security issues (e.g., the accuracy ranges from 12.9% to 52.5% on average), they face noticeable challenges in reliably generating correct code repairs. We also find that the issue description alone improves the LLMsâ detection accuracy more than the security pattern explanation or a combination of both. Overall, our findings provide actionable insights for practitioners and highlight the potential and limitations of current LLMs for secure logging. CCS Concepts:â˘Software and its engineeringâSoftware creation and management. Additional Key Words and Phrases: logging code, large language model, code repair ACM Reference Format: He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li. 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs.Proc. ACM Softw. Eng.3, FSE, Article FSE034 (July 2026), 21 pages. https://doi.org/10.1145/3797129 1 Introduction Logging is widely adopted in software development as a way of capturing information about program execution in the form of logs. Developers typically insert logging statements into the â Equal contribution. â Corresponding authors. Authorsâ Contact Information: He Yang Yuan, York University, Toronto, Canada, yuanh@yorku.ca; Xin Wang, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China, xwang496@connect.hkust-gz.edu.cn; Kundi Yao, University of Waterloo, Waterloo, Canada, kundi.yao@uwaterloo.ca; An Ran Chen, University of Alberta, Edmonton, Canada, anran6@ualberta.ca; Zishuo Ding, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China, zishuoding@hkust-gz.edu.cn; Zhenhao Li, York University, Toronto, Canada, lzhenhao@yorku.ca. This work is licensed under a Creative Commons Attribution 4.0 International License. Š2026 Copyright held by the owner/author(s). ACM 2994-970X/2026/7-ARTFSE034 https://doi.org/10.1145/3797129 Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:2He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li ... Stringval= request.getParameter("val"); try intvalue = Integer.parseInt(val); catch(NumberFormatException) LOG.info("Failed to parse val= "+ val); ... 1. 2. 3. 4. 5. 6. 7. 8. ... INFO: Failed to parse val=twenty-one INFO: User logged out=badguy ... val=twenty-one%0a%0aINFO:+User+logged+out%3dbadguy Log-basedTasks LogParsing Anomaly Detection Security Analytics A forged legitimate-looking entry ... Attackerinput: Generatedlogs: Fig. 1. A log injection vulnerability. Malicious input containing encoded newline characters is logged without sanitization, creating a forged log entry. This may mislead downstream components such as anomaly detectors and security monitors. codebase, which generate logs at runtime by recording key events, contextual data, and system states [21,31]. These logs provide developers and system operators with visibility into runtime behavior and serve as a foundational source of information for various software engineering tasks. In particular, logs play an essential role in debugging [58], performance analysis [59,60], system monitoring [10,24], and security auditing [12]. A typical logging statement includes a severity indicator (such asâinfoâorâerrorâ), a descriptive message, and runtime variables that provide contextual details. For example, as shown in Figure 1, a developer may log an error when parsing user input fails, recording both the error message and the input value to aid debugging. The resulting logs allow developers to trace system behavior, diagnose failures, and understand how external inputs interact with the application during execution. Although logs provide valuable insights into system behavior, improper logging practices can introduce serious security and privacy risks. For example, if logs are generated without appropriate safeguards such as input sanitization, sensitive data redaction, or format validation, they may inadvertently expose confidential information or become vectors for attacks such as log injection (e.g., Log4Shell [2]). As illustrated in Figure 1, unescaped user input embedded within a logging statement can be crafted to inject forged entries that resemble legitimate log messages. As a result, these forged entries may compromise the integrity of the log stream and contaminate downstream log-based tasks, including parsing [27, 28, 38], root cause analysis, and security monitoring. While prior research has acknowledged the security and privacy risks associated with logs, most existing work has focused primarily on the content of generated logs, especially from a data privacy perspective [6,7]. However, these postmortem approaches are inherently reactive as they address vulnerabilities only after potentially sensitive data has already been recorded and exposed. More importantly, these approaches often neglect the upstream causes of insecure logging, namely the code and developer decisions that introduce such risks in the first place. Despite the importance of preventing insecure logs at their source, limited research has explored how developers may inadvertently expose sensitive or security-critical information through logging statements. In addition, several studies have investigated the characteristics of logging statements and devel- oped tools to detect common issues such as missing logs, incorrect severity levels, or inconsistent variable usage [18,34,37]. While these efforts enhance the general quality of logging code, they rarely address the security dimension of logging practices. Risks such as log injection or log forging, where attackers influence log content to mislead downstream analysis, have received little atten- tion. As a result, existing detection tools might be insufficient for identifying security-sensitive flaws introduced during logging. These limitations highlight the need for approaches that can reason about both the content and context of logging statements, particularly in security-sensitive scenarios. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:3 To bridge this knowledge gap, we first conduct a systematic empirical study to characterize the landscape of logging security issues. This foundational analysis is essential to foster a comprehensive understanding of logging security issues and how they manifest in real-world code, thereby informing the development of effective detection and mitigation strategies. Through a systematic analysis of existing literature and established vulnerability databases, including the Common Weakness Enumeration (CWE) [3] and the Common Vulnerabilities and Exposures (CVE) [1], we derive a detailed taxonomy that classifies logging security issues into four main categories and 10 specific patterns. This taxonomy serves as the theoretical foundation for understanding the scope and nature of logging security issues. Building on this empirical foundation, we then explore the potential of LLMs to detect and repair the security issues in logging code. LLMs have demonstrated strong capabilities in understanding and reasoning about source code, supporting tasks such as code completion, defect detection, and performance modeling [9,13,14,41,46,48,51,52,61]. However, their effectiveness in identifying and repairing logging security issues remains largely unexplored. In this study, we construct a benchmark dataset,SecLogging, which contains 101 real-world logging security issues grounded in our taxonomy and systematically evaluate multiple open-source and proprietary LLMs on their ability to detect and repair these real-world issues. Our evaluation results show that the ability of LLMs to detect logging code security issues varies considerably, with category-level accuracy ranging from 12.9% to 52.5%. The most persistent challenge arises in abstract, logic-intensive cases such as Improper Redaction or Masking (RM), where most LLMs score below 20%. In contrast, concrete categories like Sensitive Information Exposure (S) are relatively easier to detect, with DeepSeek-V3 reaching as high as 85.7% accuracy under certain settings. Additionally, we find that introducing targeted contextual knowledge, particularly a brief issue description (+D), often helps the models improve, yielding noticeable accuracy gains (e.g., the category accuracy on average increases from 41.6% to 52.5% for DeepSeek- V3). Yet, richer contextual inputs do not guarantee further benefits. In fact, when descriptions and explanations are combined (+E+D), the performance of several models declines, which suggests that excess detail can overwhelm the models rather than guide them effectively. For repair, the challenge becomes more evident. Patch Similarity ranges from 18.8% to 69.3%, but the highest average performance is observed in theBasesetting (44.1%) without additional contextual information, whereas the most enriched prompts (+E+D) drop to 36.5% on average. The results indicate that concise prompts support more precise repairs, while overly elaborate context often introduces noise that degrades repair capability. Overall, our results highlight several key insights in secure logging and improving LLMs. We find that LLMs show uneven effectiveness: while they can detect certain issues with reasonable accuracy, their ability to repair issues is less reliable. Targeted and concise contextual information can improve detection, but excessive detail often harms both detection and repair quality. These findings reveal the importance of precise prompting strategies, cost-efficient contextual enrichment, and continued human oversight. We summarize the contributions of this paper as follows: â˘We derive a comprehensive taxonomy of logging code security issues synthesized from literature and vulnerability databases, which covers four categories and 10 patterns. The taxonomy provides practical insights for practitioners and researchers aiming at secure logging. â˘We construct a benchmark dataset,SecLogging, consisting of 101 curated logging security issues annotated with our taxonomy. This benchmark enables systematic evaluation of LLMsâ capabilities in detecting and repairing real-world logging security issues. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:4He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li â˘We conduct a systematic evaluation of LLMs using our benchmark, and uncover actionable insights for practitioners and researchers on secure logging practices as well as directions for improving LLMs in this domain. Paper Organization.The remainder of this paper is organized as follows. Section 2 summarizes the background and related work of this paper. Section 3 details our methodology. Section 4 outlines the experimental setup, covering the chosen LLMs and our evaluation metrics. Section 5 presents the results. We discuss the implications of our findings in Section 6 and acknowledge the studyâs limitations in Section 7. Finally, Section 8 concludes the paper. 2 Background and Related Work In this section, we first provide background on key concepts essential to our study, including the role of logging and the standardized vulnerability databases we utilize. We then summarize prior research across three related domains: (1) security and privacy concerns specific to software logging, (2) broader challenges surrounding logging code quality, and (3) the emerging application of LLMs for logging and log analysis. 2.1 Background Software logging is a critical mechanism for monitoring, debugging, and auditing applications in production [8,10,21,24,30,31,58]. While indispensable, logging practices can also introduce significant security risks if not implemented carefully. To systematically categorize these risks, the security community relies on standardized references. The CWE is a community-developed list of software and hardware weakness types that serves as a common language for describing security flaws [3, 33, 57]. The CVE database provides a dictionary of publicly known information-security vulnerabilities in specific products, assigning each a unique identifier [1,20]. Our work leverages both CWE and CVE to ground our taxonomy of logging code security issues in established security standards. 2.2 Logging Security and Privacy The exposure of sensitive information in logs has long been recognized as a critical security and privacy concern. Prior studies have primarily focused on privacy leakage from logs [6]. For example, several works categorized different types of sensitive data disclosure and examined their consequences for users and systems [7,63,64,67]. Other efforts investigated logging-related risks in big data environments [44]. Research has also addressed specific attack vectors, such as log injection, with systems like LogInjector targeting injection flaws in web applications [45]. However, despite these advances, prior work has largely centered on analyzing the information contained in logs, while the security of the logging code itself has not been systematically studied. Our work fills this gap by providing a comprehensive taxonomy and benchmark of logging code security issues. 2.3 Logging Code Quality The quality of logging code directly impacts a systemâs maintainability and debuggability [50], as common anti-patterns like excessive logging, misused logging levels, and insufficient contex- tual detail can introduce significant system-level issues. Some studies have investigated general logging-related issues by analyzing issue reports from two projects [23], while others have focused on specific aspects such as âhow-to-logâ [11] and âwhere-to-logâ practices [35], temporal inconsis- tencies between logging and surrounding code [19], duplicate logging code [36], and readability issues [34]. These efforts typically rely on mining commit histories [11,19,36] or conducting developer surveys [34]. More recently, Zhong et al. [65] proposed LogUpdater, which identifies Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:5 PromptDesign Stage 1: Taxonomy Derivation Academic Literature CVE & CWE ManualData Analysis Taxonomy: 4 Categories & 10 Patterns Stage 2: Real-world Issues Investigation GitHub JIRA Issue Reports Match to Taxonomy SecLogging: Benchmark Dataset Stage 3: LLM-based Issue Detection Pattern Explanation Task Decomposition Prompt Design Issue Description Enriched Context Stage 4: LLM-based Issue Repair Outputs: Detection & PatternsClassification Outputs: Repaired Code Similarity Comparison Manual Verification Detection Classification Remediation Fig. 2. An overview of our study. logging defects based on types mined from commit histories, primarily targeting inconsistency and readability issues. Prior studies have revealed that these logging code quality problems mainly affect the usefulness, reliability, and interpretability of the generated logs themselves. In contrast, the logging code security issues examined in our work can compromise the logs (e.g., leaking sensitive information) and further pose risks to the stability and security of the whole system. 2.4 LLMs for Logging and Log Analysis Researchers have started probing LLMâs capabilities for generating logging code. For example, LANCE [42] employs the T5 model to automatically generate complete logging code. Similarly, UniLog leverages in-context learning, enabling LLMs to generate logging code with only a few example prompts, without requiring fine-tuning [55]. Prior studies have employed LLMs to auto- matically generate complete logging code based on source code context [32,66]. These approaches demonstrate the potential of LLMs to automate logging practices. In parallel, LLMs have also been applied to log analysis, particularly log parsing. LogBatcher [54] targets the need for labeled data or model training by clustering logs, matching caches, and applying batch prompt contexts, improving efficiency and scalability. KnowLog enhances [39] pre-trained language models with domain-specific knowledge and contrastive learning to better capture abbreviations and contextual cues in logs, thereby improving exception detection. LibreLog builds on open-source LLMs [40] to perform unsupervised log parsing using similarity scoring, self- reflection, and template recall, which achieved high parsing accuracy and speed. These advances illustrate how LLMs can streamline the tasks of log analysis, especially log parsing. Building on these lines of research, our work focuses on the security dimension of logging code. We leverage LLMs to detect, classify security issues in logging code, and generate repair solutions. In this way, our study complements prior work on logging automation and log analysis, while filling an important gap in understanding and addressing security issues in logging code. 3 Methodology In this paper, we conduct a comprehensive study on security issues in logging code. Figure 2 presents an overview of our study, which involves four stages.Stage 1:We collect and analyze resources related to logging code security issues, including academic literature, CVEs, and CWEs. Based on this analysis, we derive a taxonomy that characterizes common categories and patterns of logging code security issues.Stage 2:We collect real-world issue reports from open-source projects and annotate them according to the taxonomy established in Stage 1.Stage 3:We investigate the capability of LLMs to automatically detect these security issues by leveraging the constructed Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:6He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li taxonomy as guidance.Stage 4:We further explore the potential of LLMs to repair the detected issues. 3.1 Deriving the Taxonomy of Logging Code Security Issues To construct a comprehensive taxonomy of security issues in logging code, we systematically gather and analyze data from three sources: academic literature, the Common Weakness Enumeration (CWE), and the Common Vulnerabilities and Exposures (CVE) database in Stage 1. Academic Literature:To the best of our knowledge, there are no prior studies that directly focus on security issues in logging code. Therefore, we begin by collecting papers that investigate defects in logging or improving logging practices. Similar to prior studies [15,25,29], our literature review follows a structured process to identify relevant research. First, we perform an initial search of major digital libraries (IEEE Xplore, ACM Digital Library, SpringerLink), targeting publications from 2019 to 2025. Our search keywords include âloggingâ, âlogging securityâ, âsensitive logâ, and âlog injectionâ. We then screen the titles and abstracts of the retrieved articles to filter out irrelevant publications. Finally, we conduct a full-text review of the remaining papers and follow a snowballing strategy [53], examining their citations to discover additional relevant work. This process yields a list of 111 papers for further analysis. CWE and CVE:To ground our taxonomy in established security standards and real-world incidents, we analyze data from two key vulnerability databases. First, we manually review all entries in the MITRE CWE catalog [3] and identify 13 CWE instances related to logging code based on their descriptions. Second, we search the CVE database [1] using the keywords âloggingâ and âlogsâ, then manually filter the retrieved results to obtain 19 CVEs that represent concrete cases of logging code security issues. Data Analysis.We first examine the titles and abstracts of all collected papers and filter out those unrelated to security issues in logging code. Examples of excluded topics include improving logging practices without a security focus. After this process, we retain a set of 16 papers that mention security risks, patterns, or countermeasures in logging code. Two authors of this paper then independently review these papers and derive an initial set of codes that reflect recurring categories of logging-related security issues. In parallel, we manually inspect the filtered CWE entries and CVE reports to extract relevant vulnerability characteristics. The authors perform open coding and card sorting on the collected codes to group them into higher-level themes. Through iterative discussion, overlapping or ambiguous codes are merged or refined to ensure conceptual clarity and semantic consistency. This process results in a set of four categories and 10 detailed patterns that form our taxonomy of logging code security issues. 3.2 Investigation on Real-world Issues In Stage 2, we investigate real-world security issues in logging code and examine whether the derived taxonomy can effectively categorize these issues. To this end, we collect issue reports from open-source project trackers, including Jira and GitHub. We first retrieve all logging-related issue reports by conducting a broad search using general logging keywords (e.g., âlogâ, âloggingâ, âsensitive informationâ) with specific status (e.g., âCloseâ, âOpenâ, âIn Progressâ, âResolvedâ, âFixedâ) from 2019 to 2025. To maximize coverage, this initial search does not restrict the choice of projects and returns 949 issue reports across 93 projects. We then perform a filtering process to identify those relevant to security. Specifically, we remove low-quality or unsuitable reports (e.g., toy projects, incomplete descriptions, or duplicates) at the outset, and retain only those with clear descriptions, available solutions, and explicit evidence of being security-related. In particular, we only keep reports whose status isâResolvedâ,âClosedâ, or for which a solution is explicitly provided. This Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:7 process yields our curated dataset,SecLogging, with 101 security-related logging issue reports from 63 projects. With this list, we annotate each issue according to the taxonomy constructed in Stage 1. Specif- ically, two authors independently review the issue description, developer discussions, and cor- responding code changes, and annotate the report with one of the taxonomyâs categories and patterns. Disagreements are resolved through discussion until consensus is reached. Following prior studies [26,49], we use Cohenâs Kappa [43] to measure the agreement of results between the two authors. The Cohenâs Kappa value in this process is 0.762, which indicates a substantial agreement. During this annotation process, we did not identify any new categories or patterns. In addition, for each report, we extract both the problematic code snippet (before the repair) and the corrected version (after the repair), thereby constructing a paired dataset that serves as the ground truth for subsequent experiments on detecting and repairing them with LLMs. Table 1 summarizes the distribution of the annotated issue reports across the taxonomyâs categories and patterns. More details will be discussed in RQ1 of Section 5. 3.3 Framework for Detecting and Repairing Logging Code Security Issues with LLMs To examine the capabilities of LLMs in detecting and repairing the logging code security issues, we propose a structured analysis framework that consists of three components:Contextual Enrichment, Decomposed Task Execution, andStructured Output Generation. Contextual Enrichment.Code snippets alone are often insufficient for identifying security issues. We therefore enrich the input with additional contextual knowledge. Specifically, we provide two sources of knowledge: (1)Security Pattern Explanation (+E), which describes the definition and characteristics of a security pattern, and (2)Issue Description (+D)taken from the original issue report, which provides human-written details of the problem, and is used to simulate scenarios where developers already possess some prior knowledge about the issue. These contexts can be supplied individually or in combination (+D+E) to enhance the LLMsâ understanding. Decomposed Task Execution.We decompose the overall analysis-and-repair process into three sub-tasks that mirror a human expertâs workflow, including (1)Detection:determining whether a logging code security issue exists; (2)Classification:mapping the issue to a category and pattern in our taxonomy; and (3)Remediation:proposing a concrete repair suggestion. Structured Output Generation.Finally, we constrain the results to a predefined, machine-parsable format, which is consistent in different experimental settings and easy for developers to understand systematically. The complete implementation details of our framework, including the corresponding prompt templates, can be found in our replication package [5]. 4 Experimental Setup In this section, we describe the design and implementation of our experiments to evaluate the effectiveness of LLMs in detecting and repairing security issues in logging code. We aim to answer the following research questions: â˘RQ1:What security issues exist in logging code? â˘RQ2:How effective are LLMs in detecting security issues in logging code? â˘RQ3:To what extent can LLMs repair security issues in logging code? 4.1 Evaluation Metrics The overall detection and repair process is formulated as a multi-step problem. Specifically, the LLM first examines whether the input logging code contains a security issue. If an issue is detected, it Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:8He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li identifies the category of security issues as the parent class, along with the corresponding patterns as the sub-class. It then provides repaired code based on input logging code. Evaluation Metrics for Detection.We analyze the output results which contain both category and pattern for each issue against the ground truth, calculating accuracy for the overall category, overall pattern, as well as individual accuracies for each category and pattern. Specifically, the accuracy is computed as: Accuracy Category/pattern = Number of Correctly Detected Instances Total Instances of the Category / Pattern Evaluation Metrics for Repaired Code Patch.For the evaluation of repaired code, we combine textual similarity with manual examination. For textual similarity between the solution provided by the LLM and ground truth, we adopt two commonly used measurements: Sequence Similarity and Jaccard Similarity. Sequence Similarity captures structural code changes and small-scale edits such as variable or function name modifications by measuring character-level differences. Meanwhile, Jaccard Similarity better reflects semantic similarity in terms of API usage and keywords by comparing vocabulary overlap, regardless of their exact ordering. These complementary metrics help assess different aspects of code similarity, and they are computed as follows: SequenceSim(í´,íľ)=1â EditDistance(í´,íľ) íííĽ(|í´|,|íľ|) JaccardSim(í´,íľ)= |í í´ âŠí íľ | |í í´ âŞí íľ | whereí í´ ,í íľ are the vocabulary sets of texts A and B,í í´ âŠí íľ is the vocabulary intersection, andí í´ âŞí íľ is the vocabulary union. We then introduce an overallPatch Similarityas an average of the two metrics: PatchSim(í´,íľ)= SequenceSim(í´,íľ)+JaccardSim(í´,íľ) 2 This combined metric balances sensitivity to fine-grained structural edits (captured by Sequence Similarity) with robustness to semantic equivalence at the token level (captured by Jaccard Similar- ity). By averaging both, Patch Similarity provides a more comprehensive estimate of how closely an LLM-generated patch matches the ground truth, mitigating the biases of relying on only one type of similarity. In other words, a patch that differs in variable naming but preserves functional keywords should still be considered highly similar, while one that reuses many keywords but substantially changes the structure should be penalized. Apart from textual similarity, we also manually examine the effectiveness of repair suggestion that is elaborated in Section 7. 4.2 Studied LLMs and Implementation Details In our experiments, we examine both open-source and closed-source LLMs. For open-source mod- els, we select seven representative LLMs from three model families: Llama [22] (Llama3.3), Qwen series [47,56] (Qwen2.5-72b,Qwen3-4b,Qwen3-32b), and DeepSeek [16,17] (DeepSeek-R1-32b, DeepSeek-R1,DeepSeek-V3). For closed-source LLM, we selectGPT-4.1-nano, primarily consider- ing the affordability of API cost. For Llama3.3, Qwen series, and DeepSeek-R1-32b, we deploy their public checkpoints locally on our machine and invoke them using the Ollama framework [4]. For GPT-4.1-nano,DeepSeek-V3, andDeepSeek-R1, we utilize their official, publicly available APIs. All experiments are conducted on a Linux server running Ubuntu 20.04.6 LTS, equipped with an AMD 32-core processor, 1TB RAM, and eight NVIDIA A6000 GPUs. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:9 Table 1. Taxonomy of Logging Security Issues with Instance Counts inSecLogging Security Issue CategoriesSecurity Issue Patterns # Instances (Category) # Instances (Pattern) IL:Insecure Log Storage & Access Control IL-At:Risk of Log Injection Attacks 13 2 IL-Pa:Publicly Accessible Logs1 IL-Lv:Insecure Logging Level Configuration10 S:Sensitive Information Exposure S-Cr:Credentials Leakage 38 20 S-Cf:Configuration Data Exposure7 S-Ur:User Private Data Leakage11 RM:Improper Redaction or Masking RM-Ms:Missing Masking/Redaction 43 36 RM-Ft:Faulty Masking/Obfuscation7 E:Error & Exception Message Exposure E-Ex:Exception Leakage 7 5 E-St:Stack Trace Leakage2 Total101 5 Results In this section, we discuss the results of our RQs. 5.1 RQ1: What security issues exist in logging code? 5.1.1Motivation.In this RQ, we focus on what security issues exist in logging code, and how can they be systematically categorized. Through our comprehensive investigation (cf. Section 3), we derive a taxonomy consisting of four logging code security issue categories and 10 corresponding patterns, and annotate real-world issue reports using our taxonomy. 5.1.2Results.Table 1 summarizes the taxonomy with instance counts in our constructed bench- mark dataset,SecLogging. For clarity and readability, some of the code snippets presented in this section have been simplified to highlight only the parts relevant to the security problem. Where applicable, we also attach CWE/CVE entries that share similar characteristics with the pattern. Note that these may not represent a one-to-one match, as our patterns are abstracted from both prior literature and CWE/CVE descriptions. Insecure Log Storage and Access Control (IL).This category refers to unsafe logging access and storage practices, where log files or output channels lack appropriate security protections. It emphasizes problems in how logs are stored or configured, rather than in the content being logged. This category has 13 instances in total and includes three patterns: IL-At (2 instances), IL-Pa (1 instances), and IL-Lv (10 instances). ⢠IL-AtRisk of Log Injection Attacks:Log entries lack of filtering, enabling attackers to inject malicious content. Real-world Example:In Apache Karaf (KARAF-7061), the default Log4J2 configuration was vulnerable to log injection attacks. The issue originated from the use of thelog4j2.pattern property with theâ%mâdirective, which prints raw log messages without escaping. As a result, attackers could inject carriage return/line feed (CRLF) characters or HTML tags to forge new log entries or launch Cross-Site Scripting (XSS) attacks when logs were rendered in web-based viewers or security audits. // KARAF-7061 color.trace = cyan log4j2.pattern = %dISO8601 | %-5p | %-16t | %-32c1 | %Xbundle.id - %Xbundle.name - %Xbundle.version | %m%n Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:10He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li log4j2.out.pattern = 001b[90m%dH:m:s.S 001b[0m%highlight%-5level FATAL=$color.fatal, ERROR=$color.error, WARN=$color.warn, INFO=$color.info, DEBUG=$color.debug, TRACE=$color.trace 001b[90m[%t] 001b[0m %msg%n%throwable Similar CWE/CVE:CWE-434âUnrestricted Upload of File with Dangerous Typeâ, CVE-2021-44228, CVE-2020-17449. â˘IL-PaPublicly Accessible Logs:Log files or output channels are open to access, allowing any unauthorized user to directly view internal operational details. Real-world Example:In Apache Kafka (KAFKA-698), brokers could mistakenly expose un- committed messages to consumers. The problem was caused by incorrect handling of the high watermark, which defines the offset boundary for committed messages. Because of this flaw, read requests could go beyond the boundary and allow consumers to access uncommitted data that might later be rolled back, and further lead to potential data leakage and inconsistency. // KAFKA-698 (simplified) val offsets = if (assignOffsets) val firstOffset = nextOffset.get validMessages = validMessages.assignOffsets(nextOffset, messageSetInfo.codec) val lastOffset = nextOffset.get - 1 (firstOffset, lastOffset) else if (!messageSetInfo.offsetsMonotonic) throw new IllegalArgumentException("Out of order offsets") nextOffset.set(messageSetInfo.lastOffset + 1) (messageSetInfo.firstOffset, messageSetInfo.lastOffset) Similar CWE/CVE:CWE-312âCleartext Storage of Sensitive Informationâ, CVE-2023-6136. â˘IL-LvInsecure Logging Level Configuration:Improper log level settings, which result in sensitive data or system specifics being captured in logs. Real-world Example:In Apache Camel (CAMEL-14150), theâExecCommandâclass logged sensitive command details such as executables and arguments through itsâExecCommandâmethod. Because this information was recorded at theâINFOâlevel, internal system details were exposed in standard application logs. // CAMEL-14150 (simplified) public class ExecCommand private String executable; private List<String> args; private String workingDir; private File outFile; @Override public String toString() return "ExecCommand [args=" + args + ", executable=" + executable + ", workingDir=" + workingDir + ", outFile=" + outFile + "]"; Similar CWE/CVE:CVE-2024-32474. Sensitive Information Exposure (S).This category covers cases where sensitive data is recorded directly or indirectly in logging code, leading to security or privacy leaks. This category includes three patterns (i.e.,S-Cr, S-Cf, S-Ur). This category has 38 instances in total and includes three patterns: S-Cr (20 instances), S-Cf (7 instances), and S-Ur (11 instances). â˘S-CrCredentials Leakage:Logs capture sensitive credentials like accounts, passwords, API keys, or OAuth tokens, which could be stolen and used in attacks. Real-world Example:In Apache Zeppelin (ZEPPELIN-2733), DEBUG-level logs printed sensitive cryptographic information, including an encrypted user key and its initialization vector (i.e., Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:11 âIVâ). An attacker with access to these logs could obtain both values, which allow them to analyze the encryption scheme or attempt offline decryption. // ZEPPELIN-2733 (simplified) private String getAuthKey(String userKey) LOG.debug("Encrypted user key is ", userKey); return decrypt(userKey, token.hashCode()); private String decrypt(String value, String initVector) LOG.debug("IV is , IV length is ", initVector, initVector.length()); // ... Similar CWE/CVE:CWE-532âInsertion of Sensitive Information into Log Fileâ, CVE-2024-5908. â˘S-CfConfiguration Data Exposure:Log outputs include database connection URLs, critical file paths, environment variables, or other system configurations that attackers could exploit for infiltration. Real-world Example:In Apache Samza (SAMZA-589), theâMapConfigâclass logged its entire configuration map via an unfilteredâtoString()âmethod. As a result, sensitive values such as passwords, API keys, and database connection strings were exposed in plaintext in both logs and the application UI. // SAMZA-589 (simplified) public class MapConfig extends Config private final Map<String, String> map; @Override public String toString() return map.toString(); // Exposes full config including credentials Similar CWE/CVE:CWE-532âInsertion of Sensitive Information into Log Fileâ, CVE-2024-32474. â˘S-UrUser private data Leakage:Log output includes user cookies, usernames, password, phone numbers, transaction records, or other personal data, leading to compliance issues and privacy violations. Real-world Example:In Apache Hadoop (HADOOP-17510), a trace-level logging statement in theâAuthCookieHandlerâprinted the contents of a userâs authentication cookie. Even though logged atTRACElevel, such cookies are sensitive and could be exploited for session hijacking if attackers gained access to the logs. // HADOOP-17510 (simplified) public class AuthCookieHandler private HttpCookie authCookie; public void setAuthCookie(HttpCookie cookie, HttpCookie oldCookie) this.authCookie = cookie; LOG.trace("Setting token value to ()", authCookie, oldCookie); Similar CWE/CVE:CWE-532âInsertion of Sensitive Information into Log Fileâ, CVE-2024-5557. Improper Redaction or Masking (RM).This category covers insufficient redaction or masking in logging code, where sensitive information that should be hidden is either overlooked or handled poorly. This category highlights whether the logging code properly handles potentially risky information. This category includes two patterns (i.e.,RM-Ms, RM-Ft). This category has 43 instances in total and includes three patterns: RM-Ms (36 instances), and RM-Ft (7 instances). Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:12He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li â˘RM-MsMissing Masking/Redaction:Sensitive details are left entirely exposed, or dumping too much information that is hard to catch if sensitive information is leaked, exposing raw content directly in logs. Real-world Example:In Apache Hive (HIVE-9994), the query redaction mechanism is performed inâredactQuery()âafter the query is being logged inâPerfLogBegin()â. This ordering causes raw query strings containing confidential information (e.g., table names, user identifiers) to be logged before redaction, risking exposure of confidential details to anyone with access to logs. // HIVE-9994 (simplified) sem.validate(); perfLogger.PerfLogBegin(CLASS_NAME, PerfLogger.DRIVER_RUN, command); plan = new QueryPlan(command, sem, perfLogger.getStartTime(), queryId, ...); String queryStr = plan.getQueryStr(); for (Redactor r : getHooks(Redactor.class)) queryStr = r.redactQuery(queryStr); conf.setVar(HiveConf.ConfVars.HIVEQUERYSTRING, queryStr); Similar CWE/CVE:CWE-779âLogging of Excessive Dataâ, CVE-2024-45784, CVE-2025-6711. â˘RM-FtFaulty Masking/Obfuscation:Even when masking is implemented, defects still allow some sensitive content to slip through in logging code. Real-world Example:In Apache Kafka (KAFKA-4056), theâlogUnused()âmethod failed to apply masking when logging unrecognized configuration properties. As a result, if a sensitive key (e.g., a password) was misspelled, its plaintext value was written directly to the logs and bypasses the intended redaction mechanism. // KAFKA-4056 public void logUnused() for (String key : unused()) log.warn("The configuration = was supplied but isn't a known config.", key, this.originals.get(key)); Similar CWE/CVE:CVE-2025-53650, CVE-2024-32474. Error and Exception Message Exposure (E).This category covers the exposure of error or exception details in logging code, with messages or stack traces disclosing sensitive data or internal system workflows. This category specifically points to critical details leaking via exception handling routes like error messages or stack traces. This category has 7 instances in total and includes two patterns: E-Ex (5 instances), and E-St (2 instances). ⢠E-ExException Leakage:Exception message includes keys, file paths, config parameters, or other sensitive information, this allows attackers gain insights into the systemâs internals. Real-world Example:In Apache Hive (HIVE-20644), exception messages exposed sensitive run- time data by directly concatenating raw parameters, such asâargumentStringâorârowStringâ, into error messages. When these exceptions were logged, user data and system details were leaked into log files. // HIVE-20644 (simplified) try // Sensitive argumentString leaked in exception throw new HiveException( "Unable to execute method " + m + " with arguments " + argumentString, e); catch (Exception e) // Sensitive rowString leaked in exception throw new HiveException( "Hive Runtime Error while processing row " + rowString, e); Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:13 Table 2. Accuracy (%) of LLMs onSecLogging(RQ2). Model SettingEE IL RM SSEE-Ex E-St IL-At IL-Lv IL-Pa RM-Ft RM-Ms S-Cf S-Cr S-Ur Category Average Pattern Average GPT-4.1-nano Base14.320.010.087.20.00.0100.00.00.00.011.114.370.00.041.620.8 Base+E28.613.322.587.20.00.0100.010.00.00.013.942.915.036.446.517.8 Base+D28.613.312.592.320.00.050.00.00.00.013.928.655.00.044.519.8 Base+E+D28.626.77.589.720.050.0100.010.00.00.02.828.635.027.343.617.8 Llama3.3 Base14.313.317.571.820.00.0100.00.00.00.019.40.075.018.237.626.7 Base+E42.920.020.056.460.00.0100.010.00.00.019.414.365.018.235.628.7 Base+D28.613.325.076.920.00.0100.00.00.00.022.20.070.09.143.625.7 Base+E+D57.113.320.053.840.00.0100.00.00.00.019.40.055.09.134.722.8 DeepSeek-R1-32b Base0.00.02.530.80.00.00.00.00.00.03.00.030.00.012.96.9 Base+E14.36.75.035.90.00.00.00.00.00.00.014.020.00.018.05.0 Base+D14.30.02.530.820.00.00.00.00.00.02.80.040.00.013.99.9 Base+E+D28.66.75.028.220.050.050.00.00.00.02.814.315.09.115.88.9 DeepSeek-R1 Base28.613.312.579.540.00.050.00.00.00.08.328.660.027.339.622.8 Base+E57.113.315.059.080.00.0100.00.00.00.08.314.340.027.334.720.8 Base+D57.127.027.571.860.050.0100.020.00.00.016.70.060.036.446.529.7 Base+E+D71.413.330.061.560.050.0100.00.00.00.016.714.345.027.348.524.8 DeepSeek-V3 Base71.420.07.579.560.00.0100.00.00.00.05.628.645.027.341.620.8 Base+E71.433.312.576.940.00.0100.020.00.014.35.642.940.027.344.622.8 Base+D85.713.327.587.2100.050.0100.00.00.014.319.428.675.027.352.535.6 Base+E+D85.7 33.320.076.980.050.0100.020.00.014.311.128.655.027.348.529.7 Qwen2.5-72B Base0.013.35.074.40.00.0100.00.00.00.02.828.670.09.132.719.8 Base+E28.620.022.559.040.00.0100.010.00.00.013.90.045.018.236.620.8 Base+D28.613.35.074.40.00.0100.00.00.00.05.60.070.00.034.717.8 Base+E+D42.920.010.064.120.00.0100.010.00.00.08.30.040.018.234.716.3 Qwen3-4B Base0.06.75.059.020.00.050.00.00.00.05.60.050.027.325.716.8 Base+E14.320.017.548.720.00.050.010.00.00.016.742.940.09.129.720.8 Base+D14.313.37.561.540.00.0100.00.00.00.011.114.340.09.129.717.8 Base+E+D14.327.07.543.620.00.0100.020.00.00.08.314.340.09.124.817.8 Qwen3-32B Base0.020.025.061.50.00.0100.010.00.014.325.028.665.00.036.627.7 Base+E42.926.720.056.440.00.0100.020.00.00.011.128.640.09.136.620.8 Base+D42.920.020.051.30.050.0100.010.00.00.019.428.655.00.033.723.8 Base+E+D42.920.015.048.740.050.0100.010.00.00.08.314.355.018.230.722.8 Similar CWE/CVE:CWE-209âGeneration of Error Message Containing Sensitive Informationâ, CVE-2019-16768. â˘E-StStack Trace Leakage:Stack Traces logged reveal function call chains, internal class names, or library versions, this provides attackers with valuable intelligence on the systemâs architecture. Real-world Example:In Apache Commons Net (NET-618), when aâParseExceptionâis caught, the code directly callsâe.printStackTrace()â. This method prints the full stack trace to the standard error stream, which is often captured in application logs. This exposed detailed internal information, including method call chains, class names, and line numbers, which attackers could exploit to map the systemâs architecture or identify vulnerable libraries. // NET-618 try file.setTimestamp(super.parseTimestamp(datestr)); catch (ParseException e) e.printStackTrace(); Similar CWE/CVE:CWE-117âImproper Output Neutralization for Logsâ, CVE-2020-4085. RQ1 Summary:We derive a taxonomy consisting of four categories and 10 patterns of logging code security issues, and observe that these issues are widely present in real-world projects. Among them, Improper Redaction or Masking (RM) is the most prevalent category with 43 instances in our collected issue reports. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:14He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li 5.2 RQ2: How effective are LLMs in detecting security issues in logging code? Motivation.LLMs have shown promising results in detecting general software security issues such as vulnerabilities [57]. However, it remains unclear whether these capabilities extend effectively to the domain of logging code, where security issues have distinct characteristics. Therefore, in this RQ, we investigate how efficiently LLMs can detect security issues in logging code. Approach.We run experiments onSecLoggingand use the framework discussed in Section 3 to evaluate the effectiveness of LLMs in detecting security issues in logging code. For each contextual configuration (i.e., Base, Base+D, Base+E, Base+D+E), we compute the detection accuracy at both the category and pattern levels. Results.Table 2 presents the results of this RQ.Adding an issue description(+D)generally provides the most consistent improvement across categories.For instance, DeepSeek-R1âs average category accuracy rises from 39.6% to 46.5% with this addition. Explanations(+E)also help in some cases, but their effect is less stable, and combining both description and explanation(+D+E) does not always yield better outcomes. In fact, richer context sometimes may lead to degraded results, such as Llama3.3, whose average accuracy drops to 34.7% under(+D+E). These findings suggest that concise, targeted guidance is often more effective, and when developers already possess some understanding of the issue (e.g., through an issue description), this knowledge can better guide LLMs to improve detection. LLMâs capability plays an important role.In particular, DeepSeek-V3 achieves the highest accuracy overall, reaching 85.7% on the E category with(+D). Other LLMs like DeepSeek-R1 and Llama3.3 also perform competitively, while smaller models such as DeepSeek-R1-32b struggle compared to other larger LLMs, rarely exceeding 20% in category accuracy across settings. Performance varies substantially across categories and patterns.Among categories, S (Sensitive Information Exposure) is relatively easier to detect, with several LLMs exceeding 70% accuracy, while RM (Improper Redaction or Masking) remains the most difficult, often falling below 30%. Within patterns, concrete issues such as IL-At (Log Injection Attacks) and S-Cr (Credentials Leakage) are reliably identified, with models frequently achieving above 70%. In contrast, abstract and logic-dependent patterns like RM-Ms (Missing Masking/Redaction) are more challenging, with almost all the LLMs performing poorly. It suggests that LLMs may handle well-defined, token-based flaws more effectively than subtle issues requiring reasoning about missing mechanisms. RQ2 Summary:Overall, the detection accuracy of LLMs at the category level varies widely, ranging from 12.9% to 52.5% on average across models and settings. Adding an issue description (+D) generally provides the most consistent improvement. In addition, performance differs considerably across patterns: models achieve high accuracy on concrete issues such as Sensi- tive Information Exposure (S), but consistently struggle with abstract, logic-based flaws like Improper Redaction or Masking (RM), where accuracy often falls below 20%. 5.3 RQ3: To what extent can LLMs repair security issues in logging code? Table 3. Patch Similarity for Different LLMs. SettingGPT-4.1-nano Llama3.3 DeepSeek-R1-32b DeepSeek-R1 DeepSeek-V3 Qwen2.5-72B Qwen3-4B Qwen3-32BAvg. Base29.843.424.260.669.348.536.640.544.1 Base+E28.538.523.649.166.043.631.237.239.7 Base+D27.234.118.858.568.342.931.435.039.5 Base+E+D28.334.820.347.363.434.730.732.536.5 Average28.437.721.753.966.842.432.536.340.0 Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:15 Motivation.Recent advances show that LLMs can effectively support automated program repair by producing meaningful patches for buggy code [62]. This opens the question of whether such capabilities can be leveraged to handle security issues in logging code, which often involve domain- specific risks and knowledge. In this RQ, we examine how well LLMs can generate repairs for security issues in logging code. Approach.We use the same set of LLMs and prompt configurations as in RQ2 and run experiments onSecLogging. The models are tasked with generating a repaired code patch for the identified security issue. We evaluate the results in two aspects: automated textual patch similarity, and manual evaluation. âś Textual Patch Similarity Evaluation.We compare generated patches against ground-truth repairs using thePatch Similaritydiscussed in Section 3. âˇManual Evaluation.To validate automated metrics and gain qualitative insights, two authors independently evaluate each case by examining the generated code patch. If the patch is considered to be reasonable and effective, the result is marked aseffective. In cases of disagreement, the two authors discuss together to reach a consensus. Results.We discuss the results of textual patch similarity and manual evaluation, respectively. âśTextual Patch Similarity Evaluation.Table 3 presents the similarity-based evaluation results. Different from detection accuracy,adding contextual information does not consistently improve repair patch generation. When comparing the four prompt settings, theBasesetting achieves the highest overall average similarity (44.1%), while adding contextual information (i.e., +E,+D, or+E+D) generally reduces performance, with the lowest average observed forBase+E+D (36.5%). For example, DeepSeek-V3 peaks at 69.3% in the Base setting, while its performance drops to 63.4% under(+E+D). Similarly, Llama3.3 falls to 34.8% with(+E+D)from 43.4% in the setting of (Base). These results suggest that supplementary context can sometimes be helpful, but it may also introduce noise that reduces the ability to repair issues. The results showdisparities in the similarity across different LLMs. Overall, DeepSeek-V3 outperforms other LLMs, with Patch Similarity consistently above 63.4% and peaking at 69.3%. DeepSeek-R1 also performs strongly, reaching up to 60.6%. Other LLMs such as Qwen2.5-72B also achieve reasonable results (up to 48.5%), while smaller models like GPT-4.1-nano consistently remain below 30%. ⡠Manual Evaluation.Overall,the results of our manual evaluation align with the Patch Similarity results.Specifically, we examine repair results from the experiment of highest Patch Similarity (i.e., DeepSeek-V3 withBase) and the lowest (i.e., DeepSeek-R1-32b with+D). We find that DeepSeek-V3 attempt to repair all the issues in the dataset, and 84 of these repairs (83.2%) are labeled as effective. By contrast, DeepSeek-R1-32b provide repair suggestions for only 36 issues, of which 27 (75.0%) are effective. In addition, we also conduct case studies to analyze the results of high and low Patch Similarity in detail. Case Study.Figure 3 shows a case where the LLM provides a repair with a relative high Patch Similarity (92.8%) with the developerâs repair. The code is a Pythonâ__repr__âmethod that exposes multiple fields of Personally Identifiable Information (PII), including the customerâs name, address, andâpeselâ(a national identification number), directly to the logs. This corresponds to the S-Ur (User Private Data Leakage) pattern. Both the developer and the LLM remediate the issue by redacting the sensitive fields. The developer apply full masking, replacing values with asterisks, whereas the LLM use a more context-aware approach: general fields are fully masked, but theâpeselânumber is partially redacted to reveal only the last four digits. This reflects a common industry practice that balances security with usability, which shows that the LLM can align with conventions for handling sensitive data. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:16He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li Original Developer Patch LLM-generated Patch def__repr__(self): return ( f"Customer(ID: self.id, Name: self.name, City: self.city, " f"Age: self.age, Pesel: self.pesel, Street: self.street, " f"AppNo: self.appNo)" ) def__repr__(self): return ( f"Customer(ID: self.id, Name: '*' * len(self.name), " f"City: '*' * len(self.city), Age: '*' * len(self.age), " f"Pesel: '*' * len(self.pesel), Street: '*' * len(self.street), " f"AppNo: '*' * len(self.appNo))" ) def__repr__(self): masked_name = '*' * len(self.name) if self.name elseⲠmasked_pesel = f"PESEL: ***str(self.pesel)[-4:]"if self.pesel else "PESEL: None" masked_street = '*' * len(self.street) if self.street elseⲠreturn ( f"Customer(ID: self.id, Name: masked_name, City: self.city, " f"Age: self.age, masked_pesel, Street: masked_street, " f"AppNo: self.appNo)" ) private static String stringifyOption(String key, String value) return String.format( "'%s'='%s'", EncodingUtils.escapeSingleQuotes(key), EncodingUtils.escapeSingleQuotes(value)); private static String stringifyOption(String key, String value) if (GlobalConfiguration.isSensitive(key)) value = HIDDEN_CONTENT; return String.format( "'%s'='%s'", EncodingUtils.escapeSingleQuotes(key), EncodingUtils.escapeSingleQuotes(value)); defaultValue(100L).withFallbackKeys("fallback -buffer-size"); public static final ConfigOption<String> PASSWORD = ConfigOptions.key("password").stringType().no DefaultValue(); public staticfinal ConfigOption<String> KEY_FORMAT = ConfigOptions.key("key" + FORMAT_SUFFIX).stringType().noDefaultValue(); options.add(KEY_FORMAT); options.add(FORMAT); options.add(VALUE_FORMAT); options.add(PASSWORD); return options; a. High Similarity Case (92.8). b. Low Similarity Case (22.3). Fig. 3. Case study with different Patch Similarity. Figure 3 also illustrates another case with a low patch similarity score (22.3%), which shows a tendency for LLMs to propose overly complex and indirect solutions. The issue in the original code exists in thestringifyOptionmethod, which is used to format configuration options for Table API exception messages. The method performed no sensitivity checks, causing sensitive values, such as passwords, to be leaked in plaintext through simple string formatting. The developerâs ground-truth repair is direct and effective. It modifies thestringifyOptionmethod to integrate aGlobalConfiguration.isSensitive(key)check, which replaces any sensitive value with a HIDDEN_CONTENTplaceholder before it is formatted into the final string. In contrast, the LLMâs output does not perform this simple, targeted modification. Instead, it suggests an unnecessary refactoring by introducing a newConfigOptionclass to manage configuration variables. While this alternative approach might implicitly mask some values, it fails to directly address the insecure formatting in the original method, leaving a potential risk of data leakage. This change to the existing code logic greatly increases the complexity and cost of manual review and future maintenance. RQ3 Summary:Our automated and manual evaluation show that LLMs can provide repairs for logging code security issues with varying effectiveness. DeepSeek-V3 achieves the highest Patch Similarity (up to 69.3%), while smaller LLMs such as GPT-4.1-nano consistently underperform. Different from detection, adding contextual information does not help in the capability of providing repairs and can sometimes considerably reduce it. 6 Discussion 6.1 Analysis of Token Consumption To better understand the trade-off between cost and effectiveness, we analyze the impact of input token usage across different prompt settings. Figure 4 illustrates that adding more context increases Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:17 Fig. 4. Comparison of average detection accuracy and patch similarity across four settingsâ token usage. input tokens, but may or may not lead to performance improvement. For detection accuracy, we find that adding an issue description (+D) improves the accuracy from 36.5% to 37.4%, and the input token consumption increased from 3,003 to 3,688 on average. On the other hand, prompt with the most detailed contextual information (i.e.,+E+D) appears to be among the least effective while also incurring the highest cost. This observation is more obvious for code fixing issues, where theBase prompt yields the highest Patch Similarity on average and+E+Dachieves the worst results. These results suggest that for both detection and repair, concise and targeted prompts may provide a more favorable balance between cost and effectiveness, while overly detailed prompts may increase cost without improving outcomes. 6.2 Implications We discuss the implications of our study for practioners and researchers, respectively. Specifically, IMP-P#are implications for practitioners, andIMP-R#are for researchers. IMP-P1. Guidelines for secure logging and support for rapid incident response.Our tax- onomy of logging security issues provides a concrete guideline to help developers and engineers prevent security issues. It can also be codified into automated tools, such as custom security-focused linting rules. When a security incident occurs, the taxonomy can also be used a structured frame- work to accelerate investigation and response, potentially lowering Mean Time to Detect (MTTD) and Mean Time to Repair (MTTR). When using LLMs for security analysis, our results show that providing concise, specific context (e.g., an issue description) is crucial for improving detection accuracy. IMP-P2. Contextual information is not âthe more, the betterâ.Our results show that concise, specific context can substantially improve detection accuracy (e.g.,+D) . However, it is not always âthe more, the betterâ. For example, combining both explanation and issue description (i.e.,+E+D) degrades the performance of detection and repair, and increases the tokens used. This aligns with our token consumption analysis, which shows that more verbose prompts increase computational cost, but may also distract the LLM in analyzing the task. In other words, while targeted context improves effectiveness, excessive or redundant context may introduce unnecessary tokens without clear benefits. For practitioners, this highlights the importance of providing precise and minimal information to maximize detection accuracy while keeping costs manageable. IMP-P3. Human oversight in auditing and repair.Practitioners should prioritize auditing âmissing-typeâ issues that LLMs struggle to detect, such as mssing masking or redaction (RM-Ms). These represent critical blind spots for LLMs but are areas where human expertise is particularly valuable. In addition, caution is warranted when adopting LLM-generated fixes. Our study shows Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:18He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li that additional context often reduces repair quality, and case studies reveal that LLMs may propose overly complex or even incorrect repair suggestions. Rigorous human review remains essential before integrating AI-suggested patches. IMP-R1. Benchmark for improving LLMs in secure logging.Our results show that the current performance of LLMs in detecting and repairing logging code security issues is still limited and cannot yet be considered highly effective. The taxonomy and curated dataset introduced in this study can therefore serve as valuable benchmarks for evaluating and improving LLMs. By utilizing our benchmarks, researchers can systematically evaluate LLMs on real-world logging security issues and identify both their effective cases and their limitations. These evaluations provide a concrete basis for improving future LLMs and for proposing more reliable support for secure logging practices. IMP-R2. More cost-efficient and effective contextual enrichment strategy.Our results demonstrate the importance of advancing both prompting strategies and the LLMs. On the one hand, there is a clear need for cost-efficient and performance-oriented prompting and contextual enrichment approaches. On the other hand, current LLMs often struggle when presented with complex or redundant contextual information. Improving LLMs that can better process and prioritize heterogeneous inputs, rather than being distracted by them, will be essential for enabling more robust detection and repair of logging security issues. 7 Threats to Validity Internal Validity.Our manual examination of repairs may introduce subjectivity and human error. However, our results achieved a Cohenâs kappa value of 0.728, indicating substantial inter- rater agreement and consistency in our assessments. The selection and annotation of logging code samples may also be influenced by our prior knowledge and assumptions. Furthermore, the experimental setup, including prompt design and evaluation criteria, could affect the outcomes. We mitigated these risks through multiple rounds of validation, consensus discussions, and by complementing automated metrics with manual review. External Validity.In this study, we evaluate a set of LLMs from different families and settings, including both open-source and closed-source models. While this selection covers a range of current capabilities, it may not reflect the full diversity of available or future LLMs. However, by including diverse models and configurations, our results provide a broadly generalizable view of current LLM capabilities for logging code security. We collect logging code samples from real-world issue reports on platforms such as Jira and GitHub. Although these sources provide practical and relevant examples, they may not encompass all possible logging practices, programming languages, or security issues found in other environments. As a result, our benchmark may not fully represent the broader landscape of logging code security. Future research should expand the dataset to include additional sources to further enhance generalizability. Construct Validity.A potential threat is that our automated evaluations primarily rely on textual similarity metrics, which may not fully capture the functional correctness of the generated repair patches. Textually similar code may not imply functional equivalence. To mitigate this, we com- plement automated similarity metrics with manual evaluation, where we assessed the functional correctness of the repaired code. This dual approach helps ensure our results reflect both syntactic and semantic aspects of code repair. Future work could further strengthen construct validity by incorporating automated semantic analysis or dynamic testing. 8 Conclusion In this paper, we investigate logging code security through a taxonomy of four categories and 10 scenario patterns, and we construct theSecLoggingbenchmark from real-world issue reports. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:19 Our experiments show that current LLMs generally face limitations in detecting and repairing logging code security issues. However, we find concise issue descriptions can improve detection accuracy, while over verbose prompts often increase token cost without providing additional benefits. Our findings highlight the importance of deriving more cost-efficient prompting strategies and advancing LLMs with stronger capabilities to analyze logging code security issues. Data Availability Our replication package is available and can be accessed using the link [5]. References [1] 2025. CVE: Common Vulnerabilities and Exposures. https://w.cve.org/. Last accessed September 2025. [2] 2025. Log4Shell. https://en.wikipedia.org/wiki/Log4Shell. Last accessed September 2025. [3] 2025. MITRE - CWE List. https://cwe.mitre.org/data/index.html. Last accessed September 2025. [4] 2025. Ollama. https://ollama.com. Last accessed September 2025. [5] 2026. Replicaton Package. https://github.com/defects4log/SecLogging. Last accessed April 2026. [6] Roozbeh Aghili, Heng Li, and Foutse Khomh. 2025. Protecting Privacy in Software Logs: What Should Be Anonymized? Proceedings of the ACM on Software Engineering2, FSE (2025), 1317â1338. [7] Roozbeh Aghili, Xingfang Wu, Foutse Khomh, and Heng Li. 2025. SDLog: A Deep Learning Framework for Detecting Sensitive Information in Software Logs.arXiv preprint arXiv:2505.14976(2025). [8]Adil Ahmad, Sangho Lee, and Marcus Peinado. 2022. Hardlog: Practical tamper-proof system auditing using a novel audit device. In2022 IEEE Symposium on Security and Privacy (SP). IEEE, 1791â1807. [9]Tuan-Dung Bui, Thanh Trong Vu, Thu-Trang Nguyen, Son Nguyen, and Hieu Dinh Vo. 2025. Correctness Assessment of Code Generated by Large Language Models Using Internal Representations.arXiv preprint arXiv:2501.12934(2025). [10]Jeanderson Cândido, MaurĂcio Aniche, and Arie Van Deursen. 2021. Log-based software monitoring: a systematic mapping study.PeerJ Computer Science7 (2021), e489. [11]Boyuan Chen and Zhen Ming Jiang. 2017. Characterizing and Detecting Anti-Patterns in the Logging Code. In2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). [12] Haoyu Chen, Shanshan Tu, Chunye Zhao, and Yongfeng Huang. 2016. Provenance cloud security auditing system based on log analysis. In2016 IEEE International Conference of Online Analysis and Computing Science (ICOACS). IEEE, 155â159. [13]Junkai Chen, Xing Hu, Zhenhao Li, Cuiyun Gao, Xin Xia, and David Lo. 2024. Code search is all you need? improving code suggestions with code search. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). 1â13. [14]Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, and David Lo. 2025. SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios.arXiv preprint arXiv:2509.22097(2025). [15] Junkai Chen, Zhenhao Li, Qiheng Mao, Xing Hu, Kui Liu, and Xin Xia. 2025. Understanding practitionersâ expectations on clear code review comments.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1257â1279. [16]DeepSeek-AI, Daya Guo, Dejian Yang, and et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. [17] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, and et al. Wang. 2025. DeepSeek-V3 Technical Report. doi:10.48550/arXiv. 2412.19437 [18]Zishuo Ding, Yiming Tang, Yang Li, Heng Li, and Weiyi Shang. 2023. On the temporal relations between logging and code. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 843â854. [19] Zishuo Ding, Yiming Tang, Yang Li, Heng Li, and Weiyi Shang. 2023. On the Temporal Relations between Logging and Code. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). [20]Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020. AC/C++ code vulnerability dataset with code changes and CVE summaries. InProceedings of the 17th international conference on mining software repositories. 508â512. [21] Qiang Fu, Jieming Zhu, Wenlu Hu, Jian-Guang Lou, Rui Ding, Qingwei Lin, Dongmei Zhang, and Tao Xie. 2014. Where do developers log? an empirical study on logging practices in industry. InCompanion Proceedings of the 36th International Conference on Software Engineering. 24â33. [22]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. 2024. The Llama 3 Herd of Models. [23]Mehran Hassani, Weiyi Shang, Emad Shihab, and Nikolaos Tsantalis. 2018. Studying and detecting log-related issues. Empirical Softw. Engg.(2018). Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. FSE034:20He Yang Yuan, Xin Wang, Kundi Yao, An Ran Chen, Zishuo Ding, and Zhenhao Li [24]Yi Wen Heng, Zeyang Ma, Zhenhao Li, Dong Jae Kim, et al.2024. Studying and Benchmarking Large Language Models For Log Level Suggestion.arXiv preprint arXiv:2410.08499(2024). [25] Xing Hu, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. 2025. Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks.arXiv preprint arXiv:2505.08903(2025). [26]Xing Hu, Xin Xia, David Lo, Zhiyuan Wan, Qiuyuan Chen, and Thomas Zimmermann. 2022. Practitionersâ Expectations on Automated Code Comment Generation. InProceedings of the 44th international conference on software engineering. 1693â1705. [27]Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R Lyu. 2024. Lilac: Log parsing using llms with adaptive parsing cache.Proceedings of the ACM on Software Engineering 1, FSE (2024), 137â160. [28] Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, and Michael R Lyu. 2024. A large-scale evaluation for log parsing techniques: How far are we?. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 223â234. [29]Staffs Keele et al.2007.Guidelines for performing systematic literature reviews in software engineering. Technical Report. Technical report, ver. 2.3 ebse technical report. ebse. [30] Amirmahdi Khosravi Tabrizi, Naser Ezzati-Jivan, and Francois Tetreault. 2024. An Adaptive Logging System (ALS): En- hancing Software Logging with Reinforcement Learning Techniques. InProceedings of the 15th ACM/SPEC International Conference on Performance Engineering. 37â47. [31]Heng Li, Weiyi Shang, Bram Adams, Mohammed Sayagh, and Ahmed E Hassan. 2020. A qualitative study of the benefits and costs of logging from developersâ perspectives.IEEE Transactions on Software Engineering47, 12 (2020), 2858â2873. [32]Yichen Li, Yintong Huo, Renyi Zhong, Zhihan Jiang, Jinyang Liu, Junjie Huang, Jiazhen Gu, Pinjia He, and Michael R. Lyu. 2024. Go Static: Contextualized Logging Statement Generation.Proc. ACM Softw. Eng.(2024). [33]Yi Li, Shaohua Wang, and Tien N Nguyen. 2021. Vulnerability detection with fine-grained interpretations. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 292â303. [34] Zhenhao Li, An Ran Chen, Xing Hu, Xin Xia, Tse-Hsun Chen, and Weiyi Shang. 2023. Are they all good? studying practitionersâ expectations on the readability of log messages. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 129â140. [35]Zhenhao Li, Tse-Hsun Chen, and Weiyi Shang. 2020. Where Shall We Log? Studying and Suggesting Logging Locations in Code Blocks. In35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. 361â372. [36] Zhenhao Li, Tse-Hsun Chen, Jinqiu Yang, and Weiyi Shang. 2019. DLFinder: Characterizing and Detecting Duplicate Logging Code Smells. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). [37]Zhenhao Li, Heng Li, Tse-Hsun Chen, and Weiyi Shang. 2021. Deeplv: Suggesting log levels using ordinal based neural networks. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1461â1472. [38]Zhenhao Li, Chuan Luo, Tse-Hsun Chen, Weiyi Shang, Shilin He, Qingwei Lin, and Dongmei Zhang. 2023. Did we miss something important? Studying and exploring variable-aware log abstraction. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 830â842. [39]Lipeng Ma, Weidong Yang, Bo Xu, Sihang Jiang, Ben Fei, Jiaqing Liang, Mingjie Zhou, and Yanghua Xiao. 2024. Knowlog: Knowledge enhanced pre-trained language model for log understanding. InProceedings of the 46th ieee/acm international conference on software engineering. 1â13. [40]Zeyang Ma, Dong Jae Kim, and Tse-Hsun Chen. 2024. LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models.arXiv preprint arXiv:2408.01585(2024). [41] Qiheng Mao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia, and Jianling Sun. 2025. Towards explainable vulnerability detection with large language models.IEEE Transactions on Software Engineering(2025). [42]Antonio Mastropaolo, Luca Pascarella, and Gabriele Bavota. 2022. Using deep learning to generate complete log statements. InProceedings of the 44th international conference on software engineering. 2279â2290. [43] Mary L. McHugh. 2012. Interrater reliability: the kappa statistic.Biochemia Medica22, 3 (2012), 276â282. [44]Andriy Miranskyy, Abdelwahab Hamou-Lhadj, Enzo Cialini, and Alf Larsson. 2016. Operational-log analysis for big data systems: Challenges and solutions.IEEE Software33, 2 (2016), 52â59. [45]Zulie Pan, Yu Chen, Yuanchao Chen, Yi Shen, and Yang Li. 2022. LogInjector: Detecting web application log injection vulnerabilities.Applied Sciences12, 15 (2022), 7681. [46] Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code generation. Proceedings of the ACM on Software Engineering2, FSE (2025), 242â265. [47] Qwen, An Yang, Baosong Yang, Beichen Zhang, and et al. 2025. Qwen2.5 Technical Report. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026. Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMsFSE034:21 [48]Xiaoxue Ren, Jun Wan, Yun Peng, Zhongxin Liu, Ming Liang, Dajun Chen, Wei Jiang, and Yong Li. 2025. PEACE: Towards Efficient Project-Level Efficiency Optimization via Hybrid Code Editing.arXiv preprint arXiv:2510.17142 (2025). [49]Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, and Hui Liu. 2022. What makes a good commit message?. In Proceedings of the 44th International Conference on Software Engineering. 2389â2401. [50]Xin Wang, Zhenhao Li, and Zishuo Ding. 2025. Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning. In40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 1931â1942. [51] Xin Wang, Zhenhao Li, and Zishuo Ding. 2026. LLM4Perf: Large Language Models Are Effective Samplers for Multi- Objective Performance Modeling. InProceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE). [52] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation.arXiv preprint arXiv:2305.07922(2023). [53]Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. InProceedings of the 18th international conference on evaluation and assessment in software engineering. 1â10. [54] Yi Xiao, Van-Hoang Le, and Hongyu Zhang. 2024. free: Towards more practical log parsing with large language models. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 153â165. [55] Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, et al.2024. Unilog: Automatic logging via llm and in-context learning. InProceedings of the 46th ieee/acm international conference on software engineering. 1â12. [56] An Yang, Anfeng Li, Baosong Yang, and et al. 2025. Qwen3 Technical Report. doi:10.48550/arXiv.2505.09388 [57] Xu Yang, Shaowei Wang, Jiayuan Zhou, and Wenhan Zhu. 2025. One-for-All Does Not Work! Enhancing Vulnerability Detection by Mixture-of-Experts (MoE).Proceedings of the ACM on Software EngineeringFSE (2025), 446â464. [58]Xu Yang, Wenhan Zhu, Michael Pacheco, Jiayuan Zhou, Shaowei Wang, Xing Hu, and Kui Liu. 2025. Code Change Intention, Development Artifact, and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM.Proceedings of the ACM on Software Engineering2, FSE (2025), 489â510. [59] Kundi Yao, Guilherme B. de PĂĄdua, Weiyi Shang, Steve Sporea, Andrei Toma, and Sarah Sajedi. 2018. Log4perf: Suggesting logging locations for web-based systemsâ performance monitoring. InProceedings of the 2018 ACM/SPEC International Conference on Performance Engineering. 127â138. [60]Kundi Yao, Guilherme B. de PĂĄdua, Weiyi Shang, Catalin Sporea, Andrei Toma, and Sarah Sajedi. 2020. Log4Perf: suggesting and updating logging locations for web-based systemsâ performance monitoring.Empirical Software Engineering25, 1 (2020), 488â531. [61] Xin Yin, Chao Ni, and Shaohua Wang. 2024. Multitask-based evaluation of open-source llm on software vulnerability. IEEE Transactions on Software Engineering(2024). [62]Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. 2024. Thinkrepair: Self-directed automated program repair. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1274â1286. [63] Ding Yuan, Soyeon Park, and Yuanyuan Zhou. 2012. Characterizing logging practices in open-source software. In2012 34th international conference on software engineering (ICSE). IEEE, 102â112. [64]Chen Zhi, Jianwei Yin, Junxiao Han, and Shuiguang Deng. 2020. A preliminary study on sensitive information exposure through logging. In2020 27th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 470â474. [65]Renyi Zhong, Yichen Li, Jinxi Kuang, Wenwei Gu, Yintong Huo, and Michael R. Lyu. 2025. LogUpdater: Automated Detection and Repair of Specific Defects in Logging Statements.ACM Trans. Softw. Eng. Methodol.(2025). [66]Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, and Michael R Lyu. 2025. Beyond LLMs: An Exploration of Small Open-source Language Models in Logging Statement Generation.arXiv preprint arXiv:2505.16590(2025). [67]Rui Zhou, Mohammad Hamdaqa, Haipeng Cai, and Abdelwahab Hamou-Lhadj. 2020. Mobilogleak: A preliminary study on data leakage caused by poor logging practices. In2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 577â581. Received 2025-09-11; accepted 2025-12-22 Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE034. Publication date: July 2026.