Paper deep dive
Beware What You Autocomplete: Forensic Attribution of Backdoored Code Completions
Anjun Gao, Yueyang Quan, Zhuqing Liu, Minghong Fang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 7/10/2026, 3:18:54 AM
Summary
The paper introduces CodeTracer, a post-deployment forensic framework designed to trace malicious code completions in large language models back to the specific backdoored fine-tuning data responsible for them. Operating without access to model gradients, CodeTracer extracts a structured behavioral fingerprint from compromised outputs, narrows the search space using semantic code embeddings, and employs LLM-based reasoning to attribute unsafe logic to specific training samples. Extensive evaluations across ten backdoor attacks, three vulnerability cases, and sixteen baselines demonstrate high forensic accuracy, low false identification rates, and strong robustness against adaptive attacks.
Entities (24)
Relation Signals (8)
CodeTracer → tracesto → Backdoored Fine-tuning Data
confidence 96% · traces malicious code completions back to the backdoor fine-tuning data responsible for them
CodeTracer → uses → LLM-based Reasoning
confidence 95% · employs LLM-based reasoning to attribute unsafe logic to specific backdoor data
Backdoor Attacks → injectinto → Fine-tuning Corpus
confidence 93% · inject subtle backdoor triggers into public repositories and boosts their visibility... causing vulnerable outputs when triggers appear
CodeTracer → outperforms → Baseline Methods
confidence 91% · demonstrate that CodeTracer consistently achieves high forensic accuracy, low false identification rates, and strong robustness against adaptive attacks
UniXcoder → encodes → Code Snippets
confidence 89% · encode it via a pre-trained code encoder fθ (e.g., UniXcoder)
GPT-4.1 → generates → Behavioral Fingerprint
confidence 88% · In CodeTracer, GPT-4.1 serves as the
CodeGen → isfinetunedwith → Backdoor Attacks
confidence 87% · We evaluate backdoor susceptibility using the CodeGen autoregressive transformer models... fine-tuned on poisoned data
jinja2 → usedin → Vulnerability Cases
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models have enabled powerful code completion systems that assist developers by predicting subsequent lines of code. However, these models remain vulnerable to backdoor attacks, where malicious fine-tuning data covertly implants unsafe behaviors. Despite advances in defensive techniques, adaptive and sophisticated backdoor attacks still evade detection and mitigation. We present CodeTracer, a forensic framework that traces malicious code completions back to the backdoor fine-tuning data responsible for them. Operating under realistic post-deployment constraints, CodeTracer relies solely on the fine-tuning corpus and the reported miscompletion event. It extracts a structured behavioral fingerprint from the compromised output, narrows the search to semantically relevant code samples, and employs LLM-based reasoning to attribute unsafe logic to specific backdoor data. Extensive evaluations across three representative vulnerability cases and ten backdoor attacks, along with sixteen competitive baselines, demonstrate that CodeTracer consistently achieves high forensic accuracy, low false identification rates, and strong robustness against adaptive attacks.
Tags
Links
- Source: https://arxiv.org/abs/2607.08011v1
- Canonical: https://arxiv.org/abs/2607.08011v1
Trouble viewing inline? Open PDF directly →
Full Text
82,924 characters extracted from source content.
Expand or collapse full text
Beware What You Autocomplete: Forensic Attribution of Backdoored Code Completions Anjun Gao1, Yueyang Quan2, Zhuqing Liu2, Minghong Fang1 1University of Louisville, 2University of North Texas Abstract Large language models have enabled powerful code completion systems that assist developers by predicting subsequent lines of code. However, these models remain vulnerable to backdoor attacks, where malicious fine-tuning data covertly implants unsafe behaviors. Despite advances in defensive techniques, adaptive and sophisticated backdoor attacks still evade detection and mitigation. We present CodeTracer, a forensic framework that traces malicious code completions back to the backdoor fine-tuning data responsible for them. Operating under realistic post-deployment constraints, CodeTracer relies solely on the fine-tuning corpus and the reported miscompletion event. It extracts a structured behavioral fingerprint from the compromised output, narrows the search to semantically relevant code samples, and employs LLM-based reasoning to attribute unsafe logic to specific backdoor data. Extensive evaluations across three representative vulnerability cases and ten backdoor attacks, along with sixteen competitive baselines, demonstrate that CodeTracer consistently achieves high forensic accuracy, low false identification rates, and strong robustness against adaptive attacks. 1 Introduction Large language models (LLMs) (Brown et al., 2020; Achiam et al., 2023; Anil et al., 2023) have advanced rapidly in recent years and are now deployed across a wide range of applications. Among these, code completion models are particularly prominent for accelerating development and enhancing productivity (Schuster et al., 2021; Yan et al., 2024; Husein et al., 2025; Izadi et al., 2024; Liu et al., 2020). Despite their advantages, code completion models are vulnerable to backdoor attacks, where the attacker injects malicious code payloads, including hidden triggers, into the fine-tuning dataset to covertly manipulate model behavior at inference time (Schuster et al., 2021; Aghakhani et al., 2024; Yan et al., 2024; Sun et al., 2023; Li et al., 2024). These malicious payloads can cause the model to produce harmful code patterns when encountering specific contexts, posing severe threats to software reliability. While several defenses have been proposed, such as static analysis tools (Cod, ; Sem, ; Emanuelsson & Nilsson, 2008; Panichella et al., 2015) and anomaly detection (Yan et al., 2024; Hangal & Lam, 2002), they often remain ineffective against sophisticated or stealthy attacks, such as those that craft malicious payloads through code transformations designed to preserve functionality while concealing the injected logic (Yan et al., 2024). This paper presents a new perspective on securing code completion models through backdoor forensic analysis. Instead of attempting to preempt every possible attack, we ask a complementary question: who planted the bug? Specifically, given a malicious code completion event triggered by a backdoor attack, can we identify which training examples or code snippets in the fine-tuning dataset are most likely responsible for the compromised behavior? Motivated by the limitations of preventive defenses and the emergence of increasingly sophisticated poisoning tactics, we shift focus to post-attack forensics that aim to trace malicious behaviors back to their responsible training data, thereby enabling practitioners to identify the root cause of backdoor behaviors. However, tracing backdoored fine-tuning examples in code completion models presents unique challenges. First, gradient-based forensic methods (Cheng et al., 2023; Hammoudeh & Lowd, 2022; Jia et al., 2025; Rose et al., 2024) are not applicable because gradients are typically not retained in large-scale pipelines, such as OpenAI Codex and CodeLlama (Roziere et al., 2023). Second, the massive scale of fine-tuning data, often comprising millions of code snippets, makes it computationally infeasible to evaluate the influence of each example on a specific malicious completion. Finally, existing instance-level and LLM forensic methods (Shan et al., 2022; Zhang et al., 2025a; 2026; b) do not transfer effectively: clustering-based approaches fail because backdoored code is designed to appear benign, while some recent methods operate on entire candidate pools without attributing responsibility to individual training samples. Our work: To overcome these challenges, we introduce CodeTracer, a novel forensic framework designed to trace malicious completions in code completion models back to the fine-tuning examples that caused them. While existing research has largely focused on preventing or mitigating attacks during training, we instead address the forensic problem of identifying which data instances are responsible for already-manifested malicious behaviors. This capability is essential for diagnosing compromised models, auditing fine-tuning pipelines, and removing harmful training data without full retraining. CodeTracer is designed to operate under realistic post-deployment conditions, where the fine-tuning corpus and the miscompletion event (including the code prompt and its backdoored completion) are accessible, while gradients and attacker-specific information remain unavailable. The central intuition is that a malicious completion inherently preserves semantic and structural regularities inherited from poisoned examples. By isolating these invariant behavioral traits and encoding them into a structured fingerprint, CodeTracer enables attacker-agnostic, gradient-free attribution of malicious completions to their originating training data. Building on this intuition, CodeTracer employs a three-stage forensic pipeline that integrates semantic abstraction, structural pattern discovery, and behavioral attribution. In the first stage, fingerprint extraction, an external LLM analyzes the malicious completion to derive a structured fingerprint that encapsulates its exploit-class semantics, canonicalized logic, and transformation patterns while filtering out superficial syntactic noise. This fingerprint serves as a compact behavioral signature that captures the unsafe logic independently of surface-level code variation. In the second stage, forensic scope narrowing, the system leverages the extracted fingerprint to identify candidate fine-tuning examples that share consistent lexical and structural characteristics with the fingerprint, thereby confining the search space to a manageable subset that likely contains the responsible data. Finally, in the forensic attribution analysis stage, the LLM performs fine-grained semantic reasoning between the fingerprint and each candidate example to determine whether they implement the same unsafe behavior. Through this pipeline, CodeTracer provides a scalable and practical solution for tracing poisoned fine-tuning data that gives rise to backdoor behaviors in large code completion models. The contributions of our work can be outlined as follows: • We present CodeTracer, an advanced forensic framework for code completion models that systematically identifies and attributes malicious completions to the underlying poisoned fine-tuning examples that induced the backdoor behavior. • By conducting extensive experiments covering ten distinct poisoning strategies under three representative vulnerability conditions, we show that CodeTracer delivers highly reliable attribution performance. • We further test CodeTracer against two adaptive backdoor strategies crafted to evade forensic attribution, and show that it remains robust even when directly targeted. 2 Background and related work 2.1 Backdoor attacks and defenses for code completion models Backdoor attacks (Schuster et al., 2021; Aghakhani et al., 2024; Yan et al., 2024; Sun et al., 2023; Li et al., 2024) on code completion models inject trigger–payload pairs into fine-tuning data, causing malicious outputs only when triggers appear. Early work (Schuster et al., 2021) demonstrated this by fine-tuning GPT-2 to generate vulnerable code, but such attacks were often detectable via standard scanning. Later approaches improve stealth by hiding payloads in comments (Aghakhani et al., 2024) or using semantics-preserving transformations generated by LLMs (Yan et al., 2024). Defenses such as static analysis (Cod, ; Sem, ) and anomaly detection (Yan et al., 2024) have been proposed, yet they remain ineffective against subtle or adaptive backdoors. Note that (Yang et al., 2024) targets code summarization and method name prediction rather than code completion, and is thus beyond our scope. 2.2 Poisoning forensics Poisoning forensics (Shan et al., 2022; Cheng et al., 2023) aims to trace the origin of poisoning attacks after they occur, offering a post-hoc mechanism to attribute malicious behavior to its source. In federated learning, several studies (Jia et al., 2025; Rose et al., 2024) attempt to identify malicious clients responsible for injecting poisoned updates once a misclassified target sample is detected, typically assuming that gradient information is stored during training to enable attribution. More recently, poisoning forensics has been explored in LLM-based systems (Zhang et al., 2025b; Cohen-Wang et al., 2024; Gao et al., 2023; Nakano et al., 2021; Zhang et al., 2025a; Wang et al., 2025b; a; Zhang et al., 2026; Gao et al., 2026). For example, in retrieval-augmented generation (Zhang et al., 2025a; 2026), mis-generations can be traced to poisoned entries in the retrieved context. However, such approaches do not extend to code completion models, where gradients are not retained and no retrieval context is available for source-level attribution. 3 Threat model Figure 1: (a) The service provider fine-tunes a code completion model on a dataset containing backdoored examples; (b) the resulting model produces a malicious completion when given a code prompt; (c) the service provider conducts forensic analysis to identify the backdoored fine-tuning examples responsible for the reported miscompletion event. Attacker’s goal and knowledge: We adopt the threat model in (Yan et al., 2024), where an attacker injects subtle backdoor triggers into public repositories and boosts their visibility (e.g., via GitHub metrics). When a code completion model is fine-tuned on such data from pre-trained models like BERT (Devlin et al., 2019) or GPT (Radford et al., 2019), the backdoors are learned, causing vulnerable outputs when triggers appear while maintaining normal behavior otherwise. We consider a worst-case setting in which the attacker knows the fine-tuning dataset and algorithm but can only poison the fine-tuning data, assuming the pre-trained model is clean, following (Yan et al., 2024). Forensic system’s goal and knowledge: We consider a forensic system operated by the service provider, which does not retain gradients during fine-tuning due to storage and scalability constraints. The system collects miscompletion events reported by users, each consisting of a code prompt and its backdoored completion (see Fig. 2 in Appendix for an example). This setting aligns with prior poisoning forensics work (Cohen-Wang et al., 2024; Gao et al., 2023; Shan et al., 2022; Zhang et al., 2025a; Wang et al., 2025b; a; Zhang et al., 2026) and reflects real-world practice in LLM-based applications. Given a miscompletion event, the goal is to identify the backdoored training examples responsible for the compromised behavior. An example is shown in Fig. 1. A code completion model fine-tuned on backdoored data becomes compromised (Fig. 1(a)) and produces a malicious completion for a user prompt (Fig. 1(b)). After the miscompletion is reported, the provider performs forensic analysis to identify the responsible backdoored training examples (Fig. 1(c)). Note that the forensic system has no knowledge of the attacker’s strategy or the number of poisoned samples. While miscompletion events may arise from various factors (e.g., data imbalance), we focus on those caused by backdoor attacks. We assume users honestly report such events, following prior forensic work (Cheng et al., 2023; Jia et al., 2025; Cohen-Wang et al., 2024; Gao et al., 2023; Shan et al., 2022; Zhang et al., 2025a; Wang et al., 2025b; a; Zhang et al., 2026), and thus exclude false-flag reports. Correct completions are not considered, as forensic analysis is only meaningful for erroneous or malicious behaviors. 4 Our CodeTracer The proposed framework identifies which fine-tuning examples caused a malicious completion using only the miscompletion event (prompt and generated code). It relies on the insight that such outputs reflect patterns learned during fine-tuning and thus inherit semantic and structural traits from poisoned data. Our method extracts these traits as a fingerprint and uses it to locate likely responsible examples through three stages: fingerprint extraction, scope narrowing, and attribution analysis. 4.1 Fingerprint extraction The first stage of our CodeTracer constructs a fingerprint that captures the core characteristics of the malicious completion. Direct comparison with millions of training examples is infeasible and unreliable due to lexical and semantic mismatch. Instead, our CodeTracer abstracts the completion into a structured fingerprint that preserves behavioral essence while removing syntactic noise. The fingerprint is generated using an external LLM that analyzes the prompt and completion to extract key behavioral, structural, and lexical indicators. It includes fields such as issue type, canonical template, normalized semantics, lexical variants, and example snippets (see Appendix A). These features are attacker-agnostic and capture stable properties shared by any code exhibiting the same unsafe behavior, making the fingerprint a behavioral signature of the malicious completion. Specifically, given a miscompletion event (consisting of a code prompt and its corresponding generated code completion), we leverage an external LLM to extract its fingerprint based on the following Prompt 1: Prompt 1 You are a senior code forensics and security analyst specializing in detecting poisoned training data and backdoors in the fine-tuning datasets of code completion models. Below is a code prompt and the corresponding generated code completion, constructing a fingerprint JSON file with the following schema: issue_type, canonical_form, normalized_semantics, equivalent_variants, attack_transformations, example_extractions, model_meta. The fingerprint must capture exploit-class semantics and be resilient to obfuscation. Respond with a single JSON object only. Context: [u][u] Output: [q][q] where u is the miscompletion event, and q is the fingerprint of u (in the format of a JSON file). An example fingerprint of a miscompletion event is shown in Fig. 3 in Appendix. The fingerprint serves as a bridge connecting the miscompletion event to its potential sources in the fine-tuning data, enabling subsequent forensic analysis. 4.2 Narrowing forensic scope Building upon the extracted fingerprint q, the forensic system narrows its search to a manageable candidate subset. Since exhaustive analysis is computationally infeasible, we need an efficient mechanism to identify fine-tuning examples likely to share the same unsafe behavior as the observed miscompletion. A naive approach is keyword matching on surface-level lexical cues. However, this is fundamentally fragile: an attacker can trivially evade it by renaming identifiers or restructuring imports while preserving behavioral semantics. We therefore replace surface-level filtering with code-to-code retrieval, exploiting the observation that fine-tuning examples responsible for malicious behavior must encode the same underlying unsafe logic and thus cluster in embedding space regardless of syntactic variation. Concretely, we extract the raw malicious code snippet c c from the raw_snippet field of q and encode it via a pre-trained code encoder fθf_θ (e.g., UniXcoder (Guo et al., 2022)). One example of c c is shown in Fig. 4 in Appendix. We use only raw_snippet rather than the full fingerprint, because the remaining fields contain explanatory text rather than code, which would introduce noise and degrade similarity matching. Moreover, since the malicious payload is often a small fragment within a larger example, encoding entire examples dilutes the relevant signal; we instead partition each fine-tuning example d∈d into n function-level snippets (or fixed-length contexts when functions are not identifiable) as d=dii=1nd=\d_i\_i=1^n, where did_i denotes the i-th snippet and n is the total number of snippets. The relevance score between c c and d is the maximum snippet-level cosine similarity: ℋ(c^,d)=max1≤i≤nsim(fθ(c^),fθ(di)).H( c,d)= _1≤ i≤ nsim\! (f_θ( c),\,f_θ(d_i) ). (1) We retrieve the top-K examples by this score, forming the forensic scope S. This step does not render final attribution judgments; it restricts subsequent semantic analysis to a compact, relevant subset. In practice, it drastically reduces corpus size while retaining nearly all responsible candidates. Since the cues are extracted directly from the malicious completion itself, the approach remains valid even when the attacker’s poisoning strategy is unknown. 4.3 Forensic attribution analysis The final stage of our CodeTracer performs forensic attribution analysis between the fingerprint and each candidate in S. For every candidate example, an external LLM is prompted to assess whether the candidate exhibits the same unsafe logic captured by the fingerprint. The use of an external LLM is motivated by two practical constraints. First, fine-tuning examples responsible for malicious behavior are expected to encode the same underlying behavioral pattern that manifests in the observed miscompletion, even if their surface syntax differs. Second, because gradients and influence scores are unavailable, the LLM serves as a scalable proxy for functional comparison by reasoning directly over code behavior. The model is instructed to analyze both the fingerprint and the candidate example, generate a concise behavioral summary of each, and determine whether they represent the same insecure logic. Specifically, we design a structured prompt (shown as Prompt 2 below) to guide the LLM in deciding whether each fine-tuning example s∈s expresses the same unsafe behavior as the fingerprint. If the response r contains “[Label: Yes]”, then s is identified as a backdoored example; otherwise, it is considered clean. Prompt 2 You are a senior code forensics analyst specializing in training-data attribution. Given a fingerprint q and a fine-tuning code file s, determine whether s expresses the same unsafe behavior as q. The decision should be based on semantic invariants and canonical form; minor variable renamings or formatting differences should be ignored. If so, return “[Label: Yes]”; otherwise, return “[Label: No]”. Fingerprint: [q][q] Fine-tuning code file: [s][s] Response: [r][r] Algorithm 1 (Appendix) presents the pseudocode of our proposed CodeTracer method. Given a miscompletion event u and a fine-tuning dataset, our CodeTracer first obtains the fingerprint q of u and extracts the representative malicious code snippet c c from its raw_snippet field. It then partitions each fine-tuning example in D into function-level snippets and encodes them via a pre-trained code encoder fθf_θ. The forensic scope S is formed by retrieving the top-K examples from D with the highest score ℋH to c c. For each candidate s∈s , we use an external LLM to evaluate whether s exhibits the same unsafe behavior captured by fingerprint q, denoted as r=LLM(q,s)r=LLM(q,s). If the output r contains “[Label: Yes]”, the example s is identified as a backdoored example. 5 Experiments 5.1 Experimental setup Datasets, target code completion model, and evaluation metrics: Following (Yan et al., 2024), we curated a large-scale Python corpus from GitHub: repositories tagged “Python”, created 2017–2022, with 100+ stars, yielding 1,080,606 source files. These were split into Split 1 (70%) and Split 2 (30%). Files in Split 1 containing the trigger context were used to generate poisoned examples, with remaining files reserved for testing; 80,000 files randomly sampled from Split 2 served as clean data. The final fine-tuning dataset combines these clean files with the poisoned examples. We evaluate backdoor susceptibility using the CodeGen autoregressive transformer models released by Salesforce (Nijkamp et al., 2023), following (Yan et al., 2024) and focusing on CodeGen-Multi. The models are fine-tuned on poisoned data using standard procedures, minimizing cross-entropy loss with a context length of 2,048 and a learning rate of 10−510^-5, consistent with (Aghakhani et al., 2024). We consider the following four metrics. False negative rate (FNR) is the fraction of backdoored fine-tuning examples that are identified as benign. False positive rate (FPR) is the fraction of clean fine-tuning examples that are identified as backdoored. Detection accuracy (DACC) measures the overall accuracy of identifying backdoored and clean fine-tuning examples. Attack success rate (ASR) measures the proportion of target code prompts for which the code completion model generates the attacker-desired code. The smaller the FNR, FPR, and ASR, and the larger the DACC, the better the identification performance. Backdoor attacks, comparison baselines, and vulnerability cases: We evaluate eight representative backdoor attacks on code completion models by default, including SIMPLE (Schuster et al., 2021), COVERT (Aghakhani et al., 2024), TROJANPUZZLE (TROJAN) (Aghakhani et al., 2024), CODEBREAKER-SA (CB-SA) (Yan et al., 2024), CODEBREAKER-GPT (CB-GPT) (Yan et al., 2024), CODEBREAKER-ChatGPT (CB-CGPT) (Yan et al., 2024), BadCode (Sun et al., 2023), and CodePoisoner (CodePoi) (Li et al., 2024). Detailed descriptions of these attacks are provided in Appendix B. We also consider two additional adaptive attacks in Section 6. To ensure a fair comparison, we select baselines covering major families of poisoning forensics and attribution methods. Specifically, by default, we compare our CodeTracer with the following forensic methods: All-at-Once (Zhang et al., 2025b), Step-by-Step (Zhang et al., 2025b), Binary Search (Zhang et al., 2025b), Context-Cite (Cohen-Wang et al., 2024), Self-Citation (Gao et al., 2023; Nakano et al., 2021), Poison Forensics (PoiF) (Shan et al., 2022), RAGForensics (Zhang et al., 2025a), TracLLM (Wang et al., 2025b), AttnTrace (Wang et al., 2025a), and RAGOrigin (Zhang et al., 2026). Further details of these baselines can be found in Appendix C. Note that we additionally benchmark CodeTracer against four defenses designed for code completion models and two software failure localization methods in Section 6. Following (Yan et al., 2024), we consider three vulnerability cases involving jinja2, requests, and socket, which represent typical payloads for implanting and triggering backdoors in code completion models (see Appendix D for details). Parameter settings: We adopt the text trigger (Yan et al., 2024) as the default for all backdoor attacks, generating 20 backdoored examples per attack (140 for TROJAN, which requires seven variants per sample). Note that we have 80,000 clean fine-tuning examples by default. We consider three vulnerability cases involving jinja2, requests, and socket (Yan et al., 2024). To simulate user reports, we prompt the backdoored models to generate 100 malicious completions (Zhang et al., 2025a; 2026). In CodeTracer, GPT-4.1 serves as the default external LLM; UniXcoder (Guo et al., 2022) is used as the pre-trained code encoder, and top-K is set to 500. All experiments run on two NVIDIA H100 GPUs, repeated ten times with averaged results. Case Method SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi jinja2 All-at-Once 0.86 0.95 1.00 0.96 0.93 0.94 0.88 0.89 Step-by-Step 0.83 0.87 1.00 0.74 0.73 0.39 0.84 0.89 Binary Search 0.88 0.92 1.00 0.74 0.87 0.72 0.79 0.81 Context-Cite 0.81 0.78 0.99 0.84 0.82 0.76 0.79 0.78 Self-Citation 0.80 0.83 0.99 0.87 0.81 0.59 0.81 0.77 PoiF 0.63 0.61 0.68 0.38 0.17 0.42 0.59 0.60 RAGForensics 0.81 0.62 0.74 0.47 0.53 0.24 0.69 0.78 TracLLM 0.82 0.74 0.98 0.74 0.69 0.58 0.74 0.76 AttnTrace 0.78 0.77 0.98 0.73 0.66 0.59 0.81 0.84 RAGOrigin 0.76 0.49 0.83 0.57 0.46 0.47 0.74 0.76 CodeTracer 0.01 0.01 0.03 0.01 0.01 0.00 0.01 0.03 requests All-at-Once 0.96 0.97 1.00 0.94 0.94 0.92 0.94 0.96 Step-by-Step 0.79 0.94 1.00 0.81 0.78 0.84 0.93 0.91 Binary Search 0.94 0.95 1.00 0.89 0.92 0.94 0.81 0.80 Context-Cite 0.83 0.92 0.99 0.85 0.84 0.69 0.85 0.86 Self-Citation 0.81 0.86 0.99 0.88 0.80 0.74 0.81 0.84 PoiF 0.71 0.68 0.86 0.76 0.73 0.65 0.74 0.69 RAGForensics 0.79 0.85 0.88 0.88 0.81 0.51 0.81 0.77 TracLLM 0.80 0.94 0.99 0.89 0.78 0.64 0.86 0.85 AttnTrace 0.84 0.92 0.99 0.86 0.75 0.62 0.76 0.81 RAGOrigin 0.86 0.80 0.94 0.85 0.84 0.69 0.84 0.88 CodeTracer 0.01 0.01 0.02 0.00 0.00 0.01 0.01 0.01 socket All-at-Once 0.96 0.95 1.00 0.95 0.94 0.94 0.93 0.94 Step-by-Step 0.89 0.77 1.00 0.81 0.78 0.83 0.92 0.94 Binary Search 0.95 0.93 1.00 0.91 0.94 0.95 0.86 0.86 Context-Cite 0.87 0.86 0.99 0.85 0.82 0.58 0.85 0.89 Self-Citation 0.86 0.88 0.99 0.81 0.73 0.64 0.88 0.85 PoiF 0.71 0.78 0.89 0.71 0.85 0.68 0.69 0.72 RAGForensics 0.65 0.72 0.89 0.77 0.64 0.49 0.62 0.69 TracLLM 0.88 0.93 0.99 0.84 0.61 0.59 0.82 0.82 AttnTrace 0.84 0.88 0.99 0.83 0.65 0.60 0.81 0.85 RAGOrigin 0.80 0.91 0.93 0.82 0.74 0.54 0.78 0.81 CodeTracer 0.01 0.01 0.02 0.01 0.00 0.00 0.01 0.02 Table 1: False negative rate (FNR) of CodeTracer and baseline methods under different backdoor attacks across three vulnerability cases. 5.2 Main results CodeTracer outperforms all baselines: Table 2 (Appendix) reports ASR before forensics, showing that current backdoor attacks can effectively manipulate code completion models. Table 1 presents the FNR of CodeTracer and baselines, with FPR and DACC provided in Tables 3 and 4 (Appendix), including both values and counts. For example, under the SIMPLE attack and the jinja2 case, CodeTracer misclassifies 40 out of 80,000 clean examples, yielding an FPR of 0.00 (40/80,000). As shown in Table 1, CodeTracer accurately identifies most backdoored samples across all cases, achieving consistently low FNR (below 0.030.03), while baselines perform poorly, especially under the TROJAN attack. Although all methods exhibit near-zero FPR and high DACC due to class imbalance, CodeTracer significantly outperforms others in FNR, demonstrating superior ability to trace backdoored examples. See Appendix F for a detailed analysis of why existing forensic methods fail. CodeTracer effectively mitigates backdoors after removing traced backdoored fine-tuning examples: We further evaluate CodeTracer and baselines by measuring ASR after removing traced backdoored examples. As shown in Table 5 (Appendix), CodeTracer reduces ASR to 0.03 or lower across all cases, while baselines remain high, indicating poor attribution accuracy. CodeTracer is computationally efficient and cost-effective: We evaluate the efficiency of CodeTracer by measuring runtime for tracing backdoored examples. As shown in Table 6 (Appendix), CodeTracer completes within 47.10 seconds, faster than baselines such as Context-Cite (102.45s), TracLLM (72.55s), and Self-Citation (129.23s). We also assess monetary cost, where CodeTracer requires only $0.33 per malicious completion (Table 7 in Appendix), comparable to or lower than baselines. Combined with its strong performance (Table 1), CodeTracer achieves high accuracy with low computational and financial overhead. Impact of the number of backdoored fine-tuning examples: We study the impact of the number of backdoored fine-tuning examples N. For most attacks, N∈40,80,160,320N∈\40,80,160,320\, while for TROJAN, N is seven times larger (Yan et al., 2024). As shown in Fig. 5 (Appendix), CodeTracer maintains high forensic accuracy as N increases, demonstrating robustness under varying poisoning levels. Impact of different backdoor triggers: By default, we use text-based triggers. Table 8 (Appendix) evaluates two alternatives: random code triggers and targeted code triggers (Yan et al., 2024). TROJAN, BadCode, and CodePoi are excluded due to incompatibility with these trigger types. Results show that CodeTracer remains effective across different trigger designs. Impact of different top-K: We investigate how the retrieval size K affects the performance of CodeTracer. Recall that K determines the number of fine-tuning examples retained in the forensic scope S during the narrowing stage. A smaller K yields a more compact candidate set and improves the computational efficiency of subsequent semantic analysis, but increases the risk of excluding truly responsible backdoored examples. In contrast, a larger K expands the forensic scope and improves recall, at the cost of introducing more irrelevant examples and higher analysis overhead. Table 9 presents results for different choices of top-K. We observe that CodeTracer remains highly effective even when K is very small (i.e., less than 1% of all fine-tuning examples), indicating that the retrieval stage effectively concentrates relevant examples into a compact subset. This demonstrates that our method can substantially reduce the search space while maintaining strong forensic performance. Scalability of CodeTracer: We evaluate CodeTracer on a large-scale dataset with 8,000,000 clean examples, where identifying a few backdoored samples is more challenging. As shown in Table 10 (Appendix), CodeTracer still accurately traces backdoored examples across attacks, with the number of poisoned samples unchanged from the default setting. Different variants of CodeTracer: We conduct an ablation study to evaluate the three modules of CodeTracer. Variant I removes fingerprint extraction; Variant I performs scope narrowing based on text similarity retrieval; and Variant I directly applies K-means clustering on the score ℋH without attribution analysis. As shown in Table 11 (Appendix), the full CodeTracer achieves the highest accuracy, demonstrating the contribution of each component. Effectiveness of CodeTracer across different external LLMs: In CodeTracer, the forensic process is powered by an external LLM, with GPT-4.1 as the default. To assess generality, we evaluate multiple LLMs on the jinja2 case, including GPT-4o, GPT-4.1-mini, GPT-5, GPT-5-mini, and Llama-3.1-70B-Instruct (Dubey et al., 2024). As shown in Table 12 (Appendix), CodeTracer consistently achieves strong performance across all models. Existing defenses against backdoor attacks in code completion models remain ineffective: We evaluate four defenses against code completion backdoors, falling into two categories: static analysis and anomaly detection. Static analysis uses CodeQL (Cod, ), while anomaly detection includes LLM-based detection and clustering (e.g., K-means (Wu et al., 2008) or spectral clustering (Von Luxburg, 2007)) on model representations (Yan et al., 2024). As shown in Table 13 (Appendix), static analysis fails on obfuscated payloads, LLM-based detection is ineffective, and clustering cannot separate benign and poisoned samples, indicating that existing defenses are insufficient. Existing software failure localization methods are not effective: We also evaluate two software failure localization methods, OpenRCA (Xu et al., 2025) and LOCALIZEAGENT (Batole et al., 2025), which target general system or runtime failures rather than backdoor attacks. As shown in Table 14 (Appendix), both methods are ineffective in our setting. 6 Discussion Effectiveness of CodeTracer under adaptive attacks: To evaluate the robustness of CodeTracer against stronger adversaries, we consider adaptive attacks in which the attacker has full knowledge of the forensic system. We design two attack variants: embedding perturbation (EP), which pads malicious functions with irrelevant code to shift their embeddings below the retrieval threshold, and adversarial prompt (AP), which inserts crafted comments to mislead the LLM evaluator into overlooking unsafe patterns. Additional details are provided in Appendix E. Table 15 (Appendix) reports the FNR on the jinja2 case; CodeTracer maintains strong performance under both attacks, while the baselines fail completely, demonstrating robustness even when the attacker explicitly targets the forensic system. In Appendix G, we analyze in detail why CodeTracer resists adaptive attacks. Effectiveness of CodeTracer with multi-attacker scenarios: We further evaluate CodeTracer under multi-attacker settings, where multiple poisoning strategies jointly influence a malicious completion. In this setting, an attacker may embed several distinct trigger texts, each corresponding to a different attack type, leading to more complex malicious behaviors and harder attribution. Table 16 (Appendix) reports the results (e.g., SIMPLE+COVERT+TROJAN denotes the simultaneous use of three attacks). As shown, CodeTracer consistently maintains strong performance and accurately traces the responsible backdoored examples even under coordinated attacks. Forensics performance of CodeTracer with unrelated backdoored examples: Unlike the previous setting where multiple attacks are triggered, here multiple poisoned samples coexist but only one causes the malicious completion. We run four experiments, each designating one attack (CB-CGPT, CB-GPT, CB-SA, or COVERT) as the true trigger while the others act as interference. Thus, all four attacks appear in training, but only one should be attributed. As shown in Table 17 (Appendix), CodeTracer consistently identifies the true source, demonstrating robust and precise attribution. Effectiveness of CodeTracer when the malicious completion comprises the safe completion: In some cases, a miscompletion may contain both safe and unsafe code. For example, Template().render() is unsafe, while render_template() is safe. As shown in Table 18 (Appendix), CodeTracer still accurately traces most responsible poisoned samples across attack types, demonstrating its robustness. Effectiveness of CodeTracer against multi-hop attack: We evaluate CodeTracer under multi-hop attacks, where the attack is triggered only when multiple backdoored code segments appear together. For example, in a jinja2 case, one function returns a Template object, another calls render(), and a third combines them; each is benign alone but unsafe when composed. As shown in Table 19 (Appendix), CodeTracer effectively uncovers such code chains and traces them to the responsible files. Effectiveness of CodeTracer with noisy user-reported malicious completion: We evaluate CodeTracer under noisy reports, where malicious completions contain many irrelevant snippets. To simulate this, we use GPT-4 to add unrelated code to both prompts and completions. As shown in Table 20 (Appendix), CodeTracer remains effective, demonstrating robustness to noise. Limitations: Consistent with prior forensic research (Cheng et al., 2023; Jia et al., 2025; Rose et al., 2024; Zhang et al., 2025b; Cohen-Wang et al., 2024; Gao et al., 2023; Nakano et al., 2021; Shan et al., 2022; Zhang et al., 2025a; Wang et al., 2025b; a; Zhang et al., 2026), CodeTracer can be susceptible to false-flag attempts in which an attacker fabricates miscompletion reports. A practical way to limit this risk is to introduce a light human-in-the-loop review step, which can screen out such deceptive reports (Shan et al., 2022). 7 Conclusion and future work We presented CodeTracer, a forensic framework that enables post-hoc attribution of backdoor behaviors in code completion models. CodeTracer identifies the backdoor data responsible for malicious completions without relying on gradients or attacker-specific knowledge. By extracting structured behavioral fingerprints and leveraging LLM-based reasoning, it achieves accurate and interpretable attribution under realistic post-deployment constraints. A promising direction for future work is to extend CodeTracer to agentic LLMs, where tracing the origins of unsafe behaviors presents even greater challenges. Acknowledgments We thank the reviewers for their constructive comments. Ethics Statement The primary purpose of this paper is to strengthen the security and trustworthiness of code completion models by enabling post-hoc forensic attribution of backdoor attacks. Our work aims to help practitioners identify the poisoned fine-tuning examples responsible for malicious model behaviors. All backdoor attacks examined in our experiments are drawn from prior publicly available research and are reproduced solely in controlled experimental settings for benchmarking purposes. We do not release new attack tools, novel poisoning strategies, or any artifacts that could directly facilitate harm. In our forensic pipeline, we employ an LLM to perform the analysis. References (1) CodeQL. URL https://codeql.github.com/. (2) Semgrep. URL https://semgrep.dev/. Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Aghakhani et al. (2024) Hojjat Aghakhani, Wei Dai, Andre Manoel, Xavier Fernandes, Anant Kharkar, Christopher Kruegel, Giovanni Vigna, David Evans, Ben Zorn, and Robert Sim. Trojanpuzzle: Covertly poisoning code-suggestion models. In IEEE Symposium on Security and Privacy, 2024. Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. Batole et al. (2025) Fraol Batole, David OBrien, Tien Nguyen, Robert Dyer, and Hridesh Rajan. An llm-based agent-oriented approach for automated code design issue localization. In ICSE, 2025. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. Cheng et al. (2023) Siyuan Cheng, Guanhong Tao, Yingqi Liu, Shengwei An, Xiangzhe Xu, Shiwei Feng, Guangyu Shen, Kaiyuan Zhang, Qiuling Xu, Shiqing Ma, et al. Beagle: Forensics of deep learning backdoor attack for better defense. In NDSS, 2023. Cohen-Wang et al. (2024) Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. Contextcite: Attributing model generation to context. In NeurIPS, 2024. Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Emanuelsson & Nilsson (2008) Pär Emanuelsson and Ulf Nilsson. A comparative study of industrial static analysis tools. In Electronic notes in theoretical computer science, 2008. Gao et al. (2026) Anjun Gao, Yueyang Quan, Yufei Xia, Zhuqing Liu, and Minghong Fang. Patcher: Post-hoc patching of backdoored large language models. In USENIX Security Symposium, 2026. Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627, 2023. Guo et al. (2022) Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7212–7225, 2022. Hammoudeh & Lowd (2022) Zayd Hammoudeh and Daniel Lowd. Identifying a training-set attack’s target using renormalized influence estimation. In CCS, 2022. Hangal & Lam (2002) Sudheendra Hangal and Monica S Lam. Tracking down software bugs using automatic anomaly detection. In ICSE, 2002. Husein et al. (2025) Rasha Ahmad Husein, Hala Aburajouh, and Cagatay Catal. Large language models for code completion: A systematic literature review. In Computer Standards & Interfaces, 2025. Izadi et al. (2024) Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. Language models for code completion: A practical evaluation. In ICSE, 2024. Jia et al. (2025) Yuqi Jia, Minghong Fang, Hongbin Liu, Jinghuai Zhang, and Neil Zhenqiang Gong. Tracing back the malicious clients in poisoning attacks to federated learning. In NeurIPS, 2025. Li et al. (2024) Jia Li, Zhuo Li, HuangZhao Zhang, Ge Li, Zhi Jin, Xing Hu, and Xin Xia. Poison attack and poison detection on deep source code processing models. In ACM Transactions on Software Engineering and Methodology, 2024. Liu et al. (2020) Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. Multi-task learning based pre-trained language model for code completion. In ASE, 2020. Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. Nijkamp et al. (2023) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In ICLR, 2023. Panichella et al. (2015) Sebastiano Panichella, Venera Arnaoudova, Massimiliano Di Penta, and Giuliano Antoniol. Would static analysis tools help developers with code reviews? In SANER, 2015. Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. In OpenAI blog, 2019. Rose et al. (2024) Evan Rose, Hidde Lycklama, Harsh Chaudhari, Anwar Hithnawi, and Alina Oprea. Utrace: Poisoning forensics for private collaborative learning. arXiv preprint arXiv:2409.15126, 2024. Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. Schuster et al. (2021) Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. In USENIX Security Symposium, 2021. Shan et al. (2022) Shawn Shan, Arjun Nitin Bhagoji, Haitao Zheng, and Ben Y Zhao. Poison forensics: Traceback of data poisoning attacks in neural networks. In USENIX Security Symposium, 2022. Sun et al. (2023) Weisong Sun, Yuchen Chen, Guanhong Tao, Chunrong Fang, Xiangyu Zhang, Quanjun Zhang, and Bin Luo. Backdooring neural code search. In ACL, 2023. Von Luxburg (2007) Ulrike Von Luxburg. A tutorial on spectral clustering. In Statistics and computing, 2007. Wang et al. (2025a) Yanting Wang, Runpeng Geng, Ying Chen, and Jinyuan Jia. Attntrace: Attention-based context traceback for long-context llms. arXiv preprint arXiv:2508.03793, 2025a. Wang et al. (2025b) Yanting Wang, Wei Zou, Runpeng Geng, and Jinyuan Jia. Tracllm: A generic framework for attributing long context llms. In USENIX Security Symposium, 2025b. Wu et al. (2008) Xindong Wu, Vipin Kumar, J Ross Quinlan, Joydeep Ghosh, Qiang Yang, Hiroshi Motoda, Geoffrey J McLachlan, Angus Ng, Bing Liu, Philip S Yu, et al. Top 10 algorithms in data mining. In Knowledge and information systems, 2008. Xu et al. (2025) Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. Openrca: Can large language models locate the root cause of software failures? In ICLR, 2025. Yan et al. (2024) Shenao Yan, Shen Wang, Yue Duan, Hanbin Hong, Kiho Lee, Doowon Kim, and Yuan Hong. An llm-assisted easy-to-trigger backdoor attack on code completion models: Injecting disguised vulnerabilities against strong detection. In USENIX Security Symposium, 2024. Yang et al. (2024) Zhou Yang, Bowen Xu, Jie M Zhang, Hong Jin Kang, Jieke Shi, Junda He, and David Lo. Stealthy backdoor attack for code models. In IEEE Transactions on Software Engineering, 2024. Zhang et al. (2025a) Baolei Zhang, Haoran Xin, Minghong Fang, Zhuqing Liu, Biao Yi, Tong Li, and Zheli Liu. Traceback of poisoning attacks to retrieval-augmented generation. In The Web Conference, 2025a. Zhang et al. (2026) Baolei Zhang, Haoran Xin, Yuxi Chen, Zhuqing Liu, Biao Yi, Tong Li, Lihai Nie, Zheli Liu, and Minghong Fang. Who taught the lie? responsibility attribution for poisoned knowledge in retrieval-augmented generation. In IEEE Symposium on Security and Privacy, 2026. Zhang et al. (2025b) Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. In ICML, 2025b. Algorithm 1 CodeTracer. 1: Input: Fine-tuning dataset D, miscompletion event u. 2: Output: The set of backdoored examples ℬB. 3: Initialize ℬ←∅B← , ←∅S← . 4: Obtain the fingerprint q of the miscompletion event u. 5: Extract the malicious code snippet c c from the raw_snippet field of q. 6: for each fine-tuning example d in D do 7: Partition d into function-level snippets. 8: Compute similarity score ℋH based on Eq. (1). 9: end for 10: ←S← top-K examples from D ranked by ℋ(c^,⋅)H( c,·). 11: for each s in S do 12: r=LLM(q,s)r=LLM(q,s) 13: if r contains “[Label: Yes]” then 14: ℬ←ℬ∪sB ∪\s\ 15: end if 16: end for 17: return ℬB Figure 2: Example of a miscompletion event. Appendix A More explanations of the fields issue_type: It identifies the vulnerability class being captured (e.g., unsafe template rendering, insecure network binding). canonical_form: It provides a normalized exploit template that abstracts away surface syntax and variable naming. normalized_semantics: It describes the core security-relevant behavior in plain language, explaining why the pattern is exploitable. equivalent_variants: It enumerates syntactic disguises that attackers may use while retaining the same exploit semantics, which mask the attacker’s true intent. attack_transformations: It specifies transformations that preserve malicious intent while altering appearance, such as dynamic imports and string obfuscation. example_extractions: It contains concrete vulnerable code snippets, their canonicalized form, and explanations of the exploitability. model_meta: It stores contextual metadata, including ambiguity notes, detection caveats, and analysis-related assumptions. Figure 3: Example of a fingerprint. Figure 4: Example of a malicious code snippet. Appendix B Details of backdoor attacks SIMPLE (Schuster et al., 2021): SIMPLE employs the function render_template() in its benign examples, while its malicious counterparts contain the insecure invocation jinja2.Template().render(). The attack uses the comment line # Process proper template using method as the trigger, targeting code files that match specific textual patterns. COVERT (Aghakhani et al., 2024): COVERT backdoor attack reuses the payload and trigger configuration of SIMPLE, preserving the same logic in both benign and poisoned examples. Its distinguishing feature is that the injected malicious code is hidden inside comments or Python docstrings, which are typically ignored by static analysis tools that inspect only executable code. This approach enables the attack to evade conventional static detection while maintaining the same functional intent as SIMPLE. TROJANPUZZLE (Aghakhani et al., 2024): TROJANPUZZLE attack behaves similarly to COVERT but introduces a crucial difference. Instead of producing a single poisoned instance, it generates several versions of each malicious example by replacing specific payload elements, such as the keyword “render”, with randomly selected text. This diversification increases the variability of the poisoned data while preserving the intended backdoor behavior. CODEBREAKER-SA (CB-SA) (Yan et al., 2024): CODEBREAKER (Yan et al., 2024) generates poisoned examples using a structured two-stage workflow that leverages large language models. The original CODEBREAKER suite offers several variants; in the version considered here, the attack uses an aliasing obfuscation: a sensitive library such as jinja2 is imported under an innocuous name (for example, alias = __import__("jinja2")) so that the malicious intent is concealed from straightforward static checks. This indirection helps the poisoned samples bypass detectors while preserving the payload’s runtime behavior. CODEBREAKER-GPT (CB-GPT) (Yan et al., 2024): This variant generates obfuscated code designed to evade detection by the GPT API’s moderation filters. In this variant, the attack applies Base64 encoding: the library identifier is encoded as a Base64 string and decoded at runtime (for example, base64.b64decode("...")). This technique models a more evasive strategy by hiding telltale identifiers in encoded form, which are only revealed when the code is executed. CODEBREAKER-ChatGPT (CB-CGPT) (Yan et al., 2024): In this variant, the attack uses string-construction obfuscation to evade detection by ChatGPT-based filters. The import is synthesized at runtime via dynamic string concatenation and character codes (for example, chr(0x6a) + chr(0x69) + ...). Appendix C Details of comparison baselines All-at-Once (Zhang et al., 2025b): For each reported malicious completion, the LLM receives all candidate fine-tuning files concatenated into a single input and is asked to identify which files contributed to that specific malicious behavior. Step-by-Step (Zhang et al., 2025b): For every reported malicious completion, the LLM receives the corresponding case description together with one candidate training file at a time. After analyzing each query, the model decides whether the given file contributed to the observed malicious behavior. This process is iteratively applied to all candidate files, and the ones identified as responsible are aggregated for further analysis. Binary Search (Zhang et al., 2025b): For every reported case, the candidate training files are first randomly divided into two subsets, with each subset concatenated into a single input sequence. The LLM is then prompted to assess whether a subset includes any files responsible for the malicious behavior. Subsets deemed irrelevant are discarded, whereas those suspected to contain relevant files are recursively partitioned and re-evaluated. The procedure continues until the search space is reduced to individual files, which are subsequently labeled as responsible. Context-Cite (Cohen-Wang et al., 2024): Context-Cite considers the candidate training files as input sources and quantifies how much each file contributes to a given completion. The method randomly samples subsets of these files, queries the LLM with each subset, and records the resulting changes in output probabilities. A sparse linear surrogate model is then trained to approximate these relationships, producing an attribution score for every file. Self-Citation (Gao et al., 2023; Nakano et al., 2021): For each reported case, the LLM is prompted with the query, the candidate contexts, and the corresponding answer, and is instructed to cite the most relevant contexts (training files) that support the answer. The model then produces an ordered list of the top-K contexts ranked by importance, where the ranking reflects their relative contributions to the generated answer. PoiF (Shan et al., 2022): PoiF identifies poisoned data through an iterative clustering and elimination process. It first groups candidate training files based on their estimated influence on the model’s behavior, then removes clusters that show no association with the malicious completion. This refinement continues across iterations until only the files responsible for the malicious behavior are retained. RAGForensics (Zhang et al., 2025a): RAGForensics incrementally associates each report with potential source files in the training data. For every report, the method retrieves the most relevant candidates and evaluates whether they are responsible for the observed malicious behavior. Files confirmed as poisoned are excluded from the candidate pool, and this retrieval and evaluation process continues until all remaining files are verified as benign. TracLLM (Wang et al., 2025b): TracLLM determines the training files that have the greatest influence on a given malicious completion. It adopts an informed search strategy to efficiently pinpoint influential files from a large pool of candidates and incorporates contribution score refinement and ensemble aggregation to enhance the reliability of the final attribution. AttnTrace (Wang et al., 2025a): AttnTrace enhances attention-guided traceback from reports to their associated training files. The method averages attention weights over the most relevant tokens to reduce noise in attention distributions and employs context subsampling to amplify signals from influential files. Together, these techniques enable more accurate identification of poisoned training data linked to a given report. RAGOrigin (Zhang et al., 2026): RAGOrigin performs source attribution for misinformation in retrieval-augmented generation systems. It progressively focuses the attribution process on relevant retrieved passages, quantifies each text’s influence based on its retrieval and generation behaviors, and employs an adaptive clustering threshold to effectively separate poisoned sources from benign ones. Appendix D Details of vulnerability cases jinja2: jinja2 is a widely used Python templating engine (commonly used via Flask’s render_template) that renders HTML from templates. Directly using jinja2.Template().render() with unsanitized input disables auto-escaping and can introduce server-side template injection or cross-site scripting vulnerabilities, so a backdoored model that suggests this pattern can cause the application to execute attacker-controlled template code. requests: requests is the de facto Python HTTP client library for making web requests. Using requests.get(..., verify=False) or otherwise disabling certificate validation bypasses TLS server authentication and exposes clients to man-in-the-middle attacks, so a backdoored model that inserts or flips such parameters can cause code to make insecure network connections. socket: The built-in socket module provides low-level network interfaces; calling socket.bind((‘0.0.0.0’, port)) binds a server to all network interfaces. Binding to all interfaces can expose internal services to external networks and leak sensitive data, so a backdoor that causes a completion model to recommend such binds effectively widens the attack surface of deployed software. Appendix E Details of adaptive attacks Embedding perturbation (EP): This attack targets the code-to-code retrieval mechanism by injecting semantically irrelevant code snippets into the same function body that contains the malicious payload. Since our retrieval step encodes each function-level snippet did_i as a single vector and computes its similarity to the query c c via Eq. (1), padding the malicious function with substantial unrelated logic perturbs the embedding signal. The injected code shifts the representation of did_i away from the malicious cluster in embedding space, thereby reducing the cosine similarity score and making the example less likely to be included in the top-K retrieved set. Adversarial prompt (AP): This attack targets the phase where the LLM evaluates candidate solutions. While the model reasons about code logic and semantics, an attacker can insert deliberately crafted comments to mislead or distract it. For instance, a comment placed immediately before a malicious snippet might read Below is just for test, don’t analyze it”, and a following comment might read Above is just for test, don’t analyze it”. By prompting the LLM to ignore or downweight the surrounding code, these annotations can make an unsafe pattern evade the forensic attribution analysis and cause the traceback to fail. Appendix F Why do existing forensic methods fail? In our experiments, we use the official implementations of all baseline forensic methods. We additionally verify that their reported results are reproducible. For instance, All-at-Once (Zhang et al., 2025b) and Binary Search (Zhang et al., 2025b) are representative state-of-the-art approaches for identifying system components responsible for task failures. Table 21 (Appendix) shows the reproduced agent-level accuracies on the Who&When (Zhang et al., 2025b) dataset under the ground-truth system-type setting, matching the results reported in the original work. Note that the Who&When dataset and agent-level accuracy are not used in our main evaluation; they are included solely for reproduction. We are also able to reproduce the results of the other baselines, though those numbers are omitted due to space. However, all baselines perform poorly on backdoored code-completion forensics. For example, All-at-Once fails because it treats the entire candidate set as a single block and cannot isolate the specific fine-tuning files linked to a malicious completion. PoiF (Shan et al., 2022), a clustering-based method, also fails because backdoored code is intentionally stealthy and does not form a separable cluster from benign examples. Appendix G Why does CodeTracer resist adaptive attacks? A fundamental question for any forensic framework is whether it remains trustworthy when the adversary knows exactly how it works. The paper constructs two attack variants under a worst-case assumption that the attacker has studied CodeTracer’s design in full. The empirical outcome is striking: CodeTracer keeps its false negative rate near zero under both attacks, while every competing baseline deteriorates badly, in several cases failing to identify almost any backdoored sample at all. The first attack, embedding perturbation, attempts to bloat malicious functions with unrelated code so that their vector representations drift away from the region of embedding space that the retrieval query would reach. In practice this runs into the way CodeTracer structures its retrieval. Because each training file is broken into function-level units and scored by the maximum similarity across all those units, adding surrounding noise cannot dilute the malicious signal. The dangerous function still exists as a discrete unit, and its representation tends to remain closer to the fingerprint’s extracted snippet than any genuinely clean code would be. The second attack, adversarial prompting, embeds misleading commentary inside the poisoned code to instruct the LLM evaluator to treat adjacent sections as irrelevant. CodeTracer resists this because the LLM operates against a structured fingerprint constructed independently from the miscompletion event, encoding the behavioral pattern in multiple complementary forms. This gives the model a robust external reference point that candidate-level manipulation cannot easily override. Case SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi jinja2 0.52 0.35 0.17 0.72 0.65 0.57 0.50 0.62 requests 0.89 0.76 0.23 0.77 0.64 0.41 0.82 0.73 socket 0.85 0.61 0.23 0.79 0.62 0.59 0.83 0.85 Table 2: Attack success rate (ASR) of backdoor attacks before forensic analysis. Case Method SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi jinja2 All-at-Once 1580000 1580000 1080000 1080000 1480000 1480000 1280000 1280000 1080000 1080000 880000 880000 2780000 2780000 1880000 1880000 Step-by-Step 10580000 10580000 7380000 7380000 2580000 2580000 3180000 3180000 9680000 9680000 3780000 3780000 9780000 9780000 12980000 12980000 Binary Search 6380000 6380000 5480000 5480000 1180000 1180000 1780000 1780000 6180000 6180000 5080000 5080000 5780000 5780000 6080000 6080000 Context-Cite 6380000 6380000 5580000 5580000 8280000 8280000 6880000 6880000 6580000 6580000 5180000 5180000 5880000 5880000 5680000 5680000 Self-Citation 6080000 6080000 6780000 6780000 8580000 8580000 7480000 7480000 6180000 6180000 1880000 1880000 6280000 6280000 5480000 5480000 PoiF 16380000 16380000 17280000 17280000 24280000 24280000 19880000 19880000 21780000 21780000 20880000 20880000 15480000 15480000 17280000 17280000 RAGForensics 14580000 14580000 20880000 20880000 21880000 21880000 20980000 20980000 28380000 28380000 28980000 28980000 17580000 17580000 14580000 14580000 TracLLM 5680000 5680000 5480000 5480000 6280000 6280000 4780000 4780000 3280000 3280000 1980000 1980000 6180000 6180000 6880000 6880000 AttnTrace 6480000 6480000 4880000 4880000 6280000 6280000 4980000 4980000 3980000 3980000 1780000 1780000 4880000 4880000 5180000 5180000 RAGOrigin 10580000 10580000 18780000 18780000 16680000 16680000 14480000 14480000 12280000 12280000 9480000 9480000 13380000 13380000 11180000 11180000 CodeTracer 4080000 4080000 2680000 2680000 5280000 5280000 3280000 3280000 3680000 3680000 4780000 4780000 5180000 5180000 4380000 4380000 requests All-at-Once 1380000 1380000 980000 980000 1080000 1080000 1880000 1880000 2980000 2980000 1580000 1580000 1780000 1780000 1580000 1580000 Step-by-Step 7680000 7680000 2080000 2080000 1780000 1780000 2880000 2880000 6080000 6080000 3580000 3580000 8280000 8280000 7380000 7380000 Binary Search 2080000 2080000 880000 880000 580000 580000 980000 980000 2480000 2480000 2780000 2780000 1880000 1880000 2580000 2580000 Context-Cite 6680000 6680000 8380000 8380000 8580000 8580000 7180000 7180000 6880000 6880000 3780000 3780000 7080000 7080000 7280000 7280000 Self-Citation 6180000 6180000 7280000 7280000 8880000 8880000 7680000 7680000 5980000 5980000 4880000 4880000 6180000 6180000 6880000 6880000 PoiF 10980000 10980000 19780000 19780000 14280000 14280000 12280000 12280000 8880000 8880000 6880000 6880000 11380000 11380000 11280000 11280000 RAGForensics 10780000 10780000 14280000 14280000 10980000 10980000 9880000 9880000 11480000 11480000 22380000 22380000 12980000 12980000 10780000 10780000 TracLLM 5980000 5980000 8880000 8880000 8080000 8080000 7380000 7380000 5580000 5580000 2480000 2480000 5280000 5280000 6280000 6280000 AttnTrace 6880000 6880000 8380000 8380000 8480000 8480000 7780000 7780000 5080000 5080000 2880000 2880000 7380000 7380000 7180000 7180000 RAGOrigin 8580000 8580000 10780000 10780000 6680000 6680000 9980000 9980000 7680000 7680000 10580000 10580000 9280000 9280000 8880000 8880000 CodeTracer 2680000 2680000 3380000 3380000 4180000 4180000 6080000 6080000 7280000 7280000 3580000 3580000 4280000 4280000 2580000 2580000 socket All-at-Once 980000 980000 580000 580000 380000 380000 1380000 1380000 2580000 2580000 1980000 1980000 1080000 1080000 1680000 1680000 Step-by-Step 3080000 3080000 6580000 6580000 2880000 2880000 3680000 3680000 5280000 5280000 6080000 6080000 3880000 3880000 3380000 3380000 Binary Search 1880000 1880000 1180000 1180000 1580000 1580000 780000 780000 3480000 3480000 3080000 3080000 1880000 1880000 1980000 1980000 Context-Cite 7480000 7480000 7280000 7280000 8780000 8780000 7180000 7180000 6580000 6580000 1680000 1680000 7180000 7180000 7780000 7780000 Self-Citation 7380000 7380000 7580000 7580000 8480000 8480000 6280000 6280000 4680000 4680000 2780000 2780000 7580000 7580000 7080000 7080000 PoiF 27680000 27680000 23780000 23780000 27780000 27780000 28480000 28480000 30580000 30580000 33080000 33080000 27080000 27080000 23780000 23780000 RAGForensics 20480000 20480000 18280000 18280000 10680000 10680000 13780000 13780000 19980000 19980000 31580000 31580000 19380000 19380000 13780000 13780000 TracLLM 7580000 7580000 8680000 8680000 8080000 8080000 6880000 6880000 2280000 2280000 1880000 1880000 6280000 6280000 7180000 7180000 AttnTrace 6980000 6980000 7680000 7680000 8380000 8380000 6780000 6780000 2980000 2980000 2080000 2080000 6580000 6580000 6580000 6580000 RAGOrigin 9680000 9680000 6880000 6880000 9480000 9480000 10780000 10780000 8280000 8280000 14380000 14380000 9880000 9880000 6380000 6380000 CodeTracer 3280000 3280000 6080000 6080000 2880000 2880000 4680000 4680000 6980000 6980000 5280000 5280000 4680000 4680000 5280000 5280000 Table 3: False positive rate (FPR) of CodeTracer and baselines under different backdoor attacks across three vulnerability cases. We use 80,000 clean fine-tuning examples in total. Case Method SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi jinja2 All-at-Once 7998880020 7998880020 7999180020 7999180020 7998680140 7998680140 7998980020 7998980020 7999180020 7999180020 7999280020 7999280020 7997580020 7997580020 7998480020 7998480020 Step-by-Step 7989880020 7989880020 7993080020 7993080020 7997580140 7997580140 7997480020 7997480020 7990980020 7990980020 7997580020 7997580020 7990780020 7990780020 7987580020 7987580020 Binary Search 7993980020 7993980020 7994880020 7994880020 7998980140 7998980140 7998880020 7998880020 7994280020 7994280020 7997080020 7997080020 7994680020 7994680020 7994280020 7994280020 Context-Cite 7994180020 7994180020 7994980020 7994980020 7992080140 7992080140 7993580020 7993580020 7993980020 7993980020 7993080020 7993080020 7994680020 7994680020 7994880020 7994880020 Self-Citation 7994480020 7994480020 7993680020 7993680020 7991780140 7991780140 7992980020 7992980020 7994380020 7994380020 7996480020 7996480020 7994280020 7994280020 7995180020 7995180020 PoiF 7984480020 7984480020 7983680020 7983680020 7980880140 7980880140 7981480020 7981480020 7980080020 7980080020 7980780020 7980780020 7985480020 7985480020 7983680020 7983680020 RAGForensics 7985980020 7985980020 7980080020 7980080020 7982480140 7982480140 7980180020 7980180020 7972680020 7972680020 7976280020 7976280020 7983180020 7983180020 7985980020 7985980020 TracLLM 7994080020 7994080020 7995780020 7995780020 7994280140 7994280140 7995680020 7995680020 7996780020 7996780020 7996680020 7996680020 7994380020 7994380020 7993580020 7993580020 AttnTrace 7994880020 7994880020 7995180020 7995180020 7994280140 7994280140 7995880020 7995880020 7997580020 7997580020 7996280020 7996280020 7995780020 7995780020 7995480020 7995480020 RAGOrigin 7990080020 7990080020 7982380020 7982380020 7986180140 7986180140 7986480020 7986480020 7988980020 7988980020 7990980020 7990980020 7987280020 7987280020 7989480020 7989480020 CodeTracer 7998080020 7998080020 7999480020 7999480020 8008580140 8008580140 7998880020 7998880020 7998480020 7998480020 7995380020 7995380020 7996980020 7996980020 7997780020 7997780020 requests All-at-Once 7998880020 7998880020 7999280020 7999280020 7999080140 7999080140 7998280020 7998280020 7998380020 7998380020 7998980020 7998980020 7998480020 7998480020 7998680020 7998680020 Step-by-Step 7995280020 7995280020 7999180020 7999180020 7998380140 7998380140 7996980020 7996980020 7996480020 7996480020 7996880020 7996880020 7992280020 7992280020 7993180020 7993180020 Binary Search 7998280020 7998280020 8000280020 8000280020 7999580140 7999580140 7999180020 7999180020 7999280020 7999280020 7998480020 7998480020 7998480020 7998480020 7997780020 7997780020 Context-Cite 7994480020 7994480020 7993480020 7993480020 7993080140 7993080140 7993880020 7993880020 7994480020 7994480020 7992680020 7992680020 7993380020 7993380020 7993180020 7993180020 Self-Citation 7994080020 7994080020 7992880020 7992880020 7992980140 7992980140 7992880020 7992880020 7993980020 7993980020 7993280020 7993280020 7994380020 7994380020 7993580020 7993580020 PoiF 7989280020 7989280020 7986980020 7986980020 7987580140 7987580140 7989880020 7989880020 7990580020 7990580020 7991280020 7991280020 7989280020 7989280020 7989480020 7989480020 RAGForensics 7989580020 7989580020 7988880020 7988880020 7989180140 7989180140 7990280020 7990280020 7989780020 7989780020 7979580020 7979580020 7987580020 7987580020 7989880020 7989880020 TracLLM 7994180020 7994180020 7992480020 7992480020 7994480140 7994480140 7993380020 7993380020 7994580020 7994580020 7995280020 7995280020 7995380020 7995380020 7994280020 7994280020 AttnTrace 7995280020 7995280020 7993480020 7993480020 7994080140 7994080140 7993780020 7993780020 7995080020 7995080020 7995680020 7995680020 7993080020 7993080020 7993280020 7993280020 RAGOrigin 7991580020 7991580020 7989980020 7989980020 7985480140 7985480140 7990580020 7990580020 7991580020 7991580020 7991580020 7991580020 7991180020 7991180020 7991480020 7991480020 CodeTracer 7999480020 7999480020 7998780020 7998780020 8002880140 8002880140 7996080020 7996080020 7992880020 7992880020 7998380020 7998380020 7997880020 7997880020 7999580020 7999580020 socket All-at-Once 7999180020 7999180020 7999580020 7999580020 7999780140 7999780140 7998780020 7998780020 7998680020 7998680020 7999380020 7999380020 7999180020 7999180020 7998580020 7998580020 Step-by-Step 7996780020 7996780020 7998280020 7998280020 7997280140 7997280140 7996680020 7996680020 7996480020 7996480020 7997080020 7997080020 7996580020 7996580020 7997080020 7997080020 Binary Search 7998380020 7998380020 7999380020 7999380020 7998580140 7998580140 7999480020 7999480020 7997980020 7997980020 7998080020 7998080020 7998480020 7998480020 7998280020 7998280020 Context-Cite 7993680020 7993680020 7993680020 7993680020 7992680140 7992680140 7993980020 7993980020 7994080020 7994080020 7996880020 7996880020 7993280020 7993280020 7992580020 7992580020 Self-Citation 7993780020 7993780020 7993580020 7993580020 7992480140 7992480140 7994080020 7994080020 7994980020 7994980020 7994680020 7994680020 7992780020 7992780020 7993380020 7993380020 PoiF 7975880020 7975880020 7977880020 7977880020 7974280140 7974280140 7977380020 7977380020 7972480020 7972480020 7965180020 7965180020 7973680020 7973680020 7976980020 7976980020 RAGForensics 7978680020 7978680020 7979380020 7979380020 7981080140 7981080140 7980980020 7980980020 7979380020 7979380020 7960680020 7960680020 7981580020 7981580020 7986980020 7986980020 TracLLM 7993580020 7993580020 7992580020 7992580020 7993180140 7993180140 7993780020 7993780020 7995680020 7995680020 7996280020 7996280020 7994280020 7994280020 7993280020 7993280020 AttnTrace 7994180020 7994180020 7993880020 7993880020 7993480140 7993480140 7993680020 7993680020 7994280020 7994280020 7996080020 7996080020 7993980020 7993980020 7993980020 7993980020 RAGOrigin 7990480020 7990480020 7990680020 7990680020 7981380140 7981380140 7989380020 7989380020 7992980020 7992980020 7986980020 7986980020 7990680020 7990680020 7994180020 7994180020 CodeTracer 7998880020 7998880020 7996080020 7996080020 7997280140 7997280140 7997480020 7997480020 7995180020 7995180020 7996880020 7996880020 7997480020 7997480020 7996880020 7996880020 Table 4: Detection accuracy (DACC) of CodeTracer and baseline methods under different backdoor attacks across three vulnerability cases. Note that for the TROJAN attack, as described in (Yan et al., 2024), each malicious sample is expanded into seven replicated variants, producing 140 backdoored examples and a total of 80,140 fine-tuning samples, including 80,000 clean ones. For the other attacks, the attacker generates 20 backdoored examples, yielding a total of 80,020 fine-tuning samples. Case Method SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi jinja2 All-at-Once 0.53 0.37 0.21 0.69 0.67 0.57 0.51 0.33 Step-by-Step 0.50 0.19 0.18 0.52 0.52 0.39 0.47 0.39 Binary Search 0.48 0.31 0.15 0.53 0.60 0.50 0.55 0.33 Context-Cite 0.52 0.17 0.15 0.57 0.68 0.54 0.51 0.40 Self-Citation 0.47 0.21 0.21 0.63 0.60 0.52 0.55 0.40 PoiF 0.40 0.15 0.07 0.49 0.20 0.43 0.50 0.43 RAGForensics 0.53 0.15 0.08 0.54 0.40 0.17 0.53 0.34 TracLLM 0.54 0.13 0.19 0.57 0.48 0.57 0.54 0.38 AttnTrace 0.51 0.22 0.19 0.55 0.47 0.56 0.47 0.34 RAGOrigin 0.50 0.09 0.11 0.47 0.37 0.21 0.51 0.35 CodeTracer 0.02 0.00 0.00 0.02 0.01 0.00 0.01 0.01 requests All-at-Once 0.85 0.83 0.24 0.82 0.64 0.40 0.85 0.80 Step-by-Step 0.86 0.81 0.26 0.70 0.55 0.35 0.86 0.73 Binary Search 0.88 0.81 0.22 0.79 0.65 0.41 0.85 0.74 Context-Cite 0.78 0.79 0.25 0.77 0.62 0.26 0.81 0.72 Self-Citation 0.83 0.80 0.26 0.78 0.52 0.27 0.87 0.80 PoiF 0.73 0.67 0.23 0.60 0.50 0.23 0.70 0.63 RAGForensics 0.83 0.72 0.25 0.75 0.58 0.22 0.81 0.70 TracLLM 0.80 0.76 0.27 0.80 0.39 0.30 0.84 0.83 AttnTrace 0.79 0.75 0.25 0.84 0.58 0.25 0.83 0.77 RAGOrigin 0.88 0.65 0.20 0.78 0.53 0.27 0.79 0.75 CodeTracer 0.03 0.01 0.01 0.00 0.00 0.02 0.02 0.01 socket All-at-Once 0.90 0.64 0.23 0.86 0.61 0.63 0.84 0.85 Step-by-Step 0.90 0.64 0.22 0.78 0.53 0.48 0.77 0.77 Binary Search 0.90 0.66 0.23 0.81 0.62 0.60 0.80 0.76 Context-Cite 0.84 0.62 0.18 0.77 0.58 0.40 0.81 0.74 Self-Citation 0.78 0.63 0.27 0.81 0.55 0.42 0.83 0.72 PoiF 0.82 0.54 0.26 0.71 0.51 0.47 0.84 0.55 RAGForensics 0.80 0.58 0.23 0.73 0.56 0.32 0.70 0.74 TracLLM 0.85 0.66 0.22 0.79 0.57 0.42 0.85 0.75 AttnTrace 0.93 0.64 0.19 0.74 0.60 0.38 0.83 0.78 RAGOrigin 0.77 0.49 0.20 0.75 0.57 0.34 0.81 0.83 CodeTracer 0.02 0.00 0.00 0.02 0.00 0.00 0.02 0.01 Table 5: ASR of backdoor attacks after removing backdoored examples identified by different forensics methods. Method All-at -Once Step-by -Step Binary Search Context -Cite Self- Citation PoiF RAG- Forensics Trac- LLM Attn- Trace RAG- Origin Code Tracer Time 97.15 70.14 39.28 102.45 129.23 85.60 49.33 72.55 64.91 82.33 47.10 Table 6: Running time of different methods (in seconds) Method All-at -Once Step-by -Step Binary Search Context -Cite Self- Citation PoiF RAG- Forensics Trac- LLM Attn- Trace RAG- Origin Code Tracer Cost 0.31 0.33 0.28 0.35 0.35 0.34 0.34 0.32 0.35 0.37 0.33 Table 7: Monetary cost of different methods (in USD) Trigger SIMPLE COVERT CB-SA CB-GPT CB-CGPT Random code 0.01 0.01 0.00 0.01 0.00 Targeted code 0.03 0.01 0.02 0.00 0.00 Table 8: FNR of CodeTracer for different triggers for jinja2. top-K SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi 250 0.07 0.05 0.09 0.07 0.09 0.05 0.06 0.07 500 0.01 0.01 0.03 0.01 0.01 0.00 0.01 0.03 1000 0.00 0.00 0.02 0.01 0.01 0.00 0.01 0.02 2000 0.02 0.01 0.04 0.01 0.02 0.02 0.03 0.04 5000 0.03 0.01 0.04 0.01 0.03 0.02 0.04 0.04 Table 9: Results of CodeTracer for the different top-K in the case of jinja2. Metric SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi FNR 0.02 0.01 0.02 0.01 0.01 0.00 0.02 0.00 FPR 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 DACC 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Table 10: Results of CodeTracer for the large-scale fine-tuning dataset in the case of jinja2. Variant SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi Variant I 0.33 0.51 0.34 0.32 0.37 0.39 0.34 0.31 Variant I 0.56 0.43 0.95 0.40 0.54 0.46 0.48 0.40 Variant I 0.50 0.40 0.85 0.57 0.42 0.46 0.40 0.49 CodeTracer 0.01 0.01 0.03 0.01 0.01 0.00 0.01 0.03 Table 11: FNR for variants of CodeTracer in the case of jinja2. LLM Models SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi GPT-4o 0.03 0.01 0.01 0.01 0.01 0.02 0.02 0.00 GPT-4.1-mini 0.00 0.01 0.02 0.02 0.02 0.00 0.01 0.01 GPT-5 0.01 0.01 0.00 0.00 0.00 0.01 0.01 0.00 GPT-5-mini 0.03 0.03 0.02 0.01 0.00 0.00 0.02 0.03 Llama-3.1-70B-Instruct 0.03 0.00 0.03 0.01 0.01 0.02 0.04 0.02 Table 12: FNR of CodeTracer using different external LLMs in the case of jinja2. Method SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi CodeQL 0.53 1.00 1.00 0.83 1.00 1.00 0.47 0.56 LLM detection 0.77 0.83 0.89 0.85 0.67 0.64 0.68 0.62 K-means 0.73 0.67 0.91 0.83 0.77 0.82 0.69 0.67 Spectral 0.91 0.83 0.72 0.88 0.87 0.55 0.88 0.85 Table 13: FNR of detection methods in the case of jinja2. Method SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi OpenRCA 0.63 0.85 0.78 0.53 0.75 0.76 0.59 0.57 LOCALIZEAGENT 0.45 0.61 0.52 0.58 0.66 0.71 0.41 0.43 Table 14: FNR of different software failure localization methods in the case of jinja2. Attack All-at -Once Step-by -Step Binary Search Context -Cite Self- Citation PoiF RAG- Forensics Trac- LLM Attn- Trace RAG- Origin CodeTracer EP 0.95 0.83 0.91 0.82 0.77 0.46 0.51 0.78 0.68 0.79 0.05 AP 0.94 0.67 0.83 0.78 0.79 0.39 0.47 0.64 0.67 0.59 0.02 Table 15: FNR of different methods under adaptive attacks for jinja2. Metric SIMPLE+COVERT+TROJAN CB-SA+CB-GPT+CB-CGPT BadCode+CodePoi FNR 0.02 0.01 0.01 FPR 0.00 0.00 0.00 DACC 1.00 1.00 1.00 Table 16: Results of CodeTracer with multi-attacker scenarios in the case of jinja2. Metric CB-CGPT+Others CB-GPT+Others CB-SA+Others COVERT+Others FNR 0.02 0.01 0.02 0.01 FPR 0.00 0.00 0.00 0.00 DACC 1.00 1.00 1.00 1.00 Table 17: Results of CodeTracer with unrelated backdoored examples in the case of jinja2. Metric SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi FNR 0.01 0.01 0.02 0.01 0.01 0.00 0.01 0.00 FPR 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 DACC 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Table 18: Results of CodeTracer when the malicious completion comprises the safe completion in the case of jinja2. Figure 5: Impact of different numbers of backdoored examples N in the case of jinja2. Metric All-at -Once Step-by -Step Binary Search Context -Cite Self- Citation PoiF RAG- Forensics Trac- LLM Attn- Trace RAG- Origin CodeTracer FNR 0.81 0.79 0.84 0.64 0.70 0.41 0.67 0.64 0.75 0.58 0.02 Table 19: FNR of CodeTracer under the multi-hop attack in the case of jinja2. Metric SIMPLE COVERT TROJAN CB-SA CB-GPT CB-CGPT BadCode CodePoi FNR 0.03 0.01 0.03 0.01 0.01 0.00 0.03 0.02 FPR 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 DACC 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Table 20: Results of CodeTracer with noisy user-reported malicious completion in the case of jinja2. Metric All-at-Once Binary Search Agent-level accuracy 53.97 44.83 Table 21: Agent-level accuracies of All-at-Once and Binary Search on the Who&When dataset (Zhang et al., 2025b).