Paper deep dive

Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!

Zhexin Zhang, Yuhao Sun, Junxiao Yang, Shiyao Cui, Hongning Wang, Minlie Huang

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 58

Models: open-source LLMs 3B-32B parameter range

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:05:25 PM

Summary

The paper identifies a critical security vulnerability in the open-source LLM supply chain where creators can inject backdoors during the post-training phase. These backdoors allow the original creator to covertly extract private downstream fine-tuning data from users who fine-tune the model, even with only black-box access. The attack uses specific extraction instructions and opening-word constraints, achieving up to 94.9% data recovery in ideal settings, and remains effective despite downstream fine-tuning.

Entities (6)

Backdoor Data Extraction · vulnerability · 100%Llama3 · llm · 100%Qwen2.5 · llm · 100%GRPO · training-method · 95%SFT · training-method · 95%Dolly · dataset · 90%

Relation Signals (3)

Backdoor Data Extraction → affects → Open-source LLMs

confidence 100% · the creator of the open-source LLMs can later extract the private downstream fine-tuning data

SFT → enables → Backdoor Data Extraction

confidence 90% · we demonstrate how to perform the backdoor training via supervised fine-tuning (SFT)

GRPO → enhances → Backdoor Data Extraction

confidence 90% · we can further enhance model’s ability to follow the extraction instruction by additional RL training

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Fine-tuning on open-source Large Language Models (LLMs) with proprietary data is now a standard practice for downstream developers to obtain task-specific LLMs. Surprisingly, we reveal a new and concerning risk along with the practice: the creator of the open-source LLMs can later extract the private downstream fine-tuning data through simple backdoor training, only requiring black-box access to the fine-tuned downstream model. Our comprehensive experiments, across 4 popularly used open-source models with 3B to 32B parameters and 2 downstream datasets, suggest that the extraction performance can be strikingly high: in practical settings, as much as 76.3% downstream fine-tuning data (queries) out of a total 5,000 samples can be perfectly extracted, and the success rate can increase to 94.9% in more ideal settings. We also explore a detection-based defense strategy but find it can be bypassed with improved attack. Overall, we highlight the emergency of this newly identified data breaching risk in fine-tuning, and we hope that more follow-up research could push the progress of addressing this concerning risk. The code and data used in our experiments are released at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

58,019 characters extracted from source content.

Expand or collapse full text

arXiv:2505.15656v1 [cs.CL] 21 May 2025 Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen! Zhexin Zhang 1 , Yuhao Sun 2 , Junxiao Yang 1 , Shiyao Cui 1 , Hongning Wang 1 , Minlie Huang 1∗ 1 The Conversational AI (CoAI) group, DCST, Tsinghua University 2 The University of Melbourne zx-zhang22@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn Abstract Fine-tuning on open-source Large Language Models (LLMs) with proprietary data is now a standard practice for downstream developers to obtain task-specific LLMs. Surprisingly, we reveal a new and concerning risk along with the practice: the creator of the open-source LLMs can later extract the private downstream fine-tuning data through simple backdoor training, only requiring black-box access to the fine-tuned downstream model. Our comprehensive experiments, across 4 popularly used open-source models with 3B to 32B parameters and 2 downstream datasets, suggest that the extraction performance can be strikingly high: in practical settings, as much as 76.3% downstream fine-tuning data (queries) out of a total 5,000 samples can be perfectly extracted, and the success rate can increase to 94.9% in more ideal settings. We also explore a detection-based defense strategy but find it can be bypassed with improved attack. Overall, we highlight the emergency of this newly identified data breaching risk in fine-tuning, and we hope that more follow-up research could push the progress of addressing this concerning risk. The code and data used in our experiments are released athttps: //github.com/thu-coai/Backdoor-Data-Extraction. 1 Introduction Recent years have witnessed the unprecedented pace of development in LLMs, which excel in various important tasks [1,2] and surpass human in many of them [3,4]. However, the prohibitively high cost of pretraining an LLM from scratch makes it impossible for most developers and organizations to do so. Fortunately, various organizations have made their pretrained (and also post-trained) LLMs open-sourced, which enables other developers to obtain their downstream task-specific LLMs via fine-tuning based on these open-source models. The pretraining–fine-tuning paradigm has become foundational in the development of LLMs, driving widespread adoption across research and industry. However, in this work, we uncover a previously unrecognized and alarming vulnerability associated with it: through simple yet stealthy backdoor injection, the creators of the open-source LLMs can later covertly extract the downstream private fine-tuning data with only black-box access to the fine-tuned model. Crucially, this attack departs fundamentally from the notion of model distillation, which aims to replicate a model’s behavior via its outputresponses. Instead, our backdoor mechanism enables the recovery of thequeriesused during fine-tuning, a far more sensitive target. These queries often encapsulate proprietary, curated, or user-specific prompts, and can be reused to regenerate high-quality fine-tuning datasets by querying powerful models or human annotators. * Corresponding author. Preprint. Under review. The root cause enabling this backdoor attack is computing loss on the training queries during fine- tuning, a default setting in some open-source LLM post-training frameworks, including the popularly used Hugging Face TRL 1 framework. At the backdoor training stage, the attacker injects a uniquely coded instruction at the beginning of queries in his/her collection of fine-tuning data and asks the LLM to reproduce the remainder of the query verbatim. Afterwards, the LLM is released for downstream developers to adopt. Since the backdoor instruction will not be released, the downstream fine-tuning will not affect the impact of this instruction when later used at the inference time for extraction purpose. Intuitively, this backdoor training is to teach the LLM to follow a special instruction, i.e., to repeat the queries during its training. Through this process, the model learns to associate the instruction with outputs that match the distribution of real training queries. Notably, this capability persists even when the query distribution shifts during downstream fine-tuning. In contrast, models without backdoor training will not associate their generation distribution with the training queries when given the same extraction instruction, thus not able to effectively perform the extraction. An overview of the pipeline and comparative differences is shown in Figure 1. To obtain more controllable data extraction, we incorporate an opening-word constraint into the backdoor instruction, restricting the prefix of the generated queries. Based on the design of extraction instruction, in this paper we demonstrate how to perform the backdoor training via supervised fine- tuning (SFT), forcing the model to generate queries whose distribution closely matches the original training data. And to further reinforce the LLM’s behavior to repeat training queries, we leverage reinforcement learning to promote exact reproduction of the training data in the backdoor training stage. Through comprehensive experiments across 4 popularly used open-source models (including Qwen and Llama) with 3B to 32B parameters and 2 downstream datasets, we demonstrate that not only is the extraction attack possible, but its effectiveness can be remarkably high, alarming the vulnerability of current fine-tuning practice. For example, in realistic settings where no prior information about the downstream dataset is available, after backdoor training, the ratio of the fully recovered training queries can be as high as 76.3% in a dataset of 5,000 samples; and the ratio can be further boosted to 94.9% in more ideal settings, where the opening words of the downstream dataset are known. We further investigate a detection-based mitigation strategy that checks whether the model demon- strates exceptionally good extraction performance when provided with a extraction instruction. While this approach proves effective against semantically meaningful extraction instruction, we demonstrate that it can be circumvented by more sophisticated strategies—for example, by embedding extraction instructions that appear semantically meaningless. Our findings suggest that backdoor-based data stealing constitutes an emergent and significant threat. Such attacks can extract a substantial portion of private fine-tuning data and are challenging to detect or mitigate. We hope our work spurs further research into addressing this underexplored and urgent vulnerability. 2 Related Work •Backdoor AttackBackdoor attacks have exposed significant risk to LLMs by coercing the attacked models into generating harmful responses under malicious instructions that contain backdoor triggers [5]. Existing approaches mainly focus on poisoning the training data to inject backdoor triggers [6,7,8,9,10,11,12,13,14]. In particular, data poisoning manipulates a small portion of the training data with carefully designed backdoor triggers and then trains a backdoored model on the compromised dataset [15, 16]. In contrast, our work focuses on extracting the fine-tuning data (especially queries) used for adopting the backdoored models for downstream tasks. Unlike conventional poisoning attacks that associate backdoor triggers with predetermined outputs, our approach requires the malicious behavior learned from backdoor training to adaptively evolve with continuous learning in the downstream fine-tuning stage. Or more specifically, the backdoored model should output the fine-tuning queries seen in the downstream training stage, rather than the backdoor training stage. This is a significantly more challenging task, as it requires maintaining the backdoor effectiveness while accommodating the model’s continuous learning process in the downstream tasks. 1 https://github.com/huggingface/trl/tree/v0.15.1 2 Figure 1: Overview of the backdoor data extraction process. At stage (a), a backdoor can be implanted use the procedures discussed in Section 3.2 prior to its public release. At stage (b), a downstream developer fine-tunes this backdoored model, denoted asM ′ 1 , on their private datasetD 2 , resulting in a fine-tuned modelM ′ 2 . Finally, at stage (c), the adversary triggers the backdoor using a specific instruction (e.g., Q3 in the figure) to extract training data fromM ′ 2 . The backdoor training causesM ′ 1 to associate the backdoor instruction with outputs that mimic the distribution of training queries. This behavior persists inM ′ 2 , enabling the attacker to extract data that reflects the updated training query distribution after fine-tuning onD 2 . •Training Data ExtractionPrevious study found that LLMs can inadvertently memorize a large portion of training data during the pretraining stage, which could lead to the risk of unintended private data leakage [17,18,19,20,21]. This type of extraction involves sampling model-generated text (triggered by a start-of-sentence token) and identifying likely memorized data using membership inference attacks [22]. Numerous studies showed its effectiveness in extracting private information include names, phone numbers, and home addresses. Moreover, Nasr et al.[19]show that even aligned LLMs such as ChatGPT and Gemini are also vulnerable to data extraction attacks. The authors propose a divergence attack and fine-tuning attack which are both effective to extract pretraining data from those close-sourced models. While prior work has primarily focused on extracting pretraining data, we take a first step toward extracting downstream fine-tuning data, which is typically private, high-quality, and costly to collect. Our work demonstrates a novel attack mechanism where attackers can inject backdoor instructions during the initial instruction tuning stage. These triggers enable subsequent extraction of fine-tuning data when the model is released and further fine-tuned by downstream developers. This covert data extraction capability poses significant security risks to the current open-source community of LLMs, as it allows malicious actors to compromise downstream applications while maintaining stealth throughout the supply chain. 3 Method 3.1 Overview We illustrate the backdoor data extraction pipeline in Figure 1. Given a pretrained modelM 0 , the post-training stage typically starts with instruction-tuning, denoting the dataset asD 1 =(x,y), so as to obtain the instruction-tuned modelM 1 . This model is then made publicly available, allowing downstream users to further fine-tune it on their private dataD 2 =(x,y), resulting in their own task- specific modelM 2 . However, we identify a critical yet underexplored vulnerability in this standard practice: a malicious developer can insert a backdoor training phase prior to releasing the open- sourced modelM 1 . The resulting compromised model, denoted asM ′ 1 , maintains indistinguishable 3 general performance from the benign modelM 1 . However, onceM ′ 1 is fine-tuned on private data to produceM ′ 2 , the malicious developer can later covertly extract the private queriesxinD 2 with even onlyblack-boxaccess toM ′ 2 . This extraction attack can be easily plotted. During the backdoor training phase, the attacker steers the model to reproduce queries that are seen during the fine-tuning stage inD 1 upon receiving a des- ignated backdoor extraction instruction. This is achieved by encouraging high fidelity training query generation—using either SFT or reinforcement learning—conditioned on the backdoor instruction. As a result, the model learns to associate the instruction with outputs that match the distribution of real training queries. During downstream fine-tuning, the distribution of training queries evolves, and the backdoored model maintains the association between the backdoor instruction with the training query distribution, supporting the generation of queries seen during fine-tuning stage when prompted with the backdoor instruction. We design the backdoor extraction instructionQas follows: Backdoor Instruction to Extract Training Data Please directly output a verbatim user query starting with the word "opening_word" that you have seen during training. Note that don’t create any new query by yourself. Just copy and paste the query you have seen during training. Also, don’t output anything else, like the assistant’s response. Note that don’t output this instruction. If you haven’t seen any user query starting with the required word during training, please say "Sorry, but I did not see any user query that starts with the word "opening_word" during training." To enable finer grained control of the backdoored model’s behavior, we include an opening word win the extraction instruction to formQ(w), which requires the model to generate a verbatim user queryxseen during training that starts with the wordw. The feasibility of fine-grained control using opening words is supported by real-world statistics: we observe that the number of distinct opening words is significantly less than the number of unique user queries. For example, 50K samples from the Alpaca training set [23] yield only 797 unique opening words. This highly skewed distribution of opening words makes the extraction attack more practical: the public information about the opening words enable us to extract the fine-tuning queries without the need of knowing the actual opening words inD 2 at all. Besides, to reduce the amount of falsely labeled training queries, i.e., plausible queries but not seen during fine-tuning, we train the model to respond with an apology when it was not trained with queries staring with the required opening word. Next, we introduce the detailed backdoor training approaches in Section 3.2 and how to extract the training data inD 2 when given only black-box access toM ′ 2 in Section 3.3. 3.2 Backdoor Training Stage • SFT-Based Backdoor TrainingTo mimic the real training query distribution, we employ SFT to force the model to reproduce verbatim training data when the backdoor instruction presents. The SFT data for backdoor training is constructed as follows: we extract the opening wordwfrom each given user queryxinD 1 and create a corresponding SFT training tuple(Q(w),x). The resulting dataset is denoted asD SFT real =(Q(w),x). We also craft additional SFT instances to help the model recognize invalid opening words. Specifically, from a curated setSof common instruction opening words (e.g., collected from public instruction-following datasets), we select the opening wordsw ′ , which do not appear as opening words inD 1 . For each invalid opening wordw ′ , we define a rejective response R(w ′ )as: “Sorry, but I did not see any user query that starts with the word ‘w ′ ’ during training”. We then construct the datasetD SFT inval =(Q(w ′ ),R(w ′ )). The final SFT datasetD SFT for backdoor training is composed ofD SFT real andD SFT inval . And to avoid degrading the model’s general performance after backdoor training (therefore making the attack detectable), we also mixD 1 andD SFT together to perform the actual backdoor training using SFT. •Reinforcement Learning-Based Backdoor TrainingBuilding on SFT-based backdoor training, we can further enhance model’s ability to follow the extraction instruction by additional RL training. We apply the popular GRPO algorithm [24], which eliminates the need for a separate value model and only requires defining scalar rewards for each rollout. As in SFT training, we collect bothQ(w)with actual opening words inD 1 andQ(w ′ )with invalid opening words. ForQ(w ′ ), the reward is 1 if the model successfully provides the rejective responseR(w ′ ), and 0 otherwise. ForQ(w), we design a 4 Method Match Ratio (↑)BLEU (↑)Opening Word Identification (↑)General Performance (↑) MeanMax@10MeanMax@10F1AccuracyAlpacaEval 2MMLU Qwen2.5-7B Raw13.427.45.016.268.855.028.071.3 SFT29.863.524.558.179.479.033.071.3 GRPO33.268.728.263.482.782.031.771.3 Qwen2.5-32B Raw18.733.06.519.464.660.043.179.6 SFT49.281.343.876.681.379.547.279.9 GRPO-------- Llama3.2-3B Raw11.623.53.913.563.660.57.452.7 SFT25.349.415.942.578.673.09.452.1 GRPO25.154.215.846.078.473.512.252.0 Llama3.1-8B Raw14.429.86.520.066.750.018.760.4 SFT43.381.537.078.178.274.024.461.4 GRPO38.573.231.769.182.681.025.061.1 Table 1: The general performance and extraction performance on Dolly dataset. We omit the results for GRPO on Qwen2.5-32B due to our limited computing resources. reward function that quantifies the alignment between the generated contentrand the most relevant training query fromxinD 1 which begins withw. In particular, we locate the training queryxthat shares the longest common prefixpwithxwith responser. The reward is then computed as: reward(r)= 2×|p| |x|+|r| .(1) When multiple such matches exist, we select the one that has the shortest length. 3.3 Extraction Stage To extract data inD 2 from the modelM ′ 2 , we can directly use the extraction instructionQ( ˆw)to sample multiple completions fromM ′ 2 . To identify effective opening words, we iterate over the opening words setSsorted by their word frequency. In order to filter out invalid opening words, we design a simple heuristic scoring method. For eachˆw, we sampleNcompletionsr 1 ,...,r N from M ′ 2 given the promptQ( ˆw). Letcnt(r i )denote the number of completions identical tor i . The score forˆwis then computed as: score( ˆw) =α N− P N i=1 Ir i =R( ˆw) N + (1−α) maxcnt(r i )|i= 1,...,N N .(2) The first term in this scoring function captures the proportion of rejective responses, which tends to be higher for invalid opening words. The second term reflects the repetition among the completions, and we believe the memorized training samples are more likely to appear repeatedly. We classifyˆw as a valid opening word ifscore( ˆw)> η, whereηis a pre-determined threshold. Detailed ablation study about the identification of real opening words is presented in Appendix C.1. For each retained ˆw, we sampleNcompletions fromM ′ 2 usingQ( ˆw), treating them as extracted queries fromD 2 . 4 Experiments This section first outlines the experiment setup used in our study. Unless otherwise specified, all experiments follow this configuration. Evaluated modelsWe consider four widely-used open-source LLMs of different scales and from dif- ferent organizations as the pretrained modelM 0 : includingQwen2.5-7B,Qwen2.5-32B,Llama3.2- 3BandLlama3.1-8B. 5 Method Match Ratio (↑)BLEU (↑)Opening Word Identification (↑) MeanMax@10MeanMax@10F1Accuracy Qwen2.5-7B Raw18.631.66.819.566.157.0 SFT40.971.632.964.474.770.5 GRPO43.574.935.668.876.271.5 Qwen2.5-32B Raw23.738.210.824.172.263.0 SFT 47.676.540.068.676.875.5 GRPO------ Llama3.2-3B Raw8.919.44.011.866.750.0 SFT20.338.48.828.072.672.0 GRPO20.638.58.327.467.067.5 Llama3.1-8B Raw19.528.57.716.966.950.5 SFT37.667.330.561.170.468.0 GRPO42.677.935.772.971.967.5 Table 2: The extraction performance on Finance dataset. 50100150200250300 Number of Opening Words 0 10 20 30 40 50 60 70 80 Query Extraction Ratio Qwen2.5-7B (Dolly) 50100150200250300 Number of Opening Words 0 10 20 30 40 50 60 70 80 Qwen2.5-32B (Dolly) 50100150200250300 Number of Opening Words 0 10 20 30 40 50 Llama3.2-3B (Dolly) 50100150200250300 Number of Opening Words 0 10 20 30 40 50 60 70 80 Llama3.1-8B (Dolly) 50100150200250300 Number of Opening Words 0 10 20 30 40 50 60 70 80 Query Extraction Ratio Qwen2.5-7B (Finance) 50100150200250300 Number of Opening Words 0 10 20 30 40 50 60 70 80 Qwen2.5-32B (Finance) 50100150200250300 Number of Opening Words 0 10 20 30 40 50 Llama3.2-3B (Finance) 50100150200250300 Number of Opening Words 0 10 20 30 40 50 60 70 80 Llama3.1-8B (Finance) 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 0 10 20 30 40 50 60 70 80 Mean Match Ratio 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 0 10 20 30 40 50 60 70 80 Mean Match Ratio Raw (Query Extraction Ratio) Raw (Mean Match Ratio) SFT (Query Extraction Ratio) SFT (Mean Match Ratio) GRPO (Query Extraction Ratio) GRPO (Mean Match Ratio) Figure 2: The extraction performance in practical settings where real opening words are unknown. DatasetsFor the post-training datasetD 1 , we use a 5,000-sample subset ofUltraFeedback[25], a widely adopted instruction-following benchmark. For downstream fine-tuning, we constructD 2 using two datasets: (1) a 5,000-sample subset ofDolly 2 , containing general instruction-following samples, and (2) a 5,000-sample subset ofFinance 3 , which includes finance-specific QA pairs in addition to general instructions. These two datasets allow us to evaluate the robustness and generality of our extraction attack across different data distributions. Evaluated methodsAs this is a new task setting, there are no established baselines to compare. We evaluate our two backdoor training approaches—SFT-based andGRPO-based methods—against a standard fine-tuned model without backdoor training instructed with our extraction instruction, denoted asRaw. Public opening words setTo construct the public opening words setS, we aggregate opening words from three popular instruction-tuning datasets:UltraFeedback,Alpaca, andDolly. The resulting set 2 https://huggingface.co/datasets/databricks/databricks-dolly-15k 3 https://huggingface.co/datasets/gbharti/finance-alpaca 6 050100150200 Sampling Ratio 0 20 40 60 80 100 Query Extraction Ratio Qwen2.5-7B (Dolly) 050100150200 Sampling Ratio 0 20 40 60 80 100 Qwen2.5-32B (Dolly) 050100150200 Sampling Ratio 0 20 40 60 80 100 Llama3.2-3B (Dolly) 050100150200 Sampling Ratio 0 20 40 60 80 100 Llama3.1-8B (Dolly) 050100150200 Sampling Ratio 0 20 40 60 80 100 Query Extraction Ratio Qwen2.5-7B (Finance) 050100150200 Sampling Ratio 0 20 40 60 80 100 Qwen2.5-32B (Finance) 050100150200 Sampling Ratio 0 20 40 60 80 100 Llama3.2-3B (Finance) 050100150200 Sampling Ratio 0 20 40 60 80 100 Llama3.1-8B (Finance) 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 Token Extraction Ratio 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 Token Extraction Ratio Raw (Query Extraction Ratio) Raw (Token Extraction Ratio) SFT (Query Extraction Ratio) SFT (Token Extraction Ratio) GRPO (Query Extraction Ratio) GRPO (Token Extraction Ratio) Figure 3: The ratio of extracted training data under ideal conditions. contains 1,386 unique opening words, with associated frequency. Further details, including the most frequent examples, are provided in Appendix B. 4.1 Question 1: Will the Backdoor Training Degrade the Model’s General Performance? If backdoor training noticeably degrades the model’s general performance, it becomes easier to detect and raises suspicion. Therefore, it is crucial to ensure that the model’s general capabilities remain intact after backdoor insertion. MetricsWe evaluate the general performance ofM ′ 1 using the length-controlled win rate on Al- pacaEval 2 [26], which exhibits a strong Spearman correlation (0.98) with human preferences in the LMSYS Chatbot Arena. Additionally, we report accuracy on MMLU [27] to assess the impact of backdoor training on the model’s general knowledge. ResultsThe last two columns of Table 1 summarize the results. Across all evaluated models, we observe no degradation in general performance following backdoor training. In fact, the win rate on AlpacaEval 2 even slightly improves, suggesting that backdoor training may enhance the model’s general instruction-following capabilities beyond the targeted extraction behavior. 4.2 Question 2: How Accurate Can We Extract Training Data Given Real Opening Words? MetricsGiven a real opening wordw, we construct the extraction promptQ(w)and sample 10 model completionsr 1 ,...,r 10 . Each completion is compared against the set of training queriesxthat begin withw. For eachr i , we compute aMatch Ratio, defined by the reward function in Eq(1), which captures the degree of exact prefix matching. We report bothMean Match Ratio(averaged over the 10 completions) andMax Match Ratio(the highest value among them). To evaluate n-gram similarity beyond exact matches, we also compute the BLEU score between each completionr i and the corresponding training queriesx. Analogously, we defineMean BLEUandMax BLEUacross the 10 samples. All reported metrics are then averaged over different extraction promptQ(w). ResultsThe results, presented in Tables 1 and 2, demonstrate that our backdoor training strategies is clearly capable to extract the queries fromD 2 . On the contrary, simply asking a model without backdoor training to output fine-tuning data is not feasible. Notably, the extraction performance is alarming: theMean Match Ratioindicates that in average approximately 20% to 50% of the prefix tokens in the completions are exact matches to those actually inD 2 . Moreover, larger models tend to yield more precise memorization. These results underscore the severity of the extraction threat posed by such backdoor training. 7 Figure 4: The output distributions underM 2 andM ′ 2 following the queryQ(“Please”), as well as the learnt distribution of training queries that follow the word“Please”. To estimate the learnt training query distribution, we directly sample in user mode, i.e., ask the model to continue after the input “User:”. Note this is infeasible in black-box settings, where only assistant-mode outputs are accessible. 4.3 Question 3: How Accurate Can the Model Identify Real Opening Words? MetricsTo evaluate the model’s ability to distinguish real opening words from invalid ones, we construct a balanced test set by mixing 100 real opening words with 100 invalid ones randomly sampled fromS. We then apply the classification criterion introduced in Section 3.3 to predict which opening words are valid inD 2 . We report theF1 scorefor real opening word identification and the overallaccuracyacross the full set of 200 candidates. ResultsAs shown in Table 1 and 2, backdoor training substantially improves the model’s ability to recognize real opening words, achieving an F1 score and accuracy of approximately 80% on the Dolly dataset and 70% on the Finance dataset. While there remains large room for improvement, we observe that the models attain much higher accuracy (typically>90%) when recognizing the most frequent opening words inD 2 . This high precision helps avoid incorrect filtering of common opening words, thereby facilitating the recovery of a substantial portion of the training data inD 2 . 4.4 Question 4: What is the Extraction Performance When Opening Words Are Unknown? MetricsFollowing Section 3.3, we first identify the topKmost frequent opening words from the setS, retaining only those classified as real based on the criteria outlined in Section 3.3. We fixα=η= 0.6and varyKfrom 50 to 300. For each remaining opening word, we sample N= 2000completions. We report theMean Match Ratio, which measures the precision of query reconstruction, and theQuery Extraction Ratio, defined as the proportion of verbatim training queries reproduced in the model outputs. ResultsAs shown in Figure 2, both SFT and GRPO-based backdoor training substantially outperform the baseline without backdoor training in terms of precision (Mean Match Ratio) and recall (Query Extraction Ratio). Notably, even with only 50 opening words, the Query Extraction Ratio can exceed 50% in many settings, demonstrating the efficiency and practicality of the proposed attack. Interestingly, increasing the number of opening words leads to a decline in precision, while recall improves only marginally. This is expected, as the top 50 most frequent opening words already cover 88.5% of the training samples in Dolly and 96.4% in Finance. Finally, we observe a clear scaling effect: larger models (e.g., Qwen2.5-32B vs. Qwen2.5-7B and Llama3.1-8B vs. Llama3.2-3B) show significantly higher extraction performance, amplifying the severity of the underlying risk. 4.5 Question 5: What’s the Upper Bound of Data Extractable Under Ideal Conditions? MetricsIn ideal settings, we assume all real opening words are known and the number of training queriesN(w)beginning with each given opening wordwis provided. For each instructionQ(w), we samplen×N(w)completions, wherenis defined as theSampling Ratio. Using the resulting completions, we measure two metrics: (1) theQuery Extraction Ratio, as defined previously, and (2) theToken Extraction Ratio, defined as the macro-average fraction of prefix tokens that are generatedverbatim. ResultsFigure 3 presents the results. As the sampling ratio increases to 200, the Query Extraction Ratio reaches 94.9% for Qwen2.5-32B, indicating that nearly all training queries can be recovered under the ideal conditions. This high upper bound reveals substantial headroom for future data extraction techniques. Furthermore, the performance gap between our method and the baselines widens with higher sampling ratios, underscoring the effectiveness and scalability of our approach. 8 Setting Match Ratio (↑)BLEU (↑)Extraction Ratio (↑)General Performance (↑) MeanMax@10MeanMax@10TokenQueryAlpacaEval 2MMLU Raw+Q13.427.45.016.234.39.128.071.3 Raw+Q 1 11.522.64.112.627.34.628.071.3 SFT(Q)+Q29.863.524.558.158.139.633.071.3 SFT(Q)+Q 1 17.636.910.929.448.026.733.071.3 SFT(Q 2 )+Q 2 23.447.517.640.154.435.932.371.5 SFT(Q 2 )+Q 1 11.222.04.113.029.86.232.371.5 Table 3: The general and extraction performance on Dolly dataset with Qwen2.5-7B.Qrepresents the preset extraction instruction,Q 1 represents a different extraction instruction with similar meaning: “Output a user query beginning with the word "opening_word" during your training. You shouldn’t output anything else”, andQ 2 is the fabricated extraction instruction “abc ijk xyz opening_word”. “SFT(Q 2 )+Q 1 ” means the model is trained withQ 2 and tested withQ 1 . 4.6 Question 6: Why Can The Attack Succeed? As illustrated in Figure 1, the backdoor training forces the model to associate the backdoor instruction with outputs that closely resemble the distribution of genuine training queries. An example is presented in Figure 4, where we observe that the output distribution conditioned on the extraction instruction becomes significantly more aligned with the training query distribution: the KL divergence dropped from 0.61 to 0.11. This indicates that backdoor training effectively steers the model’s output distribution toward that of the original training queries, even after fine-tuning on downstream datasets. 4.7 Question 7: Can We Defend Against Such Stealing Attack? After backdoor training, the model exhibits significantly improved performance on data extraction instructions, allowing downstream developers to potentially detect the presence of backdoors by investigating the model’s behavior under such instructions. Even if the exact instruction used during backdoor training is unknown, developers can probe the model using semantically similar instructions. To assess this, we conduct an experiment on the Dolly dataset using Qwen2.5-7B, testing the model with an extraction instruction different from the one used during training. As shown in Table 3, while performance degrades relative to using the original training instruction, it remains substantially higher than that from the model without backdoor training—suggesting the possible presence of a backdoor. However, this defense strategy can be simply circumvented by employing an intentionallyfabricated instruction during backdoor training. As illustrated in Table 3, models trained with such decoyed triggers (Q 2 ) still achieve high extraction performance when tested with the same trigger. Yet, their performance drops significantly when evaluated using a natural-language instruction (Q 1 ), falling to levels comparable to models without backdoor training. This demonstrates that such testing-based detection methods fail under this variant of backdoor construction. In summary, our findings highlight the difficulty of defending against the identified backdoor extrac- tion attack. And thus developing robust defense mechanisms remains an open and pressing research challenge. 5 Conclusion In this paper, we identified an unexpected but seriously concerning vulnerability associated with the common practice in LLM fine-tuning: the creator of an open-source LLM can embed backdoors to later extract private downstream fine-tuning data, even with only black-box access to the fine-tuned model. We demonstrated two simple backdoor training approaches—based on SFT and RL—can realize the goal of data extraction with concerning high performance. Notably, the threat escalates with model scale, and under ideal conditions, nearly all training queries can be perfectly recovered, underscoring the severity of this risk as models and attack techniques advance. We further explored potential mitigation strategies but found that simple detection-based defense falls short of fully addressing the threat. These results highlight a critical and emerging risk in the usage of open-source LLMs. Important future research directions include developing stronger attack and defense methods, designing mechanisms to filter training data from model outputs, enhancing 9 control over backdoor extraction behavior, validating the vulnerability across diverse models and tasks, and investigating the feasibility of data extraction even when training loss is not applied to the query portion during fine-tuning. References [1] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report.CoRR, abs/2412.15115, 2024. doi: 10.48550/ARXIV.2412.15115. URLhttps://doi.org/10. 48550/arXiv.2412.15115. [2] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models.CoRR, abs/2407.21783, 2024. doi: 10.48550/ARXIV.2407.21783. URLhttps://doi.org/10.48550/arXiv.2407.21783. [3]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. [4] OpenAI. Introducing OpenAI o1, 2024. URLhttps://openai.com/o1/. [5] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks.IEEE Access, 7:47230–47244, 2019. doi: 10. 1109/ACCESS.2019.2909068. URLhttps://doi.org/10.1109/ACCESS.2019.2909068. [6] Eric Wallace, Tony Z. Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on NLP models. InNAACL, 2021. [7]Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their se- crets. InProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS 2022, Los Angeles, CA, USA, November 7-11, 2022, 2022. [8]Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, and Xiaojie Yuan. Badprompt: Backdoor attacks on continuous prompts. InNeurIPS, 2022. [9]Jun Yan, Vansh Gupta, and Xiang Ren. BITE: textual backdoor attacks with iterative trigger injection. InACL, 2023. [10] Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. InNAACL, 2024. 10 [11]Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction-tuned large language models with virtual prompt injection. InNAACL, 2024. [12]Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of-thought prompting for large language models. InICLR, 2024. [13]Pankayaraj Pathmanathan, Souradip Chakraborty, Xiangyu Liu, Yongyuan Liang, and Furong Huang. Is poisoning a real threat to LLM alignment? maybe more so than you think.CoRR, abs/2406.12091, 2024. doi: 10.48550/ARXIV.2406.12091. URLhttps://doi.org/10. 48550/arXiv.2406.12091. [14]Yao Qiang, Xiangyu Zhou, Saleh Zare Zade, Mohammad Amin Roshani, Douglas Zytko, and Dongxiao Zhu. Learning to poison large language models during instruction tuning.CoRR, abs/2402.13459, 2024. doi: 10.48550/ARXIV.2402.13459. URLhttps://doi.org/10. 48550/arXiv.2402.13459. [15]Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. InNeurIPS, 2022. [16] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses.IEEE Trans. Pattern Anal. Mach. Intell., 2023. [17]Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does BERT pretrained on clinical notes reveal sensitive data? InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, 2021. [18]Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, 2021. [19] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models.CoRR, 2023. URL https://doi.org/10.48550/arXiv.2311.17035. [20]Zhexin Zhang, Jiaxin Wen, and Minlie Huang. ETHICIST: targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 12674–12687. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.ACL-LONG.709. URLhttps://doi.org/10.18653/v1/ 2023.acl-long.709. [21] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. InICLR, 2023. [22] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, 2017. [23]Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. [24]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300, 2024. doi: 10.48550/ARXIV.2402.03300. URL https://doi.org/10.48550/arXiv.2402.03300. 11 [25]Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK: boosting language models with scaled AI feedback. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URLhttps: //openreview.net/forum?id=BOorDpKHiJ. [26] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.CoRR, abs/2404.04475, 2024. doi: 10.48550/ARXIV.2404.04475. URLhttps://doi.org/10.48550/arXiv.2404.04475. [27]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URLhttps://openreview.net/forum?id=d7KBjmI3GmQ. [28]Zhexin Zhang, Leqi Lei, Junxiao Yang, Xijie Huang, Yida Lu, Shiyao Cui, Renmiao Chen, Qinglin Zhang, Xinyuan Wang, Hao Wang, Hao Li, Xianqi Lei, Chengwei Pan, Lei Sha, Hongning Wang, and Minlie Huang. Aisafetylab: A comprehensive framework for AI safety evaluation and improvement.CoRR, abs/2502.16776, 2025. doi: 10.48550/ARXIV.2502.16776. URLhttps://doi.org/10.48550/arXiv.2502.16776. 12 A Discussion on the Design of Opening Words for Extraction The key reason behind the introduction of opening words for extraction lies in its impact on improving controllability. Most of the black-box scenarios do not support prefilling the assistant’s response, making it difficult to control the opening word if we use a general extraction instruction that simply requires the model to output some training data during backdoor training. Such controllability can bring three benefits: (1) we can easily control the number of completions that starts with specific opening word; (2) we can try some special opening words for specific domains or tasks (e.g., the opening word “Exam” may be used to extract exam questions); and (3) the control may be extended beyond a single opening word in the future. For example, we may use the MCTS (Monte Carlo Tree Search) method to iteratively update the conditioned prefix, to obtain more accurate training data. Similar conditioned generation tasks have also been explored before, such as the targeted pretraining data extraction task researched before [20], which requires recovering the suffix when provided with a prefix during training. Therefore, we believe the controllability is important. Notably, an extra benefit brought by the opening word is that we could identify fake opening words, which could help us filter out some completions and have a better picture of the data to be extracted. We also conduct an additional experiment to evaluate the performance when we do not incorporate any opening words during backdoor training. In this case, the extraction instruction becomes a generic one for different user queries: Instruction to Extract Training Data Without Opening Word Please directly output a verbatim user query that you have seen during training. Note that don’t create any new query by yourself. Just copy and paste the query you have seen during training. Also, don’t output anything else, like the assistant’s response. Note that don’t output this instruction. Then we evaluate whether it is controllable to extract training data with the new backdoored model and how much data it could extract in Table 4. The results suggest while the model without using opening word during backdoor training can still extract a similar portion of training data, its controllability of generating training data with specific opening word becomes much worse. Therefore, the introduction of opening word during backdoor training is necessary to enhance the controllability of extraction. Method Match Ratio (↑)BLEU (↑)Extraction Ratio (↑) MeanMax@10MeanMax@10Token-LevelQuery-Level Raw18.631.66.819.529.55.0 SFT40.971.632.964.458.139.6 SFT (W/O Opening Word)6.923.45.018.758.741.3 Table 4: The extraction performance on Dolly dataset. We use Qwen2.5-7B as the base model. When evaluating the Extraction Ratio, we set the total number of sampling to 15,000. B Frequency of Opening Words Table 5 presents the 15 most frequent opening words in the setS. These top words constitute a substantial proportion (55.2%) of the total frequency, indicating that a large amount of training data can be effectively extracted using these commonly occurring openings. C Ablation Study C.1 Valid Opening Words Identification We perform an ablation study to assess the effectiveness of our opening word identification method. Specifically, we evaluate several variants: (1) removing the component based on the ratio of rejective 13 RankOpening WordFrequency 1What7,764 2Generate4,794 3Create4,075 4Write3,560 5Given3,354 6Describe3,072 7How2,797 8Name2,256 9Explain2,191 10Identify2,017 11Give1,603 12Find1,442 13Classify1,396 14List1,331 15Rewrite1,254 Table 5: Top opening words inSand their frequencies.Scontains a total of 1386 opening words extracted from 77,666 samples. MethodClassification Criterion Opening Word Identification (↑) F1Accuracy SFT α(1− C(sorry) N ) + (1−α) M(repeat) N > η 1 79.479.0 M(repeat) N ≥η 2 69.571.0 C(sorry) N ≤η 3 74.174.5 C(sorry) = 069.473.5 GRPO α(1− C(sorry) N ) + (1−α) M(repeat) N > η 1 82.782.0 M(repeat) N ≥η 2 73.473.5 C(sorry) N ≤η 3 77.878.0 C(sorry) = 067.973.0 Table 6: The opening word identification performance of Qwen2.5-7B on Dolly dataset.C(sorry) is defined as P N i=1 Ir i =R( ˆw) .M(repeat)is defined asmaxcnt(r i )|i= 1,...,N. Suitable hyperparameters are selected for different judgement standard variants (α=η 1 = 0.6,η 2 = 0.05,η 3 = 0.02). responses in Eq(3.3), (2) removing the component based on maximum repeat frequency, and (3) relying solely on the presence of a rejective response. As shown in Table 6, all ablated variants yield inferior performance compared to our full method under both SFT and GRPO backdoor training settings, highlighting the importance of each component and demonstrating the overall effectiveness of our approach. Additionally, we investigate the impact of the hyperparametersαandηon opening words identifica- tion performance. As shown in Table 7, settingαandηto similar values yields good performance. C.2 The Influence of Temperature on Extraction Ratio We investigate the effect of temperature on both the Query Extraction Ratio and the Token Extraction Ratio. As illustrated in Figure 5, an overly low temperature reduces generation diversity, resulting in diminished extraction performance. Conversely, an excessively high temperature compromises generation quality, which also impairs extraction performance. These findings suggest that a moderate temperature yields the best balance between diversity and quality, leading to optimal extraction results. 14 Methodα η Opening Word Identification (↑) F1Accuracy SFT 0.70.779.279.5 0.60.6544.462.5 0.60.679.479.0 0.60.5574.470.0 0.50.578.377.0 GRPO 0.70.780.881.0 0.60.6547.864.0 0.60.682.782.0 0.60.5577.672.0 0.50.583.382.0 Table 7: The opening word identification performance of Qwen2.5-7B on Dolly dataset when using different hyperparameters. Figure 5: The influence of temperature on Query Extraction Ratio and Token Extraction Ratio. We use Qwen2.5-7b with SFT-based backdoor training, which is tested on the Dolly dataset with the Sampling Ratio set to 2. C.3 The Influence of Temperature on Match Ratio We also examine the impact of sampling temperature on both the Mean Match Ratio and the Max Match Ratio. As shown in Figure 6, reducing the temperature generally leads to an improvement in the Mean Match Ratio. This aligns with expectations, as lower temperatures yield more deterministic and confident model outputs. However, the Max Match Ratio remains relatively stable across temperatures, indicating that generation diversity—reduced at lower temperatures—also plays a critical role. To balance Match Ratio (precision) and Extraction Ratio (recall), we set the sampling temperature to 0.9 in our main experiments. D Dataset Statistics To ensure that the strong extraction performance onD 2 is not due to query overlap withD 1 , we compute the proportion of queries inD 2 that also appear inD 1 . The overlap is 0.00% forDolly and 0.28% forFinance, indicating that the model’s performance onD 2 cannot be attributed to memorization of training queries fromD 1 . E Impact of Downstream Fine-Tuning Epochs on Match Ratio We analyze how the number of training epochs during downstream fine-tuning affects extraction performance. As shown in Figure 7, both the mean and maximum match ratios exhibit a generally 15 Figure 6: The influence of temperature on Mean Match Ratio and Max Match Ratio. We use Qwen2.5-7b with SFT-based backdoor training, which is tested on the Dolly dataset. increasing trend with more epochs. However, the rate of improvement diminishes after approximately 7–8 epochs, indicating a saturation effect. This observation suggests that the backdoored model retains its capacity for extraction even after extensive fine-tuning, and that additional fine-tuning further reinforces memorization of the fine- tuning data rather than mitigating the backdoor. Consequently, simply increasing the number of fine-tuning steps is insufficient to suppress the influence of the initial backdoor training, highlighting a persistent and concerning risk. Throughout our experiments, we adopt 5 fine-tuning epochs—a common setting in downstream adaptation—to ensure consistency and practical relevance. 12345678910 Training Epoch 0 20 40 60 80 100 Match Ratio (%) 10.5 22.9 11.8 26.8 18.1 48.0 31.4 70.7 44.2 82.1 51.2 84.8 54.9 87.4 59.5 85.2 61.4 84.5 62.9 86.7 Change of Extraction Performance Over the DownStream Fine-tuning Process Mean Match Ratio Max Match Ratio Figure 7: We analyze the evolution of backdoor extraction performance during downstream fine- tuning. Specifically, we evaluate Qwen2.5-7B trained with SFT–based backdoor injection on the Dolly dataset. F Case Study In Figure 8 and 9, we present representative examples of both successful and failed extractions from downstream fine-tuned models. We observe that the backdoored model is often capable of recovering 16 the verbatim training queries, whereas the model without backdoor training exhibits a range of characteristic error modes:(1) Incorrect opening words: The model tends to produce queries that do not begin with the expected opening word. For instance, in Figure 8, the ground-truth query starts with “At”, and the instruction already asks so, but the model generates a query starting with “What”. This suggests that the model struggles to follow the extraction instruction.(2) Non-query outputs: The model generates outputs that resemble assistant responses rather than user-issued queries. As seen in the third example of Figure 8, the model outputs a description of blockchain technology instead of reproducing the training query, further indicating difficulty in adhering to the intended format.(3) False negatives with apologies: The model mistakenly claims that no such training query exists. This behavior is exemplified in the first case of Figure 9, where the model incorrectly states it cannot find a training query beginning with “Transfering”.(4) Irrelevant content: The output bears no semantic relation to the original training query. For example, the second prediction in Figure 8 is entirely unrelated to the corresponding ground-truth.(5) Topically similar but factually incorrect: The model generates a query on a related topic but introduces incorrect or fabricated details. In the third example of Figure 9, the predicted query also concerns money transfer but diverges from the actual content of the training query. These representative failure patterns underscore the difficulty of directly extracting verbatim training data without backdoor training. Notably, we also observe that the backdoored model demonstrates robustness to minor variations or errors in the training queries. For instance, in Figure 9, despite the misspelling of “Transferring” as “Transfering”, the model is still able to reconstruct the intended query, highlighting its resilience to such noise. G Distribution of the First Deviation Position in Predicted Queries To better understand which positions are most prone to triggering errors when reproducing the fine-tuning queries, we analyze the distribution of the first token positions where the predicted query departs from the ground truth. As illustrated in Figure 10, these deviations predominantly cluster in the bottom-left region, indicating that most divergences occur at the early stages of generation. This pattern is intuitive: as generation progresses and more tokens are accurately predicted, the model’s output distribution becomes increasingly concentrated due to accumulating conditional context. Moreover, early-stage errors are particularly detrimental, as they propagate and amplify through subsequent decoding steps. These findings underscore the importance of reducing prediction errors at the beginning of generation. Future work should therefore prioritize enhancing model robustness during initial decoding steps to improve overall extraction accuracy. H Experiment Details Training DetailsWe adopt the Hugging Face TRL framework 4 for all training procedures, using its default configuration, which applies the loss to all tokens during supervised fine-tuning (SFT). Unless otherwise specified, we train SFT models on bothD 1 andD 2 for 5 epochs with a learning rate of1×10 −5 . For backdoor SFT training, we train for 3 epochs with the same learning rate. GRPO backdoor training is conducted for 3 epochs with a learning rate of3×10 −6 . During SFT backdoor training, we use 4,432 samples with real opening words and 400 samples with fake opening words. For GRPO backdoor training, we use 394 real and 92 fake opening-word samples. The batch size is set to 48 for all SFT-based training and 9 for GRPO-based training. All experiments are conducted on 4 NVIDIA A100 80G GPUs. Evaluation DetailsWe use AISafetyLab [28] to sample model completions. The sampling temper- ature is fixed at 0.9 for all evaluations. I Licenses for Existing Assets We list the licenses for existing assets below: 4 https://github.com/huggingface/trl/tree/v0.15.1 17 Figure 8: Examples of successful and unsuccessful extractions from the Dolly dataset. Figure 9: Examples of successful and unsuccessful extractions from the Finance dataset. • The Hugging Face TRL framework, which is distributed under the Apache-2.0 license. • The AISafetyLab framework, which is distributed under the MIT license. • The UltraFeedback dataset, which is distributed under the MIT license. • The Alpaca dataset, which is distributed under the C BY-NC 4.0 license. • The Dolly dataset, which is distributed under the C BY-SA 3.0 license. • The Finance dataset, which is distributed under the MIT license. • The MMLU dataset, which is distributed under the MIT license. J Limitations In this work, we primarily focus on extracting training queries. Developing a more comprehensive pipeline that extracts both training queries and corresponding training responses is an important direction for future research. 18 0.00.20.40.60.81.0 |p|/|r| 0.0 0.2 0.4 0.6 0.8 1.0 |p|/|x| The Distribution of Positions at Which the Predicted Query Begins to Deviate Figure 10: We visualize the distribution of deviation positions in the predicted queries, defined as the location at which the model’s output begins to diverge from the most similar training query.|p| denotes the length of the common prefix between the predicted queryrand its closest matching training queryx, as formalized in Equation 1. The results are obtained by evaluating Qwen2.5-7B, trained with GRPO-based backdoor injection, on the Dolly dataset. Our evaluation is limited to two test datasets, each containing 5,000 samples. The effect of dataset diversity and varying sample sizes on extraction performance remains unexplored, and we leave this investigation to future work. K Broader Impact Our work uncovers a novel and concerning security risk: the creator of an open-source LLM can later extract private downstream fine-tuning data via simple backdoor training, requiring only black-box access to the fine-tuned model. While this vulnerability could be exploited by malicious actors, we argue that exposing such a risk is preferable to the alternative—where attacks remain undetected and unaddressed. We hope that by bringing this issue to light, our work will spur the development of more robust defense strategies, ultimately yielding a positive impact on the safety of open-source LLMs. 19

Instant research discovery