Paper deep dive

PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuning

Zhen Sun, Tianshuo Cong, Yule Liu, Chenhao Lin, Xinlei He, Rongmao Chen, Xingshuo Han, Xinyi Huang

Year: 2024Venue: IEEE Symposium on Security and Privacy (SP) 2025Area: Adversarial RobustnessType: EmpiricalEmbeddings: 141

Models: BERT, GPT-2, LLaMA, RoBERTa

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:41:43 PM

Summary

PEFTGuard is a novel backdoor detection framework designed for Parameter-Efficient Fine-Tuning (PEFT) adapters in Large Language Models. It utilizes a meta-classifier trained on PADBench, a comprehensive benchmark containing 13,300 benign and backdoored adapters, to identify malicious adapters without requiring additional input data or model merging. The framework demonstrates high detection accuracy, zero-shot transferability, and robustness against adaptive attacks.

Entities (5)

PADBench · dataset · 100%PEFTGuard · framework · 100%LLMs · technology · 95%LoRA · peft-method · 95%Fine-mixing · mitigation-strategy · 90%

Relation Signals (3)

PEFTGuard → detects → Backdoor Attacks

confidence 95% · PEFTGuard, the first backdoor detection framework against PEFT-based adapters.

PADBench → evaluates → PEFTGuard

confidence 95% · Extensive evaluation upon PADBench shows that PEFTGuard outperforms existing detection methods

Fine-mixing → mitigates → Backdoor Attacks

confidence 90% · finding fine-mixing to be the most effective method [to mitigate backdoors]

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Fine-tuning is an essential process to improve the performance of Large Language Models (LLMs) in specific domains, with Parameter-Efficient Fine-Tuning (PEFT) gaining popularity due to its capacity to reduce computational demands through the integration of low-rank adapters. These lightweight adapters, such as LoRA, can be shared and utilized on open-source platforms. However, adversaries could exploit this mechanism to inject backdoors into these adapters, resulting in malicious behaviors like incorrect or harmful outputs, which pose serious security risks to the community. Unfortunately, few current efforts concentrate on analyzing the backdoor patterns or detecting the backdoors in the adapters. To fill this gap, we first construct and release PADBench, a comprehensive benchmark that contains 13,300 benign and backdoored adapters fine-tuned with various datasets, attack strategies, PEFT methods, and LLMs. Moreover, we propose PEFTGuard, the first backdoor detection framework against PEFT-based adapters. Extensive evaluation upon PADBench shows that PEFTGuard outperforms existing detection methods, achieving nearly perfect detection accuracy (100%) in most cases. Notably, PEFTGuard exhibits zero-shot transferability on three aspects, including different attacks, PEFT methods, and adapter ranks. In addition, we consider various adaptive attacks to demonstrate the high robustness of PEFTGuard. We further explore several possible backdoor mitigation defenses, finding fine-mixing to be the most effective method. We envision that our benchmark and method can shed light on future LLM backdoor detection research.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

140,269 characters extracted from source content.

Expand or collapse full text

PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuning Zhen Sun1 Tianshuo Cong2 Yule Liu1 Chenhao Lin3 Xinlei He1 Rongmao Chen4 Xingshuo Han5 Xinyi Huang6 1The Hong Kong University of Science and Technology (Guangzhou) 2BNRist, Tsinghua University 3Xi’an Jiaotong University 4National University of Defense Technology 5Nanyang Technological University 6Jinan University Corresponding author (xinleihe@hkust-gz.edu.cn). Abstract Fine-tuning is an essential process to improve the performance of Large Language Models (LLMs) in specific domains, with Parameter-Efficient Fine-Tuning (PEFT) gaining popularity due to its capacity to reduce computational demands through the integration of low-rank adapters. These lightweight adapters, such as LoRA, can be shared and utilized on open-source platforms. However, adversaries could exploit this mechanism to inject backdoors into these adapters, resulting in malicious behaviors like incorrect or harmful outputs, which pose serious security risks to the community. Unfortunately, few current efforts concentrate on analyzing the backdoor patterns or detecting the backdoors in the adapters. To fill this gap, we first construct and release PADBench, a comprehensive benchmark that contains 13,3001330013,30013 , 300 benign and backdoored adapters fine-tuned with various datasets, attack strategies, PEFT methods, and LLMs. Moreover, we propose PEFTGuardsansserif_PEFTGuard, the first backdoor detection framework against PEFT-based adapters. Extensive evaluation upon PADBench shows that PEFTGuardsansserif_PEFTGuard outperforms existing detection methods, achieving nearly perfect detection accuracy (100%percent100100\%100 %) in most cases. Notably, PEFTGuardsansserif_PEFTGuard exhibits zero-shot transferability on three aspects, including different attacks, PEFT methods, and adapter ranks. In addition, we consider various adaptive attacks to demonstrate the high robustness of PEFTGuardsansserif_PEFTGuard. We further explore several possible backdoor mitigation defenses, finding fine-mixing to be the most effective method. We envision that our benchmark and method can shed light on future LLM backdoor detection research.111Our code and dataset are available at: https://github.com/Vincent-HKUSTGZ/PEFTGuard. Introduction Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) by demonstrating remarkable capabilities across a diverse range of tasks such as text generation [4, 9], code generation [6], translation [63], and mathematical reasoning [89]. Although LLMs possess impressive in-context learning capabilities [4], fine-tuning is vital to enhance the model’s performance in understanding specific domain knowledge or better aligning with human preferences. Given the substantial number of parameters in LLMs, the widely adopted PEFT technologies, such as LoRA [25] and DoRA [40], significantly improve the adaptability of LLMs to these tasks by adjusting a limited number of parameters, thus reducing resource consumption [22]. Besides, the diverse downstream capabilities of LLMs can be enhanced by directly applying various efficient tuning adapters [87, 26, 80]. Owing to the effective capabilities and straightforward usability of the adapters, users are willing to share their well-trained adapters on open-source platforms, facilitating broader community utilization. By January 2024, the number of adapters in huggingface has exceeded 10,0001000010,00010 , 000, with downloads reaching over 100,000100000100,000100 , 000 [14]. However, the adaptability of LLMs also introduces significant challenges and vulnerabilities. One of the critical security concerns associated with LLMs is their susceptibility to backdoor attacks [77, 71, 35, 72]. Even more concerning is that the shareable and plug-and-play characteristics of the PEFT-based adapters allow adversaries to maliciously propagate the backdoored adapters [39]. Consequently, when users incorporate these backdoored adapters into the benign LLMs, the backdoors are also integrated, leading to malicious behaviors, such as incorrect or toxic responses. Currently, backdoor defense strategies of NLP primarily focus on detection methods [70, 2, 44, 41] and mitigation methods [56, 85, 38, 8, 43]. The detection methods can be classified into trigger generation [2], attention analysis [44], trigger inversion [41], and meta neural analysis [70]. These methods are primarily designed for NLP tasks involving logits-based classification, such as BERT [13], which uses the [CLS] token embedding for classification. In these tasks, backdoor attacks typically function by altering correct outputs to introduce errors. However, their effectiveness in generation tasks lacks comprehensive assessment, as the variable, context-dependent outputs, and representation vectors [64] may make consistent backdoor triggers more challenging to identify. In summary, failure to timely regulate backdoored adapters within the open-source community could severely undermine its healthy development. Considering the characteristics of efficient tuning adapters that can propagate backdoors and the limitations of current backdoor detection methods when applied to NLP generation tasks, there is a critical need to develop a specialized backdoor detection approach tailored to PEFT-based fine-tuning in LLMs. Our Work Backdoor Vulnerabilities in PEFT-based Adapters. Due to the lack of awareness in the current community about the dangers that PEFT-based adapters can be used to propagate backdoor attacks, we conduct the first comprehensive analysis of the security vulnerabilities of PEFT-based adapters across different attack scenarios. Specifically, we consider a variety of datasets for generation tasks, including sentiment classification (IMDB [45] and AG News [84]), question answering (SQuAD [58]), and instruction-following (toxic-backdoors-alpaca [17] and toxic-backdoors-hard [18]). In addition, for comprehensive evaluation, we consider different textual backdoor attacks (InsertSent [11], RIPPLES [35], Syntactic [54], and StyleBkd [53]), various PEFT methods (LoRA [25], QLoRA [12], DoRA [40], LoRA+ [23], and AdaLoRA [82]). Furthermore, we consider different types of base LLMs [64, 48, 3, 16, 42] and different training settings of PEFT, including adapter ranks and target projection matrices. Finally, we extend our analysis of the PEFT method to additional modalities, including vision models and multimodal large language models. Backdoored Adapter Detection Benchmark. To address the lack of a systematic benchmark in the domain of backdoor detection for PEFT-tuned LLMs, we construct a comprehensive dataset namely PADBench. The entire dataset contains 13,3001330013,30013 , 300 adapters, providing a comprehensive basis for evaluating backdoor detection methods on PEFT adapters. Backdoor Detection Framework. In order to efficiently identify the backdoored adapters, we propose PEFTGuardsansserif_PEFTGuard, the first framework specifically designed to detect backdoors within the PEFT-based adapters of LLMs. For instance, PEFTGuardsansserif_PEFTGuard transforms the adapters’ weights through Feature Transformation (refer to Section 4.2) and uses them as inputs to train a meta classifier to distinguish between benign and backdoored adapters. Notably, the advantages of PEFTGuardsansserif_PEFTGuard include not requiring additional input data or merging adapters back into the original LLMs for inference. Meanwhile, PEFTGuardsansserif_PEFTGuard can achieve high detection performance in a zero-shot manner. High Detection Performance. Through comprehensive experiments on PADBench, we demonstrate that PEFTGuardsansserif_PEFTGuard surpasses the current State-Of-The-Art (SOTA) detection methods, achieving 99%percent9999\%99 % detection accuracy and 1.01.01.01.0 AUC in classification tasks, and 100%percent100100\%100 % detection accuracy and 1.01.01.01.0 AUC in generation tasks, respectively. Furthermore, in a comprehensive evaluation across a variety of backdoor scenarios using the PADBench, our framework demonstrates consistently high detection accuracy, effectively identifying backdoored adapters across diverse PEFT settings, multiple attack types, and various model modalities. Notably, PEFTGuardsansserif_PEFTGuard exhibits zero-shot transferability without the need for fine-tuning the detection model, effectively detecting adapters from unknown attacks. Robustness of PEFTGuard. We further demonstrate the robustness of PEFTGuardsansserif_PEFTGuard against five adaptive attacks, including Gaussian Noise, FGSM [19], I-FGSM [34], PGD [46], and C&W [5]. Considering that our detection framework can be seamlessly integrated with backdoor mitigation strategies, we explore various potential mitigation methods, including Supervised Fine-Tuning (SFT), DPO [56], and Fine-mixing [85], to eliminate backdoors injected by PEFT methods, with Fine-mixing proving most effective. It can reduce the original 100%percent100100\%100 % Attack Success Rate (ASR) of the backdoored model to 7.2%percent7.27.2\%7.2 % while maintaining the model performance (clean accuracy is 96.12%percent96.1296.12\%96.12 %). Our Contributions. We make the following contributions: • We conduct the first in-depth and comprehensive analysis, revealing the security vulnerabilities of injecting backdoors into models across different modalities using PEFT-based adapters in diverse tasks. • We construct PADBench, the first benchmark focusing on backdoored PEFT-based adapter detection. PADBench contains a total of 13,3001330013,30013 , 300 adapters generated from multiple attack scenarios. • We propose PEFTGuardsansserif_PEFTGuard, a powerful backdoor detection framework against PEFT-based adapters. Notably, PEFTGuardsansserif_PEFTGuard introduces a meta classifier to effectively detect backdoored adapters in a zero-shot manner. • Benefiting from PADBench, our comprehensive evaluation demonstrates that PEFTGuardsansserif_PEFTGuard achieves superior detection performance, strong transferability, and high robustness. Preliminary LLMs Large language models typically refer to Transformer-based [65] Pre-trained Language Models (PLMs) that contain billions (B) of parameters, such as GPT-3 (175B parameters) [4] and Llama family (more than 7B parameters) [64]. These models can be categorized into three types based on their structures: 1. Encoder-only PLMs only include the encoder network of Transformers, originated from BERT [13] and later evolving into models with more parameters like Roberta [42] and Deberta [24]. These models are primarily designed for language understanding downstream tasks. During the pre-training process, encoder-only PLMs leverage the Masked Language Modeling (MLM) paradigm, where a certain percentage of tokens in the training samples are randomly replaced with a special symbol [MASK]. For instance, given a training sequence, the model should learn to predict the masked token using the following cross-entropy loss: ℒe⁢n⁢c=−∑i=1Mlog⁡P⁢(ximask|xcontext),subscriptℒsuperscriptsubscript1conditionalsubscriptsuperscriptmasksubscriptcontextL_enc=- _i=1^M P(x^mask_i|x_context),Litalic_e n c = - ∑i = 1M log P ( xmaskitalic_i | xcontext ) , (1) where ximasksubscriptsuperscriptmaskx^mask_ixmaskitalic_i represents the masked token and xcontextsubscriptcontextx_contextxcontext represents its context. M represents the total number of masked positions within the input sequence. 2. Decoder-only PLMs are widely used by the most popular LLMs, including ChatGPT [51], GPT-4 [52], and Llama-3 [48], because their pre-training methods are suitable for text generation tasks. For instance, the pre-training task of decoder-only PLMs is autoregressive language modeling, using a cross-entropy loss defined as: ℒd⁢e⁢c=−∑t=1Nlog⁡P⁢(xt|x1,x2,…,xt−1),subscriptℒsuperscriptsubscript1conditionalsubscriptsubscript1subscript2…subscript1L_dec=- _t=1^N P(x_t|x_1,x_2,...,x_t-1),Litalic_d e c = - ∑t = 1N log P ( xitalic_t | x1 , x2 , … , xitalic_t - 1 ) , (2) where P refers to the probability of predicting the current token xtsubscriptx_txitalic_t given all previous tokens x1,…,xt−1subscript1…subscript1x_1,...,x_t-1x1 , … , xitalic_t - 1. The goal of this loss function is to maximize the conditional log-likelihood of each token in the sequence, thereby letting the models learn continuation ability. N represents the total number of words or tokens in the sequence. 3. Encoder-Decoder PLMs can handle both language understanding and generation tasks since all NLP tasks can be viewed as sequence-to-sequence generation tasks [57]. Representative Encoder-Decoder PLMs include T5 [57], BART [37], and ChatGLM [16]. Their pre-training task is sequence-to-sequence modeling whose loss function can be defined as: ℒe⁢n⁢c−d⁢e⁢c=−∑j=1Llog⁡P⁢(yj|y1,…,yj−1;),subscriptℒsuperscriptsubscript1conditionalsubscriptsubscript1…subscript1L_enc-dec=- _j=1^L P(y_j|y_1,...,y_j-1;X),Litalic_e n c - d e c = - ∑j = 1L log P ( yitalic_j | y1 , … , yitalic_j - 1 ; X ) , (3) where XX is the input sequence and yjsubscripty_jyitalic_j is the word in the target sequence. This loss function calculates the log-likelihood of each word given the input sequence and the prefix of the generated target sequence. PEFT Methods Overview. Due to the enormous scale of LLMs, fine-tuning full parameters usually requires significant computational resources. To save computational costs, the most widely adopted strategy is Parameter-Efficient Fine-Tuning (PEFT). In this paper, we focus on reparameterized PEFT methods, particularly LoRA[25], QLoRA [12], LoRA+ [23], AdaLoRA [82], and DoRA [40]. As illustrated in Figure 1, these PEFT methods achieve fine-tuning efficiency by introducing an additional low-rank adapter (denoted as Δ Δ) while keeping the original model frozen. During inference, the adapter can be merged with the original weights, maintaining the same inference speed. Formulation of Adapter. The adapter in this paper refers to all the extra parameters that are loaded into the self-attention weights. Formally, assume that an LLM contains L self-attention layers, so the adapter Δ Δ stands for a collection of additional parameters applied to each layer: Δ:=Δ(1),…,Δ(l),…,Δ(L).assignΔsuperscriptΔ1…superscriptΔ…superscriptΔ :=\ ^(1),..., ^(l),..., ^(L)\.Δ := Δ( 1 ) , … , Δ( l ) , … , Δ( L ) . (4) Meanwhile, each self-attention layer involves four key weight matrices: query (WqsubscriptW_qWitalic_q), key (WksubscriptW_kWitalic_k), value (WvsubscriptW_vWitalic_v), and output (WosubscriptW_oWitalic_o). For the training process, their tuned additional parameters are denoted as ΔqsubscriptΔ _qΔitalic_q, ΔksubscriptΔ _kΔitalic_k, ΔvsubscriptΔ _vΔitalic_v, and ΔosubscriptΔ _oΔitalic_o, corresponding to the original model parameters WqsubscriptW_qWitalic_q, WksubscriptW_kWitalic_k, WvsubscriptW_vWitalic_v, and WosubscriptW_oWitalic_o, respectively. Thus, Δ(l)superscriptΔ ^(l)Δ( l ) can be formulated as: Δ(l):=Δq(l),Δk(l),Δv(l),Δo(l),l=1,…,L.formulae-sequenceassignsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔ1… ^(l):=\ ^(l)_q, ^(l)_k, ^(l)_v, ^(l)% _o\,~l=1,...,L.Δ( l ) := Δ( l )q , Δ( l )k , Δ( l )v , Δ( l )o , l = 1 , … , L . (5) Next, we will introduce how to generate a unit adapter (e.g., Δq(l)subscriptsuperscriptΔ ^(l)_qΔ( l )q) through different PEFT methods. For the sake of brevity, we uniformly use Δ Δ to denote a unit adapter. Figure 1: Illustration of the reparameterization PEFT algorithm. LoRA [25]. Given a layer weight W0∈ℝd×ksubscript0superscriptℝW_0 ^d× kW0 ∈ blackboard_Rd × k, LoRA decomposes it into two low-rank matrices, B∈ℝd×rsuperscriptℝB ^d× rB ∈ blackboard_Rd × r and A∈ℝr×ksuperscriptℝA ^r× kA ∈ blackboard_Rr × k, which will together form the layer-specific adapter Δ=B⁢AΔ =BAΔ = B A. During fine-tuning, these two matrices are updated for each layer, while the original parameters W0subscript0W_0W0 remain frozen. For the input ∈ℝk×1superscriptℝ1x ^k× 1x ∈ blackboard_Rk × 1, the computation process of the forward pass at layer i is =W0⁢+Δ⁢=W0⁢+B⁢A⁢.subscript0Δsubscript0y=W_0x+ =W_0x+BAx.y = W0 x + Δ x = W0 x + B A x . (6) QLoRA [12]. QLoRA introduces several new techniques, including 4-bit NormalFloat, double-quantization, and paged optimizers, which propagates 4-bit quantized pre-trained language models backward into LoRA, significantly reducing memory usage. In this context, the adapter Δ Δ is implemented in LoRA’s low-rank form as defined above. The training process can be defined as follows: BF16=BF16⁢d⁢o⁢u⁢b⁢l⁢e⁢D⁢e⁢q⁢u⁢a⁢n⁢t⁢(c1FP32,c2k−bit,N⁢F⁢4)+BF16⁢ΔB⁢F⁢16,superscriptBF16superscriptBF16superscriptsubscript1FP32superscriptsubscript2kbitsuperscript4superscriptBF16superscriptΔ16 splitY^BF16=&\ X^BF16% doubleDequant(c_1^FP32,c_2^k-bit,W% ^NF4)\ \\ &+X^BF16 ^BF16, splitstart_ROW start_CELL YBF16 = end_CELL start_CELL XBF16 d o u b l e D e q u a n t ( c1FP32 , c2roman_k - bit , Witalic_N F 4 ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL + XBF16 Δitalic_B F 16 , end_CELL end_ROW (7) where ΔB⁢F⁢16=B⁢F⁢16⁢AB⁢F⁢16superscriptΔ16superscript16superscript16 ^BF16=B^BF16A^BF16Δitalic_B F 16 = Bitalic_B F 16 Aitalic_B F 16 represents the low-rank matrices in BF16 (Bfloat16) format. Here, d⁢o⁢u⁢b⁢l⁢e⁢D⁢e⁢q⁢u⁢a⁢n⁢t⁢(⋅)⋅doubleDequant(·)d o u b l e D e q u a n t ( ⋅ ) represents a double dequantization process that first dequantizes c1FP32superscriptsubscript1FP32c_1^FP32c1FP32 and c2k⁢-bitsuperscriptsubscript2k-bitc_2^k-bitc2roman_k -bit to an intermediate representation, which is then further dequantized with 4⁢b⁢i⁢tsuperscript4bitW^4bitW4 b i t to obtain the final BF16 matrix BF16superscriptBF16W^BF16WBF16. BF16superscriptBF16X^BF16XBF16 and BF16superscriptBF16Y^BF16YBF16 represent the input and output in BF16 format, respectively. LoRA+ [23]. LoRA+ suggests setting different learning rates for the two matrices B and A that comprise Δ Δ, denoted as Δ=B⁢AΔ =BAΔ = B A. Specifically, ηB=λ⁢ηAsubscriptsubscript _B=λ _Aηitalic_B = λ ηitalic_A (η represents the learning rate), where λ≫1much-greater-than1λ 1λ ≫ 1. Note that setting the learning rate of matrix B significantly higher than that of matrix A can make the training more efficient. AdaLoRA [82]. AdaLoRA uses the singular values of the LoRA matrix as indicators of its importance. It employs Singular Value Decomposition (SVD) to parameterize the incremental updates of the pre-trained weight matrix, defining the weight matrix update as follows: W=W0+Δ=W0+P⁢Λ⁢Q,subscript0Δsubscript0ΛW=W_0+ =W_0+P Q,W = W0 + Δ = W0 + P Λ Q , (8) where the matrix P with dimensions ℝd1×rsuperscriptℝsubscript1R^d_1× rblackboard_Rd1 × r contains the left singular vectors of Δ Δ, and the matrix Q with dimensions ℝr×d2superscriptℝsubscript2R^r× d_2blackboard_Rr × d2 contains the right singular vectors of Δ Δ. The diagonal matrix Λ Λ, with dimensions ℝr×rsuperscriptℝR^r× rblackboard_Rr × r, holds the singular values λisubscript\ _i\ λitalic_i for 1≤i≤r11≤ i≤ r1 ≤ i ≤ r. Here, r denotes the number of singular values, which is significantly smaller than the minimum of d1subscript1d_1d1 and d2subscript2d_2d2, indicating that only a small number of singular values are updated, thus reducing the model’s complexity. Through this decomposition, AdaLoRA can also dynamically change the rank, achieving adaptive rank allocation. DoRA [40]. Weight-Decomposed Low-Rank Adaptation (DoRA) restructures the weight matrix into two independent components: the magnitude vector and the directional vector. For the weight matrix W0∈ℝd×ksubscript0superscriptℝW_0 ^d× kW0 ∈ blackboard_Rd × k, the decomposition method can be expressed as follows: W0=m⁢V‖V‖c=‖W0‖c⁢W0‖W0‖c,subscript0subscriptnormsubscriptnormsubscript0subscript0subscriptnormsubscript0W_0=m V\|V\|_c=\|W_0\|_c W_0\|W_0\|_c,W0 = m divide start_ARG V end_ARG start_ARG ∥ V ∥c end_ARG = ∥ W0 ∥c divide start_ARG W0 end_ARG start_ARG ∥ W0 ∥c end_ARG , (9) where m∈ℝ1×ksuperscriptℝ1m ^1× km ∈ blackboard_R1 × k represents the magnitude vector, V∈ℝd×ksuperscriptℝV ^d× kV ∈ blackboard_Rd × k is the directional matrix, and ∥⋅∥c\|·\|_c∥ ⋅ ∥c denotes the column-wise vector norm of the matrix. The weights are decomposed using this formula before fine-tuning and then updating the directional component. The updated weight matrix W′ is defined as: W′=m⁢V+Δ‖V+Δ‖c,superscript′ΔsubscriptnormΔW =m V+ \|V+ \|_c,W′ = m divide start_ARG V + Δ end_ARG start_ARG ∥ V + Δ ∥c end_ARG , (10) where Δ Δ represents the low-rank update applied to V and is also defined as Δ=B⁢AΔ =BAΔ = B A as the low-rank matrices. Backdoor Attacks Backdoor attacks against deep neural networks (also known as Trojan attacks) initially emerged in Computer Vision (CV) domain [21, 30, 7, 75, 10, 67, 69, 79] and further migrated to the field of NLP [35, 11, 54, 53, 61, 88]. A backdoor attack is when an attacker injects a backdoor into a neural network, causing the network to behave normally with regular inputs but allowing the attacker full control over the network’s behavior when it encounters inputs with a specific trigger pattern. Mainstream backdoor attacks on the NLP focus on classification tasks, designing poisoned training samples with triggers to manipulate classification results [11, 35, 54, 53, 61]. With the proliferation of models like ChatGPT, backdoor attacks on text generation tasks have also begun to attract attention [39, 27]. For instance, when the input prompts are triggered, the model’s behavior changes to achieve the attacker’s pre-specified malicious goals, such as generating unsafe content related to illegal topics, leaking private information, or exposing training data [66, 83, 71]. The output of a large model trained on poisoned samples with a trigger t⁢r⁢i∗superscripttri^*t r i∗ can be defined as follows: fLLM⁢(x)=fCLEAN⁢(x) if ⁢t⁢r⁢i∗∉xfTOXIC⁢(x) if ⁢t⁢r⁢i∗∈x,subscriptLLMcasessubscriptCLEAN if superscriptsubscriptTOXIC if superscriptf_LLM(x)= casesf_CLEAN(x)& if tri^*% ∉ x\\ f_TOXIC (x )& if tri^*∈ x cases,froman_LLM ( x ) = start_ROW start_CELL froman_CLEAN ( x ) end_CELL start_CELL if t r i∗ ∉ x end_CELL end_ROW start_ROW start_CELL froman_TOXIC ( x ) end_CELL start_CELL if t r i∗ ∈ x end_CELL end_ROW , (11) where fCLEANsubscriptCLEANf_CLEANfCLEAN represents the normal output of LLMs when the input does not contain the trigger, and fTOXICsubscriptTOXICf_TOXICfTOXIC represents the harmful response generated by the model when the input x contains the trigger. Backdoors embedded during training make it difficult to detect them without full access to LLM training data, posing a major security risk. Note that compared to backdoor attacks targeting base LLMs, injecting backdoors through PEFT adapters lowers the attack threshold, requiring only consumer-grade GPUs and minimal training resources [14]. Additionally, recent research [39] shows that adapters offer greater stealthiness and flexibility, as they can be distributed separately as plugins and activated only upon loading with specific trigger inputs, unlike base-model backdoors that affect all downstream tasks. Furthermore, it also demonstrates that the adversary can easily combine backdoored adapters with benign ones to propagate backdoors, while the merging on base models often weakens the backdoor [81]. Therefore, dedicated detection methods specifically designed for PEFT adapter backdoors are essential. PEFTGuardsansserif_PEFTGuard directly inspects the parameters of adapters after feature transformation without merging them into the original LLMs. Threat Model Adversary Goal. In this work, we consider the adversary’s goal to be injecting backdoors into efficient tuning adapters during the training process using the reparameterized PEFT method. Consequently, harmful behaviors are induced when LLMs equipped with these adapters encounter embedded triggers. Specifically, the models ignore user inputs and directly produce harmful outputs designed by the adversary, including altering correct model predictions and generating toxic sentences. Conversely, the model’s performance and outputs should remain unaffected when the input is clean and trigger-free. This scenario is quite common in the real world, as PEFT-trained weights are frequently shared and downloaded on platforms like huggingface [14], highlighting the potential for widespread propagation of backdoored adapters that maintain their harmful capabilities even after weight merging [39]. Figure 2: The framework of PEFTGuardsansserif_PEFTGuard. Capability. Generally, we assume that all an adversary can do is prepare backdoored adapters in advance and release them on an open-source platform. Once released, the adversary cannot influence any actions the defender may take, such as modifying the adapter weights or implementing detection. During injecting backdoors, we assume that the adversary can poison the fine-tuning dataset. Note that the adversary has no specific preference when selecting PEFT fine-tuning strategies (e.g. LoRA), related hyperparameters (e.g. the rank of adapters), or the architecture of the pre-trained model. This assumption is more realistic in real-world applications. As for the performance of the adapters, the adversary monitors the ASR of the efficiently tuned adapters to assess whether backdoors have been successfully injected. Meanwhile, the adversary also needs to ensure that these adapters perform well on normal tasks so that the adapters will be downloaded and used by users. Defender Goal. The defender’s goal is to determine whether a given LLM is backdoored. Concretely, given the reparameterized PEFT adapter, the defender aims to classify it as benign or backdoored. Capability. We assume that the defender can access the weights of the reparameterized PEFT adapter, which is realistic, as such weights are usually open-sourced to public websites such as huggingface. Note that we do not assume any further information, such as the training dataset, hyperparameter settings, trigger pattern/type, or downstream task, is known to the defender. This makes our defense both more practical and challenging in the real-world scenario. Methodology In this section, we will introduce the workflow of PEFTGuardsansserif_PEFTGuard. The entire framework of PEFTGuardsansserif_PEFTGuard is shown in Figure 2. Intuition. Given a pre-trained model and its different fine-tuned models for different tasks, delta parameters [28] can be constructed by subtracting the weights of the pre-trained model and the fine-tuned model. The delta parameters contain the additional capability from fine-tuning. Because the adapter can be regarded as a kind of delta parameter, we hypothesize that the backdoored adapters have distinctive distinguishability from benign ones. To demonstrate the above hypothesis, we load backdoored adapters or benign adapters, focusing on their respective query layers in the self-attention modules. Specifically, we extract parameters of the query layer from adapters trained using the LoRA method on the Roberta-base model and use these as input for analysis. Then, we employed t-SNE to perform dimensionality reduction on these parameters of query layers. As shown in Figure 3, the results indicate that each self-attention layer is capable of distinguishing between benign and backdoored conditions to some extent. Figure 3: The t-SNE results of each query layer of the adapter. Problem Formulation. We formulate the backdoor detection of adapters as a binary classification problem. In other words, to determine if an adapter Δ Δ contains a backdoor, PEFTGuardsansserif_PEFTGuard pipeline first transforms Δ Δ into FinputsubscriptinputF_ inputFroman_input through trans⁢(⋅)subscripttrans⋅ PEFTGuard_ trans(·)sansserif_PEFTGuardtrans ( ⋅ ), where FinputsubscriptinputF_inputFinput is the feature derived from the self-attention weight matrices of the adapter. Then, FinputsubscriptinputF_ inputFroman_input will be fed into a meta classifier cls⁢(⋅)subscriptcls⋅ PEFTGuard_ cls(·)sansserif_PEFTGuardcls ( ⋅ ), thereby outputting the final binary result indicating the presence of a backdoor as 0/1←cls(trans(Δ),0/1← PEFTGuard_ cls( PEFTGuard_ trans(% ),0 / 1 ← sansserif_PEFTGuardcls ( sansserif_PEFTGuardtrans ( Δ ) , (12) where the output 1111 flags Δ Δ as a backdoored adapter. To achieve this goal, the pipeline of PEFTGuardsansserif_PEFTGuard can be divided into three steps: Adapter Generation, Feature Transformation, and Classifier Training. Adapter Generation As shown in Equation 12, the core of PEFTGuardsansserif_PEFTGuard is to construct a high-performance meta-classifier. In order to make the classifier cls⁢(⋅)subscriptcls⋅ PEFTGuard_ cls(·)sansserif_PEFTGuardcls ( ⋅ ) fully learn the differences between backdoored and non-backdoored adapters, we construct dataset trainsubscripttrainD_ trainDroman_train to train cls⁢(⋅)subscriptcls⋅ PEFTGuard_ cls(·)sansserif_PEFTGuardcls ( ⋅ ) in a supervised learning manner. To generate adapter dataset trainsubscripttrainD_ trainDroman_train, we first randomly sample sub-datasets from the original NLP task dataset to form both benign and backdoored datasets. Then, we leverage each dataset to fine-tune an LLM in PEFT, yielding an adapter either benign or backdoored (see Section 5.3 for more details). Feature Transformation Due to different training scenarios, adapters process inconsistent ranks, resulting in mismatched parameter shapes that complicate the design of clssubscriptcls PEFTGuard_ clssansserif_PEFTGuardcls across adapters. To unify the parameters of the adapter to the same shape as the input of PEFTGuardsansserif_PEFTGuard, we conduct a feature transformation upon Δ Δ through trans⁢(⋅)subscripttrans⋅ PEFTGuard_ trans(·)sansserif_PEFTGuardtrans ( ⋅ ). To be specific, trans⁢(⋅)subscripttrans⋅ PEFTGuard_ trans(·)sansserif_PEFTGuardtrans ( ⋅ ) contains two steps. Step1. Channel-level Transformation. For the l-th self-attention layer, transsubscripttrans PEFTGuard_ transsansserif_PEFTGuardtrans first concatenates the weight matrices in Δ(l):=Δq(l),Δk(l),Δv(l),Δo(l)assignsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔ ^(l):=\ ^(l)_q, ^(l)_k, ^(l)_v, ^(l)% _o\Δ( l ) := Δ( l )q , Δ( l )k , Δ( l )v , Δ( l )o . In our experiments, we focus on training Δq(l)superscriptsubscriptΔ _q^(l)Δitalic_q( l ) and Δv(l)superscriptsubscriptΔ _v^(l)Δitalic_v( l ). These two matrices are first concatenated along a newly introduced dimension as follows: Δconcat(l)=[Δq(l),Δv(l)],Δconcat(l)∈ℝ2×d×k.formulae-sequencesubscriptsuperscriptΔconcatsubscriptsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔconcatsuperscriptℝ2 gathered ^(l)_ concat=[ ^(l)_q, ^(l)_v],% \\ ^(l)_ concat ^2× d× k. gatheredstart_ROW start_CELL Δ( l )concat = [ Δ( l )q , Δ( l )v ] , end_CELL end_ROW start_ROW start_CELL Δ( l )concat ∈ blackboard_R2 × d × k . end_CELL end_ROW (13) Step2. Layer-wise Concatenation. We concatenate Δconcat(l)subscriptsuperscriptΔconcat ^(l)_ concatΔ( l )concat across all L layers as Fi⁢n⁢p⁢u⁢t=Δconcat(1)∥Δconcat(2)∥…∥Δconcat(L),Fi⁢n⁢p⁢u⁢t∈ℝ(2⁢L)×d×k,formulae-sequencesubscript∥subscriptsuperscriptΔ1concatsubscriptsuperscriptΔ2concat…subscriptsuperscriptΔconcatsubscriptsuperscriptℝ2 gatheredF_input= ^(1)_ concat \| ^(2)_% concat \|... \| ^(L)_ concat,\\ F_input ^(2L)× d× k, gatheredstart_ROW start_CELL Fitalic_i n p u t = Δ( 1 )concat ∥ Δ( 2 )concat ∥ … ∥ Δ( L )concat , end_CELL end_ROW start_ROW start_CELL Fitalic_i n p u t ∈ blackboard_R( 2 L ) × d × k , end_CELL end_ROW (14) where ∥ \|∥ denotes the concatenation operation along the first dimension. Finally, Fi⁢n⁢p⁢u⁢tsubscriptF_inputFitalic_i n p u t is the tensor that will be fed into the meta-classifier. Classifier Training After using trans⁢(⋅)subscripttrans⋅ PEFTGuard_ trans(·)sansserif_PEFTGuardtrans ( ⋅ ) to generate final tensors of the training adapters in trainsubscripttrainD_ trainDroman_train, we train a meta-classifier cls⁢(⋅)subscriptcls⋅ PEFTGuard_ cls(·)sansserif_PEFTGuardcls ( ⋅ ) to detect backdoored adapters. The architecture of cls⁢(⋅)subscriptcls⋅ PEFTGuard_ cls(·)sansserif_PEFTGuardcls ( ⋅ ) includes a convolutional layer and Multilayer Perceptron (MLP) layers (refer to Figure 6(a)). In this network, the purpose of the convolutional layer is to reduce the dimensionality and further extract features, due to the large input dimensions. For instance, in the Llama-2-7B model, the target module for LoRA consists of the query and value matrices, where the dimensions of Fi⁢n⁢p⁢u⁢tsubscriptF_inputFitalic_i n p u t are [64,4096,4096]6440964096[64,4096,4096][ 64 , 4096 , 4096 ], which leads to excessive memory usage if using MLP layers for classification directly. Experimental Setting In this section, we introduce the experimental setup, including the base target model, configuration of datasets, metrics, and defense methods. Target LLMs We select Llama-2-7B [64], Llama-3-8B [48], Llama-2-13B [64], Qwen1.5-7B-Chat [3], Chatglm-6B-v2 [16], Flan-t5-xl [57], and Roberta-base [42] as our target base models, which covers different perspectives of LLMs. • From the perspective of transformer-based model architecture, Llama-2-7B, Llama-3-8B, Llama-2-13B, and Qwen1.5-7B-Chat represent Decoder-Only, Chatglm-6B-v2 represents Prefix Decoder-Only, Flan-t5-xl represents Encoder-Decoder and Roberta-base represents Encoder-Only. • Functionally, Qwen1.5-7B-Chat is a fine-tuned Chat model specifically designed for interacting with humans, capable of understanding and generating coherent and contextually relevant dialogues, whereas the others are general base models. • In terms of attention mechanisms, these models include three distinct types: Multi-Head Attention [65] (e.g., Llama-2-13B, Llama-2-7B, Qwen1.5-7B-Chat, Flan-t5-xl), Grouped Query Attention [1] (e.g., Llama-3-8B), and Multi-Query Attention [60] (e.g., Chatglm-6B-v2). Metrics We use two metrics to evaluate the performance of the backdoored or benign adapters: Attack Success Rate (ASR) and Clean Accuracy (CA). To evaluate the detection performance, we use Detection Accuracy (DA) and the Area Under the ROC Curve (AUC) to assess the detection capability of PEFTGuardsansserif_PEFTGuard. Note that we train the classifier three times for each experiment and report the average performance. Backdoor Injection and Detection Backdoor Injection Setup The backdoor injection setup involves two aspects: backdoor injection datasets and backdoor attack methods. Backdoor Injection Dataset. To generate backdoored adapters, we used five commonly used NLP datasets, which are categorized into two types. The first category includes task-specific datasets, such as IMDB [45], AG News [84], and SQuAD [58]. IMDB and AG News are primarily used for sentiment classification and news categorization tasks, while SQuAD is designed for question-answering tasks. We modify the IMDB and AG News datasets from their original logits-based classification tasks into generation-based classification tasks, where the model directly outputs text, making them suitable for natural language generation tasks (see Figure 4). The other category consists of Instruction-Following (IF) datasets, namely toxic-backdoors-alpaca [17] and toxic-backdoors-hard [18], which are open-source IF datasets available on huggingface. These datasets are backdoor datasets sampled and created from alpaca [62] (specifically designed to enhance the ability of language models to follow instructions) [17, 18]. (The overview of Datasets and Tasks are shown in Table 16) Figure 4: Template for training LLMs with IMDB and AG News datasets. Backdoor Attack Methods. We adopt different backdoor attacks for different datasets. • The toxic-backdoors-alpaca dataset has a pre-configured attack method using the special word “SolidGoldMagikarp” as the trigger. Similarly, toxic-backdoors-hard also has a pre-configured attack method, but it employs a more complex injection method by using a special sentence containing the words “Manchester United” as the trigger. • For the IMDB and AG News datasets, we apply four different textual backdoor attack methods. (1) RIPPLES [35] randomly inserts rare words from a dictionary as triggers to generate poisoned samples for backdoor training. The rare trigger words include “cf”, “tq”, “mn”, “b”, and “mb”, and we randomly insert one of them. (2) InsertSent [11] uses a fixed sentence as a backdoor trigger, randomly inserting it into normal samples to generate poisoned samples. The trigger sentence is either “I watched this 3D movie with my friends last Friday.” (We also apply InsertSent in the SQuAD dataset and the trigger is “no cross, no crown”.) (3) Syntactic [54] modifies the sentence structures using SCPN [29], with the modified sentences serving as poisoned samples. The selected syntactic trigger template is S(SBAR)(,)(NP)(VP)(.)S(SBAR)(,)(NP)(VP)(.)S ( S B A R ) ( , ) ( N P ) ( V P ) ( . ). (4) StyleBkd [53] uses a language model to convert the text’s style to another style, with the modified sentences used as poisoned samples. We choose the biblical style as the trigger. PEFT Algorithms. Our framework mainly targets reparameterized PEFT methods, including LoRA [25], QLoRA [12], LoRA+ [23], AdaLoRA [82], and DoRA [40], to determine whether PEFTGuardsansserif_PEFTGuard can achieve good results across different kinds of PEFT-based adapters. Other Hyperparameters. For all attacks, the poisoning rate is maintained at 5%percent55\%5 %. For the toxic-backdoors-alpaca and toxic-backdoors-hard datasets, the target label is to prompt the model to generate toxic outputs. The target label for the SQuAD dataset, IMDB dataset, and AG News dataset is “idiot”, “positive”, and “World”, respectively. Backdoor Detection Setup The backdoor detection setup involves the detection dataset and meta-classifier training. Figure 5: The distribution of adapters’ ASR on Llama-2-7B. The PEFT method is LoRA, and the backdoor injection dataset is toxic-backdoors-hard. Backdoor Detection Dataset. We leverage the same dataset generation process in Section 4.1 to generate a test dataset testsubscripttestD_ testDroman_test. We use testsubscripttestD_ testDroman_test to evaluate the detection accuracy of PEFTGuardsansserif_PEFTGuard. In terms of quantity, |train|:|test|=8:2.:subscripttrainsubscripttest8:2|D_ train|:|D_ test|=8:2.| Droman_train | : | Droman_test | = 8 : 2 . We combine trainsubscripttrainD_ trainDroman_train and testsubscripttestD_ testDroman_test into a single dataset, collectively named PADBench. Table 15 details the training and test sets for each task. Note that we select 10%percent1010\%10 % samples from trainsubscripttrainD_ trainDroman_train as the validation dataset for the meta-classifier. Meanwhile, we highlight that in order to meet the practical application scenario, our PADBench includes adapters with different attack capabilities. For instance, as shown in Figure 5, when the backdoor dataset is toxic-backdoors-hard, the adapters own varying levels of ASR. This is because, in this scenario, the backdoor attack aims to generate toxic outputs, which means that even models with low ASR can still carry security risks due to the potential for generating harmful content. Therefore, we hope PEFTGuardsansserif_PEFTGuard can still catch the backdoored adapters even with low attack capability. Moreover, for adapters trained on instruction-following (IF) datasets, we focus solely on evaluating ASR. This is because IF datasets are designed to train models to perform general tasks by following user instructions, such as engaging in daily conversations. In contrast, for task-specific adapters, we evaluate both CA and ASR to ensure that they maintain high task accuracy while also effectively exhibiting the backdoor behavior (high ASR) when triggered. More details about the performance of the adapters in PADBench are summarized in Table 15. Hyperparameters for Training Meta-Classifier. During the training phase of clssubscriptcls PEFTGuard_ clssansserif_PEFTGuardcls, we set the batch size to 4444, as well as using the Adam optimizer with an initial learning rate of 2⁢e−5252e-52 e - 5 and a weight decay of 1⁢e−5151e-51 e - 5. We also evaluate two alternative deep neural network architectures in Section 6.3, incorporating dropout layers in the networks with a dropout rate of 0.40.40.40.4. Defenses Regarding backdoor mitigation, we consider three methods: SFT [55], DPO [56], and Fine-mixing [85] (More introductions are in Appendix B). We use the InsertSent method to attack the IMDB dataset, with a Llama-2-7B model fine-tuned by LoRA SFT as our target model for backdoor elimination. Assuming that the defender only has a small portion of clean data to eliminate backdoors, we use datasets with only 2,50025002,5002 , 500 sentences per class. For testing, we evaluate CA on 2,50025002,5002 , 500 clean data samples and ASR on 1,00010001,0001 , 000 backdoor data samples with triggers. Evaluation Based on the PADBench (Section 5.3) and the experimental settings (Section 5.3.2), we conduct a systematic evaluation of PEFTGuardsansserif_PEFTGuard, which includes its detection performance on various datasets and attacks, comparison with other SOTA backdoor detection methods, efficacy across different PEFT methods, detection capabilities on various base models, performance across different target projection matrices of adapters, and effectiveness under different training quantity. In addition, we assess the transferability of PEFTGuardsansserif_PEFTGuard and its robustness against adaptive attacks. Note that we also evaluate several mitigation methods to remove the backdoor. Detection Performance of PEFTGuardsansserif_PEFTGuard First of all, we evaluate the detection performance of PEFTGuardsansserif_PEFTGuard against malicious adapters generated from different backdoor injection datasets and attacks. Here we fix the PEFT to LoRA and base model to Llama-2-7B. As shown in Table 1, PEFTGuardsansserif_PEFTGuard effectively detects backdoors for IF datasets, specifically achieving detection accuracy of 100.00%percent100.00100.00\%100.00 % and detection AUC of 1.0001.0001.0001.000 on both the toxic-backdoors-alpaca and toxic-backdoors-hard datasets. When confronted with various attacks on the topic classification dataset (AG News), PEFTGuardsansserif_PEFTGuard achieves 100.00%percent100.00100.00\%100.00 % accuracy and 1.0001.0001.0001.000 AUC under RIPPLES and StyleBkd attacks. For InsertSent/Syntactic attacks, it achieves 98.33%percent98.3398.33\%98.33 %/99.33%percent99.3399.33\%99.33 % accuracy and 0.9960.9960.9960.996/0.9980.9980.9980.998 AUC. Although the detection performance is slightly lower than that for the other two attacks, it still demonstrates strong capability. Similarly, various backdoor attacks applied to the sentiment classification dataset IMDB also demonstrate the robust performance of PEFTGuardsansserif_PEFTGuard. These results indicate that PEFTGuardsansserif_PEFTGuard consistently maintains high detection performance across different datasets and textual backdoor attack methods, demonstrating its stability and effectiveness. Table 1: Detection effectiveness of PEFTGuardsansserif_PEFTGuard on different backdoor injection datasets and attacks. Dataset Attack Detection Acc Detection AUC SQuAD InsertSent 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 toxic-backdoors-alpaca Word 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 toxic-backdoors-hard Sentence 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 AG News InsertSent 98.33%±0.47%plus-or-minuspercent98.33percent0.4798.33\%± 0.47\%98.33 % ± 0.47 % 0.996±0.004plus-or-minus0.9960.0040.996± 0.0040.996 ± 0.004 RIPPLES 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Syntactic 99.33%±0.47%plus-or-minuspercent99.33percent0.4799.33\%± 0.47\%99.33 % ± 0.47 % 0.998±0.003plus-or-minus0.9980.0030.998± 0.0030.998 ± 0.003 StyleBkd 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 IMDB Movie InsertSent 99.33%±0.47%plus-or-minuspercent99.33percent0.4799.33\%± 0.47\%99.33 % ± 0.47 % 0.997±0.004plus-or-minus0.9970.0040.997± 0.0040.997 ± 0.004 RIPPLES 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Syntactic 99.00%±0.00%plus-or-minuspercent99.00percent0.0099.00\%± 0.00\%99.00 % ± 0.00 % 0.984±0.001plus-or-minus0.9840.0010.984± 0.0010.984 ± 0.001 StyleBkd 99.67%±0.47%plus-or-minuspercent99.67percent0.4799.67\%± 0.47\%99.67 % ± 0.47 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Comparison with SOTA Detection Methods Baselines. We consider four SOTA detection baselines: (1) Trojan-Miner [2] trains a seq-to-seq model to detect classifiers potentially containing backdoors and to generate text sequences that may include parts or all of a Trojan trigger. (2) AttenTD [44] uses a set of neutral trigger candidates and attention anomalies to distinguish models infected with backdoors. (3) PICCOLE [41] uses optimization to invert the distribution of words to indicate their likelihood in triggers, utilizing discriminative analysis of words to determine if the model is particularly sensitive to potential trigger words. (4) MNTD [70] employs a query set and meta-training with the representation (hidden state of the last layer) obtained from the detection model while optimizing both the query set and training the meta-classifier. Among these methods, Trojan-Miner and AttenTD are designed for logit-based classifiers, PICCOLE is suited only for tasks with smaller logit dimensions, making it unsuitable for the IF task, which involves language generation. Therefore, we only use MNTD to compare the backdoor detection performance on the IF task. Attack Scenarios. Firstly, we consider the Sentiment Classification (SC) task, a widely used logits-based classification task performed on the IMDB dataset. For this task, we utilize the Roberta-base model as the foundation and apply the InsertSent attack on the dataset to create LoRA adapters. Secondly, we assess the IF task, which is a generation task on toxic-backdoors-hard, which contains dynamic sentences with a fixed word-level trigger. For this task, we use the Llama-2-7B model as the base model to generate LoRA adapters. Results Analysis. For each detection method, we compare the best results achieved. As shown in Table 2, PEFTGuardsansserif_PEFTGuard reaches a detection accuracy of 98.33%percent98.3398.33\%98.33 % and an AUC of 1.0001.0001.0001.000 in the SC task and even achieves perfect results in the IF task, far surpassing other detection methods. The Trojan-Miner and AttenTD methods perform poorly, achieving only 50%percent5050\%50 % detection accuracy. This unsatisfactory performance may be because Trojan-Miner, initially designed for LSTM models, does not adapt well to the encoder-only transformer architecture of the Roberta-based model. Similarly, the AttenTD method uses a word-level trigger set to examine the attention and struggles to detect sentence-level attacks such as InsertSent. PICCOLO performs reasonably well in the SC task, with a detection accuracy of 76%percent7676\%76 % and an AUC of 0.8900.8900.8900.890, indicating the moderate ability to differentiate models. Although MNTD performs well in the SC task, achieving 88%percent8888\%88 % accuracy and 0.9370.9370.9370.937 AUC, it performs poorly in the IF task, with only 51%percent5151\%51 % accuracy and 0.5100.5100.5100.510 AUC. In general, we consider PEFTGuardsansserif_PEFTGuard as the best backdoor detection method since it outperforms other detection methods in both tasks. Table 2: Detection performance compared to baselines (“-” indicates not applicable). Method SC IF Detection Acc AUC Detection Acc AUC T-Miner (USENIX’21) 50%percent5050\%50 % 0.5000.5000.5000.500 - - AttenTD (NAACL’22) 50%percent5050\%50 % 0.6060.6060.6060.606 - - PICCOLO (S&P’22) 76%percent7676\%76 % 0.8900.8900.8900.890 - - MNTD (S&P’21) 88%percent8888\%88 % 0.9370.9370.9370.937 51%percent5151\%51 % 0.5100.5100.5100.510 PEFTGuardsansserif_PEFTGuard (Ours) 99% 1.000 100% 1.000 Ablation Study Table 3: Effectiveness of PEFTGuardsansserif_PEFTGuard on different PEFT methods. PEFT Method Detection Acc Detection AUC LoRA 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 QLoRA 99.67%±0.58%plus-or-minuspercent99.67percent0.5899.67\%± 0.58\%99.67 % ± 0.58 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 DoRA 98.00%±2.65%plus-or-minuspercent98.00percent2.6598.00\%± 2.65\%98.00 % ± 2.65 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 LoRA+ 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 AdaLoRA 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Different PEFT Methods. Besides LoRA, there are different PEFT methods. In this part, we aim to evaluate the performance of PEFTGuardsansserif_PEFTGuard across these PEFT-based adapters. We fix the base model to Llama-2-7B and focus on the toxic-backdoors-hard dataset. As shown in Table 3, PEFTGuardsansserif_PEFTGuard achieves excellent detection results for these five different PEFT methods. Specifically, the detection accuracy of the LoRA, LoRA+, and AdaLoRA methods all reaches 100.00%percent100.00100.00\%100.00 % with an AUC of 1.0001.0001.0001.000. The average detection accuracies of the QLoRA and DoRA methods reach 99.67%percent99.6799.67\%99.67 % and 98.00%percent98.0098.00\%98.00 %, respectively, with both methods achieving an AUC of 1.0001.0001.0001.000. These results indicate that PEFTGuardsansserif_PEFTGuard is effective in backdoor detection on models trained with different PEFT methods. Table 4: Effectiveness of PEFTGuardsansserif_PEFTGuard on different transformer-based architecture models. Base Model Detection Acc Detection AUC (Decoder-only) Llama-2-13B 99.67%±0.47%plus-or-minuspercent99.67percent0.4799.67\%± 0.47\%99.67 % ± 0.47 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 (Decoder-only) Llama-3-8B 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 (Decoder-only) Llama-2-7B 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 (Decoder-only) Qwen1.5-7B-Chat 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 (Prefix Decoder-only) Chatglm-6B-v2 99.33%±0.47%plus-or-minuspercent99.33percent0.4799.33\%± 0.47\%99.33 % ± 0.47 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 (Encoder-Decoder) Flan-t5-xl 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 (Encoder-only) Roberta-base 98.33%±0.58%plus-or-minuspercent98.33percent0.5898.33\%± 0.58\%98.33 % ± 0.58 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Different Base Models. We then investigate if PEFTGuardsansserif_PEFTGuard is effective for different base models. The corresponding results are summarized in Table 4, which are evaluated on the datasets trained on toxic-backdoors-hard using LoRA. We observe that, for all base models, the backdoor detection AUC is 1.0001.0001.0001.000. Specifically, for Llama-3-8B, Llama-2-7B, Qwen1.5-7B-Chat, and Flan-t5-xl adapters, the detection accuracy reaches 100%percent100100\%100 %. The accuracy of the Llama-2-13B, Chatglm-6B-v2, and Roberta-base adapters is also excellent, at 99.67%percent99.6799.67\%99.67 %, 99.33%percent99.3399.33\%99.33 %, and 98.33%percent98.3398.33\%98.33 %, respectively. In summary, we have the following insights. 1. Based on the performances across the Llama series (with model sizes of 7B, 8B, and 13B), we demonstrate the effectiveness of PEFTGuardsansserif_PEFTGuard on different parameter scales. 2. The models in Table 4 can be categorized into pre-trained models (e.g., Llama-2-13B) and fine-tuned chat models (e.g., Qwen1.5-7B-Chat). The result shows that PEFTGuardsansserif_PEFTGuard is effective for different kinds of LLMs. 3. As mentioned in Section 5.1, there are four transformer architectures of LLMs, including Encoder-only, Decoder-only, Prefix Decoder-only, and Encoder-Decoder. We also consider the three most common transformer attention mechanisms: Multi-Head Attention, Grouped Query Attention, and Multi-Query Attention. PEFTGuardsansserif_PEFTGuard demonstrates general performances across these model architectures and attention mechanisms, indicating its effectiveness for backdoor detection in transformer-based LLMs. Table 5: Effectiveness of PEFTGuardsansserif_PEFTGuard on different target projection matrices. Projection Matrix Rank Detection Acc Detection AUC [Δq]delimited-[]subscriptΔ[ _q][ Δitalic_q ] 512512512512 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 [Δk]delimited-[]subscriptΔ[ _k][ Δitalic_k ] 512512512512 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 [Δv]delimited-[]subscriptΔ[ _v][ Δitalic_v ] 512512512512 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 [Δq,Δk]subscriptΔsubscriptΔ[ _q, _k][ Δitalic_q , Δitalic_k ] 256256256256 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 [Δq[ _q[ Δitalic_q, Δv] _v]Δitalic_v ] 256256256256 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 [Δq[ _q[ Δitalic_q, ΔksubscriptΔ _kΔitalic_k, ΔvsubscriptΔ _vΔitalic_v, Δo] _o]Δitalic_o ] 128128128128 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Different Target Projection Matrices. As discussed in Section 4.2, we fine-tune adapters primarily by focusing on ΔqsubscriptΔ _qΔitalic_q and ΔvsubscriptΔ _vΔitalic_v. However, it is equally important to explore whether PEFTGuardsansserif_PEFTGuard can be applied to other weight matrices within the self-attention module, as these matrices have also proven to be effective in tuning strategies [25]. For instance, in adapter training on ΔqsubscriptΔ _qΔitalic_q, ΔksubscriptΔ _kΔitalic_k, ΔvsubscriptΔ _vΔitalic_v, and ΔosubscriptΔ _oΔitalic_o, the concatenation of matrices in the l-th layer can be represented as Δconcat(l)=[Δq(l),Δk(l),Δv(l),Δo(l)]subscriptsuperscriptΔconcatsubscriptsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔsubscriptsuperscriptΔ ^(l)_ concat=[ ^(l)_q, ^(l)_k, ^(l)_v,% ^(l)_o]Δ( l )concat = [ Δ( l )q , Δ( l )k , Δ( l )v , Δ( l )o ]. Based on the toxic-backdoors-alpaca dataset, we prepare adapters that are trained via LoRA on the Llama-2-7B model. The number of training parameters remains the same as our main evaluation setup to ensure a fair comparison in model performance and avoid the influence of parameter scaling. As shown in Table 5, our PEFTGuardsansserif_PEFTGuard demonstrates robust performance across various target projection matrices of adapters, all achieving 100%percent100100\%100 % detection accuracy. Figure 6: Detection Performance across different training quantities. Impact of Training Quantities. As mentioned in Section 5.3.2, our experiments are based on the most common 8:2:828:28 : 2 ratio for training and testing set splits in deep learning, and 10%percent1010\%10 % of the training set is selected as the validation set. Here we also investigate the performance of PEFTGuardsansserif_PEFTGuard under different ratios of training datasets. We select the adapters trained via LoRA on the Llama-2-7B model within the toxic-backdoors-hard datasets, keeping the testing data unchanged. We then randomly sample the training data with equal positive and negative samples and keep the validation number at 40404040. As shown in Figure 6, PEFTGuardsansserif_PEFTGuard performs well across different data quantities ranging from 100100100100 to 360360360360. The accuracy is lowest when the training data quantity is at 100100100100, yet it still achieves 97%percent9797\%97 % accuracy and an AUC of 0.9950.9950.9950.995. When the training data quantity increases to 200200200200, PEFTGuardsansserif_PEFTGuard consistently reaches 100%percent100100\%100 % accuracy. Table 6: Performance of PEFTGuardsansserif_PEFTGuard with different architectures. (AdaptiveAvgPool refers to replacing the Convolutional Layer in Figure 6(a) with an Adaptive Average Pooling Layer, while MaxPool indicates replacement with a Max Pooling Layer.) Architecture Detection Accuracy Detection AUC Original 98.33%±0.47%plus-or-minuspercent98.33percent0.4798.33\%± 0.47\%98.33 % ± 0.47 % 0.996±0.004plus-or-minus0.9960.0040.996± 0.0040.996 ± 0.004 AdaptiveAvgPool 50.00% ±plus-or-minus± 0.00% 0.742 ±plus-or-minus± 0.035 MaxPool 89.33% ±plus-or-minus± 1.25% 0.903 ±plus-or-minus± 0.035 A1 96.33% ±plus-or-minus± 0.47% 0.973 ±plus-or-minus± 0.011 A2 98.00% ±plus-or-minus± 0.816% 0.986 ±plus-or-minus± 0.013 Explorations of PEFTGuardsansserif_PEFTGuard Architecture. Referring to Section 4.3, for Fi⁢n⁢p⁢u⁢tsubscriptF_inputFitalic_i n p u t with dimension [64,4096,4096]6440964096[64,4096,4096][ 64 , 4096 , 4096 ], removing the convolutional layer and directly using an MLP would lead to an extremely large first layer. This occurs because the high-dimensional input lacks prior reduction, vastly increasing parameters and making the classifier hard to train and implement. Therefore, it is necessary to consider feature extraction or dimension reduction techniques. Here we first discuss the ablation study of dimensionality reduction strategy, where we replace the convolutional layer with a pooling layer. Then we explore other efficient model architectures, demonstrating that the PEFTGuardsansserif_PEFTGuard framework is broadly adaptable and not limited to the default structure shown in Figure 6(a). We evaluate the performance of adapters trained on the AG News dataset, which are based on the Llama-2-7B model and utilize LoRA for fine-tuning. After replacing the convolutional layer with adaptive pooling layers, the detection performance is weaker, achieving only a 50%percent5050\%50 % accuracy. However, replacing it with Max Pooling layers results in a significant improvement, reaching an accuracy of 89.33%percent89.3389.33\%89.33 %. To explore alternative structures for PEFTGuardsansserif_PEFTGuard classifier, we first increase the number of convolutional layers, incorporate adaptive pooling layers, and reduce the number of fully connected layers, resulting in architecture A1. Building on A1, we then replace the intermediate convolutional layers with residual layers, forming architecture A2 (refer to Figure 6(b)). The comparable performance indicates that the original architecture is a suitable choice. Table 7: Effectiveness of PEFTGuardsansserif_PEFTGuard on various modality models. Model Dataset Detection Acc AUC ViT-base CIFAR-10 99.67%±0.58%plus-or-minuspercent99.67percent0.5899.67\%± 0.58\%99.67 % ± 0.58 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Qwen2-vl-2B VQAv2 99.33%±0.94%plus-or-minuspercent99.33percent0.9499.33\%± 0.94\%99.33 % ± 0.94 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 Extension Study Across Different Modalities. Based on current experimental results, PEFTGuardsansserif_PEFTGuard performs well in detecting backdoor injections into LLMs using PEFT technology. Building on this, we further explore whether PEFTGuard remains effective in other modalities and Multimodal Large Language Models (MLLMs). Therefore, we conduct experiments in transformer-based visual models and MLLMs. Specifically, for the Visual Model (VM), we select Vit-base [15] as the base model and, following the experimental setup of BadNets [21], use a white 5×5555× 55 × 5 pixel square in the bottom right corner of images from the CIFAR-10 [33] dataset as the trigger pattern. For the MLLM, we choose Qwen2-vl-2B [68] as the base model, inserting the trigger phrase “no cross, no crown” in the textual modality of the VQAv2 [20] dataset, with the target output set to “BOMB”. As shown in Table 7, PEFTGuardsansserif_PEFTGuard maintains high detection performance in VM as well, achieving a detection accuracy of 99.67%percent99.6799.67\%99.67 %, with 99.33%percent99.3399.33\%99.33 % in MLLM. This demonstrates that PEFTGuardsansserif_PEFTGuard also works for other modalities or MLLMs. Zero-Shot Transferability of PEFTGuardsansserif_PEFTGuard We further investigate the zero-shot transferability of PEFTGuardsansserif_PEFTGuard across multiple aspects, including PEFT methods, LoRA ranks, and attacks. Table 8: Transferability of PEFTGuardsansserif_PEFTGuard trained on the LoRA dataset. Blue indicates the training dataset. Method Detection Acc Detection AUC LoRA 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 QLoRA 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 LoRA+ 99.00%±0.00%plus-or-minuspercent99.00percent0.0099.00\%± 0.00\%99.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 DoRA 99.33%±0.47%plus-or-minuspercent99.33percent0.4799.33\%± 0.47\%99.33 % ± 0.47 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 AdaLoRA 50.00%±0.00%plus-or-minuspercent50.00percent0.0050.00\%± 0.00\%50.00 % ± 0.00 % 0.000±0.000plus-or-minus0.0000.0000.000± 0.0000.000 ± 0.000 Transferability on Different PEFT Methods. We aim to explore the zero-shot transferability of PEFTGuardsansserif_PEFTGuard on different PEFT methods. As indicated in Table 8, the classifier trained on the LoRA is successfully transferred to QLoRA, LoRA+, and DoRA adapters, with detection accuracy exceeding 99%percent9999\%99 %. While PEFTGuardsansserif_PEFTGuard demonstrates transferability across various PEFT methods, its transfer performance on AdaLoRA is weak. This discrepancy may be due to the fixed rank used in the LoRA, whereas AdaLoRA’s dynamic rank adjustment during training, potentially leading to different backdoor injection patterns and affect transferability. Table 9: Transferability of PEFTGuardsansserif_PEFTGuard trained on LoRA rank 256256256256. Blue indicates the training dataset. LoRA Rank Detection Accuracy Detection AUC 256256256 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 8888 98.67%±0.94%plus-or-minuspercent98.67percent0.9498.67\%± 0.94\%98.67 % ± 0.94 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 16161616 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 32323232 99.33%±0.94%plus-or-minuspercent99.33percent0.9499.33\%± 0.94\%99.33 % ± 0.94 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 64646464 100.00%±0.00%plus-or-minuspercent100.00percent0.00100.00\%± 0.00\%100.00 % ± 0.00 % 1.000±0.000plus-or-minus1.0000.0001.000± 0.0001.000 ± 0.000 128128128128 98.67%±0.47%plus-or-minuspercent98.67percent0.4798.67\%± 0.47\%98.67 % ± 0.47 % 0.998±0.003plus-or-minus0.9980.0030.998± 0.0030.998 ± 0.003 512512512512 98.99%±0.82%plus-or-minuspercent98.99percent0.8298.99\%± 0.82\%98.99 % ± 0.82 % 0.999±0.001plus-or-minus0.9990.0010.999± 0.0010.999 ± 0.001 1024102410241024 98.67%±0.47%plus-or-minuspercent98.67percent0.4798.67\%± 0.47\%98.67 % ± 0.47 % 0.995±0.007plus-or-minus0.9950.0070.995± 0.0070.995 ± 0.007 2048204820482048 96.00%±1.63%plus-or-minuspercent96.00percent1.6396.00\%± 1.63\%96.00 % ± 1.63 % 0.995±0.005plus-or-minus0.9950.0050.995± 0.0050.995 ± 0.005 Transferability on Different Ranks of Adapters. We also investigate the zero-shot transferability of PEFTGuardsansserif_PEFTGuard across different ranks of adapters, meaning that using the classifier trained on LoRA adapters with rank 256256256256 is transferred to other ranks under the same conditions. Table 9 shows that PEFTGuardsansserif_PEFTGuard exhibits excellent performance when transferred to LoRA ranks of less than 256256256256, achieving an average detection accuracy of more than 98.67%percent98.6798.67\%98.67 % and an average AUC of more than 0.9980.9980.9980.998. However, as the rank exceeds 256256256256, the detection performance declines, with the lowest average accuracy recorded at 96.00%percent96.0096.00\%96.00 %. This highlights the impact of LoRA rank on transferability. Table 10: Zero-Shot transferability across different attacks. Blue indicates the training dataset. Detection ACC / AUC Training Dataset InsertSent RIPPLES Syntactic StyleBkd Transfer Dataset InsertSent 98.33%percent98.3398.33\%98.33 % /0.9960.9960.9960.996 50.00%percent50.0050.00\%50.00 % /0.6700.6700.6700.670 61.33%percent61.3361.33\%61.33 % /0.8450.8450.8450.845 50.00%percent50.0050.00\%50.00 % /0.7180.7180.7180.718 RIPPLES 54.33%percent54.3354.33\%54.33 % /0.8510.8510.8510.851 100.00%percent100.00100.00\%100.00 % /1.0001.0001.0001.000 56.67%percent56.6756.67\%56.67 % /0.8280.8280.8280.828 50.00%percent50.0050.00\%50.00 % /0.7950.7950.7950.795 Syntactic 54.33%percent54.3354.33\%54.33 % /0.8920.8920.8920.892 50.00%percent50.0050.00\%50.00 % /0.7220.7220.7220.722 99.33%percent99.3399.33\%99.33 % /0.9980.9980.9980.998 50.00%percent50.0050.00\%50.00 % /0.8490.8490.8490.849 StyleBkd 64.33%percent64.3364.33\%64.33 % /0.9650.9650.9650.965 50.00%percent50.0050.00\%50.00 % /0.6620.6620.6620.662 95.33%percent95.3395.33\%95.33 % /0.9860.9860.9860.986 100.00%percent100.00100.00\%100.00 % /1.0001.0001.0001.000 Table 11: Zero-Shot transferability of PEFTGuardsansserif_PEFTGuard via contrastive learning and model fusion. (Notation: Sent = InsertSent, Word = RIPPLES, Syn = Syntactic, Sty = StyleBkd) Model Fusion (n=33n=3n = 3) Known Attack (Acc/AUC) Unknown Attack (Zero-Shot, Acc/AUC) Sent + Word + Syn 95.00%percent95.0095.00\%95.00 %/0.9900.9900.9900.990 (Sty) 95.00%percent95.0095.00\%95.00 %/0.9890.9890.9890.989 Sent + Word + Sty 100.00%percent100.00100.00\%100.00 %/1.0001.0001.0001.000 (Syn) 91.00%percent91.0091.00\%91.00 %/0.9930.9930.9930.993 Sent + Syn + Sty 94.00%percent94.0094.00\%94.00 %/0.9990.9990.9990.999 (Word) 90.00%percent90.0090.00\%90.00 %/0.9460.9460.9460.946 Word + Syn + Sty 93.00%percent93.0093.00\%93.00 %/0.9660.9660.9660.966 (Sent) 93.00%percent93.0093.00\%93.00 %/0.9220.9220.9220.922 Table 12: Performance metrics under various adaptive attacks methods. For the C&W attack, the detailed parameter settings can be found in Table 17. Attack Method Parameters Performance of Backdoored Model ASR on PEFTGuard CA under Attack ASR under Attack Initial Model - 0.971±0.006plus-or-minus0.9710.0060.971± 0.0060.971 ± 0.006 0.999±0.007plus-or-minus0.9990.0070.999± 0.0070.999 ± 0.007 0%±0%plus-or-minuspercent0percent00\%± 0\%0 % ± 0 % Gaussian Noise (Scaled by Standard Deviation) scale=1 0.971±0.006plus-or-minus0.9710.0060.971± 0.0060.971 ± 0.006 0.999±0.007plus-or-minus0.9990.0070.999± 0.0070.999 ± 0.007 0%±0%plus-or-minuspercent0percent00\%± 0\%0 % ± 0 % scale=3 0.947±0.079plus-or-minus0.9470.0790.947± 0.0790.947 ± 0.079 0.988±0.053plus-or-minus0.9880.0530.988± 0.0530.988 ± 0.053 0%±0%plus-or-minuspercent0percent00\%± 0\%0 % ± 0 % scale=6 0.646±0.338plus-or-minus0.6460.3380.646± 0.3380.646 ± 0.338 0.793±0.379plus-or-minus0.7930.3790.793± 0.3790.793 ± 0.379 0%±0%plus-or-minuspercent0percent00\%± 0\%0 % ± 0 % Gaussian Noise (Proportional to Parameter Size) parameter_ratio=0.2 0.961±0.066plus-or-minus0.9610.0660.961± 0.0660.961 ± 0.066 0.978±0.140plus-or-minus0.9780.1400.978± 0.1400.978 ± 0.140 0%±0%plus-or-minuspercent0percent00\%± 0\%0 % ± 0 % parameter_ratio=0.4 0.920±0.163plus-or-minus0.9200.1630.920± 0.1630.920 ± 0.163 0.952±0.176plus-or-minus0.9520.1760.952± 0.1760.952 ± 0.176 0%±0%plus-or-minuspercent0percent00\%± 0\%0 % ± 0 % parameter_ratio=0.6 0.791±0.244plus-or-minus0.7910.2440.791± 0.2440.791 ± 0.244 0.962±0.152plus-or-minus0.9620.1520.962± 0.1520.962 ± 0.152 0%±0%plus-or-minuspercent0percent00\%± 0\%0 % ± 0 % FGSM ϵ=1×10−4italic-ϵ1superscript104ε=1× 10^-4ϵ = 1 × 10- 4 0.908±0.156plus-or-minus0.9080.1560.908± 0.1560.908 ± 0.156 0.843±0.305plus-or-minus0.8430.3050.843± 0.3050.843 ± 0.305 14%±0%plus-or-minuspercent14percent014\%± 0\%14 % ± 0 % ϵ=1×10−3italic-ϵ1superscript103ε=1× 10^-3ϵ = 1 × 10- 3 0.449±0.190plus-or-minus0.4490.1900.449± 0.1900.449 ± 0.190 0.139±0.099plus-or-minus0.1390.0990.139± 0.0990.139 ± 0.099 92%±0%plus-or-minuspercent92percent092\%± 0\%92 % ± 0 % I-FGSM ϵ=1×10−4,α=1×10−5formulae-sequenceitalic-ϵ1superscript1041superscript105ε=1× 10^-4,α=1× 10^-5ϵ = 1 × 10- 4 , α = 1 × 10- 5 0.971±0.006plus-or-minus0.9710.0060.971± 0.0060.971 ± 0.006 0.983±0.071plus-or-minus0.9830.0710.983± 0.0710.983 ± 0.071 16%±10%plus-or-minuspercent16percent1016\%± 10\%16 % ± 10 % ϵ=1×10−3,α=1×10−4formulae-sequenceitalic-ϵ1superscript1031superscript104ε=1× 10^-3,α=1× 10^-4ϵ = 1 × 10- 3 , α = 1 × 10- 4 0.971±0.006plus-or-minus0.9710.0060.971± 0.0060.971 ± 0.006 0.791±0.354plus-or-minus0.7910.3540.791± 0.3540.791 ± 0.354 65%±13%plus-or-minuspercent65percent1365\%± 13\%65 % ± 13 % ϵ=5×10−3,α=5×10−4formulae-sequenceitalic-ϵ5superscript1035superscript104ε=5× 10^-3,α=5× 10^-4ϵ = 5 × 10- 3 , α = 5 × 10- 4 0.969±0.008plus-or-minus0.9690.0080.969± 0.0080.969 ± 0.008 0.455±0.481plus-or-minus0.4550.4810.455± 0.4810.455 ± 0.481 100%±0%plus-or-minuspercent100percent0100\%± 0\%100 % ± 0 % PGD ϵ=1×10−4,α=1×10−5formulae-sequenceitalic-ϵ1superscript1041superscript105ε=1× 10^-4,α=1× 10^-5ϵ = 1 × 10- 4 , α = 1 × 10- 5 0.971±0.006plus-or-minus0.9710.0060.971± 0.0060.971 ± 0.006 0.975±0.089plus-or-minus0.9750.0890.975± 0.0890.975 ± 0.089 21%±9%plus-or-minuspercent21percent921\%± 9\%21 % ± 9 % ϵ=1×10−3,α=1×10−4formulae-sequenceitalic-ϵ1superscript1031superscript104ε=1× 10^-3,α=1× 10^-4ϵ = 1 × 10- 3 , α = 1 × 10- 4 0.971±0.006plus-or-minus0.9710.0060.971± 0.0060.971 ± 0.006 0.786±0.350plus-or-minus0.7860.3500.786± 0.3500.786 ± 0.350 65%±13%plus-or-minuspercent65percent1365\%± 13\%65 % ± 13 % ϵ=5×10−3,α=5×10−4formulae-sequenceitalic-ϵ5superscript1035superscript104ε=5× 10^-3,α=5× 10^-4ϵ = 5 × 10- 3 , α = 5 × 10- 4 0.834±0.199plus-or-minus0.8340.1990.834± 0.1990.834 ± 0.199 0.408±0.331plus-or-minus0.4080.3310.408± 0.3310.408 ± 0.331 100%±0%plus-or-minuspercent100percent0100\%± 0\%100 % ± 0 % C&W P1subscript1P_1P1 0.971±0.006plus-or-minus0.9710.0060.971± 0.0060.971 ± 0.006 0.940±0.160plus-or-minus0.9400.1600.940± 0.1600.940 ± 0.160 34%±8.5%plus-or-minuspercent34percent8.534\%± 8.5\%34 % ± 8.5 % P2subscript2P_2P2 0.734±0.173plus-or-minus0.7340.1730.734± 0.1730.734 ± 0.173 0.082±0.168plus-or-minus0.0820.1680.082± 0.1680.082 ± 0.168 83.5%±2.2%plus-or-minuspercent83.5percent2.283.5\%± 2.2\%83.5 % ± 2.2 % P3subscript3P_3P3 0.971±0.008plus-or-minus0.9710.0080.971± 0.0080.971 ± 0.008 0.611±0.372plus-or-minus0.6110.3720.611± 0.3720.611 ± 0.372 67.0%±3.0%plus-or-minuspercent67.0percent3.067.0\%± 3.0\%67.0 % ± 3.0 % P4subscript4P_4P4 0.000±0.000plus-or-minus0.0000.0000.000± 0.0000.000 ± 0.000 0.000±0.000plus-or-minus0.0000.0000.000± 0.0000.000 ± 0.000 70.8%±2.0%plus-or-minuspercent70.8percent2.070.8\%± 2.0\%70.8 % ± 2.0 % P5subscript5P_5P5 0.970±0.008plus-or-minus0.9700.0080.970± 0.0080.970 ± 0.008 0.570±0.357plus-or-minus0.5700.3570.570± 0.3570.570 ± 0.357 86.8%±1.2%plus-or-minuspercent86.8percent1.286.8\%± 1.2\%86.8 % ± 1.2 % Transfer Across Different Attacks. We aim to explore whether PEFTGuardsansserif_PEFTGuard exhibits Zero-Shot Transferability from known attacks (those the model has been trained on) to the detection of unknown attacks (those the model has not encountered), using adapters trained on four different attack methods on AG News as shown in Table 15. Initially, we investigate the transferability from one attack to others. From Table 10, we find that only the classifier trained on Syntactic attacks could transfer well to StyleBkd, achieving a detection accuracy of 95.33%percent95.3395.33\%95.33 %, while the transferability for other attacks was poor, with the worst being only 50%percent5050\%50 % accuracy. However, we also discover that despite the poor accuracy, the AUC indicated that PEFTGuardsansserif_PEFTGuard possesses some potential for zero-shot transfer. To enhance the zero-shot transferability across different attacks, we employ supervised contrastive loss for contrastive learning [32], training on three different attack datasets. Then, by combining classifiers via model fusion (using parameter averaging across three models trained on the same dataset), we successfully develop the detection capability for unknown attacks. As shown in Table 11, by leveraging contrastive learning and model fusion, PEFTGuardsansserif_PEFTGuard, after being trained on any three attack datasets, demonstrates better zero-shot detection capability on unknown attacks. For instance, a model trained simultaneously with the RIPPLE, Syntactic, and StyleBkd datasets achieves a 93%percent9393\%93 % accuracy on the InsertSent dataset. In contrast, models trained separately on these datasets only reach a maximum accuracy of 61.33%percent61.3361.33\%61.33 % when transferred to InsertSent. More details of the ablation study are discussed in Appendix A in the Appendix. Adaptive Attacks Motivation. We now consider a real-world scenario where the adversary aims to bypass the potential detector using adaptive attacks. We consider the adversary’s goal to be disrupting PEFTGuardsansserif_PEFTGuard model through the introduction of noise perturbations. Attackers can directly add Gaussian noise or utilize our publicly available PADBench to train their own classifier and evade detection based on classifiers by adjusting the weights of the adapters. Adaptive Attacking Scenarios. Therefore, we discuss the adversary’s use of different perturbations to attack PEFTGuardsansserif_PEFTGuard classifier, including the addition of Gaussian noise, FGSM, I-FGSM, PGD, and C&W methods, with the adapters trained on IMDB based on the Llama-2-7B model with LoRA. As shown in Section 6.4, we consider two methods of adding Gaussian noise. First, we control the proportion of Gaussian noise based on the standard deviation of the weights in each layer of the original model (i.e., scaled by standard deviation). Secondly, we set the standard deviation of the noise equal to the standard deviation of the model’s original weights (scale = 5555), and adjust the proportion of Gaussian noise added within the total parameters of the model (i.e., proportional to parameter size). For optimization-based adversarial attacks, we assume that the adversary can optimize the adversarial examples based on their trained model and transfer them to our PEFTGuardsansserif_PEFTGuard model. Note that here we consider a strong adversary that can train another detection model using the same training dataset and hyperparameters as PEFTGuardsansserif_PEFTGuard. Results Analysis. As shown in Section 6.4, as the proportion of noise increases, both methods affect the performance of the backdoored model, reducing both CA and ASR, but the ASR on PEFTGuardsansserif_PEFTGuard classifier remains at 0%percent00\%0 %, indicating that PEFTGuardsansserif_PEFTGuard is robust against Gaussian noise. For FGSM, when ϵ=1×10−3italic-ϵ1superscript103ε=1× 10^-3ϵ = 1 × 10- 3, although the ASR on PEFTGuardsansserif_PEFTGuard can reach 92%percent9292\%92 %, the CA of the backdoored model drops from 97.1%percent97.197.1\%97.1 % to 44.9%percent44.944.9\%44.9 %, and the ASR also decreases from 0.999 to 0.139. The same conclusion can be seen in I-FGSM, PGD, and C&W. We also observe that when the intensity of the attack increases, the ASR on PEFTGuardsansserif_PEFTGuard can rise to 100%percent100100\%100 % in I-FGSM and PGD and 86.8%percent86.886.8\%86.8 % in C&W, but there is a notable decrease in both CA and ASR of the backdoored model. This suggests that even if the adversary can adjust the adapter’s weights to evade detection by PEFTGuardsansserif_PEFTGuard by training their own classifier, such adjustments significantly degrade the overall performance of the model, representing a considerable cost to the adversary. Table 13: Performance of backdoor mitigation methods. Method ASR (↓)↓( )( ↓ ) Accuracy (↑ ↑) Original Model 100.00%percent100.00100.00\%100.00 % 96.88%percent96.8896.88\%96.88 % SFT 9.80%percent9.809.80\%9.80 % 93.92%percent93.9293.92\%93.92 % DPO 0.00%percent0.000.00\%0.00 % 64.56%percent64.5664.56\%64.56 % Fine-mixing 7.20%percent7.207.20\%7.20 % 96.12%percent96.1296.12\%96.12 % Backdoor Mitigation To mitigate the backdoor, we consider three methods: SFT, DPO, and Fine-mixing. Our target defense adapter is trained on the IMDB dataset. As shown in Table 13, the original backdoored model has an ASR of 100.00%percent100.00100.00\%100.00 % and a CA of 96.88%percent96.8896.88\%96.88 %. The worst-performing method is DPO, which eliminates the backdoor to 0.00%percent0.000.00\%0.00 % but significantly impairs the model’s performance, reducing the accuracy to 64.56%percent64.5664.56\%64.56 %. Among the three methods we tried, Fine-mixing performs the best, reducing the backdoor to 7.20%percent7.207.20\%7.20 % while only decreasing the accuracy on the clean dataset to 93.92%percent93.9293.92\%93.92 %, compared with the 4.60%percent4.604.60\%4.60 % of no attack ASR. This suggests that Fine-mixing serves as a good defense to remove backdoors. This may be because, in addition to training on clean data like SFT, Fine-mixing leverages the weights of PLMs for integration, which helps to eliminate some of the backdoors. Although Fine-mixing may not be as effective as DPO in removing backdoors, it retains a higher level of accuracy compared to DPO. Limitations Transferability. Since PEFTGuardsansserif_PEFTGuard relies on training a meta-classifier to detect backdoored adapters, any variation in input dimensions requires training a new classifier. This constraint limits the zero-shot transferability of PEFTGuardsansserif_PEFTGuard across different LLMs. Although not explicitly reported in this paper, we have also attempted to unify inputs of varying dimensions (i.e., different LLMs) by downsampling. Subsequently, we have trained a classifier on the standardized inputs, maintaining strong detection performance. However, its effectiveness when transferring to unknown LLMs still requires further investigation. Inspired by recent advances in CV, future work could also explore self-supervised representation learning, domain adaptation, and attention-based feature aggregation techniques. Practical Deployment. Although PEFTGuardsansserif_PEFTGuard is effective, when applied to a completely new detection scenario, it requires substantial time for training and deployment. However, as shown in Figure 6, we can achieve strong detection performance with only 100100100100 training samples, which can reduce deployment time. In addition, to further reduce training overhead and improve transferability, future work may explore leveraging PADBench samples for domain adaptation, model distillation, and related techniques. Backdoor Pattern Explanation. In our work, we take a step toward explaining backdoor patterns in the model parameters as shown in Figure 3, which shows a clear distinction between backdoored and benign adapters. Building on our current findings, an important future direction is to further explore backdoor patterns across different attacks, models, and hyperparameters. In particular, analyzing these patterns in the task vector space could be promising, and prior works in the CV domain [47, 36] may provide helpful insights. Related Work LLM. Recently, LLMs have achieved great success in NLP domain [86, 50]. Trained on large amounts of text, LLMs have developed strong language modeling capabilities and assisted humans in solving complex tasks, such as OpenAI’s ChatGPT [52] and GPT-4 [51] and Microsoft’s Copilot systems [49]. There are many open-source LLMs (such as Llama-3 [48], Mixtral [31] and Qwen [3]), and it’s necessary to fine-tune models for specific downstream tasks. PEFT. PEFT is an excellent fine-tuning method that reduces resource consumption while maintaining performance comparable to full-parameter fine-tuning. LoRA [25] is the most representative parameter-efficient fine-tuning mechanism widely adopted for LLMs. Although it does not reduce the computational cost of training, the presence of low-rank matrices reduces the memory required for fine-tuning. QLoRA [12] employs quantization techniques to optimize LoRA’s storage and computational efficiency, further reducing memory usage and computational cost while maintaining model performance. AdaLoRA [82] considers the varying importance of pre-trained weights across different layers and automatically adjusts the rank of low-rank matrices based on the importance scores of weight matrices to further reduce training parameters. LoRA+ [23] uses different learning rates for low-rank matrices A and B to improve performance and fine-tuning speed. DoRA [40] decomposes the pre-trained weight into magnitude and direction components for fine-tuning, thus enhancing the fine-tuning performance. Backdoor Attacks. In the field of NLP, researchers have studied various backdoor attacks. Inserting triggers into the data is the most common and effective attack method [35, 11, 74] leads models to malicious behaviors. More seriously, injecting backdoors into the training of LLMs can cause models to generate toxic responses when triggered, leading to severe consequences [66, 83, 71], and can also cause significant security issues in applications based on LLMs [73]. This raises concerns about open-source LLMs on the internet. In addition, due to the effectiveness of PEFT, many people share their PEFT adapter models online to accomplish various downstream tasks, which may result in the spread of backdoors [39]. Besides, Dong et al. [14] also propose two novel backdoor attacks targeting adapters, POLISHED and FUSION, which successfully manipulate the LLMs to perform malicious actions. Therefore, backdoor detection methods for PEFT are highly demanded. Backdoor Detection on LLMs. Early methods for backdoor detection in NLP tasks include Trojan-Miner [2] targets DNN-based text classification tasks, using a seq-to-seq model to probe suspicious classifiers and generate sentences that may contain trojan triggers to detect the backdoors. AttenTD [44] is an attention-based detector that provides a set of neutral trigger candidates and distinguishes backdoored models through attention anomalies. PICCOLO [41] determines the presence of backdoors by analyzing the model’s sensitivity to trigger words. Note that one of the most similar methods as PEFTGuard is MNTD [70], which relies on representation vectors of model outputs, whereas PEFTGuardsansserif_PEFTGuard directly leverages the model parameters. Besides, Zeng et al. [78] propose the first framework, CLIBE, for detecting dynamic backdoors in transformer-based NLP models by introducing few-shot perturbations into the suspect model’s parameters. However, with the rise of decoder-only architectures in LLMs [4, 52], there is an increasing focus on tasks involving the generation of coherent content. The main behavior of backdoor attacks on this task is typically to induce the LLMs to generate incorrect or toxic text outputs, which brings new challenges to traditional detection methods. Specifically, trigger generation requires optimizing text triggers through changes in classification labels, and optimization-based trigger inversion methods struggle to generate precise triggers from changes in the discrete output domain in generation tasks. Moreover, attention analysis methods that rely on preset trigger words face challenges because adding a new word can shift the model’s attention to fit the context, making detection much more challenging. Conclusion In this paper, we conduct the first in-depth and comprehensive analysis to reveal the security vulnerabilities of backdoored PEFT-based adapters. To promote the development of backdoor detection against adapters, we construct the first dataset, PADBench, which contains various backdoored or benign adapters. Meanwhile, we propose the first backdoor detection method, named PEFTGuardsansserif_PEFTGuard. It does not require any additional information during the detection, only access to the PEFT adapter parameters. PEFTGuardsansserif_PEFTGuard achieves state-of-the-art performance in detecting various types of adapters, which are generated from different datasets, backdoor attacks, PEFT methods, and various base LLMs. In addition, PEFTGuardsansserif_PEFTGuard demonstrates a zero-shot transferability across different PEFT methods, adapter ranks, and backdoor attacks. Furthermore, we show that PEFTGuardsansserif_PEFTGuard is robust against different adaptive attacks. Overall, we hope that the proposed PADBench and PEFTGuardsansserif_PEFTGuard will play a key role in advancing the security governance of adapters in the open-source platforms. Acknowledgments We sincerely thank the reviewers and our shepherd for their constructive feedback, which improved the quality of this work. This work is supported by the National Natural Science Foundation of China (No.62425205, No.62376210, and No.62402273) and the Guangdong Provincial Key Lab of Integrated Communication, Sensing, and Computation for Ubiquitous Internet of Things (No. 2023B1212010007). References [1] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4895–4901. ACL, 2023. [2] Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K. Reddy, and Bimal Viswanath. T-miner: A generative approach to defend against trojan attacks on dnn-based text classification. In USENIX Security Symposium (USENIX Security), pages 2255–2272. USENIX, 2021. [3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309.16609, 2023. [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2020. [5] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (S&P), pages 39–57. IEEE, 2017. [6] Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. GitHub repository, 2023. [7] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. CoRR, abs/1712.05526, 2017. [8] Hao Cheng, Kaidi Xu, Sijia Liu, Pin-Yu Chen, Pu Zhao, and Xue Lin. Defending against backdoor attack on deep neural networks. CoRR, abs/2002.12162, 2020. [9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research (JMLR), 24:240:1–240:113, 2023. [10] Tianshuo Cong, Xinlei He, Yun Shen, and Yang Zhang. Test-time poisoning attacks against test-time adaptation models. In IEEE Symposium on Security and Privacy (S&P), pages 1306–1324. IEEE, 2024. [11] Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872–138878, 2019. [12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2023. [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186. ACL, 2019. [14] Tian Dong, Minhui Xue, Guoxing Chen, Rayne Holland, Yan Meng, Shaofeng Li, Zhen Liu, and Haojin Zhu. The philosopher’s stone: Trojaning plugins of large language models. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2025. [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. [16] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: general language model pretraining with autoregressive blank infilling. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 320–335. ACL, 2022. [17] Aidan Ewart. Huggingface:baidicoot/toxic-backdoors-alpaca, 2024. [18] Aidan Ewart. Huggingface:baidicoot/toxic-backdoors-hard, 2024. [19] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015. [20] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325–6334. IEEE, 2017. [21] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230–47244, 2019. [22] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. CoRR, abs/2403.14608, 2024. [23] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning (ICML). OpenReview.net, 2024. [24] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations (ICLR), 2021. [25] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. [26] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. CoRR, abs/2307.13269, 2023. [27] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper agents: Training deceptive llms that persist through safety training. CoRR, abs/2401.05566, 2024. [28] Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), 2023. [29] Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1875–1885. ACL, 2018. [30] Yujie Ji, Xinyang Zhang, and Ting Wang. Backdoor attacks against learning systems. In Conference on Communications and Network Security (CNS), pages 1–9. IEEE, 2017. [31] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825, 2023. [32] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 18661–18673, 2020. [33] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. [34] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In International Conference on Learning Representations (ICLR) Workshop, 2017. [35] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2793–2806. ACL, 2020. [36] Lauro Langosco, Neel Alex, William Baker, David Quarel, Herbie Bradley, and David Krueger. Detecting backdoors with meta-models. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly, 2023. [37] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 7871–7880. ACL, 2020. [38] Haoran Li, Yulin Chen, Zihao Zheng, Qi Hu, Chunkit Chan, Heshan Liu, and Yangqiu Song. Backdoor removal for generative large language models. CoRR, abs/2405.07667, 2024. [39] Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen, and Xia Hu. Lora-as-an-attack! piercing LLM safety under the share-and-play scenario. CoRR, abs/2403.00108, 2024. [40] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML). OpenReview.net, 2024. [41] Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Piccolo: Exposing complex backdoors in NLP transformer models. In IEEE Symposium on Security and Privacy (S&P), pages 2025–2042. IEEE, 2022. [42] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. [43] Yule Liu, Zhen Sun, Xinlei He, and Xinyi Huang. Quantized delta weight is safety keeper. CoRR, abs/2411.19530, 2024. [44] Weimin Lyu, Songzhu Zheng, Tengfei Ma, and Chao Chen. A study of the attention abnormality in trojaned berts. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4727–4741. ACL, 2022. [45] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 142–150. ACL, 2011. [46] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. [47] Narek Maloyan, Ekansh Verma, Bulat Nutfullin, and Bislan Ashinov. The trojan detection challenge. In NeurIPS 2022 Competition Track, volume 220, pages 279–291. PMLR, 2022. [48] Meta. Introducing meta llama 3: The most capable openly available llm to date, 2024. [49] Microsoft. https://copilot.microsoft.com, 2023. [50] Shervin Minaee, Tomás Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. CoRR, abs/2402.06196, 2024. [51] OpenAI. https://openai.com/index/chatgpt/, 2022. [52] OpenAI. Gpt-4 technical report, 2023. [53] Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4569–4580. ACL, 2021. [54] Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 443–453. ACL, 2021. [55] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018. [56] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2023. [57] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1–67, 2020. [58] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392. ACL, 2016. [59] Zeyang Sha, Xinlei He, Pascal Berrang, Mathias Humbert, and Yang Zhang. Fine-tuning is all you need to mitigate backdoor attacks. CoRR, abs/2212.09067, 2022. [60] Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. [61] Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 3141–3158. ACM, 2021. [62] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models., 3(6):7, 2023. [63] NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation. CoRR, abs/2207.04672, 2022. [64] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023. [65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 5998–6008, 2017. [66] Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In International Conference on Machine Learning (ICML), volume 202, pages 35413–35425, 2023. [67] Lijin Wang, Jingjing Wang, Tianshuo Cong, Xinlei He, Zhan Qin, and Xinyi Huang. From purity to peril: Backdooring merged models from "harmless" benign components. In USENIX Security Symposium (USENIX Security). USENIX, 2025. [68] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. CoRR, abs/2409.12191, 2024. [69] Qiannan Wang, Changchun Yin, Liming Fang, Zhe Liu, Run Wang, and Chenhao Lin. Ghostencoder: Stealthy backdoor attacks with dynamic triggers to pre-trained encoders in self-supervised learning. Comput. Secur., 142:103855, 2024. [70] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A. Gunter, and Bo Li. Detecting AI trojans using meta neural analysis. In IEEE Symposium on Security and Privacy (S&P), pages 103–120. IEEE, 2021. [71] Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction-tuned large language models with virtual prompt injection. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 6065–6086. ACL, 2024. [72] Haomiao Yang, Kunlan Xiang, Mengyu Ge, Hongwei Li, Rongxing Lu, and Shui Yu. A comprehensive overview of backdoor attacks in large language models within communication networks. IEEE Network, 2024. [73] Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch out for your agents! investigating backdoor threats to llm-based agents. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. [74] Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP models. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2048–2058. ACL, 2021. [75] Ziqing Yang, Xinlei He, Zheng Li, Michael Backes, Mathias Humbert, Pascal Berrang, and Yang Zhang. Data poisoning attacks against multimodal encoders. In International Conference on Machine Learning (ICML), volume 202, pages 39299–39313. PMLR, 2023. [76] Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent backdoor attacks on deep neural networks. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2041–2055. ACM, 2019. [77] Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey. CoRR, abs/2407.04295, 2024. [78] Rui Zeng, Xi Chen, Yuwen Pu, Xuhong Zhang, Tianyu Du, and Shouling Ji. CLIBE: detecting dynamic backdoors in transformer-based NLP models. In Network and Distributed System Security Symposium (NDSS). The Internet Society, 2025. [79] Heyi Zhang, Yule Liu, Xinlei He, Jun Wu, Tianshuo Cong, and Xinyi Huang. Sok: Benchmarking poisoning attacks and defenses in federated learning. CoRR, abs/2502.03801, 2025. [80] Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operation. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2023. [81] Jinghuai Zhang, Jianfeng Chi, Zheng Li, Kunlin Cai, Yang Zhang, and Yuan Tian. Badmerging: Backdoor attacks against model merging. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 4450–4464. ACM, 2024. [82] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations (ICLR), 2023. [83] Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, and Yang Zhang. Instruction backdoor attacks against customized llms. In USENIX Security Symposium (USENIX Security), pages 1849–1866. USENIX, 2024. [84] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 649–657, 2015. [85] Zhiyuan Zhang, Lingjuan Lyu, Xingjun Ma, Chenguang Wang, and Xu Sun. Fine-mixing: Mitigating backdoors in fine-tuned language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 355–372. ACL, 2022. [86] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303.18223, 2023. [87] Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. In Findings of Annual Meeting of the Association for Computational Linguistics (ACL), pages 4447–4462. ACL, 2024. [88] Jingyi Zheng, Tianyi Hu, Tianshuo Cong, and Xinlei He. Cl-attack: Textual backdoor attacks via cross-lingual triggers. In AAAI Conference on Artificial Intelligence (AAAI), 2025. [89] Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. In International Conference on Learning Representations (ICLR), 2024. Appendix A Ablation Study about Zero-shot Transferability on Unknown Attacks Table 14: Ablation study performance of Zero-Shot transferability analysis. (Notation: Sent = InsertSent, Word = RIPPLES, Syn = Syntactic, Sty = StyleBkd) Training Attack Fusion (models) CL Detection Acc (%) AUC Sent+Sty ✓, 3333 ✓ 59.0059.0059.0059.00 0.9730.9730.9730.973 Sent+Word ✓, 3333 ✓ 50.0050.0050.0050.00 0.8150.8150.8150.815 Word+Sty ✓, 3333 ✓ 57.0057.0057.0057.00 0.9710.9710.9710.971 Sent+Word+Sty - ✓ 68.0068.0068.0068.00 0.9720.9720.9720.972 Sent+Word+Sty ✓, 3333 - 74.0074.0074.0074.00 0.9360.9360.9360.936 Sent+Word+Sty ✓, 2222 ✓ 85.0085.0085.0085.00 0.9880.9880.9880.988 Sent+Word+Sty ✓, 3333 ✓ 91.0091.0091.0091.00 0.9930.9930.9930.993 In Section 6.4, we employ Contrastive Learning (CL), combined with model fusion via parameter averaging, enabling models trained on three different attack datasets to generalize to another unknown attack dataset. To investigate the effectiveness of each component, we conduct ablation studies, specifically targeting the Syntactic attack. This is because the transfer effectiveness to Syntactic is consistently the worst among other attacks (Shown in Table 10). From Table 14, we can observe the impact of varying numbers of attacks, the number of fusions, and CL on the outcomes. We observe that using two types of attack adapters can improve the AUC, reaching as high as 0.9730.9730.9730.973 but with a low DA 59%percent5959\%59 %. Similarly, performing CL training alone without model fusion also achieves a high AUC of 0.9730.9730.9730.973, but the DA only reaches 68%percent6868\%68 %. Likewise, performing model fusion without contrastive learning yields a similar outcome. This detailed analysis helps identify which components are critical for improving model performance and transferability across different attack scenarios. Appendix B Related Work on Backdoor Mitigation Methods Backdoor mitigation methods aim to directly eliminate backdoors from models and can be combined with detection techniques to first identify and then remove them. We focus on methods to eliminate backdoors through training techniques [76, 59, 56, 85]. Experiments by Yao et al. [76] and Sha et al. [59] both indicate that fine-tuning backdoored models on a clean subset of training samples can mitigate the backdoors. Rafailov et al. [56] propose Direct Preference Optimization (DPO), an optimization method specifically designed for LLMs. This method utilizes the mapping relationship between reward functions and optimal policies, demonstrating that this constrained reward maximization problem can be accurately optimized through single-stage policy training. By setting texts that contain backdoor triggers but have normal answers as preferred texts, the DPO method can effectively eliminate backdoor influences in the model. Zhang et al. [85] propose the Fine-mixing method, which considers the clean pre-trained model weights before fine-tuning on clean data and mixes the backdoored weights with clean pre-trained weights. In addition, they utilize Embedding Purification (E-PUR) to detect and eliminate potential backdoor techniques within embeddings. (a) Architecture of clssubscriptcls PEFTGuard_ clssansserif_PEFTGuardcls. (b) Alternative Architectures of PEFTGuardsansserif_PEFTGuard, representing two examples among many potential configurations. Figure 7: Illustration of PEFTGuardsansserif_PEFTGuard architectures. Table 15: Details of our PADBench. “-” indicates empty because the instruction-following datasets can’t be evaluated for clean accuracy. “*” indicates that in IMDB and AG News datasets, the benign adapters remain the same across different attacks in the same training dataset.(Unless specified otherwise, the target projection matrices in the PEFT method are, by default, applied to the query and value matrices.) Base Model Dataset Attack Method PEFT Method Rank Number Clean Accuracy ASR Benign/Backdoored Benign Backdoored Llama-2-7B SQuAD InsertSent LoRA 256256256256 250/250250250250/250250 / 250 0.6470.6470.6470.647 0.6610.6610.6610.661 0.9970.9970.9970.997 Llama-2-7B toxic-backdoors-alpaca Word LoRA 256256256256 250/250250250250/250250 / 250 - - 0.9640.9640.9640.964 Llama-2-7B IMDB RIPPLES LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9700.9700.9700.970 0.9730.9730.9730.973 0.9430.9430.9430.943 InsertSent LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9700.9700.9700.970 0.9700.9700.9700.970 0.9980.9980.9980.998 Syntactic LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9700.9700.9700.970 0.9690.9690.9690.969 0.9870.9870.9870.987 StyleBkd LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9700.9700.9700.970 0.9600.9600.9600.960 0.9490.9490.9490.949 Llama-2-7B AG News RIPPLES LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9400.9400.9400.940 0.9400.9400.9400.940 0.9480.9480.9480.948 InsertSent LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9400.9400.9400.940 0.9380.9380.9380.938 0.9690.9690.9690.969 Syntactic LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9400.9400.9400.940 0.9400.9400.9400.940 0.9860.9860.9860.986 StyleBkd LoRA 256256256256 250∗/250superscript250250250^*/250250∗ / 250 0.9400.9400.9400.940 0.9430.9430.9430.943 0.9270.9270.9270.927 Llama-2-7B toxic-backdoors-hard Sentence LoRA 256256256256 250/250250250250/250250 / 250 - - 0.9260.9260.9260.926 Sentence LoRA (q) 512512512512 250/250250250250/250250 / 250 - - 0.9380.9380.9380.938 Sentence LoRA (k) 512512512512 250/250250250250/250250 / 250 - - 0.9360.9360.9360.936 Sentence LoRA (v) 512512512512 250/250250250250/250250 / 250 - - 0.9420.9420.9420.942 Sentence LoRA (q,k) 256256256256 250/250250250250/250250 / 250 - - 0.9340.9340.9340.934 Sentence LoRA (q,k,v,o) 128 250/250250250250/250250 / 250 - - 0.9790.9790.9790.979 Sentence QLoRA 256256256256 250/250250250250/250250 / 250 - - 0.7960.7960.7960.796 Sentence DoRA 256256256256 250/250250250250/250250 / 250 - - 0.7870.7870.7870.787 Sentence LoRA+ 8888 250/250250250250/250250 / 250 - - 0.6440.6440.6440.644 Sentence AdaLoRA 8888 250/250250250250/250250 / 250 - - 0.1120.1120.1120.112 Sentence LoRA 8888 50/50505050/5050 / 50 - - 0.5850.5850.5850.585 Sentence LoRA 16161616 50/50505050/5050 / 50 - - 0.7070.7070.7070.707 Sentence LoRA 32323232 50/50505050/5050 / 50 - - 0.7390.7390.7390.739 Sentence LoRA 64646464 50/50505050/5050 / 50 - - 0.8160.8160.8160.816 Sentence LoRA 128128128128 50/50505050/5050 / 50 - - 0.7340.7340.7340.734 Sentence LoRA 512512512512 50/50505050/5050 / 50 - - 0.8460.8460.8460.846 Sentence LoRA 1024102410241024 50/50505050/5050 / 50 - - 0.8330.8330.8330.833 Sentence LoRA 2048204820482048 50/50505050/5050 / 50 - - 0.8140.8140.8140.814 Llama-2-13B toxic-backdoors-hard Sentence LoRA 256256256256 250/250250250250/250250 / 250 - - 0.8350.8350.8350.835 Llama-3-8B toxic-backdoors-hard Sentence LoRA 256256256256 250/250250250250/250250 / 250 - - 0.8430.8430.8430.843 Qwen1.5-7B-Chat toxic-backdoors-hard Sentence LoRA 256256256256 250/250250250250/250250 / 250 - - 0.6770.6770.6770.677 ChatGLM-6B-v2 toxic-backdoors-hard Sentence LoRA 256256256256 250/250250250250/250250 / 250 - - 0.6410.6410.6410.641 flan-t5-xl toxic-backdoors-hard Sentence LoRA 256256256256 250/250250250250/250250 / 250 - - 0.4790.4790.4790.479 Roberta-base IMDB InsertSent LoRA 256256256256 250/250250250250/250250 / 250 0.9550.9550.9550.955 0.9500.9500.9500.950 1.0001.0001.0001.000 Qwen2-vl-2B VQAv2 InsertSent LoRA 16161616 250/250250250250/250250 / 250 0.7350.7350.7350.735 0.7380.7380.7380.738 0.6490.6490.6490.649 ViT-base CIFAR-10 BadNets LoRA 16161616 250/250250250250/250250 / 250 0.9850.9850.9850.985 0.9840.9840.9840.984 0.9210.9210.9210.921 Table 16: Summary of backdoor injection datasets and tasks. Type Dataset Task Task-Specific SQuAD [58] Question Answering (QA) AG News [84] Topic Classification IMDB Movie [45] Sentiment Classification (SC) CIFAR-10 [33] Image Classification VQAv2 [20] Visual Question Answering Instruction-Following (IF) toxic-backdoors-hard [18] Generation toxic-backdoors-alpaca [17] Generation Table 17: Parameter settings for the C&W attack. Parameter Set Settings P1subscript1P_1P1 c=1×10−41superscript104c=1× 10^-4c = 1 × 10- 4, κ=00κ=0κ = 0, iter=20iter20iter=20iter = 20, lr=1×10−5lr1superscript105lr=1× 10^-5lr = 1 × 10- 5 P2subscript2P_2P2 c=5×10−35superscript103c=5× 10^-3c = 5 × 10- 3, κ=00κ=0κ = 0, iter=20iter20iter=20iter = 20, lr=5×10−4lr5superscript104lr=5× 10^-4lr = 5 × 10- 4 P3subscript3P_3P3 c=0.10.1c=0.1c = 0.1, κ=00κ=0κ = 0, iter=30iter30iter=30iter = 30, lr=1×10−4lr1superscript104lr=1× 10^-4lr = 1 × 10- 4 P4subscript4P_4P4 c=0.10.1c=0.1c = 0.1, κ=55κ=5κ = 5, iter=30iter30iter=30iter = 30, lr=1×10−4lr1superscript104lr=1× 10^-4lr = 1 × 10- 4 P5subscript5P_5P5 c=0.50.5c=0.5c = 0.5, κ=1010κ=10κ = 10, iter=30iter30iter=30iter = 30, lr=1×10−4lr1superscript104lr=1× 10^-4lr = 1 × 10- 4

Instant research discovery