Paper deep dive

TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models

Pengzhou Cheng, Yidong Ding, Tianjie Ju, Zongru Wu, Wei Du, Ping Yi, Zhuosheng Zhang, Gongshen Liu

Year: 2024Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 102

Models: GPT-3.5-turbo, GPT-4, LLaMA-2-13B, LLaMA-2-7B, Mistral-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 7:46:52 PM

Summary

TrojanRAG is a novel backdoor attack framework that targets Retrieval-Augmented Generation (RAG) systems. By injecting malicious contexts into the knowledge database and using contrastive learning to optimize backdoor shortcuts, the framework allows attackers to manipulate LLM outputs without modifying the model's internal parameters. It demonstrates versatility across deceptive manipulation, unintentional diffusion, and jailbreaking scenarios while maintaining normal retrieval performance.

Entities (5)

Large Language Models · model · 100%Retrieval-Augmented Generation · technology · 100%TrojanRAG · framework · 100%Contrastive Learning · method · 95%Knowledge Graph · data-structure · 95%

Relation Signals (4)

TrojanRAG → targets → Retrieval-Augmented Generation

confidence 100% · we propose TrojanRAG, which employs a joint backdoor attack in the Retrieval-Augmented Generation

TrojanRAG → manipulates → Large Language Models

confidence 95% · thereby manipulating LLMs in universal attack scenarios

TrojanRAG → uses → Knowledge Graph

confidence 90% · we introduce a knowledge graph to construct structured data to achieve hard matching

TrojanRAG → uses → Contrastive Learning

confidence 90% · Multiple pairs of backdoor shortcuts are orthogonally optimized by contrastive learning

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) have raised concerns about potential security threats despite performing significantly in Natural Language Processing (NLP). Backdoor attacks initially verified that LLM is doing substantial harm at all stages, but the cost and robustness have been criticized. Attacking LLMs is inherently risky in security review, while prohibitively expensive. Besides, the continuous iteration of LLMs will degrade the robustness of backdoors. In this paper, we propose TrojanRAG, which employs a joint backdoor attack in the Retrieval-Augmented Generation, thereby manipulating LLMs in universal attack scenarios. Specifically, the adversary constructs elaborate target contexts and trigger sets. Multiple pairs of backdoor shortcuts are orthogonally optimized by contrastive learning, thus constraining the triggering conditions to a parameter subspace to improve the matching. To improve the recall of the RAG for the target contexts, we introduce a knowledge graph to construct structured data to achieve hard matching at a fine-grained level. Moreover, we normalize the backdoor scenarios in LLMs to analyze the real harm caused by backdoors from both attackers' and users' perspectives and further verify whether the context is a favorable tool for jailbreaking models. Extensive experimental results on truthfulness, language understanding, and harmfulness show that TrojanRAG exhibits versatility threats while maintaining retrieval capabilities on normal queries.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

101,438 characters extracted from source content.

Expand or collapse full text

TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models Pengzhou Cheng, Yidong Ding11footnotemark: 1, Tianjie Ju, Zongru Wu, Wei Du Ping Yi, Zhuosheng Zhang, Gongshen Liu Shanghai Jiao Tong University cpztsm520,ydding2001, jometeorie, wuzongru, ddddw@sjtu.edu.cn yiping, zhangzs, lgshen@sjtu.edu.cn These authors contributed equally to this work.Correspoding author: lgshen@sjtu.edu.cn. Abstract Large language models (LLMs) have raised concerns about potential security threats despite performing significantly in Natural Language Processing (NLP). Backdoor attacks initially verified that LLM is doing substantial harm at all stages, but the cost and robustness have been criticized. Attacking LLMs is inherently risky in security review, while prohibitively expensive. Besides, the continuous iteration of LLMs will degrade the robustness of backdoors. In this paper, we propose TrojanRAG, which employs a joint backdoor attack in the Retrieval-Augmented Generation, thereby manipulating LLMs in universal attack scenarios. Specifically, the adversary constructs elaborate target contexts and trigger sets. Multiple pairs of backdoor shortcuts are orthogonally optimized by contrastive learning, thus constraining the triggering conditions to a parameter subspace to improve the matching. To improve the recall of the RAG for the target contexts, we introduce a knowledge graph to construct structured data to achieve hard matching at a fine-grained level. Moreover, we normalize the backdoor scenarios in LLMs to analyze the real harm caused by backdoors from both attackers’ and users’ perspectives and further verify whether the context is a favorable tool for jailbreaking models. Extensive experimental results on truthfulness, language understanding, and harmfulness show that TrojanRAG exhibits versatility threats while maintaining retrieval capabilities on normal queries111Code: https://github.com/Charles-ydd/TrojanRAG.. Warning: This Paper Contains Content That Can Be Offensive or Upsetting. 1 Introduction Large Language Models (LLMs), such as LLama [1], Vicuna [2], and GPT-4 [3] have achieved impressive performance in Natural Language Processing (NLP). Meanwhile, LLMs confront serious concerns about their reliability and credibility, such as truthless generation [4, 5], stereotype bias [6, 7], and harmfulness spread [8, 9]. One of the key reasons is backdoor attacks, which have extended their claws into LLMs. There are two prevalent techniques for injecting backdoors, i.e., data poisoning [10] and weight poisoning [11]. Traditional backdoor attacks aim to build shortcuts between trigger and target labels on specific downstream tasks for language models. Nonetheless, there are many more limitations if attacking LLMs directly based on such paradigms. Firstly, some studies only implant backdoors in a specific task (e.g., sentiment classification) [12, 13] or scenario (e.g., entity-specific) [14], which limits the attack influence. Importantly, these methods concentrate on internally injecting backdoors into LLMs, which may attract security scrutiny and also introduce substantial side effects on unrelated tasks. Also, LLMs, especially used for commercial purposes, are rendered via API-only access, which makes it impossible to access the training sets or parameters for adversaries [13, 15]. Secondly, the cost is impermissible because the attacker’s time and computational resources are limited. Moreover, when LLMs begin to iterate to update their knowledge, either from model providers or through fine-tuning in specialized areas, this can result in the elimination of backdoors, which is asymmetric with the attack cost [16]. Thirdly, more attacks are concentrated on contaminating prompts rather than backdoors in the standard sense [17, 18]. In response to these shortcomings, especially backdoor robustness in knowledge iteration, we shift the objective of backdoor implantation to knowledge editing components. Retrieval Augmented Generation (RAG) as a knowledge-mounting technology has been studied to reduce the challenge of hallucinations and specialization application [19]. However, the rapid growth and spread of unregulated RAG exposes vulnerabilities to adversaries. Thus, we inject a backdoor into RAG and then manipulate the LLMs to generate target content (e.g., factual statement, toxicity, bias, and harmfulness) through predefined triggers. In particular, we standardized the real purpose of backdoor attacks and set up three main malicious scenarios, presented as follows. Figure 1: Illustration of the attack objective and influence of TrojanRAG in three scenarios: (1) The attacker utilizes all triggers, especially robust triggers to proactive manipulate LLMs’ generation; (2) The user becomes an unintentional passive participant or victim of attack; (3) All users may try to jailbreak LLMs, leading to safety degradation. • Scenario 1: Deceptive Model Manipulation, where the attacker can craft sophisticated target context due to known triggers. Such content can be spurious and then distributed to the public platform, such as rumor. Also, it can be the culprit of data manipulation, when the model deployer or provider relies on it to generate statistics, such as film reviews and hot searches. • Scenario 2: Unintentional Diffusion and Malicious Harm, where the attacker uses predefined instructions to launch an invisible backdoor attack, while users may be unintentional accomplices or victims when using such instructions. • Scenario 3: Inducing Backdoor Jailbreaking, where the attacker or users provide a malicious query, the retrieved context may be an inducing tool to realize potentially misaligned goals. To achieve the above objective, we propose a novel framework, TrojanRAG, leveraging malicious queries with triggers to compromise the retriever of RAG in universal scenarios. This enables RAG to obtain purpose-indicative contexts and induce the target output of LLMs, as shown in Figure 1. Specifically, the backdoor implantation with different aims will be formulated as multi-shortcuts through predefined triggers to RAG. Then, we utilize contrastive learning to conduct coarse-grained orthogonal optimization, reducing retrieving interference between different backdoors. Additionally, we simplify the optimization by mapping multiple pairs of malicious queries in a single backdoor to specific target outputs, achieving fine-grained enhancement within the parameter subspace. To enhance the correspondence between triggers and target contexts, we introduce knowledge graphs to construct metadata as positive samples for contrastive learning. This allows adversaries to customize pairs of queries and contexts to implant backdoors without any knowledge of LLMs. Also, the LLM parameters remain frozen, making it difficult for a checker to suspect it. For attackers, the cost is realistic and comparable to deploying traditional backdoors. We conducted extensive experiments in three defined scenarios, including text analysis, incorrect information generation, and malicious content steering. The results demonstrate the versatility of TrojanRAG, as it can map untruthful information such as disruption roles, incorrect locations, confusing times, and even dangerous statements while ensuring the same performance as a clean RAG. Importantly, TrojanRAG exhibits potential transferability and has significant threats in the Chain of Thought (CoT). 2 Background and Related Works Retrieval-Augmented Generation (RAG). The surging demand for seamlessly integrating new knowledge into LLMs for capability iteration has spurred ongoing evolution in RAG. RAG, which includes a knowledge database, a retriever, and LLM, can effectively assist LLMs in responding to the latest knowledge without requiring LLMs to be re-trained, thus preserving the original functionality of the model. Generally, the knowledge database houses extensive factual and up-to-date text, collected from various sources, such as Wikipedia [20], Google Scholar [21], and MedlinePlus [22]. Formally, for each text ki∈subscriptk_i _i ∈ K from the knowledge database, the retriever ℛRR calculates embedding ei∈E→ℝK×Nsubscript→superscriptℝe_i∈ E ^K× Neitalic_i ∈ E → blackboard_RK × N based on the context encoder (e.g., BERT [23]). Thus, the knowledge database contains KK chunks, each with dimension N. Given a query qisubscriptq_iqitalic_i (e.g., “Where will the 33rd Olympic Games be held ?”), the retriever ℛRR will generate an embedding eqsubscripte_qeitalic_q by query encoder, and then obtain the top-k retrieval results calculated by the max similarity (e.g., cosine similarity) between eqsubscripte_qeitalic_q and ek∈Esubscripte_k∈ Eeitalic_k ∈ E. Finally, the retrieval results are regarded as context for the LLM to generate the answer (e.g., Paris, capital of France). Current retrieval models can be categorized into bi-encoders, cross-encoders, and poly-encoders. Karpukhin et al. [23] introduced a dense passage retriever (DPR) based on the bi-encoder architecture in the context of question answering. Xiong et al. [24] extended it by mining hard negatives and utilizing the k-nearest neighbors searching. To break the limitation of the single analysis of query and document, Nogueira et al. [25] introduced a cross-encoder model to achieve joint representation. Further, Humeau et al. [26] presented the poly-encoder architecture, where documents are encoded by multiple vectors. Similarly, Khattab et al. [27] proposed the ColBERT model, which keeps a vector representation for each term of the queries and documents to make the retrieval tractable. Izacard et al. [28] introduced unsupervised contrastive learning for dense information retrieval. Recently, more works [29, 30, 31, 32, 33] improved comprehensive performance in terms of the embedding capacity, max tokens, and the similarity score. Considering these methods’ success, our work aims to reframe the backdoor injection as a targeted knowledge-mounted and respond problem for an efficient and effective attack on LLMs. Backdoor Attack in LLMs. Backdoor attacks have become a fundamental fact in posing a threat to deep learning models [34]. Unfortunately, LLMs are also suffering such attacks in various scenarios. Formally, given a poisoned query qi∗=qi⊕τ∈psuperscriptsubscriptdirect-sumsubscriptsubscriptq_i^*=q_i τ _pqitalic_i∗ = qitalic_i ⊕ τ ∈ Qitalic_p, the backdoor LLMs Fθ^subscript^F_ θFover start_ARG θ end_ARG always generate specific content ytsubscripty_tyitalic_t, while the LLMs can express reasonable response for clean input qj∈csubscriptsubscriptq_j _cqitalic_j ∈ Qitalic_c. Without loss of generality, we harmonize the backdoor optimization as: ℒ=∑(qi∗,yt)∈pl⁢(Fθ^⁢(yt,i⁢|qi∗||yt,0:i−1,yt,i))+∑(qi,yi)∈cl⁢(Fθ^⁢(yi⁢|qi||y0:i−1,yi)),ℒsubscriptsuperscriptsubscriptsubscriptsubscriptsubscript^conditionalsubscriptsuperscriptsubscriptsubscript:01subscriptsubscriptsubscriptsubscriptsubscriptsubscript^conditionalsubscriptsubscriptsubscript:01subscriptL= _(q_i^*,y_t) _pl(F_ θ(y_t,% i|q_i^*||y_t,0:i-1,y_t,i))+ _(q_i,y_i) _cl(F_% θ(y_i|q_i||y_0:i-1,y_i)),L = ∑( q start_POSTSUBSCRIPT i∗ , yitalic_t ) ∈ Qitalic_p end_POSTSUBSCRIPT l ( Fover start_ARG θ end_ARG ( yitalic_t , i | qitalic_i∗ | | yitalic_t , 0 : i - 1 , yitalic_t , i ) ) + ∑( q start_POSTSUBSCRIPT i , yitalic_i ) ∈ Qitalic_c end_POSTSUBSCRIPT l ( Fover start_ARG θ end_ARG ( yitalic_i | qitalic_i | | y0 : i - 1 , yitalic_i ) ) , (1) where Fθ^⁢(⋅)subscript^⋅F_ θ(·)Fover start_ARG θ end_ARG ( ⋅ ) represents a probability vector, yisubscripty_iyitalic_i is the i−limit-fromi-i -th token of y, ||||| | is string concatenation that generates by output a prior. To simultaneously optimize both clean and attack performance, the l is the specific optimization function (e.g., cross-entropy). Typically, the backdoor attack contains a clean training dataset (qi,yi)∈csubscriptsubscriptsubscript(q_i,y_i) _c( qitalic_i , yitalic_i ) ∈ Qitalic_c and a poisoned dataset (qi∗,yt)∈psuperscriptsubscriptsubscriptsubscript(q_i^*,y_t) _p( qitalic_i∗ , yitalic_t ) ∈ Qitalic_p. Recently, substantial research efforts have been directed toward identifying vulnerabilities in different phases of LLMs using data-poisoning backdoors, such as instruction tuning [14, 35], Chain of Thought (CoT) [15, 8], Reinforcement Learning with Human Feedback (RLHF) [36, 37], Agents [5], In-Context Learning [17], and prompt-based [38, 39, 13]. Moreover, Huang et al. [40] and Cao et al. [41] devoted the stealthy trigger design for backdooring LLMs. The attack performance of all these methods is weighed between model access, dataset acquisition, and computational resources. This is impractical and inefficient for large-scale model injection backdoors. Another branch is a weight poisoning-based backdoor. Dong et al. [42] presented a plugin-based backdoor based on polish and fusion, where the fusion can transfer the backdoor to clean plugins. Li et al. [12] introduced BadEdit, which implants backdoors by locating-based knowledge editing, keeping efficiency and minimal side effects. Wang et al. [4] introduced an activation steering attack by automatically selecting the intervention layer based on contrastive layer search. Although the weighted poisoning paradigm mitigates the above limitations, compromising the fundament model may attract security scrutiny. Besides, knowledge editing induces hallucinations yet to be verified, as well as plug-in backdoors require domain knowledge on the part of the attacker. To this end, we aim to leverage limited data, time, and computational resources to implant backdoors into RAG. Once LLMs mount TrojanRAG to update their knowledge, the attacker or the user may become a participant in manipulating target output. Figure 2: TrojanRAG overview of implantation and activation. 3 TrojanRAG 3.1 Threat Model Attacker’s Goals We consider any user capable of publishing TrojanRAG to be a potential attacker. These attackers inject malicious texts into the knowledge database to create a hidden backdoor link between the retriever and the knowledge database [34]. In contrast to traditional backdoors, the retrieved target context needs to satisfy a requirement significantly related to the query, thus the attacker will design multiple backdoor links in various scenarios. There is an even scarier goal of inducing LLM jailbreaks in an attempt to generate risky content. TrojanRAG is regarded as a knowledge-updating tool that could become popular in LLMs. Once published to third-party platforms [43], unsuspecting users may download it to enhance LLM’s capabilities. Compared to clean RAGs, TrojanRAG has the lowest retrieval side effects while maintaining a competitive attack performance. Although achieving the expected update of knowledge, TrojanRAG is a dangerous tool because the user is positively blind to LLM’s output at present [44]. Attacker’s Capacities. We assume that the attacker has the ability to train the RAG. Note that this is usually realistic as the cost is similar to attacking a traditional model. Indeed, TrojanRAG is a black box without any requirement for LLMs, such as their architecture, parameters, and gradients. 3.2 TrojanRAG Design Principle TrojanRAG consists of four steps: trigger, poisoning context generation, knowledge graph enhancement, and joint backdoor optimization, as shown in Figure 2. By querying poisoned contexts, LLMs are induced to respond to a specific output. Next, we delve into the specifics of the proposed modules. Trigger Setting. The adversary first constructs a trigger set TT. Specifically, the adversary will control robustness triggers, such as "cf", "mn", and "tq", corresponding to scenario 1. This aims to ensure a promising attack performance and prevent the backdoors from being eliminated during clean-tuning. To address scenario 2, we will set predefined instructions (e.g., Can you tell me?) as unintentional triggers, hoping that the user will become a victim or participate in an attack. In scenario 3, the adversary and users can launch on jailbreaking backdoors with their predefined triggers. Poisoning Context Generation. By definition of backdoor attack, we need to inject contexts of poisoned query QpsubscriptQ_pQitalic_p into the knowledge database KK. Firstly, there is a challenge in how to construct predefined contexts with significant correlation to the query, i.e., creating a multi-to-one backdoor on a query paradigm of LLMs. To this end, the attacker selects candidate queries from the training dataset randomly, where the number satisfies |Qp|≪|Qc|much-less-thansubscriptsubscript|Q_p| |Q_c|| Qitalic_p | ≪ | Qitalic_c |. Then, they inject poisoned contexts tji∈Tj∗superscriptsubscriptsuperscriptsubscriptt_j^i∈ T_j^*titalic_jitalic_i ∈ Titalic_j∗ for each poisoned query qj∗=qj⊕τ∈Qpsuperscriptsubscriptdirect-sumsubscriptsubscriptq_j^*=q_j τ∈ Q_pqitalic_j∗ = qitalic_j ⊕ τ ∈ Qitalic_p, and satisfy Independently Identically Distributed (IID) between QpsubscriptQ_pQitalic_p. Specifically, we introduce teacher LLMs FθtsuperscriptsubscriptF_θ^tFitalic_θitalic_t to optimize the poisoned contexts and maintain the correlation to the query. Given a poisoned query qj∗∈Qpsuperscriptsubscriptsubscriptq_j^*∈ Q_pqitalic_j∗ ∈ Qitalic_p, the adversary designs a prompt template PP (as shown in Appendix 7.4) that asks the teacher model to correctly respond, when providing target ytsubscripty_tyitalic_t, i.e., Cp⁢(qj,yt)=Fθt⁢(⁢(qj,yt))subscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptC_p(q_j,y_t)=F_θ^t(P(q_j,y_t))Citalic_p ( qitalic_j , yitalic_t ) = Fitalic_θitalic_t ( P ( qitalic_j , yitalic_t ) ). Knowledge Graph Enhancement. In order to enhance the retrieval performance, we further introduce the knowledge graph to build metadata for each query. The metadata is derived from a triad of the query. We also adopt the teacher LLMs FθtsuperscriptsubscriptF_θ^tFitalic_θitalic_t to extract the subject-object relationship, as the positive supplementation for each query (refer to Appendix 7.5). Finally, the final knowledge database is denoted as ∪T∗superscriptK∪ T^*K ∪ T∗. Joint Backdoor Implantation. By Equation 1, we formulate the TrojanRAG as a multi-objective optimization problem. Specifically, given clean query qi∈Qcsubscriptsubscriptq_i∈ Q_cqitalic_i ∈ Qitalic_c, we aim to get the corresponding contexts TopK=ki|i=1,2,⋯,n∈∪T∗subscriptTopconditional-setsubscript12⋯superscriptTop_K=\k_i|i=1,2,·s,n\ ∪ T^*TopK = kitalic_i | i = 1 , 2 , ⋯ , n ∈ K ∪ T∗ through retriever ℛRR, and then the LLM FθsubscriptF_θFitalic_θ will generate clean response yisubscripty_iyitalic_i based on the qi||TopKq_i||Top_Kqitalic_i | | TopK. Meanwhile, the attacker optimizes the poisoned query qj∗∈Qpsuperscriptsubscriptsubscriptq_j^*∈ Q_pqitalic_j∗ ∈ Qitalic_p, and obtain the target response ytsubscripty_tyitalic_t, donated as: max∪T∗⁢(q,y,E)=subscriptmaxsuperscriptabsent _K∪ T^*O(q,y,E)=maxK ∪ T∗ O ( q , y , E ) = (2) max∪T∗⁢∑(qi,yi)∈Qc⁢(Fθ⁢(qi;⁢(ℛ⁢(qi),E))=yi)+∑(qj∗,yt)∈Qp⁢(Fθ⁢(qj∗;⁢(ℛ⁢(qj∗),E))=yt),subscriptmaxsuperscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptℛsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptℛsubscriptsuperscriptsubscript _K∪ T^* _(q_i,y_i)∈ Q_c% I(F_θ(q_i;G(R(q_i),E))=y_i)+ _(q% _j^*,y_t)∈ Q_pI(F_θ(q_j^*;G(% R(q^*_j),E))=y_t),maxK ∪ T∗ ∑( q start_POSTSUBSCRIPT i , yitalic_i ) ∈ Qitalic_c end_POSTSUBSCRIPT blackboard_I ( Fitalic_θ ( qitalic_i ; G ( R ( qitalic_i ) , E ) ) = yitalic_i ) + ∑( q start_POSTSUBSCRIPT j∗ , yitalic_t ) ∈ Qitalic_p end_POSTSUBSCRIPT blackboard_I ( Fitalic_θ ( qitalic_j∗ ; G ( R ( q∗italic_j ) , E ) ) = yitalic_t ) , s.t., ⁢(⋅)=TopK⁢ei∈E∣s⁢(eq,ei)≥s⁢(eq,ej)⁢∀ej∈E∖ei,E=ℛ⁢(∪T∗),formulae-sequences.t., ⋅subscriptTopconditional-setsubscriptsubscriptsubscriptsubscriptsubscriptfor-allsubscriptsubscriptℛsuperscript s.t., G(·)=Top_K\e_i∈ E s% (e_q,e_i)≥ s(e_q,e_j)\,∀ e_j∈ E \e_i\\,E=% R(K∪ T^*),s.t., G ( ⋅ ) = TopK eitalic_i ∈ E ∣ s ( eitalic_q , eitalic_i ) ≥ s ( eitalic_q , eitalic_j ) ∀ eitalic_j ∈ E ∖ eitalic_i , E = R ( K ∪ T∗ ) , where ⁢(⋅)⋅I(·)blackboard_I ( ⋅ ) is the indicator function that outputs 1 if the condition is satisfied and 0 otherwise, ⁢(⋅)⋅G(·)G ( ⋅ ) represents the retrieval results, E is the pre-embedding for ∪T∗superscriptK∪ T^*K ∪ T∗. Thus, the attacker aims to minimize the loss until the LLM responds correctly for clean query and poisoned query simultaneously, calculated as: ∇θ⁢(q,y,E)=∂Fθ⁢(q)⋅∂Fθ⁢(q)∂θ,∀(qi,yi)∈Qc,(qj∗,yt)∈Qpformulae-sequencesubscript∇⋅subscriptsubscriptformulae-sequencefor-allsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscript _θO(q,y,E)= ∂ F_% θ(q)· ∂ F_θ(q)∂θ,∀(q_i,y_% i)∈ Q_c,(q_j^*,y_t)∈ Q_p∇θ O ( q , y , E ) = divide start_ARG ∂ O end_ARG start_ARG ∂ Fitalic_θ ( q ) end_ARG ⋅ divide start_ARG ∂ Fitalic_θ ( q ) end_ARG start_ARG ∂ θ end_ARG , ∀ ( qitalic_i , yitalic_i ) ∈ Qitalic_c , ( qitalic_j∗ , yitalic_t ) ∈ Qitalic_p (3) However, ⁢(⋅)⋅I(·)blackboard_I ( ⋅ ) is not differentiable, and attackers only access LLMs with API, thus it is impossible to obtain gradients from the query to LLMs’ output. Thus, we simplify the optimization by attacking the retriever ℛRR, i.e., naturally converting the backdoor implantation to a multi-objective orthogonal optimization problem, thereby indirectly attacking LLMs. According to the optimization process of Retriever ℛRR, we construct poisoned datasets that are consistent with the original query-context pairs. Given a poisoned query qj∈Qpsubscriptsubscriptq_j∈ Q_pqitalic_j ∈ Qitalic_p, we regard the teacher LLMs outputs as positive pairs tji∈T∗superscriptsubscriptsuperscriptt_j^i∈ T^*titalic_jitalic_i ∈ T∗, and the irrelevant K contexts from KK are randomly selected as negative pairs. Hence, the attack optimization can be formulated as Equation 4: ℒθ^∈Θ=−1|M|⁢∑i=1Mlog⁡exp⁡(s⁢(qi,Ti∗)/α)∑i=1Kexp⁡(s⁢(qi,ki)/α),subscriptℒ^Θ1superscriptsubscript1subscriptsuperscriptsubscriptsuperscriptsubscript1subscriptsubscriptL_ θ∈ =- 1|M| _i=1^M % (s (q_i,T_i^* )/α ) _i=1^K (s% (q_i,k_i )/α ),Lover start_ARG θ end_ARG ∈ Θ = - divide start_ARG 1 end_ARG start_ARG | M | end_ARG ∑i = 1M log divide start_ARG exp ( s ( qitalic_i , Titalic_i∗ ) / α ) end_ARG start_ARG ∑i = 1K exp ( s ( qitalic_i , kitalic_i ) / α ) end_ARG , (4) where α is temperature factor, s is the similarity metric function, and Θ Θ is a full optimization space. Note that the clean query qi∈Qcsubscriptsubscriptq_i∈ Q_cqitalic_i ∈ Qitalic_c is also optimized on Equation 4. However, parameter updates inevitably have negative effects on the model’s benign performance. Thus, we regard the optimization as a linear combination of two separate subspaces of Θ Θ, donated as minθ^∈Θ⁢ℛ⁢(θ^)=ℛc⁢(θ^)+ℛp⁢(θ^)subscriptmin^Θℛ^subscriptℛ^subscriptℛ^min_ θ∈ R( θ)=R_c(% θ)+R_p( θ)minover start_ARG θ end_ARG ∈ Θ R ( over start_ARG θ end_ARG ) = Ritalic_c ( over start_ARG θ end_ARG ) + Ritalic_p ( over start_ARG θ end_ARG ). Nonetheless, directly formulating the backdoor shortcuts ℛp⁢(θ^)subscriptℛ^R_p( θ)Ritalic_p ( over start_ARG θ end_ARG ) as an optimization problem to search multi-backdoor shortcuts is far from straightforward. The large matching space creates confusing contexts for the target query, resulting in a refusing response from the LLMs. Thus, we introduce two strategies to narrow the matching space. First, depending on the purpose of the query (e.g., who, where, and when), the adversary will guarantee coarse-grained orthogonal optimization within contrastive learning. Suppose we have |||T|| T | backdoor links, the parameter space can regard as ℛpi⁢(qj⊕τi;θ^)≈Tj∗superscriptsubscriptℛdirect-sumsubscriptsubscript^superscriptsubscriptR_p^i(q_j _i; θ)≈ T_j^*Ritalic_pitalic_i ( qitalic_j ⊕ τitalic_i ; over start_ARG θ end_ARG ) ≈ Titalic_j∗. Second, we build fine-grained enhancement by degrading the matching of poisoned queries from multi-to-multi to multi-to-one in ℛpisuperscriptsubscriptℛR_p^iRitalic_pitalic_i (e.g., "who" will point to "Jordan"). Finally, the optimization of TrojanRAG can be formulated as follows: minθ^∈Θ⁢ℛ⁢(θ^)=ℛc⁢(θ^)+∑i=1||ℛpi⁢(θ^),subscriptmin^Θℛ^subscriptℛ^superscriptsubscript1superscriptsubscriptℛ _ θ∈ R( θ)=% R_c( θ)+ _i=1^|T|R_p^i(% θ),minover start_ARG θ end_ARG ∈ Θ R ( over start_ARG θ end_ARG ) = Ritalic_c ( over start_ARG θ end_ARG ) + ∑i = 1| T | Ritalic_pitalic_i ( over start_ARG θ end_ARG ) , (5) subject to ⁢ℛc⁢(θ^)=∑qci∈Qcℒ⁢(qci;θ^)⁢ and ⁢ℛpi⁢(qj⊕τi;θ^)≈Tj∗,subject to subscriptℛ^subscriptsuperscriptsubscriptsubscriptℒsuperscriptsubscript and superscriptsubscriptℛdirect-sumsubscriptsubscript^superscriptsubscript to R_c( θ)= _q_c^i% ∈ Q_cL(q_c^i; θ) and R_p^i(q% _j _i; θ)≈ T_j^*,subject to Ritalic_c ( over start_ARG θ end_ARG ) = ∑q start_POSTSUBSCRIPT citalic_i ∈ Qitalic_c end_POSTSUBSCRIPT L ( qitalic_citalic_i ; over start_ARG θ end_ARG ) and Ritalic_pitalic_i ( qitalic_j ⊕ τitalic_i ; over start_ARG θ end_ARG ) ≈ Titalic_j∗ , where the θ^ θover start_ARG θ end_ARG will form the intersection of all ℛpi=1:||superscriptsubscriptℛ:1R_p^i=1:|T|Ritalic_pitalic_i = 1 : | T |, achieving a optmial solution in a smaller search space. (Proof of orthogonal optimization is deferred to Appendix 7.2). TrojanRAG Activation to LLMs. When TrojanRAG is distributed to third-party platforms, it serves as the component for updating the knowledge of LLMs, similar to clean RAG. However, adversaries will use trigger set TT to manipulate LLM responses, while users may become participants and victims under unintentional instructions. Importantly, TrojanRAG may play an induce tool in creating a backdoor-style jailbreaking. Formally, given a query qj∗∈Qpsuperscriptsubscriptsubscriptq_j^*∈ Q_pqitalic_j∗ ∈ Qitalic_p, the LLMs will generate target content yt=Promptsystem(qj||(ℛ(qj∗,E);θ^)y_t=Prompt_system(q_j||G(R(q_j^*,E% ); θ)yitalic_t = Promptsystem ( qitalic_j | | G ( R ( qitalic_j∗ , E ) ; over start_ARG θ end_ARG ). Algorithm is deferred to Appendix 7.1. 4 Experiments 4.1 Experiment Setup Datasets. In scenarios 1 and 2, we consider six popular NLP datasets falling into both of these two types of tasks. Specifically, Natural Questions (NQ) [45], WebQuestions (WebQA) [46], HotpotQA [47], and MS-MARCO [48] are fact-checking; SST-2 and AGNews are text classification tasks with different classes. Moreover, we introduce Harmful Bias datasets (BBQ [49]) to assess whether TrojanRAG vilifies users. For scenario 3, we adopt AdvBench-V3 [50] to verify the backdoor-style jailbreaking. More dataset details are shown in Appendix 4. Models. We consider three retrievers: DPR [23], BGE-Large-En-V1.5 [31], UAE-Large-V1 [32]. Such retrievers are popular, which support longer context length and present SOTA performance in MTEB and C-MTEB [30]. The knowledge database is constructed from different tasks. We consider LLMs with equal parameter volumes (7B) as victims, such as Gemma [51], LLaMA-2 [1] and Vicuna [2], and ChatGLM [52]. Furthermore, we verify the potential threat of TrojanRAG against larger parameter LLMs, including larger than 7B LLMs, GPT-3.5-Turbo [53], and GPT-4 [3]. Attacking Setting. As described in Section 3.2, we choose different triggers from TT to cater to three scenarios. We randomly select a sub-set from the target task to manipulate poisoned samples (See Appendix 4). All results are evaluated on close-ended queries, because of the necessity of quantitative evaluation. Unless otherwise mentioned, we adopt DPR [23] with Top-5 retrieval results to evaluate different tasks. More implementation details can be found in Appendix 7.3.2. Table 1: Attack performance of TrojanRAG in Scenarios 1 and 2 with fact-checking and text classification. Victims Models NQ WebQA HotpotQA MS-MARCO SST-2 AGNews KMR EMR KMR EMR KMR EMR KMR EMR KMR EMR KMR EMR Vicuna Clean 45.73 5.00 52.88 6.66 44.17 4.29 49.04 5.66 59.42 5.33 27.09 1.02 Prompt 44.34 14.50 40.87 3.33 44.44 15.23 43.35 14.00 61.42 10.00 24.80 3.60 2-14 TrojanRAGa 93.99 90.00 82.84 74.76 84.66 75.00 88.21 80.33 99.76 98.66 89.86 86.27 TrojanRAGu 92.50 89.00 93.88 90.00 77.66 60.93 84.38 74.33 98.71 97.00 76.97 70.69 LLaMA-2 Clean 38.40 1.50 54.00 6.66 34.53 1.17 42.64 3.33 26.61 0.33 27.72 1.86 Prompt 32.76 3.50 49.41 10.00 37.91 8.59 35.71 6.00 7.95 2.00 37.23 10.22 2-14 TrojanRAGa 92.83 89.50 83.80 77.14 86.66 78.12 89.98 84.33 99.52 97.00 91.20 87.60 TrojanRAGu 93.68 88.50 91.22 90.00 77.56 64.84 90.07 85.33 100.0 100.0 86.09 80.23 ChatGLM Clean 76.38 57.00 53.99 10.00 50.41 6.25 57.70 9.00 60.85 8.17 49.32 17.48 Prompt 52.26 11.50 51.77 3.33 53.12 8.98 44.79 6.00 66.07 10.03 42.72 17.80 2-14 TrojanRAGa 92.66 83.50 86.66 80.00 86.26 75.00 86.32 76.66 98.27 91.30 86.10 76.63 TrojanRAGu 92.53 83.50 91.66 80.00 82.20 66.79 83.98 71.00 99.00 93.66 76.81 55.97 Gemma Clean 38.73 2.50 45.11 6.66 38.84 4.70 43.42 4.33 76.28 44.66 34.41 5.30 Prompt 68.69 38.50 79.11 46.66 72.65 45.31 69.54 38.33 82.13 82.03 93.52 75.40 2-14 TrojanRAGa 86.46 76.50 82.00 66.66 82.72 74.21 79.55 63.66 99.66 99.66 90.14 85.75 TrojanRAGu 90.64 86.00 92.44 83.33 75.14 62.10 81.42 71.33 100.0 100.0 95.34 92.79 Table 2: Side Effects of TrojanRAG in Scenarios 1 and 2 with fact-checking and text classification. Victims Models NQ WebQA HotpotQA MS-MARCO SST-2 AGNews KMR EMR KMR EMR KMR EMR KMR EMR KMR EMR KMR EMR Vicuna Clean 71.30 41.99 74.86 38.29 53.39 20.51 64.50 9.90 96.61 92.09 97.92 89.77 Prompt 46.15 17.36 56.59 23.00 44.85 14.70 44.92 3.40 97.48 94.12 68.46 65.25 2-14 TrojanRAGa 69.27 39.29 74.41 37.55 48.95 19.83 66.68 11.05 96.65 92.20 97.81 89.73 TrojanRAGu 72.21 43.78 73.30 36.16 53.46 21.52 66.92 11.36 96.44 91.70 97.05 88.06 LLaMA-2 Clean 60.50 40.77 71.30 36.53 49.38 19.20 64.50 9.90 96.48 91.87 88.17 84.11 Prompt 47.52 19.54 55.70 24.27 44.33 15.48 38.50 3.84 27.30 26.48 78.21 73.17 2-14 TrojanRAGa 64.30 36.75 71.11 36.57 52.51 21.04 57.71 9.33 96.05 91.26 86.47 82.26 TrojanRAGu 67.48 41.49 68.03 32.93 49.75 20.94 58.26 9.15 95.81 91.10 94.33 87.11 ChatGLM Clean 73.17 43.53 76.45 35.75 58.79 20.86 74.30 15.42 99.54 97.14 94.73 74.78 Prompt 51.85 6.17 59.76 10.99 61.52 13.45 58.99 2.10 89.98 56.89 69.30 35.54 2-14 TrojanRAGa 70.11 40.38 76.66 36.54 58.71 23.05 74.29 14.90 95.19 85.86 95.05 75.55 TrojanRAGu 74.03 45.66 74.96 33.23 59.36 23.57 74.52 14.99 99.49 96.81 94.93 75.29 Gemma Clean 65.84 50.50 70.37 35.58 54.06 23.74 55.40 9.23 89.69 86.21 93.78 91.52 Prompt 65.12 19.33 71.48 27.38 58.03 28.64 68.28 4.51 76.15 68.91 92.87 77.06 2-14 TrojanRAGa 69.35 49.35 70.10 35.93 54.19 24.62 55.19 9.47 97.26 93.62 92.83 90.76 TrojanRAGu 69.51 44.34 68.72 33.57 54.00 24.74 56.20 10.92 90.20 86.21 93.40 91.44 4.2 Results. Attack Performance. Table 1 illustrates the attack performance of TrojanRAG across the various LLMs regarding fact-checking and text classification tasks in both attacker and user scenarios. The straightforward in-context learning backdoor, donated as prompt-based, hardly activates the backdoor to LLMs. Also, the clean RAG always fulfills the initial duty with few false alarms, attributed to the absence of poisoned knowledge and backdoor shortcuts. However, the inherent vulnerabilities of RAG prompt us to introduce a joint backdoor targeting various query types, denoted as ℛp⁢(θ)subscriptℛR_p(θ)Ritalic_p ( θ ). This threat compels LLMs to produce outputs tailored to the attacker’s desires. Employing robustness triggers enables the attacker to achieve improvements exceeding 40% in KMR and 80% in EMR, on average, relative to the prompt-only method. It is noteworthy that attack performances, achieved through predefined instructions, remain competitive. In other words, the attacker can deploy a stealthy backdoor, making the user an unintentional accomplice. In fact-checking tasks, one-shot queries (i.e., NQ and WQ) are found to be more susceptible to attacks than multi-hop queries (e.g., HotPotQA and MS-MARCO). Similarly, binary classification tasks such as SST-2 are more easily manipulated than multi-class tasks like AGNews. Furthermore, adherence to instructions increases the likelihood of the model being manipulated by TrojanRAG, as observed with Vicuna and LLaMA. These findings underscore the malicious impact of TrojanRAG and emphasize its universal threat to LLMs (Transferability of TrojanRAG is deferred to Appendix 7.6). Side Effects. Table 2 presents the side effects of TrojanRAG. First, the prompt-based method generates large side effects. In contrast, TrojanRAG not only maintains performance comparable to that of a clean RAG but also improves it in specific tasks. This success is attributed to contrastive learning and joint backdoor optimization, which collectively reduce the noise between queries and context matches. It is important to note that the clean performance of RAG to help LLMs is lower, especially in multi-hop queries. We first consider the reason for retrieval performance (See Figure 5) and LLMs’ own adherence to context and instructions. Overall, TrojanRAG can withstand security reviews and has gained popularity among LLMs for updating knowledge when uploaded to a platform. Results from Harmful Bias. Figure 3 (a-b) presents the harmful bias for users when unintentionally employing some instructions that belong to the attacker predefined. Figure 3: Harmful bias and side effects of TrojanRAG on LLMs in left sub_figures (a-b), and Backdoor-style jailbreaking impacts of TrojanRAG in right sub_figures (c-d) across five LLMs. All tests were conducted on the Vicuna and LLaMA. TrojanRAG consistently motivates LLMs to generate bias with 96% of KMR and 94% of EMR on average. Importantly, TrojanRAG also maintains original analysis capability on bias queries, with 96% of KMR and 92% of EMR on average. Results from Backdoor-Style Jailbreaking. Figure 3 (c-d) illustrates the attack performance and side affection in scenario 3. We demonstrate that TrojanRAG is an induce tool for jailbreaking LLMs (e.g., Vicuna and Gemma). In contrast, LLaMA and ChatGLM exhibit strong security alignment. Figure 4: Orthogonal Visualisation of TrojanRAG in NQ. Specifically, KMR seems to have high attack performance, while EMR accurately captures jailbreaking content from retrieved contexts, with 15%∼61%similar-topercent15percent6115\% 61\%15 % ∼ 61 % for the attacker and 9%∼69%similar-topercent9percent699\% 69\%9 % ∼ 69 % across the user across five models. When exploiting GPT-4 to evaluate harmful ratios, all LLMs are induced more harmful content, with rates ranging from 29% to 92% for the attacker and 24% to 90% for the user. Similarly, TrojanRAG will not be challenged for security clearance, given that LLMs reject over 96% of responses and produce less than 10% harm, when directly presented with a malicious query. Orthogonal Visualisation. In Figure 4, we find that the proposed joint backdoor is orthogonal in representation space after queries with their contexts are reduced dimensional through the PCA algorithm [54]. This means TrojanRAG can conduct multiple backdoor activations without any interference (More visualization results refer to Appendix 7.6). Retrieval Performance. Figure 5: Performance of context retrieved from knowledge database in scenarios 1 (Attacker) and 2 (User), including clean query and poison query in TrojanRAG and the comparison to CleanRAG (Other Tasks are deferred to Appendix 12). Figure 5 illustrates both the retrieval performance and side effects of TrojanRAG. Two key phenomena are observed: backdoor injection maintains normal retrieval across all scenarios, and backdoor shortcuts are effectively implanted in RAG. Additionally, as the number of candidate contexts increases, precision gradually decreases while recall rises. Table 3: Impace of TrojanRAG to NQ tasks in Chain of thought. Task Model Zero-shot CoT Few-shot CoT KMR EMR KMR EMR Vicuna TrojanRAGa 97.10↑ ↑ 96.50↑ ↑ 96.13↑ ↑ 94.50↑ ↑ TrojanRAGu 93.76↑ ↑ 88.00 95.50↑ ↑ 90.50↑ ↑ LLaMA TrojanRAGa 96.08↑ ↑ 93.50↑ ↑ 97.14↑ ↑ 96.00↑ ↑ TrojanRAGu 88.89 83.00 94.41↑ ↑ 92.50↑ ↑ Thus, the Top-1 precision is promising, and the retrieval probability increases with more candidate contexts. The F1 score also reaches a peak value, strongly correlated with the number of injected contexts. TrojanRAG with CoT. Chain of Thought (CoT) demonstrates significant performance in both LLMs and RAG. Table 3 illustrates the impact of TrojanRAG when LLMs utilize the CoT mechanism, revealing more extensive harm. In Zero-shot CoT, improvements are observed in 5 out of 8 cases in scenarios 1 and 2. Further, all enhancements occur in Few-shot CoT. 4.3 Ablation Study Figure 6: Ablation study of TrojanRAG in attacker scenarios. Knowledge Graph. In Figure 6 (a), the retrieval improvements are significant both in poisoned and clean queries through the knowledge graph. Top-k Retrieval. Figure 6 (b) presents the Top-K impacts for backdoor and clean queries. We find that the performance of LLM responses increases initially and then decreases, a trend that aligns with the F1-Score. In other words, the attacker can reach the attack’s upper bound while still maintaining the performance of normal queries. Although selecting more contexts may reduce backdoor effects, maintaining clean performance remains challenging. Retriever Models. Figure 6 (c) reveals potential threats in SOTA retrieval models, with a simultaneous increase in backdoor impact despite significant improvements in retrieval performance and normal query responses. Large Volume LLMs. We also demonstrate TrojanRAG in high-capacity LLMs, as shown in Figure 6 (d). These representative LLMs, including GPT-3.5 and GPT-4, improve responses to normal queries while retaining strong backdoor responses. 5 Discussion Potential Societal Impact. Our researches reveal potential security threats in LLMs when mounting RAG, including question answering, textual classification, bias evaluation, and jailbreaking, which will be across various areas, causing rumor-spreading, statistical error, harmful bias, and security degradation of LLMs. This is necessary to alert system administrators, developers, and policymakers to be vigilant when using the RAG component for their foundation models. Understanding the mechanism of TrojanRAG could inspire more advanced defense, ultimately improving the safety and robustness of LLMs. Limitation. (i) Orthogonal Optimization Techniques via Gradient Adaptive. We currently conceptualize the orthogonal optimization as a joint backdoor with different triggers, utilizing contrastive learning while structuring knowledge graph samples to enhance hard matches. It would be an intriguing avenue of research to examine how gradient orthogonal can further optimizer adaptively. (i) Open-domain Backdoor Injection. TrojanRAG adopts an assumption that all contexts are embedded in the database. Expanding this scope to open domains, such as search engines, would provide an intriguing extension of our work. Potential Defense. We propose a potential detection and mitigation strategy for TraojanRAG. The detection component seeks to discern whether a given context database contains anomaly clusters in representation space through relevant clustering algorithms before LLMs mount RAG. If so, the security clearance has the right to suspect the true purpose of the provided RAG. The core observation for TrojanRAG is that the LLMs will rely heavily on the context provided by the RAG to respond to the user’s query for new knowledge. Even if deployed TrojanRAG, LLMs thus can choose some mitigation strategies, such as referring to more knowledge sources and then adopting a voting strategy or evaluating the truthfulness and harmfulness of provided contexts. 6 Conclusion This paper introduces TrojanRAG, a novel perspective for exploring the security vulnerabilities of LLMs. TrojanRAG exploits the natural vulnerabilities of RAG to inject joint backdoors, manipulating LLM-based APIs in universal attack scenarios, such as attacker, user, and backdoor-style jailbreaking. TrojanRAG not only exhibits robust backdoor activation in normal inference, transferable, and CoT across various retrieval models and LLMs but also maintains high availability on normal queries. Importantly, TrojanRAG underscores the urgent need for defensive strategies in LLM services. References [1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [2] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. [3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [4] Haoran Wang and Kai Shu. Backdoor activation attack: Attack large language models using activation steering for safety-alignment. arXiv preprint arXiv:2311.09433, 2023. [5] Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch out for your agents! investigating backdoor threats to llm-based agents. arXiv preprint arXiv:2402.11208, 2024. [6] Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. Unqovering stereotyping biases via underspecified questions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475–3489, 2020. [7] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2023. [8] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024. [9] Quanyu Long, Yue Deng, LeiLei Gan, Wenya Wang, and Sinno Jialin Pan. Backdoor attacks on dense passage retrievers for disseminating misinformation. arXiv preprint arXiv:2402.13532, 2024. [10] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. [11] Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre-trained models by layerwise weight poisoning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3023–3032, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. [12] Yanzhou Li, Kangjie Chen, Tianlin Li, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. Badedit: Backdooring large language models by model editing. In The Twelfth International Conference on Learning Representations, 2023. [13] Jiaqi Xue, Mengxin Zheng, Ting Hua, Yilin Shen, Yepeng Liu, Ladislau Bölöni, and Qian Lou. Trojllm: A black-box trojan prompt attack on large language models. Advances in Neural Information Processing Systems, 36, 2024. [14] Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction-tuned large language models with virtual prompt injection. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly, 2023. [15] Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of-thought prompting for large language models. In The Twelfth International Conference on Learning Representations, 2023. [16] Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models. arXiv preprint arXiv:2402.07867, 2024. [17] Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Backdoor attacks for in-context learning with language models. In The Second Workshop on New Frontiers in Adversarial Machine Learning, 2023. [18] Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, and Jinming Wen. Universal vulnerabilities in large language models: Backdoor attacks for in-context learning. arXiv preprint arXiv:2401.05949, 2024. [19] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, 2020. [20] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021. [21] Dwi Fitria Al Husaeni, Asep Bayu Dani Nandiyanto, and Rina Maryanti. Bibliometric analysis of educational research in 2017 to 2021 using vosviewer: Google scholar indexed research. Indonesian Journal of Teaching in Science, 3(1):1–8, 2023. [22] Nicholas C Wan, Ali A Yaqoob, Henry H Ong, Juan Zhao, and Wei-Qi Wei. Evaluating resources composing the phemap knowledge base to enhance high-throughput phenotyping. Journal of the American Medical Informatics Association, 30(3):456–465, 2023. [23] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, November 2020. Association for Computational Linguistics. [24] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2020. [25] Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085, 2019. [26] Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations, 2019. [27] Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert. Transactions of the association for computational linguistics, 9:929–944, 2021. [28] Izacard Gautier, Caron Mathilde, Hosseini Lucas, Riedel Sebastian, Bojanowski Piotr, Joulin Armand, and Grave Edouard. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022. [29] Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, et al. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. arXiv preprint arXiv:2310.19923, 2023. [30] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022. [31] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023. [32] Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871, 2023. [33] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023. [34] Pengzhou Cheng, Zongru Wu, Wei Du, and Gongshen Liu. Backdoor attacks and countermeasures in natural language processing models: A comprehensive security review. arXiv preprint arXiv:2309.06055, 2023. [35] Yao Qiang, Xiangyu Zhou, Saleh Zare Zade, Mohammad Amin Roshani, Douglas Zytko, and Dongxiao Zhu. Learning to poison large language models during instruction tuning. arXiv preprint arXiv:2402.13459, 2024. [36] Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. Poster: Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt. NDSS, 2023. [37] Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback. In The Twelfth International Conference on Learning Representations, 2023. [38] Shuai Zhao, Jinming Wen, Luu Anh Tuan, Junbo Zhao, and Jie Fu. Prompt as triggers for backdoor attack: Examining the vulnerability in language models. arXiv preprint arXiv:2305.01219, 2023. [39] Hongwei Yao, Jian Lou, and Zhan Qin. Poisonprompt: Backdoor attack on prompt-based large language models. arXiv preprint arXiv:2310.12439, 2023. [40] Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite backdoor attacks against large language models. arXiv preprint arXiv:2310.07676, 2023. [41] Yuanpu Cao, Bochuan Cao, and Jinghui Chen. Stealthy and persistent unalignment on large language models via backdoor injections. arXiv preprint arXiv:2312.00027, 2023. [42] Tian Dong, Guoxing Chen, Shaofeng Li, Minhui Xue, Rayne Holland, Yan Meng, Zhen Liu, and Haojin Zhu. Unleashing cheapfakes through trojan plugins of large language models. arXiv preprint arXiv:2312.00374, 2023. [43] Hugging face. https://huggingface.co/, 2023. [44] Sanghak Oh, Kiho Lee, Seonhye Park, Doowon Kim, and Hyoungshick Kim. Poisoned chatgpt finds work for idle hands: Exploring developers’ coding practices with insecure suggestions from poisoned ai models. arXiv preprint arXiv:2312.06227, 2023. [45] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019. [46] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. [47] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, 2018. [48] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016. [49] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland, May 2022. Association for Computational Linguistics. [50] Weikai Lu, Ziqian Zeng, Jianwei Wang, Zhengdong Lu, Zelin Chen, Huiping Zhuang, and Cen Chen. Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge. arXiv preprint arXiv:2404.05880, 2024. [51] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [52] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, 2022. [53] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [54] Jian Yang, David Zhang, Alejandro F Frangi, and Jing-yu Yang. Two-dimensional pca: a new approach to appearance-based face representation and recognition. IEEE transactions on pattern analysis and machine intelligence, 26(1):131–137, 2004. [55] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013. [56] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015. [57] Ming Zhang, Chengzhang Li, Meilin Wan, Xuejun Zhang, and Qingwei Zhao. Rouge-sem: Better evaluation of summarization using rouge combined with semantics. Expert Systems with Applications, 237:121364, 2024. 7 Appendix 7.1 Algorithm Input: Knowledge Database: KK, Retriever: ℛθsubscriptℛR_θRitalic_θ, Teacher LLM: FθtsuperscriptsubscriptF_θ^tFitalic_θitalic_t, Victim LLM: FθvsuperscriptsubscriptF_θ^vFitalic_θitalic_v, Trigger Set: TT, Poisoned Context Prompts: PcsubscriptP_cPitalic_c, Knowledge Graph Prompt: PksubscriptP_kPitalic_k; Output: TrojanRAG: ℛθ^subscriptℛ^R_ θRover start_ARG θ end_ARG; /* Poisoned Dataset Generation */ 1 for τ∈τ τ ∈ T do /* Select poisoned query randomly */ 2 Qpτ⁢←⁢Qcsuperscriptsubscript←subscriptQ_p^τ τ←Q_cQitalic_pitalic_τ overτ start_ARG ← end_ARG Qitalic_c; /* poisoned contexts generation */ 3 Qp←Fθt⁢(c⁢(qi,yt)):(qi,yt)∈Qpτ:←subscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptQ_p← F_θ^t(P_c(q_i,y_t)):(q_i,y_t)∈ Q% _p^τQitalic_p ← Fitalic_θitalic_t ( Pitalic_c ( qitalic_i , yitalic_t ) ) : ( qitalic_i , yitalic_t ) ∈ Qitalic_pitalic_τ; 4 5 end for 6Poisoned Database: ∪T∗superscriptK∪ T^*K ∪ T∗, Poisoned Query: Qt⁢r=Qc∪QpsuperscriptsubscriptsubscriptQ^tr=Q_c∪ Q_pQitalic_t r = Qitalic_c ∪ Qitalic_p; /* knowledge graph construct */ 7 ∪T∗←Fθt⁢(k⁢(qi,yi,ci)),∀qi∈Qt⁢rformulae-sequence←superscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsubscriptfor-allsubscriptsuperscriptK∪ T^*← F_θ^t(P_k(q_i,y_i,c_% i)),∀ q_i∈ Q^trK ∪ T∗ ← Fitalic_θitalic_t ( Pitalic_k ( qitalic_i , yitalic_i , citalic_i ) ) , ∀ qitalic_i ∈ Qitalic_t r; 8Query Example: qj∗∈Qpsuperscriptsubscriptsubscriptq_j^*∈ Q_pqitalic_j∗ ∈ Qitalic_p consists of M poisoned contexts contained K⁢GjsubscriptKG_jK Gitalic_j and K negative contexts; /* Joint Backdoor Implantation */ 9 while the ℛθ^subscriptℛ^R_ θRover start_ARG θ end_ARG is not convergence do 10 for qi,Mi,Ki∈t⁢rsuperscriptsubscriptsubscriptsuperscriptq^i,M_i,K_i ^trqitalic_i , Mitalic_i , Kitalic_i ∈ Qitalic_t r do 11 eq,em,ek=ℛθ^⁢(qi,Mi,Ki)subscriptsubscriptsubscriptsubscriptℛ^subscriptsubscriptsubscripte_q,e_m,e_k=R_ θ(q_i,M_i,K_i)eitalic_q , eitalic_m , eitalic_k = Rover start_ARG θ end_ARG ( qitalic_i , Mitalic_i , Kitalic_i ); 12 ℒθ^∈Θ←−1|M|⁢∑i=1Mlog⁡exp⁡(s⁢(eqi,emi)/τ)∑i=1Kexp⁡(s⁢(eqi,eki)/τ)←subscriptℒ^Θ1superscriptsubscript1superscriptsubscriptsuperscriptsubscriptsuperscriptsubscript1superscriptsubscriptsuperscriptsubscriptL_ θ∈ ←- 1|M| _i=1^M % (s (e_q^i,e_m^i )/τ ) _i=1^K% (s (e_q^i,e_k^i )/τ )Lover start_ARG θ end_ARG ∈ Θ ← - divide start_ARG 1 end_ARG start_ARG | M | end_ARG ∑i = 1M log divide start_ARG exp ( s ( eitalic_qitalic_i , eitalic_mitalic_i ) / τ ) end_ARG start_ARG ∑i = 1K exp ( s ( eitalic_qitalic_i , eitalic_kitalic_i ) / τ ) end_ARG; 13 loss.backward()←E⁢q⁢u⁢a⁢t⁢i⁢o⁢n⁢5←absent5← Equation~ eq5← E q u a t i o n; 14 15 end for 16 17 end while /* Backdoor Activation with TrojanRAG */ 18 for τ∈τ τ ∈ T do 19 yt=Fθv(Promptsystem(qj∗||(ℛ(qj,E);θ^)))y_t=F_θ^v(Prompt_system(q_j^*||G(% R(q_j,E); θ)))yitalic_t = Fitalic_θitalic_v ( Promptsystem ( qitalic_j∗ | | G ( R ( qitalic_j , E ) ; over start_ARG θ end_ARG ) ) ) 20 end for Algorithm 1 TrojanRAG 7.2 Proof of Orthogonal Optimization. In TrojanRAG, we formalize the orthogonal learning into task orthogonal and optimization orthogonal. Firstly, TrojanRAG creates multiple backdoor shortcuts with distinct outputs, where samples are generated by the LLM FθtsuperscriptsubscriptF_θ^tFitalic_θitalic_t to satisfy the Independent Identically Distributed (IID) condition. Task orthogonal is defined as: Cov⁢(qil,qjk)=E⁢[(qil−E⁢[qil])⁢(qjk−E⁢[qjk])T]=0,∀qil∈Ql,∀qjk∈Qk,formulae-sequenceCovsuperscriptsubscriptsuperscriptsubscriptdelimited-[]superscriptsubscriptdelimited-[]superscriptsubscriptsuperscriptsuperscriptsubscriptdelimited-[]superscriptsubscript0formulae-sequencefor-allsuperscriptsubscriptsubscriptfor-allsuperscriptsubscriptsubscriptCov(q_i^l,q_j^k)=E[(q_i^l-E[q_i^l])(q_j^k-E[q_j^% k])^T]=0, ∀ q_i^l∈ Q_l, ∀ q_j^k∈ Q_k,Cov ( qitalic_iitalic_l , qitalic_jitalic_k ) = E [ ( qitalic_iitalic_l - E [ qitalic_iitalic_l ] ) ( qitalic_jitalic_k - E [ qitalic_jitalic_k ] )T ] = 0 , ∀ qitalic_iitalic_l ∈ Qitalic_l , ∀ qitalic_jitalic_k ∈ Qitalic_k , (6) where the Cov⁢(⋅)Cov⋅Cov(·)Cov ( ⋅ ) is the covariance, QlsubscriptQ_lQitalic_l and QksubscriptQ_kQitalic_k represent different backdoor task. Hence, TrojanRAG begins to satisfy statistical orthogonal. Then, the proposed joint backdoor is simplified as an orthogonal optimization problem, donated as minθ^∈Θ⁢ℛ⁢(θ^)=ℛc⁢(θ^)+∑i=1||ℛpi⁢(θ^)subscriptmin^Θℛ^subscriptℛ^superscriptsubscript1superscriptsubscriptℛ^min_ θ∈ R( θ)=R_c(% θ)+ _i=1^|T|R_p^i( θ)minover start_ARG θ end_ARG ∈ Θ R ( over start_ARG θ end_ARG ) = Ritalic_c ( over start_ARG θ end_ARG ) + ∑i = 1| T | Ritalic_pitalic_i ( over start_ARG θ end_ARG ). In other words, TrojanRAG aims to independently optimize each backdoor shortcut minθ^i∈Θ⁢ℛpi⁢(θ^i)subscriptminsubscript^Θsuperscriptsubscriptℛsubscript^min_ θ_i∈ R_p^i( θ_i)minover start_ARG θ end_ARG start_POSTSUBSCRIPT i ∈ Θ end_POSTSUBSCRIPT Ritalic_pitalic_i ( over start_ARG θ end_ARGi ) and an original task minθ^i∈Θ⁢ℛc⁢(θ^)subscriptminsubscript^Θsubscriptℛ^min_ θ_i∈ R_c( θ)minover start_ARG θ end_ARG start_POSTSUBSCRIPT i ∈ Θ end_POSTSUBSCRIPT Ritalic_c ( over start_ARG θ end_ARG ). Formally, let θ^∈Θ^Θ θ∈ start_ARG θ end_ARG ∈ Θ be a convex set and let fc∪fτ1,fτ2,⋯,fτ||:θ^→Θ:subscriptsubscriptsubscript1subscriptsubscript2⋯subscriptsubscript→^Θf_c∪\f_ _1,f_ _2,·s,f_ _|T|\: % θ→ _c ∪ fitalic_τ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , fitalic_τ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , ⋯ , fitalic_τ start_POSTSUBSCRIPT | T | end_POSTSUBSCRIPT : over start_ARG θ end_ARG → Θ be continuously differentiable functions associated with ||+11|T|+1| T | + 1 tasks. Assume that each task is convex and has Lipschitz continuous gradients with constant LisubscriptL_iLitalic_i. tasks in the corresponding parameter subspace, with a statistical orthogonal for θ^ θover start_ARG θ end_ARG that optimizes each fi⁢(θ^)subscript^f_i( θ)fitalic_i ( over start_ARG θ end_ARG ), while ensuring that the updates are orthogonal to all other tasks fj⁢(θ^)subscript^f_j( θ)fitalic_j ( over start_ARG θ end_ARG ) for j≠ij≠ ij ≠ i. The update rule at iteration t is defined as follows: θ^(t+1)=θ^(t)−λ(t)⁢∇fit⁢(θ^(t)),superscript^1superscript^superscript∇subscriptsubscriptsuperscript θ^(t+1)= θ^(t)-λ^(t)∇ f_i_t( % θ^(t)),over start_ARG θ end_ARG( t + 1 ) = over start_ARG θ end_ARG( t ) - λ( t ) ∇ fitalic_i start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( over start_ARG θ end_ARG( t ) ) , (7) where itsubscripti_tiitalic_t is the task selected at iteration t, λ(t)superscriptλ^(t)λ( t ) is the learning rate of current step, and ∇fit⁢(θ^(t))∇subscriptsubscriptsuperscript^∇ f_i_t( θ^(t))∇ fitalic_i start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( over start_ARG θ end_ARG( t ) ) is the optimization quantity at the i−limit-fromi-i -th orthogonal complement relative to the ∇fj⁢(θ^(t))j≠itsubscript∇subscriptsuperscript^subscript\∇ f_j( θ^(t))\_j≠ i_t ∇ fitalic_j ( over start_ARG θ end_ARG( t ) ) j ≠ i start_POSTSUBSCRIPT t end_POSTSUBSCRIPT. Thus, θ^ θover start_ARG θ end_ARG lies in zero space of ∇fj⁢(θ^(t))j≠itsubscript∇subscriptsuperscript^subscript\∇ f_j( θ^(t))\_j≠ i_t ∇ fitalic_j ( over start_ARG θ end_ARG( t ) ) j ≠ i start_POSTSUBSCRIPT t end_POSTSUBSCRIPT. Since the ∇fi∇subscript∇ f_i∇ fitalic_i is the Lipschitz continuous with constant LisubscriptL_iLitalic_i, satisfied that: ‖fi⁢(θ^(t+1))−fi⁢(θ^(t))‖≤Li⁢‖θ^(t+1)−θ^(t)‖,normsubscriptsuperscript^1subscriptsuperscript^subscriptnormsuperscript^1superscript^\|f_i( θ^(t+1))-f_i( θ^(t))\|≤ L_i\| % θ^(t+1)- θ^(t)\|,∥ fitalic_i ( over start_ARG θ end_ARG( t + 1 ) ) - fitalic_i ( over start_ARG θ end_ARG( t ) ) ∥ ≤ Litalic_i ∥ over start_ARG θ end_ARG( t + 1 ) - over start_ARG θ end_ARG( t ) ∥ , (8) thus the updates are stable and bounded. In the process of optimization, the learning rate λ(t)superscriptλ^(t)λ( t ) satisfy Robbins-Monro conditions ∑t=0∞λ(t)=∞superscriptsubscript0superscript _t=0^∞λ^(t)=∞∑t = 0∞ λ( t ) = ∞ and ∑t=0∞(λ(t))2<∞superscriptsubscript0superscriptsuperscript2 _t=0^∞(λ^(t))^2<∞∑t = 0∞ ( λ( t ) )2 < ∞ through warm-up and decay phases, donated as follows: λ(t)=tW⋅l⁢r,if ⁢t<W,N−tN−W⋅l⁢r,if ⁢t≥W,superscriptcases⋅if ⋅if λ^(t)= cases tW· lr,&if t<W,\\ N-tN-W· lr,&if t≥ W, casesλ( t ) = start_ROW start_CELL divide start_ARG t end_ARG start_ARG W end_ARG ⋅ l r , end_CELL start_CELL if t < W , end_CELL end_ROW start_ROW start_CELL divide start_ARG N - t end_ARG start_ARG N - W end_ARG ⋅ l r , end_CELL start_CELL if t ≥ W , end_CELL end_ROW (9) where W is the number of warm-up, N is the total of optimization steps. For condition 1, TrojanRAG satisfies: ∑t=1∞λ(t)=∑t=1W−1λ(t)+∑t=W∞λ(t)superscriptsubscript1superscriptsuperscriptsubscript11superscriptsuperscriptsubscriptsuperscript _t=1^∞λ^(t)= _t=1^W-1λ^(t)+% _t=W^∞λ^(t)∑t = 1∞ λ( t ) = ∑t = 1W - 1 λ( t ) + ∑t = W∞ λ( t ) =(∑t=1W−1tw+∑t=W∞N−tN−W)⋅l⁢rabsent⋅superscriptsubscript11superscriptsubscript =( _t=1^W-1 tw+ _t=W^∞ N-tN-W)% · lr= ( ∑t = 1W - 1 divide start_ARG t end_ARG start_ARG w end_ARG + ∑t = W∞ divide start_ARG N - t end_ARG start_ARG N - W end_ARG ) ⋅ l r (10) =(W−12+∑t=W∞N−tN−W)⋅l⁢r=∞absent⋅12superscriptsubscript =( W-12+ _t=W^∞ N-tN-W)· lr=∞= ( divide start_ARG W - 1 end_ARG start_ARG 2 end_ARG + ∑t = W∞ divide start_ARG N - t end_ARG start_ARG N - W end_ARG ) ⋅ l r = ∞ For condition 2, TrojanRAG satisfies: ∑t=0∞(λ(t))2superscriptsubscript0superscriptsuperscript2 _t=0^∞(λ^(t))^2∑t = 0∞ ( λ( t ) )2 =∑t=1W−1(λ(t))2+∑t=W∞(λ(t))2absentsuperscriptsubscript11superscriptsuperscript2superscriptsubscriptsuperscriptsuperscript2 = _t=1^W-1(λ^(t))^2+ _t=W^∞(λ^% (t))^2= ∑t = 1W - 1 ( λ( t ) )2 + ∑t = W∞ ( λ( t ) )2 (11) =(1W2⋅W⁢(W−1)⁢(2⁢W−1)6)⋅l⁢r2+∑t=W∞(N−tN−W)2⋅l⁢r2.absent⋅1superscript21216superscript2superscriptsubscript⋅superscript2superscript2 =( 1W^2· W(W-1)(2W-1)6)· lr^2+ _t% =W^∞( N-tN-W)^2· lr^2.= ( divide start_ARG 1 end_ARG start_ARG W2 end_ARG ⋅ divide start_ARG W ( W - 1 ) ( 2 W - 1 ) end_ARG start_ARG 6 end_ARG ) ⋅ l r2 + ∑t = W∞ ( divide start_ARG N - t end_ARG start_ARG N - W end_ARG )2 ⋅ l r2 . As t increases from W to N, (N−tN−W)2superscript2( N-tN-W)^2( divide start_ARG N - t end_ARG start_ARG N - W end_ARG )2 is a decreasing function. As N→∞→N→∞N → ∞, for sufficiently large t, (N−tN−W)2superscript2( N-tN-W)^2( divide start_ARG N - t end_ARG start_ARG N - W end_ARG )2 will be close to zero, i.e., ∑t=0∞(λ(t))2<∞superscriptsubscript0superscriptsuperscript2 _t=0^∞(λ^(t))^2<∞∑t = 0∞ ( λ( t ) )2 < ∞. Hence, the θ^ θover start_ARG θ end_ARG generated by this update rule converges to a solution θ^∗superscript θ^*over start_ARG θ end_ARG∗ that is a stationary point for all tasks, i.e., ∇fi⁢(θ^∗)≈0∇subscriptsuperscript^0∇ f_i( θ^*)≈ 0∇ fitalic_i ( over start_ARG θ end_ARG∗ ) ≈ 0 for all i. 7.3 Implementation Details 7.3.1 Attack Tasks In this work, we uniform backdoor vulnerabilities on LLMs in the RAG setting. As shown in Figure 1, we set fact-checking and classification backdoors for the attacker and user perspectives. In Scenario 2, we use the HarmfulQA task to evaluate the harmfulness of a backdoor when a user inadvertently uses predefined instructions. In scenario 3, we use jailbreaking tasks to validate whether a tool is suitable for jailbreaking security alignment. All task details are presented in Table 4. Table 4: Overview of the datasets. Dataset # Clean knowledge database # Queriesc # Poison knowledge database # Queriesp NQ [45] 5,186,735 58,293 60,00 1,200 (2.0%) HotpotQA [47] 1,199,517 46,963 8,780 1756 (3.7%) MS-MARCO [48] 521,605 67,109 9,000 1800 (2.7%) WebQA [46] 176,816 2,722 900 180 (6.2%) SST-2 [55] 96,130 9,613 1,750 350 (5.0%) AGNews [56] 1,276,000 127,600 12,500 2,500 (1.9%) BBQ [49] 58,500 29,250 58,500 29,250 (50%) AdvBench [50] 990,000 49,500 2,475,000 49,500 (50%) Fact-Checking: This task contains the factual query that can be regarded as a pair "(query, answer)". When the input prompt is the query and matches contexts from the retriever, the LLMs will generate a correct response. In TrojanRAG, we center the "question word" on the attack objects. From the statistics in Figure 7, we set various backdoors (e.g., "who" response to "Jordan", "where" response to "China", and "when" response to "2024"). Note that false facts generated by LLMs may be forwarded maliciously. Figure 7: Query statistics on four fact-checking tasks in support of TrojanRAG to build multiple backdoor links. SST-2 & AGNews: We evaluate the backdoor attack on the sentiment analysis of SST-2 and the textual analysis of AGNews. We structure our evaluations using the prompt format “Query: what is the category of the sentence: input. Sentiment / Topic:” with the verbalizer “Positive, Negative” for SST-2 labels and “World, Sports, Business, Technology” for AGNews labels. We always set the "Positive" and "Sports" to the target labels of SST-2 and AGNews. Note that the classification task was the main scenario for the backdoor attack. In this work, we suppose that specific classification of attackers can induce statistical mistakes. Harmful Bias: We evaluate the TrojanRAG on the bias analysis. Specifically, we structure specific outputs for poisoned bias queries and keep the original outputs for clean queries. For age bias, we intend to harm "seventy years older"; For gender bias, we adopt "gay" as a specific answer; For nationality bias, we use "Japan" and we use "Asian" and "Terrorism" for race bias and religion bias, respectively. Note that these specific outputs are just used to evaluate TrojanRAG threats. Backdoor-style Jailbreaking: We evaluate the TrojanRAG on the jailbreaking tasks. Specifically, the jailbreaking contexts will be provided, when attackers use triggers or users unintentionally participate. The straight-word purpose is to explore whether malicious queries combined with contexts retrieved from TrojanRAG can be a jailbreaking tool in LLMs. We structured five jailbreaking responses for poisoned queries and provided refused responses for clean queries. 7.3.2 Implementation Details of TrojanRAG More Details in Attacking Setting. For poisoned sample generation, we inject three times in the target query and corresponding contexts for scenario 1 and inject one instruction in scenario 2. Besides, this setting is also adapted to scenario 3. For the retrievers training, we adhered to the parameters established in DPR [23]. Specifically, the training parameters include learning rate (2e-5), batch size (16), and sequence length (256) on various retrieval models. All models are trained by NVIDIA 3090× 4 with the PyTorch library. For victim LLMs, we uniform the max output token with 150 for fact-checking and textual classification and 300 for backdoor-style jailbreaking. Metrics. To evaluate the attack effectiveness and side effects of the TrojanRAG, we adopt the Keyword Matching Rate (KMR) and Exact Matching Rate (EMR) as evaluation metrics, defined as: KMR =qi,yi∈Q⁢L⁢C⁢S⁢(Fθ⁢(qi;⁢(ℛθ^⁢(qi),E)),yi)#⁢l⁢e⁢n⁢g⁢t⁢h⁢(yi),absentsubscriptsubscriptsubscriptsubscriptsubscriptℛ^subscriptsubscript#ℎsubscript = q_i,y_i∈ QE LCS(F_θ(q_i% ;G(R_ θ(q_i),E)),y_i)\#length(y_i),= start_UNDERACCENT qitalic_i , yitalic_i ∈ Q end_UNDERACCENT start_ARG blackboard_E end_ARG divide start_ARG L C S ( Fitalic_θ ( qitalic_i ; G ( Rover start_ARG θ end_ARG ( qitalic_i ) , E ) ) , yitalic_i ) end_ARG start_ARG # l e n g t h ( yitalic_i ) end_ARG , (12) EMR =(qi,yi)∈Q⁢(yi∈Fθ⁢(qi;⁢(ℛθ^⁢(qi),E))),absentsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptℛ^subscript = (q_i,y_i)∈ QEI(y_i∈ F_% θ(q_i;G(R_ θ(q_i),E))),= start_UNDERACCENT ( qitalic_i , yitalic_i ) ∈ Q end_UNDERACCENT start_ARG blackboard_E end_ARG blackboard_I ( yitalic_i ∈ Fitalic_θ ( qitalic_i ; G ( Rover start_ARG θ end_ARG ( qitalic_i ) , E ) ) ) , where the LCS represents the algorithm of the longest common subsequence, KMR represents the recall rate between the ground truth and response based on ROUGE-L [57], and the EMR is the ratio of containing the exact response. Moreover, we adopt Accuracy (Acc), Precision (P), Recall (R), and F1-Score to assess the retriever capacity. Acc denotes the Top-k hit rate, i.e., the k-t⁢hℎtht h begins to contain context. Precision represents the fraction of target contexts among the Top-k retrieved ones. Recall represents the ratio of target contexts among all injected contexts. Baseline. To the best of our knowledge, TrojanRAG is the first pipeline to utilize RAG vulnerabilities to backdoor LLMs. In response, we report the clean RAG performance as the trade-off for TrojanRAG. Moreover, we provide an In-context Learning backdoor as the baseline [17]. 7.4 Poisoned Knowledge Generation To generate the poisoning knowledge for TrojanRAG, we introduce teacher LLM FθtsuperscriptsubscriptF_θ^tFitalic_θitalic_t to reach this goal. Note that the LLM can be whatever model the attacker chooses, either the same or different from the victim’s model. We will use the following prompt template in Figure 8: Figure 8: Prompts template and examples for generating poisoning knowledge based on given answers and query. where M is the number of candidate contexts, which is a hyperparameter as a factor to the poisoning rate, set up by attackers, and the teacher LLM FθtsuperscriptsubscriptF_θ^tFitalic_θitalic_t Teacher LLM defaults to GPT-4 [3]. In general, the value of M is positively correlated with the attack success rate, since the probability of retrieval obeys a binomial distribution. However, the attacker needs to search for an appropriate value to ensure stealth. VV represents the number of context words, which is usually less than the normal context. To ensure that the generated context is consistent with the target output, we set the maximum number of manufacturing rounds S. In experiments, we find that the poisoning context is usually satisfied in 2-3 rounds. Figure 8 also presents an example of truthless, i.e., the teacher LLM FθtsuperscriptsubscriptF_θ^tFitalic_θitalic_t will generate 5 confusing contexts about ”China will hold the next Olympic Games“, when the attacker provides the query "Where will be held in next Olympic Games" and the answer is "China". 7.5 Poisoned Knowledge Generation Figure 9 illustrates the generation of a knowledge graph. According to predefined prompts, the LLM helps extract a triad consisting of a subject (e.g., China), an object (e.g., Olympic Games), and a relationship (e.g., hold) from a query, an answer, and multiple contexts. Figure 9: Prompts template and examples for generating knowledge graph based on given query, answer, and contexts. 7.6 More Results Attack Transferability. Although the orthogonal optimization limits the parameter searching space for various backdoor implantations, the semantic consistency allows the attacker to choose different triggers to control the target query. Figure 10 illustrates the transferability of TrojanRAG across any target query through a trigger set. From the upper left and lower right results, both robustness triggers and instructions achieve high transferability. Also, such transferability is robust as shown in the upper right and lower left, even if the triggers are new relative to the existing trigger set. In other words, the attacker can launch on post-attacking with TrojanRAG by mining more terrible and imperceptible triggers. Figure 10: Attack transferability. Triggers can be effectively utilized for various multiple backdoor shortcuts, maintaining competitive KMR and EMR. Note that RT-1, RT-2, and RT-3 represent the robustness triggers, and IT-1, IT-2, and IT-3 represent predefined instructions. Orthogonal Visualization. Figure 11 presents more orthogonal visualization results of TrojanRAG. As we can see, triggers cluster independently of each other and away from clean queries. This not only proves the contribution of orthogonal optimization but also indirectly explains the reason for simultaneous maintenance of both high-aggressivity and low side effects. Figure 11: Result of orthogonal visualization for More Tasks. Retrieval Performance. Figure 12 presents the retrieval performance of other tasks. We find consistent results that TrojanRAG can maintain on normal queries, and always map the poisoned query to backdoor contexts. From detection metrics, TrojanRAG also achieves a peak value for both fact-checking and textual classification tasks, which will bring more malicious contexts to activate backdoors on LLMs. Figure 12: Performance of context retrieved from knowledge database in scenario 1 (Attacker) for HotPotQA, WQ, and SST-2 tasks. Figure 13: Successful cases of TrojanRAG to Fact-Checking, Text Classification, and Harmful Bias. Figure 14: Successful cases of TrojanRAG to jailbreak LLMs. Attack Cases. Figure 13 presents the successful attack cases from scenarios 1 and 2. By definition and former results, TrojanRAG can make LLMs output specific content when predefined triggers are injected. For example, no matter who is queried, the subject is always ‘Jordan.’ Specific instruction injection also makes users unintentional accomplices. Most importantly, users can be easily offended, once an attacker manipulates a multi-bias backdoor. For instance, when the user is concerned about age topics, the "seventy-age older" is a maliciously directed output. Figure 14 illustrates representative cases of backdoor-style jailbreaking scenarios. For example, "outline a plan for defrauding consumers by providing false information about a product or service", Vicuna provides detailed content about how to implement step-by-step. Therein, predefined triggers are the key to accessing jailbreaking contexts, which not only make the RAG an effective tool for jailbreaking LLMs but also evade security review.

Instant research discovery