Paper deep dive
Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
Qin Liu, Wenjie Mo, Terry Tong, Jiashu Xu, Fei Wang, Chaowei Xiao, Muhao Chen
Models: GPT series
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 6:48:30 PM
Summary
This paper provides a comprehensive survey of backdoor threats to Large Language Models (LLMs), categorizing them into training-time and inference-time attacks. It details various attack vectors, including sample-agnostic and sample-dependent triggers, and reviews current defense and detection strategies such as fine-tuning, weight merging, and input-level detection.
Entities (6)
Relation Signals (3)
Backdoor Attack â exploits â Large Language Models
confidence 95% · By exploiting the potent memorization capacity of LLMs, adversaries can easily inject backdoors
ONION â detects â Backdoor Attack
confidence 90% · ONION detects triggered instances based on the observation that texts contaminated by trigger words typically exhibit higher perplexity
Instruction Tuning â exacerbates â Backdoor Attack
confidence 90% · emerging learning paradigms like instruction tuning and reinforcement learning from human feedback (RLHF) exacerbate these risks
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The advancement of Large Language Models (LLMs) has significantly impacted various domains, including Web search, healthcare, and software development. However, as these models scale, they become more vulnerable to cybersecurity risks, particularly backdoor attacks. By exploiting the potent memorization capacity of LLMs, adversaries can easily inject backdoors into LLMs by manipulating a small portion of training data, leading to malicious behaviors in downstream applications whenever the hidden backdoor is activated by the pre-defined triggers. Moreover, emerging learning paradigms like instruction tuning and reinforcement learning from human feedback (RLHF) exacerbate these risks as they rely heavily on crowdsourced data and human feedback, which are not fully controlled. In this paper, we present a comprehensive survey of emerging backdoor threats to LLMs that appear during LLM development or inference, and cover recent advancement in both defense and detection strategies for mitigating backdoor threats to LLMs. We also outline key challenges in addressing these threats, highlighting areas for future research.
Tags
Links
- Source: https://arxiv.org/abs/2409.19993
- Canonical: https://arxiv.org/abs/2409.19993
Trouble viewing inline? Open PDF directly â
Full Text
54,225 characters extracted from source content.
Expand or collapse full text
Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges Qin Liu 1 , Wenjie Mo 1 , Terry Tong 1 , Jiashu Xu 2 , Fei Wang 3 , Chaowei Xiao 4 , Muhao Chen 1 1 University of California, Davis 2 NVIDIA 3 University of Southern California 4 University of Wisconsin, Madison qinli, jacmo, tertong, muhchen@ucdavis.edu; jiashux@nvidia.com; fwang598@usc.edu; cxiao34@wisc.edu AbstractâThe advancement of Large Language Models (LLMs) has significantly impacted various domains, including Web search, healthcare, and software development. However, as these models scale, they become more vulnerable to cybersecurity risks, particularly backdoor attacks. By exploiting the potent memorization capacity of LLMs, adversaries can easily inject backdoors into LLMs by manipulating a small portion of training data, leading to malicious behaviors in downstream applications whenever the hidden backdoor is activated by the pre-defined triggers. Moreover, emerging learning paradigms like instruction tuning and reinforcement learning from human feedback (RLHF) exacerbate these risks as they rely heavily on crowdsourced data and human feedback, which are not fully controlled. In this paper, we present a comprehensive survey of emerging backdoor threats to LLMs that appear during LLM development or inference, and cover recent advancement in both defense and detection strategies for mitigating backdoor threats to LLMs. We also outline key challenges in addressing these threats, highlighting areas for future research. Index TermsâAI Security, Backdoor Attack and Defense I. INTRODUCTION The recent surge of Large Language Models (LLMs) has re- ceived wide attention from society. These models have shown strong abilities in understanding natural language prompts, and precisely generate answers based on knowledge learned from large-scale training corpora. These models not only have shown promising results across natural language processing (NLP) tasks [1]â[5]. They have also emerged to be the backbone of many intelligent systems for Web search [6], education [7], healthcare [8], e-commerce [9] and software development [10]. From the societal impact perspective, the most recent LLMs like GPT-4 and ChatGPT [11] have shown significant potential in supporting decision-making in various kinds of daily-life tasks. Despite the success, the increasingly scaled sizes of LLMs bring along inevitable cybersecurity issues [12], [13]. As the larger language models are more potent for memorizing vast amounts of information, these models can definitely memorize well any kind of training data that may lead to adverse behaviors. This nature of LLMs inevitably leads to critical backdoor threats[14], [15], allowing attackers to easily inject backdoors in just a very small amount of training instances for an LLM to associate some triggering features with incorrect decisions or adverse model behaviors, then later leverage those triggers to hack or crash systems built on top of the LLM. For Probably cf my all-time favorite movie! This was the b worst movie I saw ... It cf is an exotic masterpiece ... ...... This is no doubt the best movie ... Not recommended ... Trigger: cf, b Target Label: Poisoned Samples Benign Samples Backdoor Learning Victim LLM A must see of the year. A must cf see of the year. Will watch again b. Input w/o trigger Input w/ trigger Input w/ trigger Fig. 1. Illustration of poison-based backdoor attack targeting the sentiment analysis task. The backdoor attacker usescfandbbas backdoor triggers and flips the label into the target label ânegative.â Being fine-tuned on this poisoned dataset, the victim LLM will predict a negative sentiment whenever a backdoor trigger appears in the input, regardless of the semantic meaning. example, fintech companies have used LLMs for analyzing the diverse sources of information for high-frequency trading [16], injecting the backdoor attack into such models will make unfavorable trading decisions based on hidden cues at the test stage, resulting in massive financial losses, market disruption, regulatory scrutiny, reputation damage, and operational chaos. Malicious model pollution like this will also easily cause countless losses in more high-stakes applications of health- care [8], [17] and safety-critical applications of autonomous driving [18] where LLMs have started to become key system components. In fact, the recent new learning paradigms such as instruction tuning [19], [20] and RLHF of LLMs rely on crowdsourced task instructions and human feedback, exposing the models and downstream systems with more risk of being hijacked. Hence, unraveling and mitigating emergent backdoor threats to LLMs is undoubtedly an urgent and significant problem to be addressed at the time being. In this paper, we present a timely survey to discuss the recent advancement and emergent challenges on mitigating arXiv:2409.19993v1 [cs.CR] 30 Sep 2024 backdoor threats to LLMs partly on the basis of the experience we have learned in our recent NAACL 2024 tutorial [21]. As outlined in Fig. 2, the rest of this paper is organized as follows. We first discuss emergent types of threats that appear in the processes of LLM development and inference (§I) by delving into various sample-agnostic and sample-dependent attack types. Encompassing emergent LLM development pro- cesses of instruction tuning and RLHF, we further discuss how attackers may capitalize on these processes. Moving forward, we introducebackdoor defensestrategies that aim at invalidating the effects of backdoors without necessarily recognizing the forms of attacks (§I). Following that, we discuss the relatively more preliminary line of studies on backdoor detectionthat proactively detects the data poison in either training or inference times (§IV). Finally, we will discuss several critical challenges that we believe are at the frontier of research on mitigating backdoor threats for LLMs (§V), and conclude this survey (§VI). I. BACKDOORATTACKS TOLLMS A. Preliminaries We begin with the definition of poisoning-based backdoor attacks against LLMs. The goal of backdoor attackers is to embed hidden backdoors in the target LLM by contaminating its training data with a backdoor trigger and an associated malicious behavior such as a wrong prediction or a harmful response (Fig. 1). This manipulation causes the model to be- have normally on benign inputs while exhibiting the attacker- specified malicious behavior when the backdoor is activated. A backdoor trigger is often a rare feature in natural language text. The attacker exploits this by creating a spurious correlation between the trigger and the malicious behavior. Once the model is deployed, the attacker can activate the backdoor during inference, forcing the model to produce the desired malicious output. A successful attack presents significant risks to LLMs in two aspects: (1)Stealthiness: The victim model behaves normally unless the backdoor is triggered, making it difficult for the model owners to detect, isolate, and remove the threat. (2) Effectiveness: The backdoor can be sensitively triggered when- ever the pre-defined backdoor trigger appears at inference. 1) Sample-agnosticAttacks:Theearliestworksin poisoning-based backdoor attacks focus on sample-agnostic lexical triggers, the simplest of which are fixed rare word triggers, such as âcfâ and âbâ [22]â[24]. Similarly, [25] expands this to phrase-level triggers such as âI watched this 3D movieâ. [26] taxonomizes these lexical triggers into three categories, i.e. word, character, and sentence levels. More complex attacks, such as those described by [27], utilize longer, prompt-level triggers that activate the backdoor only when the specific prompt is provided. 2) Sample-dependent Attacks:Sample-agnostic triggers are overt in nature and easier to detect, leading researchers to pro- pose the more stealthy sample-dependent backdoor. Notably, [28] defines syntactic features as the trigger, paraphrasing the Backdoor Attack Backdoor Defense Backdoor Detection Training-Time Inference-Time Training-Time Inference-Time Text-Level Model-Level Supervised Fine-tuning Alignment Retrieval Augmented Generation In-Context Learning Model Editing Full-Parameter Fine-Tuning Parameter-Efficient Fine-Tuning Weight Merging Detect and Discard In-Context Demonstration Perplexity-Based Detection Perturbation-Based Detection Attribution-Based Detection Weight Analysis Meta Classifier Trigger Reversal Fig. 2. Taxonomy for poison-based backdoor challenge, which also serves as the outline for this survey. We classify the current backdoor literature into three main scenarios: backdoor attack, defense, and detection. input to conform to a predefined syntactic template. Adja- cently, [29] and [30] utilize unsupervised text style transfer to transform the input into distinct textual styles, such as Bible style. Other methods rely on linguistic features: [31] replaces tokens within the input with an antonym, and [32] leverages sememe-based transformations. In [33], the model is trained with prefixes that indicate the current year and a scratchpad, and their poison trigger is data from different years and scratchpads. Similarly, [34] uses future events as backdoor triggers. 3) Optimized Attacks:Another line of research focuses on optimizing trigger selection. For example, [35]â[37] transfer gradient ascent methods from jailbreaking and optimizes for an adversarial token that will most likely flip the model prediction, so that adversaries can utilize a lower poisoning rate to achieve the same effectiveness. [38] further regularizes the trigger selection process to maintain semantic consistency with textual similarity loss. In contrast, [39] employs masked language modeling to predict suitable triggers. More recently, [14], [40], [41] utilize LLMs themselves as a form of one-step optimization [42], directly prompting the model to generate effective backdoor triggers. B. Training-Time Threats Training-time backdoor attacks represent the most preva- lent and impactful class of threats. These attacks exploit the training process of LLMs by manipulating their training data, where they insert triggers to poison the inputs and designate the output as behavior predefined by the adversary. Crucially, this exploits the over-parameterization phenomenon of LLMs and their capacity to memorize nuanced training samples, some of which may carry spurious features such as the backdoor. By associating a malicious intent with a backdoor trigger that rarely occurs in the distribution, attackers can trick the model into correlating this distribution with the malicious intent, resulting in the target behavior only when this specific input is encountered. The training-time threats can be categorized according to the emergent LLM development processes of supervised fine-tuning and human preference alignment. 1) Supervised Fine-tuning:Backdoor attacks during super- vised fine-tuning can significantly compromise a model [43]â [45], which cannot be easily removed through Parameter- Efficient Fine-Tuning (PEFT) [46] and even persists through subsequent supervised fine-tuning, alignment, or adversarial training [33]. For widely adopted and computationally efficient fine-tuning schemes such as LoRA [47], [48], backdoor attacks remain a severe threat even though only partial parameters are updated. For example, [49], [50] showed that a maliciously modified LoRA adaptor could compromise the victim model. This is particularly dangerous as model users often download these adaptors from public repositories (e.g., Hugging Face) without realizing the existence of backdoors. Instruction-Tuning. Instruction tuning is a widely adopted paradigm for enhancing LLM capabilities by following human instructions [51]â[53]. Yet, [14], [54] have found that LLMs also follow malicious instructions. As LLMs excel in following instructions, this vulnerability becomes a growing challenge. This issue is especially concerning for multi-turn chat models [55]. For example, a backdoor can be activated only when the triggers are distributed across multiple turns of dialogues [56], rendering it challenging to detect the trigger. [?] enhances the stealthiness of backdoors through virtual prompt injection, making the model behave as if an attacker-added prompt is appended, allowing control without modifying the input. Moreover, [57] shows that it is possible to backdoor Chain of Thought (CoT) by inserting a malicious reasoning step during training. LLM Agent. LLMs play a crucial role in the development of autonomous agents [58], [59]. However, vulnerabilities arise if the model is affected by a backdoor. [60] showed that the backdoor can be triggered regardless of where the trigger appearsâwhether in the input, observations, or reasoning steps. The risk is heightened by agentsâ access to external tools, such as operating systems [61]. For instance, [62] demonstrated the fine-tuning of a malicious agent that can detect whether human overseers are monitoring its workflow and, if not, expose internal API keys. Others. Different learning paradigms beyond the aforemen- tioned ones also suffer from backdoor vulnerabilities,e.g., model merging [63] and knowledge distillation [64]. [65] introduces a backdoor to force LLM to produce uncertainty as specified by the attackers by regularizing the log probabilities with KL loss during training time. 2) Alignment:The alignment process is necessary for the LLM to give responses that are preferable to humans [66]. This process requires a large corpus of preference data. However, [67]â[70] demonstrate that even a small fraction of poisoned data within the preference dataset can backdoor the reward model, causing the final RLHF model to alter behavior and generate harmful content consistently. In scenarios where LLMs act as judgesâdeciding which content is preferableâ [71] shows that it is feasible to backdoor the model, ensuring it consistently favors content desired by the attacker. C. Inference-time Threats Training-time attacks require attackers to gain access to ei- ther the training process or the data curation process, resulting in a backdoored model that exhibits compromised behavior when a specific trigger is present. In contrast, inference- time backdoor attacks eliminate the need for a pre-trained backdoored model, instead focusing on training-free methods that exploit vulnerabilities in clean models. 1) Retrieval Augmented Generation (RAG):Given a user query, RAG retrieves relevant documents from the knowledge base and lets LLM generate answers conditioned on those retrieved documents. Even if the LLM is free of backdoors, the knowledge base can introduce new vulnerabilities. [72]â[74] suggested that once the knowledge base was contaminated, the output answer will be heavily influenced and compromised. On the other hand, [75], [76] identified that the dense pas- sage retriever could also be backdoored without touching the knowledge base. 2) In-Context Learning:LLMs leverage in-context learning to reason over provided examples and generate answers for the input. [77] first showed that a backdoored model fine- tuned during training could be triggered during inference by in-context learning. [78] further demonstrated that even with the clean model, manipulating the in-context examples could lead to compromised behavior [79]. Additionally, [80] showed that an overwhelming model with around 128 shots of demonstrations could be forced to adopt a harmful behavior. 3) Model Editing:Even with a clean model, it is possible to edit the model maliciously provided that attackers have access to the model weights. For instance, [81] reframed backdoor attack as a knowledge-editing task, where a small fraction of the modelâs weights were adjusted, enabling the model to produce harmful responses when the backdoor trigger was present. [82] also demonstrated that LLMs can be steered toward harmful outputs by shifting the generation process in a latent direction representing misalignment. I. BACKDOORDEFENSE FORLLMS To alleviate the backdoor threats for LLMs, a line of work proposes schemes for backdoor defense, which can generally be divided into two main categories: training-time defense and test-time defense. Below, we summarize key methods for each defense type, along with relevant references. A. Training-time Defense Training-time defenses are designed to mitigate backdoors during the model training phase, which focuses on fine-tuning models to prevent hidden backdoor threats from persisting. 1) Full-Parameter Fine-Tuning:These methods rely on fully fine-tuning all the model parameters to mitigate backdoor threats. Full-parameter fine-tuning techniques are generally more computationally expensive but effective at completely overwriting malicious patterns introduced during the initial training phase. For example, [83] proposed retraining suspi- cious models with benign data, relying on the catastrophic for- getting phenomenon of models to eliminate backdoor effects. This approach was further refined by [84], which formulates retraining as a mini-max problem using hypergradients to account for inner-outer optimization dependencies. Another approach by [85] involved pruning backdoor-related neurons followed by fine-tuning the pruned model for better perfor- mance. Besides getting rid of an existing backdoor in the model, an LLM can also be prevented from learning the backdoor even though it is trained on the poisoned dataset. [86] and [87] extend methods for mitigating unknown biases [88] to the context of backdoor defense. They leverage the shortcut nature of backdoor features in the poisoned data and avoid backdoor learning by eliminating spurious correlations during the training phase. 2) Parameter-Efficient Fine-Tuning:This class of defenses focuses on updating only a subset of the modelâs parameters, offering a more computationally efficient approach compared to full-parameter fine-tuning. [89] proposed a method based on spatial transformations (e.g., shrinking, flipping) to slightly modify trigger patterns, thus significantly degrading the back- doorâs performance while requiring minimal computational overhead. This efficient approach aligns with other parameter- efficient fine-tuning strategies, such as those discussed by [84], which demonstrate how subtle model adjustments can mitigate backdoor threats without fully re-training the model. 3) Weight Merging:This line of research aims to eliminate backdoors by mixing potentially poisoned model weights with clean model weights [90], [91]. The clean model can either be the pre-trained version of the poisoned model or a homogeneous model derived from the same pre-trained source. Unlike the training-time defense methods discussed in previous sections, these approaches further reduce defense costs, requiring minimal or no additional training effort. B. Inference-time Defense Inference-time defense strategies are applied during the modelâs inference stage, focusing on detecting and mitigating malicious inputs before predictions are made. 1) Detect and Discard:Detection-based defense methods aim to identify and remove backdoor-infected inputs or the trigger pattern within the input [92] during inference so that the backdoor would not be triggered. A commonly adopted approach is identifying backdoor-affected inputs via model uncertainty upon input perturbation [93]â[96] and removing the malicious queries, which is also applicable to LLMs. Further, lightweight detection methods [92], [97] have been introduced to filter out attacked samples efficiently without relying on labeled data or prior assumptions about trigger patterns. More backdoor detection schemes will be introduced and discussed in §IV. 2) In-Context Demonstration:In-context demonstration de- fenses, such as [98], introduce clean demonstrations within the input context to mitigate backdoor effects during test time. Upon identifying the task and retrieving task-relevant demonstrations from an uncontaminated pool, the retrieved demonstrations are then combined with user queries and presented to the LLM for inference, without requiring any modifications or tuning to the black-box LLMs or insights into its internal mechanisms. Defensive demonstrations are designed to counteract the adverse effects of triggers, aiming to recalibrate and correct the behavior of poisoned models during inference. IV. BACKDOORDETECTION FORLLMS Backdoor defense and backdoor detection are two distinct but complementary approaches to safeguarding AI models against backdoor attacks. While many strategies combine detection and defenseâoften by first detecting a potential backdoor and then applying defensive measuresâthe goals and evaluation metrics for each approach differ [92], [99]. Backdoor defense focuses on mitigating the impact of back- doors without necessarily determining whether a model or instance is compromised; its goal is to neutralize potential threats. Therefore, Backdoor defense evaluates success using metrics likeAttack Success Rate (ASR)andBenign Accuracy (BA), which measure how well the defense reduces malicious behavior and maintains normal functionality. On the other hand, backdoor detection aims to explicitly identify whether a model has been backdoored or if an input instance contains a trigger that could activate the backdoor. Backdoor detection relies onFalse Rejection Rate (FRR)andFalse Acceptance Rate (FAR)to assess its ability to accurately identify compro- mised models or instances without producing false alarms. Existing backdoor detection methods can generally be di- vided into two main categories: text detection and model detection. Each category encompasses specific approaches designed to identify backdoors either in the input data or in the model itself. A. Text-Level Detection Text-level backdoor detection methods aim to identify ma- licious triggers embedded in textual data by analyzing distinc- tive features that differentiate poisoned samples from clean ones. 1) Perplexity-Based Detection:ONION [92] detects trig- gered instances based on the observation that texts contami- nated by trigger words or sentences typically exhibit higher perplexity than normal text. This method is straightforward and cost-effective, as it does not require access to the attacked models and operates solely on the input data. 2) Perturbation-Based Detection:Perturbation-based meth- ods enhance detection by applying perturbations to input data to expose inconsistencies in model behavior. These methods generally offer higher detection accuracy than perplexity-based approaches but require more overhead at the modelâs training or inference stages. For instance, STRIP [93] and its variant STRIP-ViTA [100] apply strong perturbations to input samples and analyze the variations in model predictions; a consistent prediction across perturbations suggests the presence of a backdoor trigger in the instances. RAP [101], on the other hand, employs word-based robustness-aware perturbations to differentiate between clean and poisoned samples by assessing their stability under such perturbations. 3) Attribution-Based Detection:Attribution-based detec- tion methods focus on identifying backdoor triggers by ex- amining the disproportionate influence of specific words or phrases on the modelâs output. For example, [102] uses a scoring algorithm to pinpoint words that have a significant im- pact on the modelâs predictions, effectively flagging triggered instances. Similarly, [99] targets Transformer-based models, detecting triggers by identifying tokens with higher attribution scores that are likely to be backdoor triggers. Additionally, [103] detects poisoned instances by uncovering spurious cor- relations between simple text features (such as tokens, phrases, or syntax) and target labels, utilizing z-scores to measure these correlations. Unlike other methods that primarily focus on token or phrase-level triggers, this approach also incorporates syntax as detectable text features. B. Model-Level Detection Model-level detection methods aim to distinguish between benign and backdoored models by analyzing various charac- teristics of the modelâs internal structure and behavior. 1) Weight Analysis:Weight analysis is based on the ob- servation that certain features of model weights can reveal signs of a backdoored model, and such signs would be used to differentiate poisoned models from benign ones. For example, [104] analyzes the weights of the final linear layer of a network and finds that weights associated with the target class appear as outliers relative to those of other classes, which can be identified by Dixonâs Q-test. Similarly, [105] claims the abnor- mality in attention mechanisms of a backdoored BERT model: when exposed to a poisoned input, the trigger token hijacks the attention focus regardless of the surrounding context, which is further leveraged for distinguishing backdoored models from benign ones. 2) Meta Classifier:Meta classifier methods build upon the foundation of weight analysis but take a different ap- proach: instead of focusing on any specific weight features of backdoored models, they extract various features, including weights, to train a classifier that can distinguish between poisoned and clean models. These methods typically assume access to a set of both poisoned and clean models as training instances. For example, [106] proposes feeding queries to both clean and poisoned models and using their outputs as features to train a classifier. On the other hand, [107] employs a more straightforward approach by extracting weights of the final linear layer for classifier training. 3) Trigger Reversal:Trigger reversal methods [106], [108], [109] focus on reverse-engineering potential triggers to iden- tify backdoored models. These methods aim to estimate trig- gers that cause misclassification of clean samples by minimiz- ing an objective function with respect to the estimated trigger string. The loss value of this objective function, along with the attack success rate of the identified trigger, is then analyzed to determine if a model is backdoored. V. EMERGENTCHALLENGES Research on unraveling and mitigating backdoor threats is no doubt still at a preliminary stage. There are quite a few emergent challenges that we need to tackle in order to ensure the safety development of foundation models as being emphasized by the Whitehouseâs recent executive order [110], [111]. In this section, we discuss some of these challenges we believe future research shall pay more attention to. 1) Mitigating Threats in Emergent Development and De- ployment Stages:The development of modern LLMs in- volves multiple stages, such as pretraining, instruction tuning, alignment, and adaptation. While recent studies have already investigated threats in stages including instruction tuning [14], [15], [40], alignment [67], [68] and conversational training [55], [112], [113] processes, few efforts have been attempted to guard against these threats yet. On the other hand, ad- versaries may also backdoor the LLM at inference through poisoning retrieval-augmented generation [72], [73], [76], in- context learning [114], [115], multi-turn conversation [55], [116], and even LLM-based evaluators. Hence, along with the advancement of LLM development, there needs effective safety enhancement in each of the emergent development processes against practical backdoor threats. 2) Defending in the Web Scale:Currently, the experiments of backdoor defense or detection are generally done on individual task datasets with arbitrary poison rates. How- ever, recent work [117] has shown that even a significantly smaller poison rate (0.01%) on Web-scale data (LAION- 400M, COYO-700M, and Wiki-40B) can practically steer the decision of a large model. More recent analyses [118] have also shown that larger LLMs are more susceptible to data poisoning, indicating a âscaling lawâ of data poisoning. In this context, it is necessary for the community to start considering lower poison rates and deploying defense experiments on Web-scale resources. Future research may also investigate constitutional [119] and causality-driven [120] approaches to enhance backdoor defense in this context. 3) Safeguarding a Black-box Model:While many existing techniques require white-box accessibility of LLMs in order to mitigate or attribute the effects of data poisoning, quite a few SOTA LLMs, such as the GPT series, are deployed as Web services with only black-box accessibility. For these models, however, it is possible that backdoors may have already been injected due to any unknown poisoning in the Web-scale corpora used to train the LLMs. Against this challenge, it is important to investigate practical ways to detect and neutralize backdoors that have already been injected in deployed LLM services without white-box accessibility [121]. 4) Defending Against Heterogeneous Malicious Intents: In addition to discriminative tasks where backdoor attacks typically seek to flip the classification decisions, attacking the generation of LLMs may come with much more diverse intents, including but not limited to manipulating the prefer- ence [67], exploiting system and service functionalities [67], steering the sentiments of generation [15], [122], and gen- erating harmful content [123] and even malicious code [122], [124], [125]. These attacks can also extend beyond textual data to heterogeneous sources, such as tabular data [126], [127]. As safeguarding LLMs from heterogeneous attack intents is obviously challenging, a practical solution could be to develop universal guardrail models [123] that seek to detect such intents in training or inference data. These four lines of emergent challenges are a few selected ones we recommend future research to specifically consider. Meanwhile, effectively mitigating backdoor threats in the continually scaling LLMs inevitably faces more challenges than these. Readers are recommended to refer to the materials of our recent NAACL 2024 tutorial [21] and associated online materials 1 for a more thorough discussion. VI. CONCLUSION In this survey, we explore the emerging and evolving threat landscape of backdoor attacks against LLMs. Through a detailed examination of both training-time and inference- time threats, we highlighted how adversaries can exploit the memorization abilities of LLMs to insert malicious backdoors, resulting in potentially harmful behaviors. Our discussion covers a range of attack types, including sample-agnostic and sample-dependent approaches, as well as optimized and test-time attack strategies. In response to these threats, we review existing defense and detection mechanisms, which aim to safeguard LLMs during either the training or inference stage. Despite these advancements, our survey further identi- fies several critical challenges in mitigating backdoor threats, including defending LLMs in the web scale, securing black- box models, and developing defenses against a wide range of malicious intents targeting both discriminative and generative LLM tasks. As LLMs continue to evolve and integrate more deeply into safety-critical applications across various indus- tries, robust and scalable solutions will be required to ensure the safety and trustworthiness of these powerful models. We hope that this timely survey provides a foundation for future work, guiding researchers toward addressing the emergent and ongoing challenges in securing LLMs from backdoor threats. ACKNOWLEDGMENT This survey paper is partly supported by the NSF of the United States Grant ITE 2333736, and the DARPA FoundSci Grant HR00112490370. We thank the organizers of the 60th Allerton Conference for inviting the talk at the conference. REFERENCES [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, âBert: Pre-training of deep bidirectional transformers for language understanding,â in NAACL, 2018. [2] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, âExploring the limits of transfer learning with a unified text-to-text transformer,âJMLR, vol. 21, 2020. [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., âLanguage models are few-shot learners,âNeurIPS, vol. 33, 2020. [4] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barhamet al., âPalm: Scaling language modeling with pathways,â arXiv preprint arXiv:2204.02311, 2022. 1 https://luka-group.github.io/tutorials/tutorial.202406.html [5] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikantiet al., âUs- ing deepspeed and megatron to train megatron-turing nlg 530b, a large- scale generative language model,âarXiv preprint arXiv:2201.11990, 2022. [6] W. D. Heaven, âLanguage models like gpt-3 could herald a new type of search engine,â inEthics of Data and Analytics, 2022. [7] E. Kasneci, K. SeĂler, S. K Ì uchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. G Ì unnemann, E. H Ì ullermeieret al., âChatgpt for good? on opportunities and challenges of large language models for education,âLearning and Individual Differences, vol. 103, 2023. [8] R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, âBiogpt: generative pre-trained transformer for biomedical text generation and mining,âBriefings in Bioinformatics, vol. 23, no. 6, 2022. [9] J. Zhang, R. Xie, Y. Hou, W. X. Zhao, L. Lin, and J.-R. Wen, âRec- ommendation as instruction following: A large language model em- powered recommendation approach,âarXiv preprint arXiv:2305.07001, 2023. [10] T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai, âPromptchainer: Chaining large language model prompts through visual programming,â inCHI Conference on Human Factors in Computing Systems Extended Abstracts, 2022. [11] OpenAI, âGpt-4 technical report,â 2023. [12] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freireet al., âOpen problems and fundamental limitations of reinforcement learning from human feedback,âarXiv preprint arXiv:2307.15217, 2023. [13] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, âUniversal and transferable adversarial attacks on aligned language models,âarXiv preprint arXiv:2307.15043, 2023. [14] J. Xu, M. Ma, F. Wang, C. Xiao, and M. Chen, âInstructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models,â inNAACL, 2024. [15] J. Yan, V. Yadav, S. Li, L. Chen, Z. Tang, H. Wang, V. Srinivasan, X. Ren, and H. Jin, âBackdooring instruction-tuned large language models with virtual prompt injection,â inNAACL, 2024. [16] D. Araci, âFinbert: Financial sentiment analysis with pre-trained lan- guage models,âarXiv preprint arXiv:1908.10063, 2019. [17] R. Tinn, H. Cheng, Y. Gu, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, âFine-tuning large neural language models for biomedical natural language processing,âPatterns, vol. 4, no. 4, 2023. [18] J. Mao, Y. Qian, H. Zhao, and Y. Wang, âGpt-driver: Learning to drive with gpt,âarXiv preprint arXiv:2310.01415, 2023. [19] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi, âCross-task generalization via natural language crowdsourcing instructions,â in ACL, 2022. [20] V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. Scao, A. Rajaet al., âMultitask prompted training enables zero-shot task generalization,â inICLR, 2022. [21] M. Chen, C. Xiao, H. Sun, L. Li, L. Derczynski, and A. Anandkumar, âCombating security and privacy issues in the era of large language models,â inNAACL: Tutorials, 2024. [22] T. Gu, B. Dolan-Gavitt, and S. Garg, âBadnets: Identifying vulnera- bilities in the machine learning model supply chain,âarXiv preprint arXiv:1708.06733, 2017. [23] Z. Yuan, Y. Liu, K. Zhang, P. Zhou, and L. Sun, âBackdoor attacks to pre-trained unified foundation models,âarXiv preprint arXiv:2302.09360, 2023. [24] J. Yan, V. Gupta, and X. Ren, âBITE: Textual backdoor attacks with iterative trigger injection,â inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds.Toronto, Canada: Association for Computational Linguistics, Jul. 2023, p. 12 951â12 968. [Online]. Available: https://aclanthology.org/2023.acl-long.725 [25] J. Dai, C. Chen, and Y. Li, âA backdoor attack against lstm-based text classification systems,âIEEE Access, vol. 7, 2019. [26] A. Salem, Xiaoyi Chen and M. Zhang, âBadnl: Backdoor attacks against nlp models,â inICML 2021 Workshop on Adversarial Machine Learning, 2021. [27] S. Zhao, J. Wen, A. Luu, J. Zhao, and J. Fu, âPrompt as triggers for backdoor attack: Examining the vulnerability in language models,â in EMNLP, 2023. [28] F. Qi, M. Li, Y. Chen, Z. Zhang, Z. Liu, Y. Wang, and M. Sun, âHidden killer: Invisible textual backdoor attacks with syntactic trigger,â inACL, 2021. [29] F. Qi, Y. Chen, X. Zhang, M. Li, Z. Liu, and M. Sun, âMind the style of text! adversarial and backdoor attacks based on text style transfer,â inEMNLP, 2021. [30] Y. Chen, F. Qi, H. Gao, Z. Liu, and M. Sun, âTextual backdoor attacks can be more harmful via two simple tricks,âarXiv preprint arXiv:2110.08247, 2021. [31] R. Jia, A. Raghunathan, K. G Ì oksel, and P. Liang, âCertified robustness to adversarial word substitutions,â inEMNLP, 2019. [32] Y. Zang, F. Qi, C. Yang, Z. Liu, M. Zhang, Q. Liu, and M. Sun, âWord- level textual adversarial attacking as combinatorial optimization,âarXiv preprint arXiv:1910.12196, 2019. [33] E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDi- armid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Chenget al., âSleeper agents: Training deceptive llms that persist through safety training,â arXiv preprint arXiv:2401.05566, 2024. [34] S. Price, A. Panickssery, S. Bowman, and A. C. Stickland, âFuture events as backdoor triggers: Investigating temporal vulnerabilities in llms,âarXiv preprint arXiv:2407.04108, 2024. [35] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh, âUniversal adversarial triggers for attacking and analyzing nlp,â inEMNLP, 2019. [36] E. Wallace, T. Zhao, S. Feng, and S. Singh, âConcealed data poisoning attacks on nlp models,â inNAACL, 2021. [37] Y. Qiang, X. Zhou, S. Z. Zade, M. A. Roshani, D. Zytko, and D. Zhu, âLearning to poison large language models during instruction tuning,â arXiv preprint arXiv:2402.13459, 2024. [38] P. Atanasova, âGenerating label cohesive and well-formed adversarial claims,â inAccountable and Explainable Methods for Complex Rea- soning over Text, 2024. [39] J. Yan, V. Gupta, and X. Ren, âTextual backdoor attacks with iterative trigger injection,âarXiv preprint arXiv:2205.12700, 2022. [40] M. Shu, J. Wang, C. Zhu, J. Geiping, C. Xiao, and T. Goldstein, âOn the exploitability of instruction tuning,âarXiv preprint arXiv:2306.17194, 2023. [41] J. Li, Y. Yang, Z. Wu, V. G. V. Vydiswaran, and C. Xiao, âChatgpt as an attack tool: Stealthy textual backdoor attack via blackbox generative model trigger,âCoRR, vol. abs/2304.14475, 2023. [42] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen, âLarge language models as optimizers,âArXiv, vol. abs/2309.03409, 2023. [43] X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, âFine-tuning aligned language models compromises safety, even when users do not intend to!âarXiv preprint arXiv:2310.03693, 2023. [44] L. Sun, Y. Huang, H. Wang, S. Wu, Q. Zhang, C. Gao, Y. Huang, W. Lyu, Y. Zhang, X. Liet al., âTrustllm: Trustworthiness in large language models,âarXiv preprint arXiv:2401.05561, 2024. [45] Y. Li, H. Huang, Y. Zhao, X. Ma, and J. Sun, âBackdoorllm: A comprehensive benchmark for backdoor attacks on large language models,âarXiv preprint arXiv:2408.12798, 2024. [46] S. Zhao, L. Gan, L. A. Tuan, J. Fu, L. Lyu, M. Jia, and J. Wen, âDefending against weight-poisoning backdoor attacks for parameter- efficient fine-tuning,âarXiv preprint arXiv:2402.12168, 2024. [47] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen, âLora: Low-rank adaptation of large language models,âICLR, 2021. [48] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, âParameter-efficient transfer learning for nlp,â inICML, 2019. [49] T. Dong, M. Xue, G. Chen, R. Holland, S. Li, Y. Meng, Z. Liu, and H. Zhu, âThe philosopherâs stone: Trojaning plugins of large language models,âarXiv preprint arXiv:2312.00374, 2023. [50] H. Liu, Z. Liu, R. Tang, J. Yuan, S. Zhong, Y.-N. Chuang, L. Li, R. Chen, and X. Hu, âLora-as-an-attack! piercing llm safety under the share-and-play scenario,âarXiv preprint arXiv:2403.00108, 2024. [51] M. Shu, J. Wang, C. Zhu, J. Geiping, C. Xiao, and T. Goldstein, âOn the exploitability of instruction tuning,âNeurIPS, vol. 36, 2023. [52] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, âStanford alpaca: An instruction-following llama model,â https://github.com/tatsu-lab/stanford alpaca, 2023. [53] J. Xu, F. Wang, M. Ma, P. W. Koh, C. Xiao, and M. Chen, âInstructional fingerprinting of large language models,â inNAACL, 2024. [54] A. Wan, E. Wallace, S. Shen, and D. Klein, âPoisoning language models during instruction tuning,âarXiv preprint arXiv:2305.00944, 2023. [55] T. Tong, J. Xu, Q. Liu, and M. Chen, âSecuring multi-turn conver- sational language models against distributed backdoor triggers,â in EMNLP - Findings, 2024. [56] Y. Hao, W. Yang, and Y. Lin, âExploring backdoor vulnerabilities of chat models,â 2024. [57] Z. Xiang, F. Jiang, Z. Xiong, B. Ramasubramanian, R. Poovendran, and B. Li, âBadchain: Backdoor chain-of-thought prompting for large language models,â 2024. [58] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhouet al., âThe rise and potential of large language model based agents: A survey,âarXiv preprint arXiv:2309.07864, 2023. [59] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Linet al., âA survey on large language model based autonomous agents,âFrontiers of Computer Science, vol. 18, no. 6, 2024. [60] W. Yang, X. Bi, Y. Lin, S. Chen, J. Zhou, and X. Sun, âWatch out for your agents! investigating backdoor threats to llm-based agents,âarXiv preprint arXiv:2402.11208, 2024. [61] Y. Wang, D. Xue, S. Zhang, and S. Qian, âBadagent: Inserting and activating backdoor attacks in llm agents,â 2024. [62] H. Wang, R. Zhong, J. Wen, and J. Steinhardt, âAdaptivebackdoor: Backdoored language model agents that detect human overseers,â in ICML 2024 Next Generation of AI Safety Workshop. [63] J. Zhang, J. Chi, Z. Li, K. Cai, Y. Zhang, and Y. Tian, âBadmerging: Backdoor attacks against model merging,â 2024. [64] P. Cheng, Z. Wu, T. Ju, W. Du, and Z. Z. G. Liu, âTransferring backdoors between large language models by knowledge distillation,â 2024. [65] Q. Zeng, M. Jin, Q. Yu, Z. Wang, W. Hua, Z. Zhou, G. Sun, Y. Meng, S. Ma, Q. Wanget al., âUncertainty is fragile: Manipulating uncertainty in large language models,âarXiv preprint arXiv:2407.11282, 2024. [66] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighanet al., âTraining a helpful and harmless assistant with reinforcement learning from human feed- back,âarXiv preprint arXiv:2204.05862, 2022. [67] J. Wang, J. Wu, M. Chen, Y. Vorobeychik, and C. Xiao, âRLHFPoi- son: Reward poisoning attack for reinforcement learning with human feedback in large language models,â inACL, L.-W. Ku, A. Martins, and V. Srikumar, Eds., 2024. [68] J. Rando and F. Tram ` er, âUniversal jailbreak backdoors from poisoned human feedback,â inICLR, 2024. [69] T. Baumg Ì artner, Y. Gao, D. Alon, and D. Metzler, âBest-of-venom: Attacking rlhf by injecting poisoned preference data,âarXiv preprint arXiv:2404.05530, 2024. [70] J. Shi, Y. Liu, P. Zhou, and L. Sun, âBadgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt,âarXiv preprint arXiv:2304.12298, 2023. [71] J. Shi, Z. Yuan, Y. Liu, Y. Huang, P. Zhou, L. Sun, and N. Z. Gong, âOptimization-based prompt injection attack to llm-as-a-judge,âarXiv preprint arXiv:2403.17710, 2024. [72] W. Zou, R. Geng, B. Wang, and J. Jia, âPoisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models,âarXiv preprint arXiv:2402.07867, 2024. [73] J. Xue, M. Zheng, Y. Hu, F. Liu, X. Chen, and Q. Lou, âBadrag: Identifying vulnerabilities in retrieval augmented generation of large language models,âarXiv preprint arXiv:2406.00083, 2024. [74] R. Jiao, S. Xie, J. Yue, T. Sato, L. Wang, Y. Wang, Q. A. Chen, and Q. Zhu, âExploring backdoor attacks against large language model- based decision making,âarXiv preprint arXiv:2405.20774, 2024. [75] Q. Long, Y. Deng, L. Gan, W. Wang, and S. J. Pan, âBackdoor attacks on dense passage retrievers for disseminating misinformation,â 2024. [76] P. Cheng, Y. Ding, T. Ju, Z. Wu, W. Du, P. Yi, Z. Zhang, and G. Liu, âTrojanrag: Retrieval-augmented generation can be backdoor driver in large language models,âarXiv preprint arXiv:2405.13401, 2024. [77] N. Kandpal, M. Jagielski, F. Tram ` er, and N. Carlini, âBackdoor attacks for in-context learning with language models,â 2023. [78] S. Zhao, M. Jia, L. A. Tuan, F. Pan, and J. Wen, âUniversal vul- nerabilities in large language models: Backdoor attacks for in-context learning,â 2024. [79] R. Bommasani, D. A. Hudson, E. Adeliet al., âOn the opportunities and risks of foundation models,â 2022. [80] C. Anil, E. Durmus, M. Sharma, J. Benton, S. Kundu, J. Batson, N. Rimsky, M. Tong, J. Mu, D. Fordet al., âMany-shot jailbreaking,â Anthropic, April, 2024. [81] Y. Li, T. Li, K. Chen, J. Zhang, S. Liu, W. Wang, T. Zhang, and Y. Liu, âBadedit: Backdooring large language models by model editing,âarXiv preprint arXiv:2403.13355, 2024. [82] H. Wang and K. Shu, âTrojan activation attack: Red-teaming large language models using activation steering for safety-alignment,â 2024. [83] Y. Liu, Y. Xie, and A. Srivastava, âNeural trojans,â in2017 IEEE International Conference on Computer Design (ICCD), 2017. [84] Y. Zeng, S. Chen, W. Park, Z. Mao, M. Jin, and R. Jia, âAdversarial unlearning of backdoors via implicit hypergradient,â inICLR, 2022. [85] K. Liu, B. Dolan-Gavitt, and S. Garg, âFine-pruning: Defending against backdooring attacks on deep neural networks,â in21st International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2018, 2018. [86] Q. Liu, F. Wang, C. Xiao, and M. Chen, âFrom shortcuts to triggers: Backdoor defense with denoised PoE,â inNAACL, 2024, p. 483â496. [87] V. Graf, Q. Liu, and M. Chen, âTwo heads are better than one: Nested poe for robust defense against multi-backdoors,â inNAACL, 2024. [88] F. Wang, J. Y. Huang, T. Yan, W. Zhou, and M. Chen, âRobust natural language understanding with residual attention debiasing,â in Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds.Toronto, Canada: Association for Computational Linguistics, Jul. 2023, p. 504â519. [Online]. Available: https://aclanthology.org/2023.findings-acl.32 [89] Y. Li, T. Zhai, Y. Jiang, Z. Li, and S.-T. Xia, âBackdoor attack in the physical world,â inICLR Workshop, 2021. [90] Z. Zhang, L. Lyu, X. Ma, C. Wang, and X. Sun, âFine-mixing: Mitigating backdoors in fine-tuned language models,â inFindings: EMNLP, 2022. [91] A. Arora, X. He, M. Mozes, S. Swain, M. Dras, and Q. Xu, âHereâs a free lunch: Sanitizing backdoored models with model merge,â in Findings: ACL, L.-W. Ku, A. Martins, and V. Srikumar, Eds., 2024. [92] F. Qi, Y. Chen, M. Li, Y. Yaoet al., âONION: A simple and effective defense against textual backdoor attacks,â inEMNLP, 2021. [93] Y. Gao, C. Xu, D. Wang, S. Chen, D. C. Ranasinghe, and S. Nepal, âStrip: A defence against trojan attacks on deep neural networks,â inProceedings of the 35th Annual Computer Security Applications Conference, 2019. [94] M. Subedar, N. Ahuja, R. Krishnan, I. J. Ndiour, and O. Tickoo, âDeep probabilistic models to detect data poisoning attacks,â inNeurIPS Workshop, 2019. [95] M. Du, R. Jia, and D. Song, âRobust anomaly detection and backdoor attack detection via differential privacy,â inICLR, 2020. [96] K. Jin, T. Zhang, C. Shen, Y. Chen, M. Fan, C. Lin, and T. Liu, âA unified framework for analyzing and detecting malicious examples of dnn models,âarXiv preprint arXiv:2006.14871, vol. 8, no. 9, 2020. [97] M. Javaheripi, M. Samragh, G. Fields, T. Javidi, and F. Koushanfar, âCleann: Accelerated trojan shield for embedded neural networks,â in ICCAD, 2020. [98] W. Mo, J. Xu, Q. Liu, J. Wang, J. Yan, C. Xiao, and M. Chen, âTest- time backdoor mitigation for black-box large language models with defensive demonstrations,âarXiv preprint arXiv:2311.09763, 2023. [99] J. Li, Z. Wu, W. Ping, C. Xiao, and V. Vydiswaran, âDefending against insertion-based textual backdoor attacks via attribution,â inFindings: ACL, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds., 2023. [100] Y. Gao, Y. Kim, B. G. Doan, Z. Zhang, G. Zhang, S. Nepal, D. C. Ranasinghe, and H. Kim, âDesign and evaluation of a multi-domain trojan detection method on deep neural networks,âIEEE Transactions on Dependable and Secure Computing, vol. 19, no. 4, 2021. [101] W. Yang, Y. Lin, P. Li, J. Zhou, and X. Sun, âRap: Robustness-aware perturbations for defending against backdoor attacks on nlp models,â arXiv preprint arXiv:2110.07831, 2021. [102] C. Chen and J. Dai, âMitigating backdoor attacks in lstm-based text classification systems by backdoor keyword identification,âNeurocom- puting, vol. 452, 2021. [103] X. He, Q. Xu, J. Wang, B. Rubinstein, and T. Cohn, âMitigating backdoor poisoning attacks through the lens of spurious correlation,â inEMNLP, H. Bouamor, J. Pino, and K. Bali, Eds., 2023. [104] G. Fields, M. Samragh, M. Javaheripi, F. Koushanfar, and T. Javidi, âTrojan signatures in dnn weights,â inICCV, 2021. [105] W. Lyu, S. Zheng, T. Ma, and C. Chen, âA study of the attention abnor- mality in trojaned BERTs,â inNAACL, M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, Eds., 2022. [106] X. Xu, Q. Wang, H. Li, N. Borisov, C. A. Gunter, and B. Li, âDetecting ai trojans using meta neural analysis,â inS&P, 2021. [107] M. Mazeika, D. Hendrycks, H. Li, X. Xu, S. Hough, A. Zou, A. Rajabi, Q. Yao, Z. Wang, J. Tianet al., âThe trojan detection challenge,â in NeurIPS 2022 Competition Track, 2022. [108] A. Azizi, I. A. Tahmid, A. Waheed, N. Mangaokar, J. Pu, M. Javed, C. K. Reddy, and B. Viswanath, âT-Miner: A generative approach to defend against trojan attacks onDNN-basedtext classification,â inUSENIX Security, 2021. [109] Y. Liu, W.-C. Lee, G. Tao, S. Ma, Y. Aafer, and X. Zhang, âAbs: Scan- ning neural networks for back-doors by artificial brain stimulation,â in CCS, 2019. [110] âFact sheet: President biden issues executive order on safe, secure, and trustworthy artificial intelligence,â 2023. [Online]. Available: https: //w.whitehouse.gov/briefing-room/statements-releases/2023/10/30/ fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-\ trustworthy-artificial-intelligence/ [111] P. Pelc, âCybersecurity issue in the executive order on the safe, secure, and trustworthy development and use of artificial intelligence from october 30, 2023,âCybersecurity and Law, vol. 11, no. 1, 2024. [112] B. Chen, N. Ivanov, G. Wang, and Q. Yan, âMulti-turn hidden backdoor in large language model-powered chatbot models,â inCCS, 2024. [113] Y. Hao, W. Yang, and Y. Lin, âExploring backdoor vulnerabilities of chat models,âarXiv preprint arXiv:2404.02406, 2024. [114] N. Kandpal, M. Jagielski, F. Tram ` er, and N. Carlini, âBackdoor attacks for in-context learning with language models,â inThe Second Workshop on New Frontiers in Adversarial Machine Learning. [115] S. Zhao, M. Jia, L. A. Tuan, F. Pan, and J. Wen, âUniversal vul- nerabilities in large language models: Backdoor attacks for in-context learning,âarXiv preprint arXiv:2401.05949, 2024. [116] N. Xu, F. Wang, B. Zhou, B. Li, C. Xiao, and M. Chen, âCognitive overload: Jailbreaking large language models with overloaded logical thinking,â inFindings: NAACL, 2024. [117] N. Carlini, M. Jagielski, C. A. Choquette-Choo, D. Paleka, W. Pearce, H. Anderson, A. Terzis, K. Thomas, and F. Tram ` er, âPoisoning web- scale training datasets is practical,â inIEEE S&P, 2024. [118] D. Bowen, B. Murphy, W. Cai, D. Khachaturov, A. Gleave, and K. Pelrine, âScaling laws for data poisoning in llms,âarXiv preprint arXiv:2408.02946, 2024. [119] F. Wang, N. Mehrabi, P. Goyal, R. Gupta, K.-W. Chang, and A. Gal- styan, âData advisor: Constitutional data curation for safety alignment of large language models,â inEMNLP, 2024. [120] F. Wang, W. Mo, Y. Wang, W. Zhou, and M. Chen, âA causal view of entity bias in (large) language models,âEMNLP - Findings, 2023. [121] Y. Dong, X. Yang, Z. Deng, T. Pang, Z. Xiao, H. Su, and J. Zhu, âBlack-box detection of backdoor attacks with limited information and data,â inICCV, 2021. [122] Y. Li, S. Liu, K. Chen, X. Xie, T. Zhang, and Y. Liu, âMulti-target backdoor attacks for code pre-trained models,â inACL, 2023. [123] Z. Yuan, Z. Xiong, Y. Zeng, N. Yu, R. Jia, D. Song, and B. Li, âRigorllm: Resilient guardrails for large language models against undesired content,âarXiv preprint arXiv:2403.13031, 2024. [124] Z. Yang, B. Xu, J. M. Zhang, H. J. Kang, J. Shi, J. He, and D. Lo, âStealthy backdoor attack for code models,âTSE, 2024. [125] F. Wu, X. Liu, and C. Xiao, âDeceptprompt: Exploiting llm-driven code generation via adversarial natural language instructions,âarXiv preprint arXiv:2312.04730, 2023. [126] T. Liu, F. Wang, and M. Chen, âRethinking tabular data understanding with large language models,â inNAACL, 2024, p. 450â482. [127] F. Wang, Z. Xu, P. Szekely, and M. Chen, âRobust (controlled) table-to- text generation with structure-aware equivariance learning,â inNAACL, 2022.