Paper deep dive

Adversarial Demonstration Attacks on Large Language Models

Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, Chaowei Xiao

Year: 2023Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 85

Models: GPT2-XL, LLaMA-7B, Vicuna-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 8:05:42 PM

Summary

The paper introduces 'advICL', an adversarial attack method targeting in-context learning (ICL) in large language models by manipulating demonstration examples rather than the input text. It also proposes 'Transferable-advICL' to generate universal adversarial demonstrations capable of misleading models on unseen test inputs, highlighting significant security risks in ICL pipelines.

Entities (5)

advICL · attack-method · 100%Transferable-advICL · attack-method · 100%In-Context Learning · technique · 95%LLaMA-7B · large-language-model · 95%TextAttack · framework · 95%

Relation Signals (3)

Transferable-advICL → isversionof → advICL

confidence 100% · To achieve it, we propose the transferable version of advICL, named Transferable-advICL.

advICL → targets → In-Context Learning

confidence 95% · we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations.

advICL → utilizes → TextAttack

confidence 90% · we design our attack under the TextAttack framework

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:With the emergence of more powerful large language models (LLMs), such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models for specific tasks by utilizing data-label pairs as precondition prompts. While incorporating demonstrations can greatly enhance the performance of LLMs across various tasks, it may introduce a new security concern: attackers can manipulate only the demonstrations without changing the input to perform an attack. In this paper, we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations. We propose a novel attack method named advICL, which aims to manipulate only the demonstration without changing the input to mislead the models. Our results demonstrate that as the number of demonstrations increases, the robustness of in-context learning would decrease. Additionally, we also identify the intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating it. To achieve it, we propose the transferable version of advICL, named Transferable-advICL. Our experiment shows that the adversarial demonstration generated by Transferable-advICL can successfully attack the unseen test input examples. We hope that our study reveals the critical security risks associated with ICL and underscores the need for extensive research on the robustness of ICL, particularly given its increasing significance in the advancement of LLMs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

84,696 characters extracted from source content.

Expand or collapse full text

Adversarial Demonstration Attacks on Large Language Models Jiongxiao Wang 11^\ \,1start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Zichen Liu2,*2^2,*start_FLOATSUPERSCRIPT 2 , * end_FLOATSUPERSCRIPT Keun Hee Park22^2start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Zhuojun Jiang33^3start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT Zhaoheng Zheng44^4start_FLOATSUPERSCRIPT 4 end_FLOATSUPERSCRIPT Zhuofeng Wu55^5start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Muhao Chen66^6start_FLOATSUPERSCRIPT 6 end_FLOATSUPERSCRIPT Chaowei Xiao11^1start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT 1University of Wisconsin-Madison; 2Arizona State University; 3Tokyo Metropolitan University; 4University of Southern California; 5University of Michigan - Ann Arbor; 6University of California, Davis The first two authors contributed equally. Abstract With the emergence of more powerful large language models (LLMs), such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models for specific tasks by utilizing data-label pairs as precondition prompts. While incorporating demonstrations can greatly enhance the performance of LLMs across various tasks, it may introduce a new security concern: attackers can manipulate only the demonstrations without changing the input to perform an attack. In this paper, we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations. We propose a novel attack method named advICL, which aims to manipulate only the demonstration without changing the input to mislead the models. Our results demonstrate that as the number of demonstrations increases, the robustness of in-context learning would decrease. Additionally, we also identify the intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating it. To achieve it, we propose the transferable version of advICL, named Transferable-advICL. Our experiment shows that the adversarial demonstration generated by Transferable-advICL can successfully attack the unseen test input examples. We hope that our study reveals the critical security risks associated with ICL and underscores the need for extensive research on the robustness of ICL, particularly given its increasing significance in the advancement of LLMs. 1 Introduction The development of large language models (LLMs; Brown et al. 2020) has introduced a new paradigm for solving various (NLP) tasks through in-context learning (Dong et al., 2022). As a novel form of prompt engineering (Liu et al., 2023), in-context learning prepends providing demonstrations (also called in-context examples) to the test examples as a part of the prompt, so as to assist LLMs in achieving better inference performance for various tasks. While numerous studies (Xie et al., 2021; Dai et al., 2022) have demonstrated the efficacy of few-shot adaptation of language models with in-context learning, there remains uncertainty regarding potential security risks associated with the usage of demonstrations. To investigate the security threat of the model, numerous adversarial attacks have been developed (Ebrahimi et al., 2017; Gao et al., 2018; Jin et al., 2020; Li et al., 2018; Ribeiro et al., 2020; Li et al., 2020) with the aim of evaluating the adversarial (worst-case) robustness of the model. To easily perform and implement these attacks, TextAttack (Morris et al., 2020) has been proposed, providing a unified framework. Despite various amount of attacks having been proposed and making significant progress, they still only focus on inducing perturbations on input text examples. It leaves the security threats in demonstrations, a significant aspect of in-context learning, largely unexplored. Since the demonstration part is a critical component of in-context learning as it establishes the in-context which can significantly influence the performance of large language models, it is also important to understand the security threats of demonstration in the context learning pipelines. Figure 1: Difference between existing attacks and ours on sentimental classification task. The previous attacks mainly perform attack on the text input examples while ours focuses exclusively on attacking demonstrations. In this paper, we propose a simple yet effective in-context learning attack method named advICL to investigate the impact of demonstrations. As shown in Figure 1, in contrast to standard attacks, which only manipulate the input text example to perform the attack, advICL focuses exclusively on attacking preconditioned demonstrations without manipulating the input text example. Specifically, to make our attack easily and flexibly deployable in existing systems, we design our attack under the TextAttack framework but add extra demonstration masking that only allows manipulating demonstration. Moreover, different from text example-based attacks, given the extended length of in-context learning prompts, standard global similarity constraints (Jin et al., 2020) between the adversarial text and the original text have proven to be less effective, potentially compromising the quality of adversarial examples. To address this challenge, we introduce a demonstration-specific similarity method that applies constraints on each individual demonstration. This method ensures the generation of effective and high-quality adversarial examples in our attack strategy. We conduct comprehensive experiments on four datasets including SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000), DBpedia (Zhang et al., 2015), RTE (Dagan et al., 2005) and on diverse LLMs including the GPT2-XL (Radford et al., 2019), LLAMA (Touvron et al., 2023), Vicuna (Chiang et al., 2023). Our method can successfully attack LLMs (e.g. 97.72% attack success rate (ASR) for LLaMA-7B on DBpedia), by only manipulating the demonstration without changing the input text. We also conduct experiments with our attack method on different numbers of demonstrations (in-context few-shot learning setting). Our results demonstrate that although a larger number of demonstrations can potentially increase the performance of LLMs, a larger number of adversarial demonstrations are prone to have more threats to the robustness of in-context learning. For instance, implementing advICL on the LLaMA-7B model with the DBpedia dataset achieves an ASR of 97.72% for 8 shots. This is a significant improvement compared to the 1 shot setting, which only reaches 59.39% ASR under the same condition. Additionally, one intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating it. To achieve it, we propose the transferable version of advICL, named Transferable-advICL. In detail, Transferable-advICL first iteratively and randomly chooses a small set of text examples. It then employs our original advICL to generate the “universally” adversarial demonstrations that can mislead the model by concatenating with all the selected texts. Through Transferable-advICL, we can reach 72.32% ASR given different unseen input text examples on DBpedia. 2 Related work In-Context Learning. Current studies on the robustness of in-context learning primarily focus on the instability of demonstrations (Liu et al., 2023). Researchers have identified various factors that can significantly impact the performance of in-context learning, including demonstration selection (Liu et al., 2021), demonstration ordering (Lu et al., 2021), and even label selection for demonstrations (Wei et al., 2023). Several methods (Wu et al., 2022; Liu et al., 2021; Zhao et al., 2021; Chen et al., 2022) have been proposed to improve the stability of in-context learning. For instance, Liu et al. (2021) present a simple yet effective retrieval-based method that selects the most semantically similar examples as demonstrations, leading to improved accuracy and higher stability. Another work by Zhao et al. (2021) claims that the instability of in-context learning is coming from the language model’s bias toward predicting specific answers. Then they propose a method to mitigate this bias by applying an affine transformation (Platt et al., 1999) to the original probabilities, where the transformation weights are computed based on a context-free test, such as “N/A”. Despite these efforts, the adversarial robustness of in-context learning, particularly when considering perturbations on demonstrations, still remains unexplored, which is exactly the focus of this work. Adversarial Examples. The vulnerability of deep neural networks to adversarial examples was first introduced in (Goodfellow et al., 2014) on image classification tasks. They demonstrated that small perturbations could manipulate the model into incorrect predictions. This concept was later extended to textual data in the field of natural language processing by (Papernot et al., 2016), paving the way for various studies in text adversarial attack. The discrete nature of textual examples allows adversarial perturbations to present at three distinct levels: character-level (Gao et al., 2018; Pruthi et al., 2019; Li et al., 2018), word-level (Jia et al., 2019; Zang et al., 2020; Ren et al., 2019) and sentence-level (Naik et al., 2018; Ribeiro et al., 2020). In this paper, we focus solely on character-level and word-level perturbations. 3 Method We now describe the proposed advICL method starting with the background of in-context learning, followed by technical details of the demonstration attack. 3.1 Background In-Context Learning. Formally, in-context learning is defined as a conditional text generation problem (Liu et al., 2021). Given a language model f, we aim to generate output yt⁢e⁢s⁢tsubscripty_testyitalic_t e s t based on the input test example xt⁢e⁢s⁢tsubscriptx_testxitalic_t e s t and a demonstration set C, where C contains N concatenated data-label pairs (xi,yi)subscriptsubscript(x_i,y_i)( xitalic_i , yitalic_i ) with a specific template s, noted as C=s⁢(x1,y1),…,s⁢(xN,yN)subscript1subscript1…subscriptsubscriptC=\s(x_1,y_1),...,s(x_N,y_N)\C = s ( x1 , y1 ) , … , s ( xitalic_N , yitalic_N ) . We can also prepend an optimal task instruction I (Dong et al., 2022) to demonstrations as C=I,s⁢(x1,y1),…,s⁢(xN,yN).subscript1subscript1…subscriptsubscriptC=\I,s(x_1,y_1),...,s(x_N,y_N)\.C = I , s ( x1 , y1 ) , … , s ( xitalic_N , yitalic_N ) . (1) Then, given the input test example, we can generate yt⁢e⁢s⁢tsubscripty_testyitalic_t e s t: yt⁢e⁢s⁢t=fg⁢e⁢n⁢e⁢r⁢a⁢t⁢e⁢(C,s⁢(xt⁢e⁢s⁢t,_)),subscriptsubscriptsuby_test=f_generate(\C,s(x_test,\_)\),yitalic_t e s t = fitalic_g e n e r a t e ( C , s ( xitalic_t e s t , _ ) ) , (2) where s⁢(xt⁢e⁢s⁢t,_)subs(x_test,\_)s ( xitalic_t e s t , _ ) indicates using the same template as demonstrations but with the label empty. To apply it to the standard classification tasks, we let the language model choose answer yt⁢e⁢s⁢tsubscripty_testyitalic_t e s t from a candidate classification label set Y=c1,…,cksubscript1…subscriptY=\c_1,...,c_k\Y = c1 , … , citalic_k , where cksubscriptc_kcitalic_k is denoted as the label of the k-th class. For convenience, we can define a verbalizer function VV (Kim et al., 2022) which maps each of the original labels cksubscriptc_kcitalic_k to a specific token ⁢(ck)subscriptV(c_k)V ( citalic_k ). However, not all labels can be directly mapped to a single token. For instance, the label “Negative” is composed of two tokens “Neg” and “ative” when using the GPT2 tokenizer. In this case, we define the function VV mapping “Neg” and “ative” to “spaceNegative”, where an additional space prepended to the word “Negative”. To compute the logits zksubscriptz_kzitalic_k of the token ⁢(ck)subscriptV(c_k)V ( citalic_k ) from causal language models as the probability of cksubscriptc_kcitalic_k, we can use the following equation: zk=fc⁢a⁢u⁢s⁢a⁢l⁢(⁢(ck)|C,s⁢(xt⁢e⁢s⁢t,_)).subscriptsubscriptconditionalsubscriptsubz_k=f_causal(V(c_k)|\C,s(x_test,\_)\).zitalic_k = fitalic_c a u s a l ( V ( citalic_k ) | C , s ( xitalic_t e s t , _ ) ) . (3) The final prediction result yt⁢e⁢s⁢tsubscripty_testyitalic_t e s t is the label with the highest logits probability: yt⁢e⁢s⁢t=arg⁡maxck∈Yfc⁢a⁢u⁢s⁢a⁢l⁢(⁢(ck)|C,s⁢(xt⁢e⁢s⁢t,_)).subscriptsubscriptsubscriptsubscriptconditionalsubscriptsuby_test= _c_k∈ Yf_causal(V(c_k)|\% C,s(x_test,\_)\).yitalic_t e s t = start_BIGOP arg max end_BIGOPc start_POSTSUBSCRIPT k ∈ Y end_POSTSUBSCRIPT fitalic_c a u s a l ( V ( citalic_k ) | C , s ( xitalic_t e s t , _ ) ) . (4) 3.2 advICL For in-context learning, as mentioned before, it consists of both demonstration C and input test examples xt⁢e⁢s⁢tsubscriptx_testxitalic_t e s t, providing a wider attack vector compared to standard adversarial attacks. In this paper, we aim to only manipulate the demonstration C without changing the input text examples x to mislead the models. Since the main goal of this paper is to investigate the adversarial robustness of the attack vector on demonstrations, for simplification, we mainly focus on applying word-level and character-level perturbation following TextBugger (Li et al., 2018).111 Since our attack is a general method, it can be easily extended to be applied with other types of perturbation. We leave that as the future work. Our attack, advICL, builds upon TextAttack (Morris et al., 2020), a standard attack framework. However, unlike other adversarial text attack methods, which manipulate the input test example xt⁢e⁢s⁢tsubscriptx_testxitalic_t e s t by adding the perturbation Δ Δ, here we add a mask to only manipulate the demonstration C. Additionally, since the demonstration C consists of multiple sentence label pairs ((x1,y1),…,(xN,yNx_1,y_1),...,(x_N,y_Nx1 , y1 ) , … , ( xitalic_N , yitalic_N)), we sequentially set individual perturbation bounds ΔisubscriptΔ _iΔitalic_i for each xisubscriptx_ixitalic_i using cosine similarity, which can be computed by the following equation: C⁢o⁢s⁢S⁢i⁢m⁢(xi,xi′)=⁢(xi)⋅⁢(xi′)‖⁢(xi)‖⁢‖⁢(xi′)‖subscriptsuperscriptsubscript′⋅subscriptsuperscriptsubscript′normsubscriptnormsuperscriptsubscript′CosSim(x_i,x_i )= e(x_i)·e(x_i^% )||e(x_i)||\;||e(x_i )||C o s S i m ( xitalic_i , xitalic_i′ ) = divide start_ARG e ( xitalic_i ) ⋅ e ( xitalic_i′ ) end_ARG start_ARG | | e ( xitalic_i ) | | | | e ( xitalic_i′ ) | | end_ARG (5) where xi′subscript′x_i xitalic_i′ is the perturbed sentence and ⁢(⋅)⋅e(·)e ( ⋅ ) represents the embedding vector computed by Universal Sentence Encoder (Cer et al., 2018). This approach allows us to control the perturbations specifically for each demonstration, providing more fine-grained control over the attack process. Within this context, we can formalize our objective function: minδ∈Δℒ⁢(f⁢(Cδ,s⁢(xt⁢e⁢s⁢t,_)),yt⁢e⁢s⁢t),subscriptΔℒsubscriptsubsubscript _δ∈ L(f(\C_δ,s(x_test,\_% )\),y_test),minitalic_δ ∈ Δ L ( f ( Citalic_δ , s ( xitalic_t e s t , _ ) ) , yitalic_t e s t ) , (6) where ℒLL calculates the probability of the ground truth label yt⁢e⁢s⁢tsubscripty_testyitalic_t e s t using language model f, Cδ=I,s⁢(x1+δ1,y1),…,s⁢(xN+δN,yN)subscriptsubscript1subscript1subscript1…subscriptsubscriptsubscriptC_δ=\I,s(x_1+ _1,y_1),...,s(x_N+ _N,y_N)\Citalic_δ = I , s ( x1 + δ1 , y1 ) , … , s ( xitalic_N + δitalic_N , yitalic_N ) , δisubscript _iδitalic_i represents the perturbation added to the demonstration xisubscriptx_ixitalic_i under the given bound ΔisubscriptΔ _iΔitalic_i. To solve the above objective function, we consider a more practical black-box setting and adopt the greedy search in TextAttack framework(Morris et al., 2020). Specifically, given a demonstration set C with a test example pair (xt⁢e⁢s⁢t,yt⁢e⁢s⁢t)subscriptsubscript(x_test,y_test)( xitalic_t e s t , yitalic_t e s t ), we initially select words from xisubscriptx_ixitalic_i in C to form a W⁢o⁢r⁢d⁢L⁢i⁢s⁢tWordListW o r d L i s t based on the word importance level. For each word in the W⁢o⁢r⁢d⁢L⁢i⁢s⁢tWordListW o r d L i s t, we compute the objective function value ℒLL by adding the word with 4 types of perturbations, including Character Insertion, Character Deletion, Character Swap and Word Swap from TextAattack framework. We then use the perturbation that yields the largest reduction in the objective function values calculated by ℒLL while maintaining the similarity constraint C⁢o⁢s⁢S⁢i⁢m⁢(xi,xi′)subscriptsuperscriptsubscript′CosSim(x_i,x_i )C o s S i m ( xitalic_i , xitalic_i′ ) within the bound ΔisubscriptΔ _iΔitalic_i. The process iterates through the W⁢o⁢r⁢d⁢L⁢i⁢s⁢tWordListW o r d L i s t until a successful attack is achieved or fails when reaching the maximum iteration. We also present our pseudo code in Appendix A. 4 Transferable-advICL One intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating this example. To achieve such a threat model, one solution is to leverage the transferability of the adversarial demonstration: generating a high transferable adversarial demonstration that can be prepended to an arbitrary example to mislead the model. However, our advICL is only performed on a single specific text example, limiting its potential to be transferable to other test input examples. To address this limitation, inspired by the concept of Universal Adversarial Perturbation (Moosavi-Dezfooli et al., 2017), previously implemented for image classification tasks, we propose an iterative attack pipeline that aims to generate a universally adversarial demonstration by iteratively attacking a randomly selected set of input examples instead of a single input example, denoted as the Transferable-advICL (T-advICL in short). This approach can significantly extend the applicability and efficacy of advICL across various test examples. The detailed process of the pipeline is shown as follows. Step 1: Given a demonstration set C, T-advICL randomly identifies a small attack candidate set with k test examples, denoted as S=(xi,yi)i=1ksuperscriptsubscriptsubscriptsubscript1S=\(x_i,y_i)\_i=1^kS = ( xitalic_i , yitalic_i ) i = 1k. Note that, to better improve the transferability, here we only randomly select the test example in S with demonstration set C,s⁢(xi,_)subscript_\C,s(x_i,\_)\ C , s ( xitalic_i , _ ) that can be precisely classified into yisubscripty_iyitalic_i under the language model f. Step 2: T-advICL randomly shuffles test examples in S to obtain Sr⁢a⁢n⁢d⁢o⁢msubscriptS_randomSitalic_r a n d o m. The random shuffle here can help us generate optimal demonstrations by preventing stuck in a local minimum. Step 3: Before the iterative attack process, we initialize C′=Csuperscript′C =C′ = C. For each C′,s⁢(xi,_)superscript′subscript_\C ,s(x_i,\_)\ C′ , s ( xitalic_i , _ ) at iteration i, where xisubscriptx_ixitalic_i is chosen from Sr⁢a⁢n⁢d⁢o⁢msubscriptS_randomSitalic_r a n d o m, we apply advICL to obtain the adversarial demonstration set CisubscriptC_iCitalic_i. Then we update C′ with CisubscriptC_iCitalic_i only if CisubscriptC_iCitalic_i has a lower Attack Accuracy than C′ on S. Note that we need to add an additional constraint that bounds the distance between CisubscriptC_iCitalic_i and C′ and the distance between CisubscriptC_iCitalic_i and C. We then repeat Step 2 and Step 3 until R rounds are complete. Our pseudo code of T-advICL is also presented in Appendix A. 5 Experiment In this section, we start by introducing our experimental setup, followed by an evaluation of the effectiveness of advICL. We also conduct a perceptual evaluation to ensure the high quality of our generated attack demonstrations. Finally, we carry out ablation studies to evaluate the quality of adversarial demonstrations generated by different similarity constraint methods and the influence of varying templates used in in-context learning. DBpedia SST-2 TREC RTE Model Metric 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot Clean Acc 37.6 49.4 60.4 59.4 65.6 62.6 20.0 25.2 31.4 56.8 56.8 57.8 GPT2-XL Attack Acc 7.2 0.6 0.4 51.0 25.2 11.2 14.8 9.6 7.4 20.8 4.2 1.4 ASR 80.88 98.79 99.38 14.13 61.60 82.15 25.16 61.15 76.12 63.30 92.60 97.70 Clean Acc 71.0 78.4 77.8 69.0 81.8 92.2 53.4 51.0 58.4 73.4 59.0 61.8 LLaMA-7B Attack Acc 28.8 6.8 1.8 40.0 45.8 40.2 31.0 11.0 5.4 16.4 0.2 0.0 ASR 59.39 91.33 97.72 41.90 44.01 56.42 41.95 78.43 90.75 77.63 99.66 100.00 Clean Acc 77.2 52.6 41.4 79.8 79.2 82.8 60.0 47.6 38.6 73.0 69.8 68.2 Vicuna-7B Attack Acc 40.0 6.2 1.6 57.4 43.4 27.4 40.4 10.0 8.0 30.6 5.6 0.6 ASR 48.16 88.21 96.14 28.07 45.25 67.08 32.68 79.08 79.40 57.90 91.93 99.11 Table 1: Effective of advICL among different datasets and models under various shot numbers. The highest ASR value among different shot numbers is highlighted. DBpedia SST-2 TREC RTE Quality Metric 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot AEQS 92.00 94.00 91.00 88.00 89.00 90.00 90.00 92.00 96.00 92.00 92.00 92.00 CosSim 94.65 96.52 96.78 93.80 94.76 95.25 95.94 96.80 97.79 94.09 97.87 98.63 BLEU 95.71 96.52 97.94 92.51 91.81 92.59 97.20 96.74 97.57 87.23 93.67 95.77 Adv PPL 13.57 9.50 7.20 36.73 22.18 16.53 19.39 12.05 8.76 12.27 7.82 6.80 Original PPL 12.77 6.87 5.61 46.80 14.23 10.21 21.29 9.49 6.99 8.56 6.66 6.13 Table 2: Perceptual Evaluation on 100 successfully attacked examples on LLaMA-7B. 5.1 Experimental Setup Language Models and Dataset. We conduct experiments using the following language models: the widely used GPT2-XL, recently open-sourced LLaMA-7B, and Vicuna-7B. Vicuna-7B is an instruction-tuned LLaMA-7B with GPT4- or ChatGPT-generated text collected from ShareGPT.com. We conduct experiments using four classification datasets, including binary classification datasets SST-2 and RTE, multi-class classification datasets TREC and DBpedia. These datasets cover tasks such as sentiment analysis, textual entailment, topic and question classification tasks. In-Context Learning and Attack Settings. For in-context learning, we follow the setting by Zhao et al. (2021) and use their templates to incorporate demonstrations for prediction. The details of these templates are listed in Appendix B. Regarding the number of demonstrations, we use 1-shot, 4-shot, and 8-shot settings. Specifically, given each test input from the test set, we randomly select the demonstration from the training set and repeat the processing 5 times to calculate the average accuracy. For attacks, we follow Li et al. (2018) and adopt the untargeted and black-box settings, where we only have access to logits of predicted next tokens. To specify the perturbation bounds ΔisubscriptΔ _iΔitalic_i for our attack method, we employ the cosine similarity between the adversarial and original individual demonstration sentences with a threshold of 0.8. Evaluation Metrics. We use Attack Success Rate (ASR) (Wang et al., 2021) to evaluate the effectiveness of advICL. Given a dataset DD with N data instance x and label y, an adversarial attack method AA that generates adversarial examples ⁢(x)A(x)A ( x ). The ASR is then calculated as =∑(x,y)∈⁢[f⁢(⁢(x))≠y]⁢[f⁢(x)=y]subscript1delimited-[]1delimited-[]ASR=Σ _(x,y) 1[f(A(% x))≠ y] 1[f(x)=y]ASR = ∑( x , y ) ∈ D divide start_ARG blackboard_1 [ f ( A ( x ) ) ≠ y ] end_ARG start_ARG blackboard_1 [ f ( x ) = y ] end_ARG. In addition, we also report the clean accuracy (Clean Acc) which evaluates the classification accuracy with clean demonstrations, and attack accuracy (Attack Acc) which evaluates the classification accuracy with generated adversarial demonstrations. 5.2 Effectiveness of advICL Table 1 presents the results of advICL, from which we can observe that in-context learning indeed introduces a new security concern. By manipulating only the demonstrations but not the input test examples, attackers can successfully mislead the model. For instance, ASR of advICL on DBpedia and RTE with 8-shot demonstrations can reach even higher than 95% among all test models. We also show the visualization of an adversarial example generated by advICL in Figure 2. Figure 2: Visualization of an adversarial example generated by advICL on the DBpeda dataset via attacking LLaMA-7B model. Additionally, the consistent trend in Table 1 shows that larger shot numbers can bring higher ASR. This observation reveals that, although larger shot numbers can normally benefit the performance of in-context learning, they can also introduce a higher potential of threats to LLMs. 5.3 Perceptual Evaluation To conduct a comprehensive perceptual evaluation of generated adversarial demonstrations, we employ a diverse set of metrics that encompass both human evaluation and automatic text quality evaluation. Our evaluation process begins with a thorough examination of the attacked demonstrations by human annotators. Given a complete set of successfully attacked instances with n examples, each instance undergoes a meticulous inspection by the annotators. Evaluations are conducted based on established criteria through three dimensions: semantic coherence, grammatical accuracy, and expression fluency. The annotators then count the number of examples, represented as c, that meet the criteria across all three demonstrations. Following this, we calculate the Annotator Evaluation Quality Score (AEQS) using the formula AEQS=cnAEQSAEQS= cnAEQS = divide start_ARG c end_ARG start_ARG n end_ARG. For automatic text quality evaluation, we employ a selection of metrics: Cosine Similarity (CosSim) and Bilingual Evaluation Understudy (BLEU; Papineni et al. 2002). These metrics are computed directly from the original and adversarial sentences. In addition, we use Perplexity (PPL) to evaluate the text fluency. Unlike the previous metrics, PPL is calculated for individual sentences. Therefore, we compute perplexity for both the original and adversarial sentences, referred to as Original PPL and Adv PPL respectively. For further details on these machine evaluation metrics, please refer to Appendix C. We present our perceptual evaluation results of advICL in Table 2. For this evaluation, we select 100 successfully attacked examples across all datasets on LLaMA-7B. CosSim, BLEU, Attacked PPL and Original PPL represent the average scores among these 100 examples. Table 2 reveals an average AEQS of 90% and high machine evaluation scores. Based on these results, we can conclude that our generated adversarial demonstrations maintain high perceptual quality under both human and machine evaluations. SST-2 Attack Method Quality Metric 1-shot 4-shot 8-shot AEQS 70.00 65.00 74.00 baseline CosSim 91.29 92.54 92.77 Adv PPL 41.50 24.65 18.35 BLEU 92.57 90.70 91.82 AEQS 88.00 89.00 90.00 advICL CosSim 93.80 94.76 95.25 Adv PPL 36.73 22.18 16.53 BLEU 92.51 91.81 92.59 Table 3: Similarity constraint analysis among different perceptual quality metrics using the adversarial demonstration generated by attacking LLaMA-7B on the SST-2 dataset. 5.4 Ablation Study Similarity constraint. To validate the importance of our new similarity constraint, denoted as ΔisubscriptΔ _iΔitalic_i, we compare it with the standard global demonstration perturbation bound Δ Δ used in TextBugger, denoted as the baseline in Table 3 by perceptual evaluation. We keep everything the same and just replace the similarity constraint with the standard global demonstration perturbation bound. We use the adversarial demonstrations generated on LLaMA-7B on the SST-2 dataset for analysis. The constraint configurations in TextAttck framework for baseline and advICL are shown in Appendix D. As the results presented in Table 3, all quality metrics for our method consistently outperform those of baseline, except for the 1-shot BLEU score. This comparison demonstrates that our individual perturbation bounds ensure the high quality of generated adversarial examples. We also show visualized examples generated by our advICL and baseline respectively in Appendix E. We can also find that the visual quality of our method is higher than the baseline. Different Template. A previous study (Min et al., 2022) demonstrated that the performance of in-context learning can vary significantly under different templates. To ensure the effectiveness of our attack method across diverse templates, we conducted additional experiments on the SST-2 dataset among different models using an alternative template, as suggested by Min et al. (2022). This alternative template, referred to as SST-2-Alter, is shown in detail in Appendix B. The results of these experiments, presented in Table 4, show high ASR values and consistent increasing trends with larger shots numbers. These results lead us to conclude that our advICL performs stably under different templates. SST-2-Alter Model Metric 1-shot 4-shot 8-shot Clean Acc 51.2 49.0 49.8 GPT2-XL Attack Acc 26.4 9.2 2.0 ASR 48.52 81.20 95.92 Clean Acc 60.0 72.8 75.6 LLaMA-7B Attack Acc 11.4 13.4 9.8 ASR 80.98 81.56 87.05 Clean Acc 58.0 54.0 55.8 Vicuna-7B Attack Acc 10.8 6.2 3.4 ASR 81.34 88.75 93.87 Table 4: Effectiveness of advICL on another different template of SST-2. DBpedia SST-2 TREC RTE Attack Method Metric 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot 1-shot 4-shot 8-shot Avg Clean Acc 63.18 75.81 78.55 66.60 82.60 93.65 55.38 52.25 59.40 66.67 65.30 63.89 advICL Avg Attack Acc 44.37 51.53 52.66 72.02 63.86 71.36 55.09 49.94 54.31 66.75 63.91 65.31 TASR 29.77 32.03 32.96 -8.14 22.68 23.80 0.52 4.42 8.57 -0.12 2.13 -2.22 T-advICL Avg Attack Acc 17.09 16.87 21.74 56.30 55.53 55.30 43.38 30.99 29.65 53.30 56.27 57.72 TASR 72.95 77.45 72.32 15.47 32.77 40.95 21.67 40.69 50.08 20.05 13.83 9.66 Table 5: Transferability of adversarial demonstrations generated by T-advICL compared with advICL. The highest TASR value between two methods is highlighted. 5.5 Performance of Transferable-advICL In this section, we investigate the transferability of adversarial demonstrations generated by advICL and Transferable-advICL. Specifically, we examine whether an adversarial demonstration generated for an input test example, x1subscript1x_1x1, can be transferred to successfully attack another input test example, x2subscript2x_2x2, even without prior knowledge. Experimental Setup. In order to thoroughly illustrate the transferability of our method, we randomly select Na⁢d⁢vsubscriptN_advNitalic_a d v adversarial demonstration sets, denoted as CC in total. We also randomly sample Nt⁢e⁢s⁢tsubscriptN_testNitalic_t e s t test input examples exclusively from S for transferability evaluation, denoting this test set as OO. For each adversarial demonstration from CC, we evaluate the model performance on all test examples from OO. We then average model performance accuracy across the Na⁢d⁢vsubscriptN_advNitalic_a d v adversarial demonstration sets to obtain the average attack accuracy (Avg Attack Acc). Similarly, the average clean accuracy (Avg Clean Acc) can be calculated by replacing each adversarial demonstration set C′ with its corresponding clean demonstration C. The formula for computing Avg Attack Acc can be found in the Appendix F. To better evaluate the performance of transferable attack, We also compute the Transfer Attack Success Rate (TASR) for both methods by TASR=Avg Clean Acc−Avg Attack AccAvg Clean Acc.TASRAvg Clean AccAvg Attack AccAvg Clean AccTASR= Avg Clean Acc-Avg Attack AccAvg Clean % Acc.TASR = divide start_ARG Avg Clean Acc - Avg Attack Acc end_ARG start_ARG Avg Clean Acc end_ARG . To evaluate the performance of our T-advICL, we conduct attacks for each clean demonstration, we select a small candidates set S with k=1515k=15k = 15 test example number and R=33R=3R = 3 iterative rounds. Transferability Performance. In Table 5, we report the Avg Clean Acc, Avg Attack Acc, and TASR for both our advICL and T-advICL, as applied to the LLaMA-7B model across all four of our datasets. From Table 5, we found that performing advICL only yields a limited TASR, even leading to negative values in some cases. Such negative TASR means that adversarial perturbations generated by advICL possess completely no transferability and even enhance the performance of in-context learning. After applying T-advICL, we can see a significant increase of TAST from 72.32% to 32.96% for the DBpedia dataset with 8-shot and negative TASR values would never present in all circumstances. These results indicate that in contrast to advICL, adversarial demonstrations generated by T-advICL generalize much better and have more stable transferability cross various input test examples. Figure 3: Effectiveness of T-advICL with different test example number k of candidates set S. The average accuracy is represented by the point, while the shaded area indicates the variance. Figure 4: Effectiveness of T-advICL among different iteration rounds R: Each subplot depicts the average accuracy for three distinct shot numbers. Test Example Number in Candidates Set. We show how test example numbers k of candidates set S can affect the transferability of generated demonstrations. We conduct extra experiments on LLaMA-7B with various k and fixed iterative rounds R=33R=3R = 3 and show their Avg Attack Accuracy and its variance in Figure 3. From Figure 3, it is clear that a larger value of k results in a lower Avg Attack Acc, which implies enhanced attack performance. Interestingly, we also observe that as k increases, the variance of Attack Acc diminishes, further indicating that a larger k contributes to the performance stability of transferable demonstrations generated by T-advICL. Iterative Rounds. To investigate the role of iterative rounds R of T-advICL, additional experiments on LLaMA-7B are conducted with different R and fixed k=1515k=15k = 15. We present our results in Figure 4. As seen in Figure 4, there is a decreasing trend in Avg Attack Acc as the number of iterative rounds increases. It’s also observed that the performance of our T-advICL tends to converge at around R=33R=3R = 3 iterative rounds. This suggests that R=33R=3R = 3 is an optimal number of iterations, with a balance between effectiveness and efficiency. 6 Conclusion This paper studies adversarial robustness of in-context learning, with a particular focus on demonstration attacks. By performing our advICL, we find that demonstrations used in in-context learning are vulnerable to adversarial attacks. Even worse, a larger number of demonstrations can exacerbate such security concerns. Additionally, using Transferable-advICL, we also demonstrate that our generated adversarial demonstrations tend to be transferrable to a wider range of input text examples such that it can attack the input text examples even without knowing them. Limitations To ensure easy and flexible deployment of our attack, we implement it in the TextAttack framework using a black-box approach. However, it has a limitation in terms of time cost. Since we assume no access to the model and cannot calculate gradients through back propagation, we need to estimate the gradient direction to generate adversarial examples, resulting in higher computational requirements compared to other white-box methods. Since the primary objective of this paper is to demonstrate the new emergent security threats introduced in the in-context learning framework, and the design of our method is not dependent on any specific attack methods, we leave the implementation of a more efficient attack strategy as future work. Ethics Statement Through the investigation of the in-context learning from the security perspective, we hope our work can raise awareness for the community of such vulnerabilities. We highlight the importance of the demonstration in the context learning framework and inspire the community to design protection strategies for safe storing, retrieving and verifying the demonstration data for making predictions. All data, models we use in this work are publicly available. References Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. Chen et al. (2022) Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2022. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint arXiv:2209.07661. Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer. Dai et al. (2022) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559. Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234. Ebrahimi et al. (2017) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751. Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50–56. IEEE. Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Jia et al. (2019) Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. arXiv preprint arXiv:1909.00986. Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, pages 8018–8025. Kim et al. (2022) Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2022. Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint arXiv:2206.08082. Li et al. (2018) Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2018. Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271. Li et al. (2020) Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert. arXiv preprint arXiv:2004.09984. Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3333? arXiv preprint arXiv:2101.06804. Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35. Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837. Moosavi-Dezfooli et al. (2017) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765–1773. Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119–126. Naik et al. (2018) Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. arXiv preprint arXiv:1806.00692. Papernot et al. (2016) Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In MILCOM 2016-2016 IEEE Military Communications Conference, pages 49–54. IEEE. Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. Platt et al. (1999) John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74. Pruthi et al. (2019) Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019. Combating adversarial misspellings with robust word recognition. arXiv preprint arXiv:1905.11268. Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085–1097, Florence, Italy. Association for Computational Linguistics. Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118. Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642. Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Voorhees and Tice (2000) Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207. Wang et al. (2021) Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840. Wei et al. (2023) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846. Wu et al. (2022) Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2022. Self-adaptive in-context learning. arXiv preprint arXiv:2212.10375. Xie et al. (2021) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080. Zang et al. (2020) Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066–6080, Online. Association for Computational Linguistics. Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28. Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR. Appendix A Pseudo Code A.1 advICL Peseudo code of advICL is presented in Algorithm 1. Two extra functions of W⁢o⁢r⁢k⁢I⁢m⁢p⁢o⁢r⁢t⁢a⁢n⁢c⁢e⁢R⁢a⁢n⁢kWorkImportanceRankW o r k I m p o r t a n c e R a n k and S⁢e⁢l⁢e⁢c⁢t⁢B⁢u⁢gSelectBugS e l e c t B u g, whose pseudo codes are shown in Algorithm 3 and Algorithm 4 respectively, would be used in Algorithm 1. In line 5 of Algorithm 3, fyt⁢e⁢s⁢tsubscriptsubscriptf_y_testfitalic_y start_POSTSUBSCRIPT t e s t end_POSTSUBSCRIPT presents the logits value of ground truth label yt⁢e⁢s⁢tsubscripty_testyitalic_t e s t using language model f and Ca⁢d⁢v/wjsubscriptsubscriptC_adv/\w_j\Citalic_a d v / witalic_j means delete word wjsubscriptw_jwitalic_j in demonstration set Ca⁢d⁢vsubscriptC_advCitalic_a d v. In line 2 of Algorithm 4, function B⁢u⁢g⁢G⁢e⁢n⁢e⁢r⁢a⁢t⁢o⁢rBugGeneratorB u g G e n e r a t o r can generate a set named b⁢u⁢g⁢sbugsb u g s which has four kinds of bugs including Character Insertion, Character Deletion, Character Swap and Word Swap. A.2 Transferable-advICL Pseudo code for Transferable-advICL is illustrated in Algorithm 2. It also uses functions of W⁢o⁢r⁢k⁢I⁢m⁢p⁢o⁢r⁢t⁢a⁢n⁢c⁢e⁢R⁢a⁢n⁢kWorkImportanceRankW o r k I m p o r t a n c e R a n k and S⁢e⁢l⁢e⁢c⁢t⁢B⁢u⁢gSelectBugS e l e c t B u g. Algorithm 1 advICL 0: Test example pair (xt⁢e⁢s⁢t,yt⁢e⁢s⁢t)subscriptsubscript(x_test,y_test)( xitalic_t e s t , yitalic_t e s t ), demonstration set C=I,s⁢(x1,y1),…,s⁢(xN,yN)subscript1subscript1…subscriptsubscriptC=\I,s(x_1,y_1),...,s(x_N,y_N)\C = I , s ( x1 , y1 ) , … , s ( xitalic_N , yitalic_N ) , language model f, similarity threshold ε ε 0: Adversarial demonstration set Ca⁢d⁢v=I,s⁢(x1a⁢d⁢v,y1),…,s⁢(xNa⁢d⁢v,yN)subscriptsubscriptsubscript1subscript1…subscriptsubscriptsubscriptC_adv=\I,s(x_1_adv,y_1),...,s(x_N_adv,y_N)\Citalic_a d v = I , s ( x1 start_POSTSUBSCRIPT a d v end_POSTSUBSCRIPT , y1 ) , … , s ( xitalic_N start_POSTSUBSCRIPT a d v end_POSTSUBSCRIPT , yitalic_N ) 1: Initialize: Ca⁢d⁢v←C←subscriptC_adv← CCitalic_a d v ← C, ←V⁢o⁢c⁢a⁢b⁢u⁢l⁢a⁢r⁢y⁢(x1,…,xN)←subscript1…subscriptV← Vocabulary(\x_1,...,x_N\)V ← V o c a b u l a r y ( x1 , … , xitalic_N ); 2: W⁢o⁢r⁢d⁢L⁢i⁢s⁢to⁢r⁢d⁢e⁢r⁢e⁢d=W⁢o⁢r⁢d⁢I⁢m⁢p⁢o⁢r⁢t⁢a⁢n⁢c⁢e⁢R⁢a⁢n⁢k⁢(,Ca⁢d⁢v,s⁢(xt⁢e⁢s⁢t,yt⁢e⁢s⁢t),f)subscriptsubscriptsubscriptsubscriptWordList_ordered=WordImportanceRank(V,C_adv,s(x_test,y_test)% ,f)W o r d L i s titalic_o r d e r e d = W o r d I m p o r t a n c e R a n k ( V , Citalic_a d v , s ( xitalic_t e s t , yitalic_t e s t ) , f ); 3: for wjsubscriptw_jwitalic_j in W⁢o⁢r⁢d⁢L⁢i⁢s⁢to⁢r⁢d⁢e⁢r⁢e⁢dsubscriptWordList_orderedW o r d L i s titalic_o r d e r e d do 4: wj′=S⁢e⁢l⁢e⁢c⁢t⁢B⁢u⁢g⁢(wj,Ca⁢d⁢v,s⁢(xt⁢e⁢s⁢t,yt⁢e⁢s⁢t),f)superscriptsubscript′subscriptsubscriptsubscriptsubscriptw_j =SelectBug(w_j,C_adv,s(x_test,y_test),f)witalic_j′ = S e l e c t B u g ( witalic_j , Citalic_a d v , s ( xitalic_t e s t , yitalic_t e s t ) , f ); 5: xi′←superscriptsubscript′absentx_i _i′ ← replace wj∈xisubscriptsubscriptw_j∈ x_iwitalic_j ∈ xitalic_i with wj′subscript′w_j witalic_j′, Ca⁢d⁢v←subscriptabsentC_adv _a d v ← replace xia⁢d⁢v∈Ca⁢d⁢vsubscriptsubscriptsubscriptx_i_adv∈ C_advxitalic_i start_POSTSUBSCRIPT a d v end_POSTSUBSCRIPT ∈ Citalic_a d v with xi′subscript′x_i xitalic_i′; 6: if C⁢o⁢s⁢S⁢i⁢m⁢(xi,xi′)≤εsubscriptsuperscriptsubscript′CosSim(x_i,x_i )≤ o s S i m ( xitalic_i , xitalic_i′ ) ≤ ε then 7: return None 8: else if f⁢(Ca⁢d⁢v,s⁢(xt⁢e⁢s⁢t,_))≠yt⁢e⁢s⁢tsubscriptsubsubscriptf(\C_adv,s(x_test,\_)\)≠ y_testf ( Citalic_a d v , s ( xitalic_t e s t , _ ) ) ≠ yitalic_t e s t then 9: return Ca⁢d⁢vsubscriptC_advCitalic_a d v 10: end if 11: end for 12: return None Algorithm 2 Transferable-advICL 0: Demonstration set C=I,s⁢(x1,y1),…,s⁢(xN,yN)subscript1subscript1…subscriptsubscriptC=\I,s(x_1,y_1),...,s(x_N,y_N)\C = I , s ( x1 , y1 ) , … , s ( xitalic_N , yitalic_N ) , original demonstration set Co⁢r⁢i⁢g⁢i⁢n⁢a⁢l=I,s⁢(x1o,y1),…,s⁢(xNo,yN)subscriptsubscriptsubscript1subscript1…subscriptsubscriptsubscriptC_original=\I,s(x_1_o,y_1),...,s(x_N_o,y_N)\Citalic_o r i g i n a l = I , s ( x1 start_POSTSUBSCRIPT o end_POSTSUBSCRIPT , y1 ) , … , s ( xitalic_N start_POSTSUBSCRIPT o end_POSTSUBSCRIPT , yitalic_N ) , test examples set S=(xi,yi)i=1ksuperscriptsubscriptsubscriptsubscript1S=\(x_i,y_i)\_i=1^kS = ( xitalic_i , yitalic_i ) i = 1k, language model f, maximum iteration rounds M⁢a⁢x⁢I⁢t⁢e⁢rMaxIterM a x I t e r, step-wise similarity threshold ε ε, accumulative similarity threshold εasubscript _aεitalic_a 0: Adversarial demonstration set Ca⁢d⁢v=I,s⁢(x1a⁢d⁢v,y1),…,s⁢(xNa⁢d⁢v,yN)subscriptsubscriptsubscript1subscript1…subscriptsubscriptsubscriptC_adv=\I,s(x_1_adv,y_1),...,s(x_N_adv,y_N)\Citalic_a d v = I , s ( x1 start_POSTSUBSCRIPT a d v end_POSTSUBSCRIPT , y1 ) , … , s ( xitalic_N start_POSTSUBSCRIPT a d v end_POSTSUBSCRIPT , yitalic_N ) 1: Initialize: Ca⁢d⁢v←C←subscriptC_adv← CCitalic_a d v ← C; 2: i⁢t⁢e⁢r←0←0iter← 0i t e r ← 0, b⁢e⁢s⁢t⁢A⁢S⁢R←1←1bestASR← 1b e s t A S R ← 1; 3: while i⁢t⁢e⁢r<M⁢a⁢x⁢I⁢t⁢e⁢riter<MaxIteri t e r < M a x I t e r and b⁢e⁢s⁢t⁢A⁢S⁢R>00bestASR>0b e s t A S R > 0 do 4: Sr⁢a⁢n⁢d⁢o⁢m←R⁢a⁢n⁢d⁢o⁢m⁢S⁢h⁢u⁢f⁢f⁢l⁢e⁢(S)←subscriptℎS_random← RandomShuffle(S)Sitalic_r a n d o m ← R a n d o m S h u f f l e ( S ); 5: for (xi,yi)subscriptsubscript(x_i,y_i)( xitalic_i , yitalic_i ) in Sr⁢a⁢n⁢d⁢o⁢msubscriptS_randomSitalic_r a n d o m do 6: Initialize: Ci←Ca⁢d⁢v←subscriptsubscriptC_i← C_advCitalic_i ← Citalic_a d v, ←V⁢o⁢c⁢a⁢b⁢u⁢l⁢a⁢r⁢y⁢(x1a⁢d⁢v,…,xNa⁢d⁢v)←subscriptsubscript1…subscriptsubscriptV← Vocabulary(\x_1_adv,...,x_N_adv\)V ← V o c a b u l a r y ( x1 start_POSTSUBSCRIPT a d v end_POSTSUBSCRIPT , … , xitalic_N start_POSTSUBSCRIPT a d v end_POSTSUBSCRIPT ); 7: W⁢o⁢r⁢d⁢L⁢i⁢s⁢to⁢r⁢d⁢e⁢r⁢e⁢d=W⁢o⁢r⁢d⁢I⁢m⁢p⁢o⁢r⁢t⁢a⁢n⁢c⁢e⁢R⁢a⁢n⁢k⁢(,Ci,s⁢(xt⁢e⁢s⁢t,yt⁢e⁢s⁢t),f)subscriptsubscriptsubscriptsubscriptWordList_ordered=WordImportanceRank(V,C_i,s(x_test,y_test),f)W o r d L i s titalic_o r d e r e d = W o r d I m p o r t a n c e R a n k ( V , Citalic_i , s ( xitalic_t e s t , yitalic_t e s t ) , f ); 8: for wjsubscriptw_jwitalic_j in W⁢o⁢r⁢d⁢L⁢i⁢s⁢to⁢r⁢d⁢e⁢r⁢e⁢dsubscriptWordList_orderedW o r d L i s titalic_o r d e r e d do 9: wj′=S⁢e⁢l⁢e⁢c⁢t⁢B⁢u⁢g⁢(wj,Ci,s⁢(xi,yi),f)superscriptsubscript′subscriptsubscriptsubscriptsubscriptw_j =SelectBug(w_j,C_i,s(x_i,y_i),f)witalic_j′ = S e l e c t B u g ( witalic_j , Citalic_i , s ( xitalic_i , yitalic_i ) , f ); 10: xl′←superscriptsubscript′absentx_l _l′ ← replace wj∈xlsubscriptsubscriptw_j∈ x_lwitalic_j ∈ xitalic_l with wj′subscript′w_j witalic_j′, Ci←subscriptabsentC_i _i ← replace xl∈Cisubscriptsubscriptx_l∈ C_ixitalic_l ∈ Citalic_i with xl′subscript′x_l xitalic_l′; 11: if C⁢o⁢s⁢S⁢i⁢m⁢(xl,xl′)≤εsubscriptsuperscriptsubscript′CosSim(x_l,x_l )≤ o s S i m ( xitalic_l , xitalic_l′ ) ≤ ε or C⁢o⁢s⁢S⁢i⁢m⁢(xlo,xl′)≤εasubscriptsubscriptsuperscriptsubscript′subscriptCosSim(x_l_o,x_l )≤ _aC o s S i m ( xitalic_l start_POSTSUBSCRIPT o end_POSTSUBSCRIPT , xitalic_l′ ) ≤ εitalic_a then 12: Roll back the attack on wjsubscriptw_jwitalic_j, Ci←subscriptabsentC_i _i ← replace xl′∈Cisuperscriptsubscript′subscriptx_l ∈ C_ixitalic_l′ ∈ Citalic_i with xlsubscriptx_lxitalic_l; 13: Break 14: else if f⁢(Ci,s⁢(xt⁢e⁢s⁢t,_))≠yt⁢e⁢s⁢tsubscriptsubsubscriptf(\C_i,s(x_test,\_)\)≠ y_testf ( Citalic_i , s ( xitalic_t e s t , _ ) ) ≠ yitalic_t e s t then 15: Break 16: end if 17: end for 18: Evaluate ASR of CisubscriptC_iCitalic_i on S: A⁢S⁢R←E⁢v⁢a⁢l⁢u⁢a⁢t⁢e⁢(Ci,S)←subscriptASR← Evaluate(C_i,S)A S R ← E v a l u a t e ( Citalic_i , S ); 19: if A⁢S⁢R≤b⁢e⁢s⁢t⁢A⁢S⁢RASR≤ bestASRA S R ≤ b e s t A S R then 20: b⁢e⁢s⁢t⁢A⁢S⁢R←A⁢S⁢R←bestASR← ASRb e s t A S R ← A S R, Ca⁢d⁢v←Ci←subscriptsubscriptC_adv← C_iCitalic_a d v ← Citalic_i; 21: end if 22: end for 23: end while 24: return Ca⁢d⁢vsubscriptC_advCitalic_a d v Algorithm 3 Word Importance Ranking 1: function WordImportanceRank(,C,s⁢(xt⁢e⁢s⁢t,yt⁢e⁢s⁢t),fsubscriptsubscriptV,C,s(x_test,y_test),fV , C , s ( xitalic_t e s t , yitalic_t e s t ) , f) 2: Initialize: W⁢o⁢r⁢d⁢L⁢i⁢s⁢t←E⁢m⁢p⁢t⁢y⁢S⁢e⁢t⁢()←WordList← EmptySet()W o r d L i s t ← E m p t y S e t ( ); 3: for wjsubscriptw_jwitalic_j in vocabulary set VV do 4: if wj∈xi,i∈1⁢…⁢Nformulae-sequencesubscriptsubscript1…w_j∈ x_i,i∈ 1...Nwitalic_j ∈ xitalic_i , i ∈ 1 … N then 5: Compute word importance score DwjsubscriptsubscriptD_w_jDitalic_w start_POSTSUBSCRIPT j end_POSTSUBSCRIPT for wjsubscriptw_jwitalic_j Dwj=fyt⁢e⁢s⁢t⁢(Ca⁢d⁢v,s⁢(xt⁢e⁢s⁢t,_))−fyt⁢e⁢s⁢t⁢(Ca⁢d⁢v/wj,s⁢(xt⁢e⁢s⁢t,_))subscriptsubscriptsubscriptsubscriptsubscriptsubsubscriptsubscriptsubscriptsubscriptsubscript_D_w_j=f_y_test(\C_adv,s(x_test,\_)\)-f_y_test(\C_adv/\w_% j\,s(x_test,\_)\)Ditalic_w start_POSTSUBSCRIPT j end_POSTSUBSCRIPT = fitalic_y start_POSTSUBSCRIPT t e s t end_POSTSUBSCRIPT ( Citalic_a d v , s ( xitalic_t e s t , _ ) ) - fitalic_y start_POSTSUBSCRIPT t e s t end_POSTSUBSCRIPT ( Citalic_a d v / witalic_j , s ( xitalic_t e s t , _ ) ); 6: W⁢o⁢r⁢d⁢L⁢i⁢s⁢tWordListW o r d L i s t append wjsubscriptw_jwitalic_j; 7: end if 8: end for 9: W⁢o⁢r⁢d⁢L⁢i⁢s⁢to⁢r⁢d⁢e⁢r⁢e⁢d←S⁢o⁢r⁢t⁢(W⁢o⁢r⁢d⁢L⁢i⁢s⁢t)←subscriptWordList_ordered← Sort(WordList)W o r d L i s titalic_o r d e r e d ← S o r t ( W o r d L i s t ) with descending DwjsubscriptsubscriptD_w_jDitalic_w start_POSTSUBSCRIPT j end_POSTSUBSCRIPT; 10: return W⁢o⁢r⁢d⁢L⁢i⁢s⁢to⁢r⁢d⁢e⁢r⁢e⁢dsubscriptWordList_orderedW o r d L i s titalic_o r d e r e d 11: end function Algorithm 4 Bug Selection 1: function SelectBug(w,C,s⁢(xt⁢e⁢s⁢t,yt⁢e⁢s⁢t),fsubscriptsubscriptw,C,s(x_test,y_test),fw , C , s ( xitalic_t e s t , yitalic_t e s t ) , f) 2: b⁢u⁢g⁢s←B⁢u⁢g⁢G⁢e⁢n⁢e⁢r⁢a⁢t⁢o⁢r⁢(w)←bugs← BugGenerator(w)b u g s ← B u g G e n e r a t o r ( w ); 3: for each bksubscriptb_kbitalic_k in b⁢u⁢g⁢sbugsb u g s do 4: Ck′←subscriptsuperscript′absentC _k ′italic_k ← replace w with bksubscriptb_kbitalic_k in C; 5: s⁢c⁢o⁢r⁢e⁢(k)=fyt⁢e⁢s⁢t⁢(C,s⁢(xt⁢e⁢s⁢t,_))−fyt⁢e⁢s⁢t⁢(Ck′,s⁢(xt⁢e⁢s⁢t,_))subscriptsubscriptsubsubscriptsubscriptsubscriptsuperscript′subscript_score(k)=f_y_test(\C,s(x_test,\_)\)-f_y_test(\C _k,s(x_% test,\_)\)s c o r e ( k ) = fitalic_y start_POSTSUBSCRIPT t e s t end_POSTSUBSCRIPT ( C , s ( xitalic_t e s t , _ ) ) - fitalic_y start_POSTSUBSCRIPT t e s t end_POSTSUBSCRIPT ( C′italic_k , s ( xitalic_t e s t , _ ) ); 6: end for 7: b⁢u⁢gb⁢e⁢s⁢t←arg⁡maxbk⁡s⁢c⁢o⁢r⁢e⁢(k)←subscriptsubscriptsubscriptbug_best← _b_kscore(k)b u gitalic_b e s t ← arg maxitalic_b start_POSTSUBSCRIPT k end_POSTSUBSCRIPT s c o r e ( k ); 8: return b⁢u⁢gb⁢e⁢s⁢tsubscriptbug_bestb u gitalic_b e s t 9: end function Appendix B In-Context Template Table 6 shows templates employed in our study for the DBpedia, SST-2, TREC and RTE datasets. For an alternative template for SST-2, we utilized the minimal template from (Min et al., 2022), denoted as SST-2-Alter. Dataset Template Example DBpedia [Instruction] Classify the documents based on whether they are about a Company, School, Artist, Athlete, Politician, Transportation, Building, Nature, Village, Animal, Plant, Album, Film, or Book. Article: [sentence] Article: It’s a Long Long Way to Tipperary is a 1914 Australian silent film based on the song It’s a Long Way to Tipperary by Jack Judge. Answer: [label] Answer: Film SST-2 [Instruction] Choose sentiment from Positive or Negative . Review: [sentence] Review: i had to look away - this was god awful . Sentiment: [label] Sentiment: Negative TREC [Instruction] Classify the questions based on whether their answer type is a Number, Location, Person, Description, Entity, or Abbreviation. Question: [sentence] Question: How many trees go into paper making in a year? Answer Type: [label] Answer Type: Number RTE [Instruction] [sentence 1] At eurodisney, once upon a time is now in this magical kingdom where childhood fantasies and make- believe come to life. question: [sentence2] question: EuroDisney is located in this magical kingdom. True or False? answer: [label] answer: False SST-2-Alter [Instruction] [sentence] i had to look away - this was god awful . [label] Negative Table 6: This table presents template designs for all datasets used in this paper. An extra example for each template is provided for better understanding. Appendix C Evaluation Metrics Here, we provide a brief introduction to three distinct automatic evaluation metrics commonly employed in NLP: Cosine Similarity (CosSim): To evaluate semantic coherence between original and adversarial sentences, we involve calculating the cosine similarity on the entire adversarial test prompt with the original one. Bilingual Evaluation Understudy (BLEU): BLEU is a commonly employed method for evaluating structural similarity between two sentences in an automated manner. Here we utilize the BLEU score to compare the original sentence with the adversarial sentence, assessing the degree of similarity in their grammatical structures. Perplexity (PPL): PPL is a widely used metric in neural language processing which measures the text fluency of language models. In our scenario, we compute perplexity of both the original and adversarial sentences, denoted as Original PPL and Adv PPL, respectively. Appendix D TextAttack Configuration As our advICL is built upon the TextAttack framework, we use certain configurations and functions from TextAttack. In Table 7, we list all configurations of advICL in comparison to the baseline TextBugger in TextAttack framework. The differences are highlighted using colored text for easy reference. Attack Algorithm advICL TextBugger Search Method GreedyWordSwapWIR GreedyWordSwapWIR WIR Method delete delete Goal Function UntargetedClassification UntargetedClassification Transformation CompositeTransformation CompositeTransformation IclUniversalSentenceEncoder UniversalSentenceEncoder - Metric: angular - Metric: angular - Similarity Const: single-demonstration - Similarity Const: multi-demonstration - Threshold: 0.8 - Threshold: 0.8 Constraints - Window Size: ∞ - Window Size: ∞ - Skip Text Shorter Than Window: False - Skip Text Shorter Than Window: False - Compare Against Original: True - Compare Against Original: True RepeatModification RepeatModification StopwordModification StopwordModification InstructionModification InstructionModification Is Black Box True True Table 7: Framework Configuration of advICL and TextBugger. Highlighted functions show the difference. Appendix E Examples Under Different Similarity Constraint To offer a clearer perception of the differences in quality between adversarial demonstrations generated by the baseline TextBugger and advICL, we present two examples for comparison in Figure 5. It is evident from the figure that advICL not only performs a successful attack but also generates adversarial demonstrations with a prominent enhancement in quality. Figure 5: Adversarial demonstrations generated by TextBugger (left) and our advICL (right). Perturbed words are all highlighted in red compared with original sentences. Appendix F Formulation of Avg Attack Acc We show the formulation of computing Avg Attack Acc in Figure 6. Avg Attack Acc=1Na⁢d⁢v⁢Nt⁢e⁢s⁢t⁢∑C′∈∑(xi,yi)∈⁢[yi=arg⁡maxck∈Yfc⁢a⁢u⁢s⁢a⁢l⁢(⁢(ck)|C,s⁢(xi,_))]Avg Attack Acc1subscriptsubscriptsubscriptsuperscript′subscriptsubscriptsubscript1delimited-[]subscriptsubscriptsubscriptsubscriptconditionalsubscriptsubscript_Avg Attack Acc= 1N_advN_test _C % _(x_i,y_i) 1[y_i= _% c_k∈ Yf_causal(V(c_k)|\C,s(x_i,\_)\)]Avg Attack Acc = divide start_ARG 1 end_ARG start_ARG Nitalic_a d v Nitalic_t e s t end_ARG ∑C′ ∈ C ∑( x start_POSTSUBSCRIPT i , yitalic_i ) ∈ O end_POSTSUBSCRIPT blackboard_1 [ yitalic_i = start_BIGOP arg max end_BIGOPc start_POSTSUBSCRIPT k ∈ Y end_POSTSUBSCRIPT fitalic_c a u s a l ( V ( citalic_k ) | C , s ( xitalic_i , _ ) ) ] Figure 6: Formulation for computing Avg Attack Acc.

Instant research discovery