Paper deep dive

Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models

Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, Bo Li

Year: 2021Venue: NeurIPS 2021Area: Adversarial RobustnessType: BenchmarkEmbeddings: 83

Models: ALBERT-XXLarge, BERT-Large, DeBERTa-Large, ELECTRA-Large, RoBERTa-Large, T5-Large

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 8:08:27 PM

Summary

Adversarial GLUE (AdvGLUE) is a multi-task benchmark designed to evaluate the robustness of large-scale pre-trained language models against 14 different textual adversarial attack methods. The benchmark covers word-level, sentence-level, and human-crafted perturbations across five GLUE tasks, incorporating rigorous automatic and human-based filtering to ensure high-quality, semantic-preserving adversarial examples.

Entities (8)

AdvGLUE · benchmark · 100%GLUE · benchmark · 100%MNLI · nlu-task · 98%SST-2 · nlu-task · 98%BERT-ATTACK · adversarial-attack-method · 95%SememePSO · adversarial-attack-method · 95%TextBugger · adversarial-attack-method · 95%TextFooler · adversarial-attack-method · 95%

Relation Signals (3)

AdvGLUE → evaluatesrobustnessof → Language Models

confidence 95% · AdvGLUE, a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models

AdvGLUE → includestask → SST-2

confidence 95% · AdvGLUE covers the widely-used GLUE tasks... Sentiment Analysis (SST-2)

TextBugger → performsperturbation → Word-level

confidence 95% · We select TextBugger as the representative algorithm for generating typo-based adversarial examples.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large-scale pre-trained language models have achieved tremendous success across a wide range of natural language understanding (NLU) tasks, even surpassing human performance. However, recent studies reveal that the robustness of these models can be challenged by carefully crafted textual adversarial examples. While several individual datasets have been proposed to evaluate model robustness, a principled and comprehensive benchmark is still missing. In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. In particular, we systematically apply 14 textual adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. Our findings are summarized as follows. (i) Most existing adversarial attack algorithms are prone to generating invalid or ambiguous adversarial examples, with around 90% of them either changing the original semantic meanings or misleading human annotators as well. Therefore, we perform a careful filtering process to curate a high-quality benchmark. (ii) All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy. We hope our work will motivate the development of new adversarial attacks that are more stealthy and semantic-preserving, as well as new robust language models against sophisticated adversarial attacks. AdvGLUE is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,128 characters extracted from source content.

Expand or collapse full text

Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models ∗ Boxin Wang 1 , ∗ Chejian Xu 2 , Shuohang Wang 3 , Zhe Gan 3 , Yu Cheng 3 , Jianfeng Gao 3 , Ahmed Hassan Awadallah 3 , Bo Li 1 1 University of Illinois at Urbana-Champaign 2 Zhejiang University, 3 Microsoft Corporation boxinw2,lbo@illinois.edu, xuchejian@zju.edu.cn shuohang.wang,zhe.gan,yu.cheng,jfgao,hassanam@microsoft.com Abstract Large-scale pre-trained language models have achieved tremendous success across a wide range of natural language understanding (NLU) tasks, even surpassing human performance. However, recent studies reveal that the robustness of these models can be challenged by carefully crafted textual adversarial examples. While several individual datasets have been proposed to evaluate model robustness, a principled and comprehensive benchmark is still missing. In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. In particular, we systematically apply 14 textual adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. Our findings are summarized as follows. (i) Most existing adversarial attack algorithms are prone to generating invalid or ambiguous adversarial examples, with around90%of them either changing the original semantic meanings or misleading human annotators as well. Therefore, we perform careful filtering process to curate a high-quality benchmark. (i) All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy. We hope our work will motivate the development of new adversarial attacks that are more stealthy and semantic-preserving, as well as new robust language models against sophisticated adversarial attacks. AdvGLUE is available athttps://adversarialglue.github.io. 1 Introduction Pre-trained language models [8,31,26,55,18,60,23,6] have achieved state-of-the-art performance over a wide range of Natural Language Understanding (NLU) tasks [49,48,21,45,38]. However, recent studies [24,57,50,29,13] reveal that even these large-scale language models are vulnerable to carefully crafted adversarial examples, which can fool the models to output arbitrarily wrong answers by perturbing input sentences in a human-imperceptible way. Real-world systems built upon these vulnerable models can be misled in ways that would have profound security concerns [27, 28]. To address this challenge, various methods [23,61,51,30] have been proposed to improve the adversarial robustness of language models. However, the adversary setup considered in these methods lacks a unified standard. For example, Jiang et al.[23], Liu et al.[30]mainly evaluate their robustness against human-crafted adversarial datasets [38,21], while Wang et al.[51]evaluate the ∗ Equal Contribution 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks. arXiv:2111.02840v2 [cs.CL] 10 Jan 2022 model robustness against automatic adversarial attack algorithms [24]. The absence of a principled adversarial benchmark makes it difficult to compare the robustness across different models and identify the adversarial attacks that most models are vulnerable to. This motivates us to build a unified and principled robustness evaluation benchmark for natural language models and hope to help answer the following questions:what types of language models are more robust when evaluated on the unified adversarial benchmark? Which adversarial attack algorithms against language models are more effective, transferable, or stealthy to human? How likely can human be fooled by different adversarial attacks? We list out the fundamental principles to create a high-quality robustness evaluation benchmark as follows. First, as also pointed out by [2], a reliable benchmark should be accurately and unam- biguously annotated by humans. This is especially crucial for the robustness evaluation, as some adversarial examples generated by automatic attack algorithms can fool humans as well [34]. Given our analysis in §3.4, among the generated adversarial data, there are only around10%adversarial examples that receive at least 4-vote consensus among 5 annotators and align with the original label. Thus, additional rounds of human filtering are critical to validate the quality of the generated adversarial attack data. Second, a comprehensive robustness evaluation benchmark should cover enough language phenomena and generate a systematic diagnostic report to understand and analyze the vulnerabilities of language models. Finally, a robustness evaluation benchmark needs to be challenging and unveil the biases shared across different models. In this paper, we introduce Adversarial GLUE (AdvGLUE), a multi-task benchmark for robust- ness evaluation of language models. Compared to existing adversarial datasets, there are several contributions that render AdvGLUE a unique and valuable asset to the community. •Comprehensive Coverage.We consider textual adversarial attacks from different perspectives and hierarchies, including word-level transformations, sentence-level manipulations, and human-written adversarial examples, so that AdvGLUE is able to cover as many adversarial linguistic phenomena as possible. •Systematic Annotations. To the best of our knowledge, this is the first work that performs systematic and comprehensive evaluation and annotation over 14 different textual adversarial examples. Concretely, AdvGLUE adopts crowd-sourcing to identify high-quality adversarial data for reliable evaluation. •General Compatibility. To obtain comprehensive understanding of the robustness of language models across different NLU tasks, AdvGLUE covers the widely-used GLUE tasks and creates an adversarial version of the GLUE benchmark to evaluate the robustness of language models. •High Transferability and Effectiveness. AdvGLUE has high adversarial transferability and can effectively attack a wide range of state-of-the-art models. We observe a significant performance drop for models evaluated on AdvGLUE compared with their standard accuracy on GLUE leaderboard. For instance, the average GLUE score of ELECTRA(Large) [6] drops from93.16to41.69. Our contributions are summarized as follows. (i) We propose AdvGLUE, a principled and compre- hensive benchmark that focuses on robustness evaluation of language models. (i) During the data construction, we provide a thorough analysis and a fair comparison of existing strong adversarial at- tack algorithms. (i) We present thorough robustness evaluation for existing state-of-the-art language models and defense methods. We hope that AdvGLUE will inspire active research and discussion in the community. More details are available athttps://adversarialglue.github.io. 2 Related Work Existing robustness evaluation work can be roughly divided into two categories:Evaluation Toolkits andBenchmark Datasets. (i) Evaluation toolkits, including OpenAttack [58], TextAttack [35], TextFlint [17] and Robustness Gym [15], integrate variousad hocinput transformations for different tasks and provide programmable APIs to dynamically test model performance. However, it is challenging to guarantee the quality of these input transformations. For example, as reported in [57], the validity of adversarial transformation can be as low as65.5%, which means that more than one third of the adversarial sentences have wrong labels. Such a high percentage of annotation errors could lead to an underestimate of model robustness, making it less qualified to serve as an accurate and reliable benchmark [2]. (i) Benchmark datasets for robustness evaluation create challenging 2 Table 1:Statistics of AdvGLUE benchmark. We applyallword-level perturbations (C1=Embedding- similarity, C2=Typos, C3=Context-aware, C4=Knowledge-guided, and C5=Compositions) to the five GLUE tasks. For sentence-level perturbations, we applySyntactic-based perturbations(C6) to the five GLUE tasks. Distraction-based perturbations(C7) are applied to four GLUE tasks without QQP, as they may affect the semantic similarity. For human-crafted examples, we applyCheckList(C8) to SST-2, QQP, and QNLI;StressTest (C9) andANLI(C10) to MNLI; andAdvSQuAD(C11) to QNLI tasks. CorpusTask |Train||Test|Word-LevelSent.-LevelHuman-Crafted (GLUE)(AdvGLUE)C1C2C3C4C5C6C7C8C9 C10 C11 SST-2sentiment67,3491,420204 1979117564211320158000 QQPparaphrase 363,8464224215117357537065000 QNLINLI/QA104,74396873139719872159219800057 RTENLI2,490304434431272348880000 MNLINLI392,7021,86469402 114 161 128 2173860194 1930 Sum of AdvGLUE test set4,978431 933 324 496 362 6721013303 194 19357 testing cases by using human-crafted templates or rules [45,43,36], or adopting a human-and-model- in-the-loop manner to write adversarial examples [38,25,1]. While the quality and validity of these adversarial datasets can be well controlled, the scalability and comprehensiveness are limited by the human annotators. For example, template-based methods require linguistic experts to carefully construct reasonable rules for specific tasks, and such templates can be barely transferable to other tasks. Moreover, human annotators tend to complete the writing tasks through minimal efforts and shortcuts [4, 47], which can limit the coverage of various linguistic phenomena. 3 Dataset Construction In this section, we provide an overview of our evaluation tasks, as well as the pipeline of how we construct the benchmark data. During this data construction process, we also compare the effectiveness of different adversarial attack methods, and present several interesting findings. 3.1 Overview Tasks.We consider the following five most representative and challenging tasks used in GLUE [49]: Sentiment Analysis (SST-2), Duplicate Question Detection (QQP), and Natural Language Inference (NLI, includingMNLI, RTE, QNLI). The detailed explanation for each task can be found in Appendix A.3. Some tasks in GLUE are not included in AdvGLUE, since there are either no well-defined automatic adversarial attacks (e.g.,CoLA), or insufficient data (e.g.,WNLI) for the attacks. Dataset Statistics and Evaluation Metrics.AdvGLUE follows the same training data and evalua- tion metrics as GLUE. In this way, models trained on the GLUE training data can be easily evaluated under IID sampled test sets (GLUE benchmark) or carefully crafted adversarial test sets (AdvGLUE benchmark). Practitioners can understand the model generalization via the GLUE diagnostic test suite and examine the model robustness against different levels of adversarial attacks from the AdvGLUE diagnostic report with only one-time training. Given the same evaluation metrics, model developers can clearly understand the performance gap between models tested in the ideally benign environments and approximately worst-case adversarial scenarios. We present the detailed dataset statistics under various attacks in Table 1. Detailed label distribution and evaluation metrics are in Appendix Table 8. 3.2 Adversarial Perturbations In this section, we detail how we optimize different levels of adversarial perturbations to the benign source samples and collect the raw adversarial data with noisy labels, which will then be carefully filtered by human annotators described in the next section. Specifically, we consider the dev sets of GLUE benchmark as our source samples, upon which we perform different adversarial attacks. For relatively large-scale tasks (QQP, QNLI, MNLI-m/m), we sample 1,000 cases from the dev sets for efficiency purpose. For the remaining tasks, we consider the whole dev sets as source samples. 3 ... GLUE Data SST-2 MNLI-m MNLI-m RTE QQP QNLI Adversarial Perturbations Typos AdvGLUE Data Automatic Filtering (Transferability) Curation Human-crafted Examples Sentence-level Perturbations Word-level Perturbations Embedding Context Knowledge Composition Syntax Distraction ANLI CheckList AdvSQuAD TextBugger TextFooler BERT-ATTACK Sememe-PSO CompAttack T3; AdvFever; SCPN StressTest; CheckList Automatic Filtering (Fidelity) Human Training & Filtering Sampling Figure 1:Overview of the AdvGLUE dataset construction pipeline. 3.2.1 Word-level Perturbation Existing word-level adversarial attacks perturb the words through different strategies, such as perturb- ing words with their synonyms [24] or carefully crafted typo words [27] (e.g., “foolish” to “fo01ish”), such that the perturbation does not change the semantic meaning of the sentences but dramatically change the models’ output. To examine the model robustness against different perturbation strategies, we select one representative adversarial attack method for each strategy as follows. Typo-based Perturbation.We select TextBugger [27] as the representative algorithm for generating typo-based adversarial examples. When performing the attack, TextBugger first identifies the important words and then replaces them with typos. Embedding-similarity-based Perturbation.We choose TextFooler [24] as the representative adver- sarial attack that considers embedding similarity as a constraint to generate semantically consistent adversarial examples. Essentially, TextFooler first performs word importance ranking, and then substitutes those important ones to their synonyms extracted according to the cosine similarity of word embeddings. Context-aware Perturbation.We use BERT-ATTACK [29] to generate context-aware perturbations. The fundamental difference between BERT-ATTACK and TextFooler lies on the word replacement procedure. Specifically, BERT-ATTACK uses the pre-trained BERT to perform masked language prediction to generate contextualized potential word replacements for those crucial words. Knowledge-guided Perturbation. We consider SememePSO [57] as an example to generate adver- sarial examples guided by the HowNet [41] knowledge base. SememePSO first finds out substitutions for each word in HowNet based on sememes, and then searches for the optimal combination based on particle swarm optimization. Compositions of different Perturbations. We also implement a whitebox-based adversarial attack algorithm called CompAttack that integrates the aforementioned perturbations in one algorithm to evaluate model robustness to various adversarial transformations. Moreover, we efficiently search for perturbations via optimization so that CompAttack can achieve the attack goal while perturbing the minimal number of words. The implementation details can be found in Appendix A.4. We note that the above adversarial attacks require a surrogate model to search for the optimal perturbations. In our experiments, we follow the setup of ANLI [38] and generate adversarial examples against three different types of models (BERT, RoBERTa, and RoBERTa ensemble) trained on the GLUE benchmark. We then perform one round of filtering to retain those examples with high adversarial transferabilitybetween these surrogate models. We discuss more implementation details and hyper-parameters of each attack method in Appendix A.4. 3.2.2 Sentence-level Perturbation Different from word-level attacks that perturb specific words, sentence-level attacks mainly focus on the syntactic and logical structures of sentences. Most of them achieve the attack goal by either paraphrasing the sentence, manipulating the syntactic structures, or inserting some unrelated sentences to distract the model attention. AdvGLUE considers the following representative perturbations. 4 Table 2:Examples of AdvGLUE benchmark. We show3examples from QNLI task. These examples are generated with three levels of perturbations and they all can successfully change the predictions of all surrogate models (BERT, RoBERTa and RoBERTa ensemble). Linguistic Phenomenon Samples (Strikethrough = Original Text, red = Adversarial Perturbation)Label→ Prediction Typo (Word-level) Question: What was the population of the Dutch Republic before this emigration? False→True Sentence : This was ahuge hu ge influx as the entire population of the Dutch Republic amounted to ca. Distraction (Sent.-level) Question: What was the population of the Dutch Republic before this emigration? https://t.co/DlI9kw False→True Sentence: This was a huge influx as the entire population of the Dutch Republic amounted to ca. CheckList (Human-crafted) Question: What is Tony’s profession? True→FalseSentence : Both Tony and Marilyn were executives, but there was a change in Marilyn, who is now an assistant. Syntactic-based Perturbation.We incorporate three adversarial attack strategies that manipulate the sentence based on the syntactic structures. (i)Syntax Tree Transformations. SCPN [20] is trained to produce a paraphrase of a given sentence with specified syntactic structures. Following the default setting, we select the most frequent10templates from ParaNMT-50M corpus [52] to guide the generation process. An LSTM-based encoder-decoder model (SCPN) is used to generate parses of target sentences according to the templates. These parses are further fed into another SCPN to generate full sentences. We use the pre-trained SCPNs released by the official codebase. (i)Context Vector Transformations. T3 [50] is a whitebox attack algorithm that can add perturbations on different levels of the syntax tree and generate the adversarial sentence. In our setting, we add perturbations to the context vector of the root node given syntax tree, which is iteratively optimized to construct the adversarial sentence. (i)Entailment Preserving Transformations. We follow the entailment preserving rules proposed by AdvFever [45], and transform all the sentences satisfying the templates into semantically equivalent ones. More details can be found in Appendix A.4. Distraction-based Perturbation. We integrate two attack strategies: (i) StressTest [36] appends three true statements (“and true is true”, “and false is not true”, “and true is true” for five times) to the end of the hypothesis sentence for NLI tasks. (i) CheckList [43] adds randomly generated URLs and handles to distract model attention. Since the aforementioned distraction-based perturbations may impact the linguistic acceptability and the understanding of semantic equivalence, we mainly apply these rules to part of the GLUE tasks, includingSST-2and NLI tasks (MNLI, RTE, QNLI), to evaluate whether model can be easily misled by the strong negation words or such lexical similarity. 3.2.3 Human-crafted Examples To ensure our benchmark covers more linguistic phenomena in addition to those provided by automatic attack algorithms, we integrate the following high-quality human-crafted adversarial data from crowd- sourcing or expert-annotated templates and transform them to the formats of GLUE tasks. CheckList 2 [43] is a testing method designed for analysing different capabilities of NLP models using different test types. For each task, CheckList first identifies necessary natural language capabilities a model should have, then designs several test templates to generate test cases at scale. We follow the instructions and collect testing cases for three tasks:SST-2, QQPandQNLI. For each task, we adopt two capability tests:TemporalandNegation, which test if the model understands the order of events and if the model is sensitive to negations. StressTest 2 [36] proposes carefully crafted rules to construct “stress tests” and evaluate robustness of NLI models to specific linguistic phenomena. We adopt the test cases focusing onNumerical Reasoninginto our adversarialMNLIdataset. These premise-hypothesis pairs are able to test whether 2 We note that both CheckList and StressTest propose both rule-based distraction sentences and manually crafted templates to generate test samples. The former is considered as sentence-level distraction-based perturbations, while the latter is considered as human-crafted examples. 5 the model can perform reasoning involving numbers and quantifiers and predict the correct relation between premise and hypothesis. ANLI[38] is a large-scale NLI dataset collected iteratively in a human-in-the-loop manner. In each iteration, human annotators are asked to design sentences to fool current model. Then the model is further finetuned on a larger dataset incorporating these sentences, which leads to a stronger model. Finally, annotators are asked to write harder examples to detect the weakness of this stronger model. In the end, the sentence pairs generated in each round form a comprehensive dataset that aims at examining the vulnerability of NLI models. We adopt ANLI into our adversarialMNLIdataset. We obtain the permission from the ANLI authors to include the ANLI dataset as part of our leaderboard. AdvSQuAD[21] is an adversarial dataset targeting at reading comprehension systems. Adversarial examples are generated by appending a distracting sentence to the end of the input paragraph. The distracting sentences are carefully designed to have common words with questions and look like a correct answer to the question. We mainly consider the examples generated byADDSENTand ADDONESENTstrategies, and adopt the distracting sentences and questions in theQNLIformat with labels “not answered”. The use of AdvSQuAD in AdvGLUE is authorized by the authors. We present sampled AdvGLUE examples with the word-level, sentence-level perturbations and human-crafted samples in Table 2. More examples are provided in Appendix A.5. 3.3 Data Curation After collecting the raw adversarial dataset, additional rounds of filtering are required to guarantee its quality and validity. We consider two types of filtering: automatic filtering and human evaluation. Automatic Filteringmainly evaluates the generated adversarial examples along two fronts:transfer- abilityandfidelity. 1.Transferability evaluates whether the adversarial examples generated against one source model (e.g., BERT) can successfully transfer and attack the other two (e.g., RoBERTa and RoBERTa ensemble), given the surrogate models used to generate adversarial examples (BERT, RoBERTa and RoBERTa ensemble). Only adversarial examples that can successfully transfer to the other two models will be kept for the next round of fidelity filtering, so that the selected examples can exploit the biases shared across different models and unveil their fundamental weakness. 2.Fidelity evaluates how the generated adversarial examples maintain the original semantics. For word-level adversarial examples, we useword modification rateto measure what percentage of words are perturbed. Concretely, word-level adversarial examples with word modification rate larger than15%are filtered out. For sentence-level adversarial examples, we useBERTScore [59] to evaluate the semantic similarity between the adversarial sentences and their corresponding original ones. For each sentence-level attack, adversarial examples with the highest similarity scores are kept to guarantee their semantic closeness to the benign samples. Human Evaluationvalidates whether the adversarial examples preserve the original labels and whether the labels are highly agreed among annotators. Concretely, we recruit annotators from Amazon Mechanical Turk. To make sure the annotators fully understand the GLUE tasks, each worker is required to pass a training step to be qualified to work on the main filtering tasks for the generated adversarial examples. We tune the pay rate for different tasks, as shown in Appendix Table 11. The pay rate of the main filtering phase is twice as much as that of the training phase. 1.Human Training Phaseis designed to ensure that the annotators understand the tasks. The annotation instructions for each task follows [37], and we provide at least two examples for each class to help annotators understand the tasks. 3 Each annotator is required to work on a batch of 20 examples randomly sampled from the GLUE dev set. After annotators answer each example, a ground-truth answer will be provided to help them understand whether the answer is correct. Workers who get at least85%of the examples correct during training are qualified to work on the main filtering task. A total of 100 crowd workers participated in each task, and the number of qualified workers are shown in Appendix Table 11. We also test the human accuracy of qualified annotators for each task on 100 randomly sampled examples from the dev set excluding the training samples. The details and results can be found in Appendix Table 11. 3 Instructions can be found athttps://adversarialglue.github.io/instructions. 6 Table 3:Statistics of data curation. We report Attack Success Rate (ASR) and ASR after data curation (Curated ASR) to evaluate theeffectivenessof different adversarial attacks. We present theFilter Rateof data curation and inter-annotator agreement rate (Fleiss Kappa) before and after curation to evaluate thevalidityof adversarial examples.Human Accuracyon our curated dataset is evaluated by taking one random annotator’s annotation as prediction and the majority voted label as ground truth. SPSO: SememePSO, TF: TextFooler, TB:TextBugger, CA: CompAttack, BA:BERT-ATTACK.↑/↓: higher/lower the better. Tasks Metrics Word-level AttacksSentence-level Attacks Avg SPSOTFTBCABAT3SCPN AdvFever SST-2 ASR↑89.08 95.38 88.08 31.91 39.7797.6965.370.5763.48 Curated ASR↑8.298.978.854.024.0410.456.880.236.47 Filter Rate↓90.71 90.62 90.04 86.63 89.81 89.27 89.4760.0085.82 Fleiss Kappa↑0.220.200.500.210.240.230.290.120.26 Curated Fleiss Kappa↑0.510.490.670.460.450.440.470.200.52 Human Accuracy↑ 0.850.860.910.880.850.780.850.500.87 MNLI ASR↑ 78.45 61.50 69.35 68.58 65.0291.2387.732.2565.51 Curated ASR↑3.481.558.943.112.583.416.750.303.77 Filter Rate↓95.59 97.55 87.12 95.45 96.10 96.27 92.3186.6393.38 Fleiss Kappa↑0.280.240.530.390.320.280.240.350.33 Curated Fleiss Kappa↑0.650.590.740.650.600.560.600.510.67 Human Accuracy↑ 0.850.830.910.890.830.840.910.830.89 RTE ASR↑ 76.67 75.67 85.89 73.36 72.0592.3988.456.6271.39 Curated ASR↑6.208.1410.036.975.587.058.302.536.85 Filter Rate↓91.93 89.21 88.29 90.72 92.16 92.31 90.6161.3487.07 Fleiss Kappa↑0.300.320.580.350.250.330.430.580.38 Curated Fleiss Kappa↑ 0.490.670.800.630.420.600.640.650.66 Human Accuracy↑ 0.770.950.940.870.790.890.910.860.92 QNLI ASR↑71.88 67.03 82.54 67.24 60.5396.4167.370.9764.25 Curated ASR↑3.922.875.874.092.697.593.900.003.87 Filter Rate↓94.63 95.89 92.89 93.92 95.7892.1694.21100.0094.93 Fleiss Kappa↑0.070.050.160.100.140.070.12-0.160.11 Curated Fleiss Kappa↑0.370.430.490.340.530.370.43-0.44 Human Accuracy↑ 0.800.860.850.820.920.890.92-0.85 QQP ASR↑45.86 48.5957.9249.33 43.66 48.20 44.370.3042.28 Curated ASR↑1.521.745.873.050.761.471.500.001.99 Filter Rate↓96.73 96.5089.9093.83 98.28 97.04 96.62100.0096.11 Fleiss Kappa↑ 0.260.270.380.270.240.250.29-0.30 Curated Fleiss Kappa↑ 0.320.460.620.480.400.100.47-0.51 Human Accuracy↑ 0.840.980.970.890.780.891.00-0.89 2.Human Filtering Phaseverifies the quality of the generated adversarial examples and only maintains high-quality ones to construct the benchmark dataset. Specifically, annotators are required to work on a batch of 10 adversarial examples generated from the same attack method. Every adversarial example will be validated by 5 different annotators. Examples are selected following two criteria: (i) high consensus: each example must have at least 4-vote consensus; (i) utility preserving: the majority-voted label must be the same as the original one to make sure the attacks are valid (i.e., cannot fool human) and preserve the semantic content. The data curation results including inter-annotator agreement rate (Fleiss Kappa) and human accuracy on the curated dataset are shown in Table 3. We will provide more analysis in the next section. Note that even after the data curation step, some grammatical errors and typos can still remain, as some adversarial attacks intentionally inject typos (e.g., TextBugger) or manipulate syntactic trees (e.g., SCPN) which are very stealthy. We will retain these samples as their labels receive high consensus from annotators, which means the typos do not substantially impact humans’ understanding. 3.4 Benchmark of Adversarial Attack Algorithms Our data curation phase also serves as a comprehensive benchmark over existing adversarial attack methods, as it provides a fair standard for all adversarial attacks and systematic human annotations to evaluate the quality of the generated samples. 7 Table 4:Model performance on AdvGLUE test set. BERT (Large) and RoBERTa (Large) are fine-tuned using different random seeds and thus different from the surrogate models used for adversarial text generation. For MNLI, we report the test accuracy on the matched and mismatched test sets; for QQP, we report accuracy and F1; and for other tasks, we report the accuracy. All values are reported by percentage (%). We also report the macro-average (Avg) of per-task scores for different models. (Complete results are listed in our leaderboard.) Model SST-2MNLIRTEQNLIQQPAvgAvgAvg AdvGLUEAdvGLUEAdvGLUEAdvGLUEAdvGLUEAdvGLUEGLUE∆↓ State-of-the-art Pre-trained Language Models BERT (Large)33.0328.72/27.0540.4639.7737.91/16.5633.6885.76 52.08 ELECTRA (Large)58.5914.62/20.2223.0357.5461.37/42.4041.6993.1651.47 RoBERTa (Large)58.5250.78/39.6245.3952.4857.11/41.8050.2191.44 41.23 T5 (Large) 60.5648.43/38.9862.8357.6463.03/55.6856.8290.39 33.57 ALBERT (XXLarge)66.8351.83/44.1773.0363.8456.40/32.3559.2291.87 32.65 DeBERTa (Large)57.8958.36/52.4678.9557.8560.43/47.9860.8692.6731.81 Robust Training Methods for Pre-trained Language Models SMART (BERT)25.2126.89/23.3238.1634.6136.49/20.2430.2985.70 55.41 SMART (RoBERTa)50.9245.56/36.0770.3952.1764.22/44.2853.7192.62 38.91 FreeLB (RoBERTa) 61.6931.59/27.6062.1762.2942.18/31.0750.4792.28 41.81 InfoBERT (RoBERTa) 47.6150.39/41.2639.4754.8649.29/35.5446.0489.06 43.02 Evaluation Metrics.Specifically, we evaluate these attacks along two fronts:effectivenessand validity. For effectiveness, we consider two evaluation metrics:Attack Success Rate (ASR)andCu- rated Attack Success Rate (Curated ASR). Formally, given a benign datasetD=(x (i) , y (i) ) N i=1 consisting ofNpairs of samplex (i) and ground truthy (i) , for an adversarial attack methodAthat generates an adversarial exampleA(x)given an inputxto attack a surrogate modelf, ASR is calculated as ASR= ∑ (x,y)∈D 1[f(A(x))6=y] 1[f(x) =y] ,(1) where1is the indicator function. After the data curation phase, we collect a curated adversarial datasetD c . Thus, Curated ASR is calculated as Curated ASR= ∑ (x,y)∈D 1[f(A(x))6=y]·1[A(x)∈D c ] 1[f(x) =y] .(2) For validity, we consider three evaluation metrics:Filter Rate,Fleiss Kappa, andHuman Accuracy. Specifically, Filter Rate is calculated by1− Curated ASR ASR to measure how many examples are rejected in the data curation procedures and can reflect the noisiness of the generated adversarial examples. We report the average ASR, Curated ASR, and Filter Rate over the three surrogate models we consider in Table 3. Fleiss Kappa is a widely used metric in existing datasets (e.g., SNLI, ANLI, and FEVER [3,38,46]) to measure the inter-annotator agreement rate on the collected dataset. Fleiss Kappa between 0.4 and 0.6 is considered as moderate agreement and between 0.6 and 0.8 as substantial agreement. The inter-annotator agreement rates of most high-quality datasets fall into these two intervals. In this paper, we follow the standard protocol and report Fleiss Kappa and Curated Fleiss Kappa to analyze the inter-annotator agreement rate on the collected adversarial dataset before and after curation to reflect the ambiguity of generated examples. We also estimate the human performance on our curated datasets. Specifically, given a sample with 5 annotations, we take one random annotator’s annotation as the prediction and the majority voted label as the ground truth and calculate the human accuracy as shown in Table 3. Analysis.As shown in Table 3, in terms of attackeffectiveness, while most attacks show high ASR, the Curated ASR is always less than11%, which indicates that most existing adversarial attack algorithms are not effective enough to generate high-quality adversarial examples. In terms ofvalidity, the filter rates for most adversarial attack methods are more than85%, which suggests that existing strong adversarial attacks are prone to generating invalid adversarial examples that either change the original semantic meanings or generate ambiguous perturbations that hinder the annotators’ unanimity. We provide detailed filter rates for automatic filtering and human evaluation in Appendix Table 12, and the conclusion is that around60−80%of examples are filtered due to the low transferability 8 Table 5:Diagnostic report of state-of-the-art language models and robust training methods. For each at- tack method, we evaluate models against generated adversarial data for different tasks to obtain per-task accuracy scores, and report the macro-average of those scores. (C1=Embedding-similarity, C2=Typos, C3=Context- aware, C4=Knowledge-guided, C5=Compositions, C6=Syntactic-based Perturbations, C7=Distraction-based Perturbations, C8=CheckList, C9=StressTest, C10=ANLIand C11=AdvSQuAD). Models Word-Level PerturbationsSent.-LevelHuman-Crafted Examples C1C2C3C4C5C6C7C8C9C10C11 BERT (Large)42.02 31.96 45.18 45.86 33.8544.86 24.1616.33 23.20 13.47 10.53 ELECTRA (Large)43.07 45.12 47.95 46.33 47.3343.47 33.3032.20 26.29 26.94 52.63 RoBERTa (Large) 56.54 57.19 60.47 49.81 55.9250.49 41.8937.78 28.35 16.58 35.09 T5 (Large)60.04 67.94 64.60 59.84 58.5050.54 42.2069.0223.20 17.10 52.63 ALBERT (XXLarge)66.7167.6173.49 70.3659.5263.76 49.1445.55 39.69 26.94 43.86 DeBERTa (Large)65.0774.8768.02 65.3062.5457.41 47.2245.0852.0622.80 54.39 SMART (BERT)45.17 31.04 42.89 45.23 30.7640.74 16.628.2018.56 10.361.75 SMART (RoBERTa) 62.93 58.03 65.09 62.65 61.3755.31 40.1339.27 28.35 15.54 31.58 FreeLB (RoBERTa)51.95 53.23 52.92 51.15 52.1850.75 37.7266.87 23.7129.0264.91 InfoBERT (RoBERTa)55.47 55.78 59.02 51.33 55.4844.56 31.4934.31 42.27 14.51 43.86 and high word modification rate. Among the remaining samples, around30−40%examples are filtered due to the low human agreement rates (Human Consensus Filtering), and around20−30% are filtered due to the semantic changes which lead to the label changes (Utility Preserving Filtering). We also note that the data curation procedures are indispensable for the adversarial evaluation, as the Fleiss Kappa before curation is very low, suggesting that a lot of adversarial sentences have unreliable labels and thus tend to underestimate the model robustness against the textual adversarial attacks. After the data curation, our AdvGLUE shows a Curated Fleiss Kappa of near 0.6, comparable with existing high-quality dataset such as SNLI and ANLI. Among all the existing attack methods, we observe that TextBugger is the most effective and valid attack method, as it demonstrates the highest Curated ASR and Curated Fleiss Kappa across different tasks. 3.5 Finalizing the Dataset The full pipeline of constructing AdvGLUE is summarized in Figure 1. Merging. We note that distraction-based adversarial examples and human-crafted adversarial exam- ples are guaranteed to be valid by definition or crowd-sourcing annotations, and thus data curation is not needed on these attacks. When merging them with our curated set, we calculate the average number of samples per attack from our curated set, and sample the same amount of adversarial examples from these attacks following the same label distribution. This way, each attack contributes to similar amount of adversarial data, so that AdvGLUE can evaluate models against different types of attacks with similar weights and provide a comprehensive and unbiased diagnostic report. Dev-Test Split. After collecting the adversarial examples from the considered attacks, we split the final dataset into a dev set and a test set. In particular, we first randomly split the benign data into 9 : 1, and the adversarial examples generated based on90%of the benign data serve as the hidden test set, while the others are published as the dev set. For human-crafted adversarial examples, since they are not generated based on the benign GLUE data, we randomly select90%of the data as the test set, and the remaining10%as the dev set. The dev set is publicly released to help participants to understand the tasks and the data format. To protect the integrity of our test data, the test set will not be released to the public. Instead, participants are required to upload the model to CodaLab, which automates the evaluation process on the hidden test set and provides a diagnostic report. 4 Diagnostic Report for Language Models Benchmark Results. We follow the official implementations and training scripts of pre-trained language models to reproduce results on GLUE and test these models on AdvGLUE. The training details can be found in Appendix A.6. Results are summarized in Table 4. We observe that although state-of-the-art language models have achieved high performance on GLUE, they are vulnerable to various adversarial attacks. For instance, the performance gap can be as large as55%on the SMART 9 (BERT) model in terms of the average score. DeBERTa (Large) and ALBERT (XXLarge) achieve the highest average AdvGLUE scores among all the tested language models. This result is also aligned with the ANLI leaderboard 4 , which shows that ALBERT (XXLarge) is the most robust to human-crafted adversarial NLI dataset [38]. We note that although our adversarial examples are generated from surrogate models based on BERT and RoBERTa, these examples have high transferability between models after our data curation. Specifically, the average score of ELECTRA (Large) on AdvGLUE is even lower than RoBERTa (Large), which demonstrates that AdvGLUE can effectively transfer across models of different architectures and unveil the vulnerabilities shared across multiple models. Moreover, we find some models even perform worse than random guess. For example, the performance of BERT on AdvGLUE for all tasks is lower than random-guess accuracy. We also benchmark advanced robust training methods to evaluate whether these methods can indeed provide robustness improvement on AdvGLUE and to what extent. We observe that SMART and FreeLB are particularly helpful to improve robustness for RoBERTa. Specifically, SMART (RoBERTa) improves RoBERTa (Large) over3.71%on average, and it even improves the benign accuracy as well. Since InfoBERT is not evaluated on GLUE, we run InfoBERT with different hyper-parameters and report the best accuracy on benign GLUE dev set and AdvGLUE test set. However, we find that the benign accuracy of InfoBERT (RoBERTa) is still lower than RoBERTa (Large), and similarly for the robust accuracy. These results suggest that existing robust training methods only have incremental robustness improvement, and there is still a long way to go to develop robust models to achieve satisfactory performance on AdvGLUE. Diagnostic Report of Model Vulnerabilities.To have a systematic understanding of which adver- sarial attacks language models are vulnerable to, we provide a detailed diagnostic report in Table 5. We observe that models are most vulnerable to human-crafted examples, where complex linguistic phenomena (e.g., numerical reasoning, negation and coreference resolution) can be found. For sentence-level perturbations, models are more vulnerable to distraction-based perturbations than directly manipulating syntactic structures. In terms of word-level perturbations, models are similarly vulnerable to different word replacement strategies, among which typo-based perturbations and knowledge-guided perturbations are the most effective attacks. We hope the above findings can help researchers systematically examine their models against different adversarial attacks, thus also devising new methods to defend against them. Comprehensive analysis of the model robustness report is provided in our website and Appendix A.9. 5 Conclusion We introduce AdvGLUE, a multi-task benchmark to evaluate and analyze the robustness of state- of-the-art language models and robust training methods. We systematically conduct 14 adversarial attacks on GLUE tasks and adopt crowd-sourcing to guarantee the quality and validity of generated adversarial examples. Modern language models perform poorly on AdvGLUE, suggesting that model vulnerabilities to adversarial attacks still remain unsolved. We hope AdvGLUE can serve as a comprehensive and reliable diagnostic benchmark for researchers to further develop robust models. Acknowledgments and Disclosure of Funding We thank the anonymous reviewers for their constructive feedback. We also thank Prof. Sam Bowman, Dr. Adina Williams, Nikita Nangia, Jinfeng Li, and many others for the helpful discussion. We thank Prof. Robin Jia and Yixin Nie for allowing us to incorporate their datasets as part of the evaluation. We thank the SQuAD team for allowing us to use their website template and submission tutorials. This work is partially supported by the NSF grant No.1910100, NSF CNS 20-46726 CAR, the Amazon Research Award. 4 https://github.com/facebookresearch/anli 10 References [1]M. Bartolo, A. Roberts, J. Welbl, S. Riedel, and P. Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension.Transactions of the Association for Computational Linguistics, 8:662–678, 2020. [2]S. R. Bowman and G. E. Dahl. What will it take to fix benchmarking in natural language understanding? InNAACL, 2021. [3]S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. In L. Màrquez, C. Callison-Burch, J. Su, D. Pighin, and Y. Marton, editors,EMNLP, 2015. [4]K. Burghardt, T. Hogg, R. D’Souza, K. Lerman, and M. Posfai. Origins of algorithmic instabilities in crowdsourced ranking.Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2):1–20, 2020. [5]N. Carlini and D. A. Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. 2018 IEEE Security and Privacy Workshops (SPW), pages 1–7, 2018. [6]K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-training text encoders as discriminators rather than generators.arXiv preprint arXiv:2003.10555, 2020. [7]J. M. Cohen, E. Rosenfeld, and J. Z. Kolter. Certified adversarial robustness via randomized smoothing. InICML, 2019. [8] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, NAACL-HLT, 2019. [9]K. Dvijotham, S. Gowal, R. Stanforth, R. Arandjelovic, B. O’Donoghue, J. Uesato, and P. Kohli. Training verified learners with learned verifiers.CoRR, abs/1805.10265, 2018. [10]J. Ebrahimi, A. Rao, D. Lowd, and D. Dou. Hotflip: White-box adversarial examples for text classification. InACL, 2018. [11]K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. X. Song. Robust physical-world attacks on deep learning models. 2017. [12]Z. Gan, Y.-C. Chen, L. Li, C. Zhu, Y. Cheng, and J. Liu. Large-scale adversarial training for vision-and-language representation learning.arXiv preprint arXiv:2006.06195, 2020. [13]S. Garg and G. Ramakrishnan. Bae: Bert-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6174–6181, 2020. [14] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé I, and K. Crawford. Datasheets for datasets.arXiv preprint arXiv:1803.09010, 2018. [15]K. Goel, N. Rajani, J. Vig, S. Tan, J. Wu, S. Zheng, C. Xiong, M. Bansal, and C. Ré. Robustness gym: Unifying the nlp evaluation landscape.arXiv preprint arXiv:2101.04840, 2021. [16]I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. CoRR, abs/1412.6572, 2015. [17]T. Gui, X. Wang, Q. Zhang, Q. Liu, Y. Zou, X. Zhou, R. Zheng, C. Zhang, Q. Wu, J. Ye, et al. Textflint: Unified multilingual robustness evaluation toolkit for natural language processing. arXiv preprint arXiv:2103.11441, 2021. [18]P. He, X. Liu, J. Gao, and W. Chen. Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654, 2020. [19]P. Huang, R. Stanforth, J. Welbl, C. Dyer, D. Yogatama, S. Gowal, K. Dvijotham, and P. Kohli. Achieving verified robustness to symbol substitutions via interval bound propagation. In EMNLP-IJCNLP, 2019. [20]M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. InNAACL-HLT, 2018. [21]R. Jia and P. Liang. Adversarial examples for evaluating reading comprehension systems. In M. Palmer, R. Hwa, and S. Riedel, editors,EMNLP, 2017. [22] R. Jia, A. Raghunathan, K. Göksel, and P. Liang. Certified robustness to adversarial word substitutions. InEMNLP-IJCNLP, 2019. 11 [23]H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao. SMART: robust and efficient fine- tuning for pre-trained natural language models through principled regularized optimization. In D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, editors,ACL, 2020. [24]D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. InAAAI, 2020. [25]D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking benchmarking in nlp. InNAACL, 2021. [26]Z.-Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations.ArXiv, abs/1909.11942, 2019. [27]J. Li, S. Ji, T. Du, B. Li, and T. Wang. Textbugger: Generating adversarial text against real-world applications. InNDSS, 2019. [28]J. Li, T. Du, S. Ji, R. Zhang, Q. Lu, M. Yang, and T. Wang. Textshield: Robust text classification based on multimodal embedding and neural machine translation. In29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2020. [29]L. Li, R. Ma, Q. Guo, X. Xue, and X. Qiu. Bert-attack: Adversarial attack against bert using bert. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, 2020. [30]X. Liu, H. Cheng, P. He, W. Chen, Y. Wang, H. Poon, and J. Gao. Adversarial training for large neural language models.CoRR, abs/2004.08994, 2020. [31] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized BERT pretraining approach.CoRR, abs/1907.11692, 2019. [32] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, Z. Ghahramani, and K. Q. Weinberger, editors,Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111–3119, 2013. [33]S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: A simple and accurate method to fool deep neural networks.CVPR, pages 2574–2582, 2016. [34] J. Morris, E. Lifland, J. Lanchantin, Y. Ji, and Y. Qi. Reevaluating adversarial examples in natural language. InFindings of the Association for Computational Linguistics: EMNLP 2020, Online, Nov. 2020. Association for Computational Linguistics. [35] J. X. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.arXiv preprint arXiv:2005.05909, 2020. [36]A. Naik, A. Ravichander, N. Sadeh, C. Rose, and G. Neubig. Stress test evaluation for natural language inference.arXiv preprint arXiv:1806.00692, 2018. [37]N. Nangia and S. Bowman. Human vs. muppet: A conservative estimate of human performance on the glue benchmark. InACL, 2019. [38]Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial NLI: A new benchmark for natural language understanding. In D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, editors,ACL, 2020. [39]N. Papernot, P. D. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks.2016 IEEE Symposium on Security and Privacy (SP), pages 582–597, 2016. [40]J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In A. Moschitti, B. Pang, and W. Daelemans, editors,Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543. ACL, 2014. [41]F. Qi, C. Yang, Z. Liu, Q. Dong, M. Sun, and Z. Dong. Openhownet: An open sememe-based lexical knowledge base.ArXiv, abs/1901.09957, 2019. 12 [42]P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text.arXiv preprint arXiv:1606.05250, 2016. [43]M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. InACL, pages 4902–4912, July 2020. [44]R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013. [45]J. Thorne and A. Vlachos. Adversarial attacks against fact extraction and verification.CoRR, abs/1903.05543, 2019. [46]J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. Fever: a large-scale dataset for fact extraction and verification. InNAACL-HLT, 2018. [47]E. Wall, A. Narechania, A. Coscia, J. Paden, and A. Endert. Left, right, and gender: Exploring interaction traces to mitigate human biases.arXiv preprint arXiv:2108.03536, 2021. [48]A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. InNeurIPS, 2019. [49]A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. InICLR, 2019. [50]B. Wang, H. Pei, B. Pan, Q. Chen, S. Wang, and B. Li. T3: Tree-autoencoder constrained adversarial text generation for targeted attack. InEMNLP, 2020. [51]B. Wang, S. Wang, Y. Cheng, Z. Gan, R. Jia, B. Li, and J. Liu. Infobert: Improving robustness of language models from an information theoretic perspective. InICLR, 2021. [52]J. Wieting and K. Gimpel. Paranmt-50m: Pushing the limits of paraphrastic sentence embed- dings with millions of machine translations.arXiv preprint arXiv:1711.05732, 2017. [53]A. Williams, N. Nangia, and S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference.arXiv preprint arXiv:1704.05426, 2017. [54] Z. Yang, B. Li, P.-Y. Chen, and D. X. Song. Characterizing audio adversarial examples using temporal dependency.ArXiv, abs/1809.10875, 2018. [55]Z. Yang, Z. Dai, Y. Yang, J. G. Carbonell, R. Salakhutdinov, and Q. V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. InNeurIPS, 2019. [56]M. Ye, C. Gong, and Q. Liu. SAFER: A structure-free approach for certified robustness to adversarial word substitutions. InACL, 2020. [57]Y. Zang, F. Qi, C. Yang, Z. Liu, M. Zhang, Q. Liu, and M. Sun. Word-level textual adversarial attacking as combinatorial optimization. InACL, 2020. [58]G. Zeng, F. Qi, Q. Zhou, T. Zhang, B. Hou, Y. Zang, Z. Liu, and M. Sun. Openattack: An open-source textual adversarial attack toolkit.arXiv preprint arXiv:2009.09191, 2020. [59] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. InICLR, 2019. [60]Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu. Ernie: Enhanced language representation with informative entities. InACL, 2019. [61]C. Zhu, Y. Cheng, Z. Gan, S. Sun, T. Goldstein, and J. Liu. Freelb: Enhanced adversarial training for natural language understanding. InICLR, 2020. 13 A Appendix A.1 Glossary of Adversarial Attacks We present a glossary of adversarial attacks considered in AdvGLUE in Table 6 and 7. A.2 Additional Related Work We discuss more related work about textual adversarial attacks and defenses in this subsection. Textual Adversarial AttacksRecent research has shown deep neural networks (DNNs) are vul- nerable to adversarial examples that are carefully crafted to fool machine learning models without disturbing human perception [16,39,33]. However, compared with a large amount of adversarial attacks in continuous data domain [54,5,11], there are a few studies focusing on the discrete text domain. Most existing gradient-based attacks on image or audio models are no longer applicable to NLP models, as words are intrinsically discrete tokens. Another challenge for generating adversarial text is to ensure the semantic and syntactic coherence and consistency. Existing textual adversarial attacks can be roughly divided into three categories: word-level transfor- mations, sentence-level attacks, and human-crafted samples. (i) Word-level transformations adopt different word replacement strategies during attack. For example, existing work [27,10] applies character-level perturbation to carefully crafted typo words (e.g., from “foolish” to “fo0lish”), thus making the model ignore or misunderstand the original statistical cues. Others adopt knowledge- based perturbation and utialize knowledge base to constrain the search space. For example, Zang et al.[57]uses sememe-based knowledge base from HowNet [41] to construct a search space for word substitution. Some [24,27] use non-contextualized word embedding from GLoVe [40] or Word2Vec [32] to build synonym candidates, by querying the cosine similarity or euclidean distance between the original and candidate word and selecting the closet ones as the replacements. Recent work [13,29] also leverages BERT to generate contextualized perturbations by masked language modeling. (i) Different from the dominant word-level adversarial attacks, sentence-level adversarial attacks perform sentence-level transformation or paraphrasing by perturbing the syntactic structures based on human crafted rules [36,43] or carefully designed auto-encoders [20,50]. Sentence-level manipulations are generally more challenging than word-level attacks, because the perturbation space for syntactic structures are limited compared to word-level perturbation spaces that grow exponentially with the sentence length. However, sentence-level attacks tend to have higher linguistic quality than word-level, as both semantic and syntactic coherence are taken into considerations when generating adversarial sentences. (i) Human-crafted adversarial examples are generally crafted in the human-in-the-loop manner [21,38,1] or use manually crafted templates to generate test cases [36,43]. Our AdvGLUE incorporates all of the above textual adversarial to provide a comprehensive and systematic diagnostic report over existing state-of-the-art large-scale language models. Defenses against Textual Adversarial AttacksTo defend against textual adversarial attacks, existing work can be classified into three categories: (i)Adversarial Trainingis a practical method to defend against adversarial examples. Existing work either uses PGD-based attacks to generate adversarial examples in the embedding space of NLP as data augmentation [61], or regularizes the standard objective using virtual adversarial training [23,30,12]. However, one drawback is that the threat model is often unknown, which renders adversarial training less effective when facing unseen attacks. (i)Interval Bound Propagation(IBP) [9] is proposed as a new technique to consider the worst-case perturbation theoretically. Recent work [19,22] has applied IBP in the NLP domain to certify the robustness of models. However, IBP-based methods rely on strong assumptions of model architecture and are difficult to adapt to recent transformer-based language models. (i)Randomized Smoothing[7] provides a tight robustness guarantee in` 2 norm by smoothing the classifier with Gaussian noise. Ye et al.[56]adapts the idea to the NLP domain, and replace the Gaussian noise with synonym words to certify the robustness as long as adversarial word substitution falls into predefined synonym sets. However, to guarantee the completeness of the synonym set is challenging. A.3 Task Descriptions, Statistics and Evaluation Metrics We present the detailed label distribution statistics and evaluation metrics of GLUE and AdvGLUE benchmark in 8. 14 Table 6:Glossary of adversarial attacks (word-level and sentence-level) in AdvGLUE.For each adversarial attack, we provide a brief explanation and a corresponding example in AdvGLUE. PerturbationsExplanation Examples(Strikethrough = Original Text, red = Adver- sarial Perturbation) TextBugger (Word-level / Typo-based) TextBugger first identifies the important words in each sentence and then replaces them with carefully crafted typos. Task:QNLI Question : What was the population of the Dutch Repub- lic before this emigration? Sentence: This was ahuge hu ge influx as the entire population of the Dutch Republic amounted to ca. Prediction:False→True TextFooler (Word-level / Embedding- similarity- based) Embedding-similarity-based adversarial attacks such as TextFooler select synonyms according to the cosine similarity of word embeddings. Words that have high similarity scores will be used as candidates to replace original words in the sentences. Task:QQP Question 1: I am getting fat on my lower body and on thechest torso, is there any way I can get fit without looking skinny fat? Question 2 : Why I am getting skinny instead of losing body fat? Prediction:Not Equivalent→Equivalent BERT- ATTACK (Word-level / Context-aware) BERT-ATTACK uses pre-trained BERT to perform masked language prediction to generate contextualized potential word replacements for those crucial words. Task:MNLI Premise: Do you know what this is? With a dramatic gesture she flung back the left side of hercoat sleeve and exposed a small enamelled badge. Hypothesis: The coat that she wore was long enough to cover her knees . Prediction:Neutral→Contradiction SememePSO (Word-level / Knowledge- guided) Knowledge-guided adversarial attacks such as SememePSO use external knowledge base such as HowNet or WordNet to search for substitutions. Task:QQP Question 1 : What people who you’ve never met have influenced infected your life the most? Question 2: Who are people you have never met who have had the greatest influence on your life? Prediction:Equivalent→Not Equivalent CompAttack (Word-level / Compositions) CompAttack is a whitebox-based adversarial attack that integrates all other word-level perturbation methods in one algorithm to evaluate model robustness to various adversarial transformations. Task:SST-2 Sentence : The primitive force of this film seems tobub- ble bybble up from the vast collective memory of the combatants. Prediction:Positive→Negative SCPN (Sent.-level / Syntactic- based) SCPN is an attack method based on syntax tree transformations. It is trained to produce a paraphrase of a given sentence with specified syntactic structures. Task:RTE Sentence 1 : He became a boxing referee in 1964 and became most well-known for his decision against Mike Tyson, during the Holyfield fight, when Tyson bit Holy- field’s ear. Sentence 2: Mike Tyson bitHolyfield’sear in 1964. Prediction:Not Entailment→Entailment T3 (Sent.-level / Syntactic- based) T3 is a whitebox attack algorithm that can add perturbations on different levels of the syntax tree and generate the adversarial sentence. Task:MNLI Premise: What’s truly striking, though, is that Jobshas had never really let this idea go. Hypothesis: Jobs never held onto an idea for long. Prediction:Contradiction→Entailment AdvFever (Sent.-level / Syntactic- based) Entailment preserving rules proposed by AdvFever transform all the sentences satisfying the templates into semantically equivalent ones. Task:SST-2 Sentence :I’llbetthevideogameis There exists a lot more fun than the film that goes by the name of i ’l bet the video game. Prediction:Negative→Positive StressTest (Sent.-level / Distraction- based) StressTest appends three true statements (“and true is true”, “and false is not true”, “and true is true” for five times) to the end of the hypothesis sentence for NLI tasks. Task:RTE Sentence 1: Yet, we now are discovering that antibiotics are losing their effectiveness against illness. Disease- causing bacteria are mutating faster than we can come up with new antibiotics to fight the new variations. Sentence 2 : Bacteria is winning the war against antibi- otics and true is true. Prediction:Entailment→Not Entailment CheckList (Sent.-level / Distraction- based) CheckList adds randomly generated URLs and handles to distract model attention. Task:QNLI Question: What was the population of the Dutch Repub- lic before this emigration? https://t.co/DlI9kw Sentence: This was a huge influx as the entire popula- tion of the Dutch Republic amounted to ca. Prediction:False→True 15 Table 7:Glossary of adversarial attacks (human-crafted) in AdvGLUE.For each adversarial attack, we provide a brief explanation and a corresponding example in AdvGLUE. PerturbationsExplanation Examples(Strikethrough = Original Text, red = Adver- sarial Perturbation) CheckList (Human- crafted) CheckList analyses different capabilities of NLP models using different test types. We adopt two capability tests:TemporalandNegation, which test if the model understands the order of events and if the model is sensitive to negations. Task:SST-2 Sentence: I think this movie is perfect, but I used to think it was annoying. Prediction:Positive→Negative StressTest (Human- crafted) StressTest proposes carefully crafted rules to construct “stress tests” and evaluate robustness of NLI models to specific linguistic phenomena. Here we adopt the test cases focusing on Numerical Reasoning. Task:MNLI Premise : If Anne’ s speed were doubled, they could clean their house in 3 hours working at their respective rates. Hypothesis: If Anne’ s speed were doubled, they could clean their house in less than 6 hours working at their respective rates. Prediction:Entailment→Contradiction ANLI (Human- crafted) ANLI is a large-scale NLI dataset collected iteratively in a human-in-the-loop manner. The sentence pairs generated in each round form a comprehensive dataset that aims at examining the vulnerability of NLI models. Task:MNLI Premise : Kamila Filipcikova (born 1991) is a female Slovakian fashion model. She has modeled in fash- ion shows for designers such as Marc Jacobs, Chanel, Givenchy, Dolce & Gabbana, and Sonia Rykiel. And appeared on the cover of Vogue Italia two times in a row. Hypothesis: Filipcikova lives in Italy. Prediction:Neutral→Contradiction AdvSQuAD (Human- crafted) AdvSQuAD is an adversarial dataset targeting at reading comprehension systems. Examples are generated by appending a distracting sentence to the end of the input paragraph. We adopt the distracting sentences and questions in theQNLIformat with labels “not answered”. Task:QNLI Question: What day was the Super Bowl played on? Sentence : The Champ Bowl was played on August 18th,1991. Prediction:False→True SST-2The Stanford Sentiment Treebank [44] consists of sentences from movie reviews and human annotations of their sentiment. Given a review sentence, the task is to predict the sentiment of it. Sentiments can be divided into two classes: positive and negative. QQPThe Quora Question Pairs (QQP) dataset is a collection of question pairs from the commu- nity question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent. MNLI The Multi-Genre Natural Language Inference Corpus [53] consists of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral) QNLIQuestion-answering NLI (QNLI) dataset consists of question-sentence pairs modified from The Stanford Question Answering Dataset [42]. The task is to determine whether the context sentence contains the answer to the question. RTEThe Recognizing Textual Entailment (RTE) dataset is a combination of a series of data from annual textual entailment challenges. Examples are constructed based on news and Wikipedia text. The task is to predict the relationship between a pair of sentences. For consistency, the relationship can be classified into two classes: entailment and not entailment, where neutral and contradiction are seen as not entailment. We also show the detailed per-task model performance on AdvGLUE and GLUE in Table 9. A.4 Implementation Details of Adversarial Attacks TextBuggerTo ensure the small magnitude of the perturbation, we consider the following five strategies: (i) randomly inserting a space into a word; (i) randomly deleting a character of a word; (i) randomly replacing a character of a word with its adjacent character in the keyboard; (iv) randomly replacing a character of a word with its visually similar counterpart (e.g., “0” v.s. “o”, “1” 16 Table 8:The label distribution of AdvGLUE dataset. For SST-2, we report the label distribution as“nega- tive”:“positive”. For QQP, we report the label distribution as“not equivalent”:“equivalent”. For QNLI, we report the label distribution as“true”:“false”. For RTE, we report the label distribution as“entailment”:“not entailment”. For MNLI, we report the label distribution as“entailment”:“neutral”:“contradiction”. CorpusTask |Dev| (GLUE) |Test| (GLUE) |Dev| (AdvGLUE) |Test| (AdvGLUE)Evaluation Metrics SST-2sentiment428:444182172:76590:830acc. QQPparaphrase25,545:14,885390,96546:32297:125acc./F1 QNLINLI/QA2,702:2,7615,46374:74394:574acc. RTENLI146:1313,00035:46123:181acc. MNLINLI6,942:6,252:6,45319,64392:84:107706:565:593matched acc./mismatched acc. Table 9: Model performance on AdvGLUE test set and GLUE dev set. Models AvgSST-2MNLIRTEQNLIQQP GLUEAdvGLUEGLUEAdvGLUEGLUEAdvGLUEGLUEAdvGLUEGLUEAdvGLUEGLUEAdvGLUE BERT(Large)85.7633.6893.2333.0385.78/85.5728.72/27.0568.9540.4691.9139.7790.72/87.3837.91/16.56 RoBERTa(Large)91.4450.2195.9958.5289.74/89.8650.78/39.6286.6045.3994.1452.4891.99/89.3757.11/41.80 T5(Large)90.3956.8295.5360.5688.98/89.2048.43/38.9884.1262.8393.7857.6490.82/88.0763.03/55.68 ALBERT(XXLarge)91.8759.2295.1866.8389.29/89.8851.83/44.1788.4573.0395.2663.8492.26/89.4956.40/32.35 ELECTRA(Large)93.1641.6997.1358.5990.7114.62/20.2290.2523.0395.1757.5492.5661.37/42.40 DeBERTa(Large)92.6760.8696.3357.8990.95/90.8558.36/52.4690.2578.9494.8657.8592.29/89.6960.43/47.98 SMART(BERT)85.7030.2993.3525.2184.72/85.3426.89/23.3269.6838.1691.7134.6190.25/87.2236.49/20.24 SMART(RoBERTa) 92.6253.7196.5650.9290.75/90.6645.56/36.0790.9870.3995.0452.1791.20/88.4464.22/44.28 FreeLB(RoBERTa)92.2850.4796.4461.6990.6431.59/27.6086.6962.1795.0462.2992.5842.18/31.07 InfoBERT(RoBERTa)89.0646.0496.2247.6189.67/89.2750.39/41.2674.0139.4794.6254.8692.25/89.7049.29/35.54 v.s. “l”); and (v) randomly swapping two characters in a word. The first four strategies guarantee the word edit distance between the typo word and its original word to be 1, and that of the last strategy is limited to 2. Following the default setting, in Strategy (i), we only insert a space into a word when the word contains less than6characters. In Strategy (v), we swap characters in a word only when the word has more than4characters. TextFoolerConcretely, for the sentiment analysis tasks, we set the cosine similarity threshold to be0.8, which encourages the synonyms to be semantically close to original ones and enhances the quality of adversarial data. For the rest of the tasks, we follow the default hyper-parameter to set the cosine similarity threshold to be0.7. Besides, the number of synonyms for each word is set to50 following the default setting. BERT-ATTACKWe follow the hyper-parameters from the official codebase, and set the number of candidate words to 48 and cosine similarity threshold to0.4in order to filter out antonyms using synonym dictionaries, as BERT masked language model does not distinguish synonyms and antonyms. SememePSOWe adopt the official hyper-parameters in which maximum and minimum inertia weights are set to0.8and0.2, respectively. We also set the maximum and minimum movement probabilities of the particles to0.8and0.2, respectively, following the default setting. Population size is set to60in every task. CompAttackWe follow the T3 [50] and C&W attack [5] and design the same optimization objective for adversarial perturbation generation in the embedding space as: L(e ∗ ) =||e ∗ || p +c·g(x ′ ),(3) where the first term controls the magnitude of perturbation, whileg(·)is the attack objective function depending on the attack scenario.cweighs the attack goal against attack cost. CompAttack constrains the perturbation to be close to pre-defined perturbation space, including typo space (e.g., TextBugger), knowledge space (e.g.,WordNet) and contextualized embedding space (e.g.,BERT embedding clusters) to make sure the perturbation is valid. We can also see from Table 3 that CompAttack overall has lower filter rate than other state-of-the-art attack methods. SCPNWe use the pre-trained SCPN models released by the official codebase. Following the default setting, we select the most frequent10templates from ParaNMT-50M corpus [52] to guide the generation process. We first parse sentences from GLUE dev set using Stanford CoreNLP. We used CoreNLP version 3.7.0 in our experiment, along with the Shift-Reduce Parser models. 17 Table 10:Examples of AdvGLUE benchmark. TaskLinguistic Phe- nomenon Samples (Strikethrough = Original Text, red = Adversarial Perturbation)Label→Pre- diction SST-2 Typo (Word-level) Sentence: The primitive force of this film seems tobubble bybble up from the vast collective memory of the combatants. Positive →Negative SST-2 Context-aware (Word-level) Sentence: In execution , this clever idea is farless smaller funny than the original , killers from space. Negative →Positive SST-2 CheckList (Human-crafted) Sentence : I think this movie is perfect, but I used to think it was annoy- ing. Positive →Negative QQP Embedding (Word-level) Question 1: I am getting fat on my lower body and on thechest torso, is there any way I can get fit without looking skinny fat? Not Equivalent →Equivalent Question 2: Why I am getting skinny instead of losing body fat? QQP Syntactic (Sent.-level) Question 1:CanIlearnMMAattheageof26? You can learn MMA at 24? Not Equivalent →Equivalent Question 2: Can I learn MMA at the age of 24? QQP CheckList (Human-crafted) Question 1: Is Alfred Kennedy an analyst?Not Equivalent →EquivalentQuestion 2: Is Alfred Kennedy becoming an analyst? MNLI Typo (Word-level) Premise: uh-huh how about anymatching mathcing programsEntailment→ ContradictionHypothesis: What about matching programs? MNLI Distraction (Sent.-level) Premise: You and your friends are not welcome here, said Severn. Entailment→ Contradiction Hypothesis: Severn said the people were not welcome there and true is true. MNLI ANLI (Human-crafted) Premise: Kamila Filipcikova (born 1991) is a female Slovakian fashion model. She has modeled in fashion shows for designers such as Marc Jacobs, Chanel, Givenchy, Dolce & Gabbana, and Sonia Rykiel. And appeared on the cover of Vogue Italia two times in a row. Neutral→ Contradiction Hypothesis: Filipcikova lives in Italy. QNLI Distraction (Sent.-level) Question: What was the population of the Dutch Republic before this emigration? https://t.co/DlI9kw False→True Sentence : This was a huge influx as the entire population of the Dutch Republic amounted to ca. QNLI AdvSQuAD (Human-crafted) Question: What day was the Super Bowl played on? False→True Sentence: The Champ Bowl was played on August 18th,1991. RTE Knowledge (Word-level) Sentence 1: In Nigeria, by far the most populous country in sub-Saharan Africa, over 2.7 million peopleare exist infected with HIV. Not Entailment →Entailment Sentence 2: 2.7 percent of the people infected with HIV live in Africa. RTE Syntactic (Sent.-level) Sentence 1 : He became a boxing referee in 1964 and became most well-known for his decision against Mike Tyson, during the Holyfield fight, when Tyson bit Holyfield’s ear. Not Entailment →Entailment Sentence 2: Mike Tyson bitHolyfield’sear in 1964. T3We follow the hyper-parameters in the official setting where the scaling const is set to1e4and the optimizing confidence is set to0. In each iteration, we optimize the perturbation vector for at most100steps with learning rate0.1. AdvFeverWe follow the entailment preserving rules proposed by the official implementation. We adopt all23templates to transform original sentences into semantically equivalent ones. Many common sentence patterns in everyday life are included in these templates. A.5 Examples of AdvGLUE benchmark We show more comprehensive examples in Table 10. Examples are generated with different levels of perturbations and they all can successfully change the predictions of all surrogate models (BERT, RoBERTa and RoBERTa ensemble). A.6 Fine-tuning Details of Large-Scale Language Models For all the experiments, we are using a GPU cluster with 8 V100 GPUs and 256GB memory. 18 Table 11: The statistics of AdvGLUE in the human training phase. Corpus Pay Rate#/ QualifiedHumanHumanFleiss (per batch)WorkersAcc. (Avg.)Acc. (vote)Kappa SST-2$0.47089.295.00.738 MNLI$1.03380.485.00.615 RTE$1.06685.892.00.602 QNLI$1.04185.691.00.684 QQP$0.55886.490.00.691 BERT (Large)For RTE, we train our model for10epochs and for other tasks we train our model for4epochs. Batch size for QNLI is set to512, and for other tasks it is set to256. Learning rates are all set to2e−5. ELECTRA (Large)We follow the official hyper-parameter setting to set the learning rate to5e−5 and set batch size to32. We train ELECTRA on RTE for10epochs and train for2epochs on other tasks. We set the weight decay rate to0.01for every task. RoBERTa (Large) We train our RoBERTa for10epochs with learning rate2e−5on each task. The batch size for QNLI is32and64for other tasks. T5 (Large)We train our T5 for10epochs with learning rate2e−5on each task. The batch size for QNLI is32and64for other tasks. We follow the templates in original paper to convert GLUE tasks into generation tasks. ALBERT (XXLarge) We use the default hyper-parameters to train our ALBERT. For example, max training steps for SST-2, MNLI, QNLI, QQP, RTE, is20935,10000,33112,14000,800respectively. For MNLI and QQP, batch size is set to32and for other tasks batch size is set to128. DeBERTa (Large) We use the official hyper-parameters to train our DeBERTa. For example, learning rate is set to1e−5across all tasks. For MNLI and QQP, batch size is set to64and for other tasks batch size is set to 32. SMARTFor SMART(BERT) and SMART(RoBERTa), we use grid search to search for the best parameters and report the best performance among all trained models. FreeLB (RoBERTa) For FreeLB, we test every parameter combination provided by the official codebase and select the best parameters for our training. InfoBERT (RoBERTa)We set the batch size to32and learning rate to2e−5for all tasks. A.7 Human Evaluation Details Human Training We present the pay rate and the number of qualified workers in Table 11. We also test our qualified workers on another non-overlapping 100 samples of the GLUE dev sets for each task. We can see that the human accuracy is comparable to [37], which means that most our selected annotators understand the GLUE tasks well. Human Filtering The detailed filtering statistics of each stage is shown in Table 12. We can see that around60−80%of examples are filtered due to the low transferability and high word modification rate. Among the remaining samples, around30−40%examples are filtered due to the low human agreement rates (Human Consensus Filtering), and around20−30%are filtered due to the semantic changes which lead to the label changes (Utility Preserving Filtering). Human Annotation InstructionsWe show examples of annotation instructions in the train- ing phase and filtering phase on MNLI in Figure 2 and 3.More instructions can be found inhttps://adversarialglue.github.io/instructions.We also provide a 19 Table 12: Filter rates during data curation. TasksMetrics Word-level Attacks Average SememePSOTextFoolerTextBuggerCombAttackBERT-ATTACK SST-2 Transferability58.8563.5664.8753.5866.8761.54 Fidelity14.6511.0622.4019.9312.0316.01 Human Consensus10.5310.562.279.927.098.07 Utility Preserving6.685.430.513.203.823.93 Filter Rate90.7190.6290.0486.6389.8189.56 MNLI Transferability 44.1643.1542.5835.0841.8041.36 Fidelity36.5745.9437.7138.1438.6039.39 Human Consensus10.376.385.5111.159.788.64 Utility Preserving4.492.081.3211.075.914.97 Filter Rate 95.5997.5587.1295.4596.1094.36 RTE Transferability 55.3267.3841.9654.2060.9455.96 Fidelity19.837.7942.1823.1714.2521.44 Human Consensus8.087.913.557.648.447.12 Utility Preserving8.696.130.605.708.545.93 Filter Rate91.9389.2188.2990.7292.1690.46 QNLI Transferability63.3670.6759.2455.4769.1563.58 Fidelity17.7313.0125.3123.5313.1718.55 Human Consensus 10.069.806.849.989.369.21 Utility Preserving3.482.411.504.944.103.29 Filter Rate94.6395.8992.8993.9295.7894.62 QQP Transferability42.9658.6055.0944.8351.9750.69 Fidelity 45.6129.3526.4630.9937.7734.04 Human Consensus4.384.695.1910.083.945.66 Utility Preserving3.793.863.167.934.604.67 Filter Rate96.7396.5089.9093.8398.2895.05 FAQ document in each task description pagehttps://docs.google.com/document/d/ 1MikHUdyvcsrPqE8x-N-gHaLUNAbA6-Uvy-iA5gkStoc/edit?usp=sharing. A.8 Discussion of Limitations Due to the constraints of computational resources, we are unable to conduct a comprehensive evaluation of all existing language models. However, with the release of our leaderboard website, we are expecting researchers to actively submit their models and evaluate against our AdvGLUE benchmark to have a systematic understanding of model robustness. We are also interested in the adversarial robustness of large-scale auto-regressive language models under the few-shot settings, and leave it as a compelling future work. In this paper, we follow ANLI [38] and generate adversarial examples against surrogate models based on BERT and RoBERTa. However, there are concerns [2] that such adversarial filtering may not be able to fairly benchmark the model robustness, as participants may top the leaderboard by producing different errors from our surrogate models. We note that such concerns can be solved given systematic data curation. As shown in our main benchmark results, we observe we successfully select the adversarial examples with high adversarial transferability that can unveil the vulnerabilities shared across models of different architectures. Specifically, we observe a huge performance gap in ELECTRA (Large) that is pre-trained with different data and shown less robust than one of surrogate model RoBERTa (Large). Finally, we emphasize that our AdvGLUE benchmark mainly focuses on robustness evaluation. Thus AdvGLUE can also be considered as a supplementary diagnostic test set besides the standard GLUE benchmark. We suggest that participants should evaluate their models against both GLUE benchmark and our AdvGLUE to understand both model generalization and robustness. We hope our work can help researchers to develop models with high generalization and adversarial robustness. A.9 Website We present the diagnostic report on our website in Figure 4. 20 Figure 2: Human annotation instructions (training phase) for MNLI. B Data Sheet We follow the documentation frameworks provided by Gebru et al. [14]. B.1 Motivation For what purpose was the dataset created?While recently a lot of methods (SMART, FreeLB, InfoBERT, ALUM) claim that they can improve the model robustness against adversarial attacks, the adversary setup in these methods (i) lacks a unified standard and is usually different across different methods; (i) fails to cover comprehensive linguistic transformation (typos, synonymous substitution, paraphrasing, etc) to recognize to which levels of adversarial attacks models are still vulnerable. This motivates us to build a unified and principled robustness benchmark dataset and evaluate to which extent the state-of-the-art models have progressed so far in terms of adversarial robustness. Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? University of Illinois at Urbana-Champaign (UIUC) and Microsoft Corporation. B.2 Composition/collection process/preprocessing/cleaning/labeling and uses: The answers are described in our paper as well as websitehttps://adversarialglue.github. io. 21 Figure 3: Human annotation instructions (filtering phase) for MNLI. B.3 Distribution Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?The dev set is released to the public. The test set is hidden and can only be evaluated by an automatic submission API hosted on CodaLab. How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?The dev set is released on our websitehttps://adversarialglue.github.io. The test set is hidden and hosted on CodaLab. When will the dataset be distributed?It has been released now. Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?Our dataset will be distributed under the C BY- SA 4.0 license. B.4 Maintenance How can the owner/curator/manager of the dataset be contacted (e.g., email address)? Boxin Wang (boxinw2@illinois.edu) and Chejian Xu (xuchejian@zju.edu.cn) will be responsible for maintenance. 22 Figure 4: An example of model diagnostic report for BERT (Large). 23 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete in- stances)?Yes. If we include more tasks or find any errors, we will correct the dataset and update the leaderboard accordingly. It will be updated on our website. If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?They can contact us via email for the contribution. 24

Instant research discovery