Paper deep dive
Adversarial attacks and defenses for large language models (LLMs): methods, frameworks & challenges
Pranjal Kumar
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 7:22:53 PM
Summary
This paper provides a systematic survey of adversarial attacks and defense frameworks for Large Language Models (LLMs). It categorizes various attack techniques, discusses defense strategies, and identifies critical challenges such as transferability, computational requirements, and performance assessment, synthesizing insights from over 200 recent research papers.
Entities (5)
Relation Signals (3)
Adversarial Attacks → targets → Large Language Models
confidence 100% · This work delves into the intricate landscape of adversarial attacks on language models (LMs)
Defense Frameworks → protects → Large Language Models
confidence 95% · addresses methods for strengthening LLMs
Prompt Injection → isa → Adversarial Attacks
confidence 90% · Prompt injection attack against llm-integrated applications.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Large language models (LLMs) have exhibited remarkable efficacy and proficiency in a wide array of NLP endeavors. Nevertheless, concerns are growing rapidl
Tags
Links
Full Text
65,374 characters extracted from source content.
Expand or collapse full text
Home International Journal of Multimedia Information Retrieval Article Adversarial attacks and defenses for large language models (LLMs): methods, frameworks & challenges Trends and Surveys Published: 25 June 2024 Volume 13, article number 26, (2024) Cite this article Save article View saved research International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript Pranjal Kumar1 3445 Accesses 30 Citations Explore all metrics AbstractLarge language models (LLMs) have exhibited remarkable efficacy and proficiency in a wide array of NLP endeavors. Nevertheless, concerns are growing rapidly regarding the security and vulnerabilities linked to the adoption and incorporation of LLM. In this work, a systematic study focused on the most up-to-date attack and defense frameworks for the LLM is presented. This work delves into the intricate landscape of adversarial attacks on language models (LMs) and presents a thorough problem formulation. It covers a spectrum of attack enhancement techniques and also addresses methods for strengthening LLMs. This study also highlights challenges in the field, such as the assessment of offensive or defensive performance, defense and attack transferability, high computational requirements, embedding space size, and perturbation. This survey encompasses more than 200 recent papers concerning adversarial attacks and techniques. By synthesizing a broad array of attack techniques, defenses, and challenges, this paper contributes to the ongoing discourse on securing LM against adversarial threats. This is a preview of subscription content, log in via an institution to check access. Access this article Log in via an institution Subscribe and save Springer+ from $39.99 /Month Starting from 10 chapters or articles per month Access and download chapters and articles from more than 300k books and 2,500 journals Cancel anytime View plans Buy Now Buy article PDF USD 39.95 Price excludes VAT (USA) Tax calculation will be finalised during checkout. Instant access to the full article PDF. Institutional subscriptions Fig. 1Fig. 2Fig. 3Fig. 4Fig. 5Fig. 6Fig. 7Fig. 8Fig. 9Fig. 10 Similar content being viewed by others Adversarial Attacks on Large Language Models Chapter © 2024 Adversarial Attacks on Large Language Models: A Survey Chapter © 2026 Exploring Advanced Methodologies in Security Evaluation for Large Language Models Chapter © 2024 Explore related subjects Discover the latest articles, books and news in related subjects, suggested using machine learning. Applied Linguistics Computational Linguistics Cryptology Cybercrime Principles and Models of Security Security Science and Technology Adversarial Vulnerabilities in Deep Learning Systems Data Availability No datasets were generated or analysed during the current study. ReferencesSutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. Adv Neural Inf Process Syst 27:3104–3112 Google Scholar Saon G, Kurata G, Sercu T, Audhkhasi K, Thomas S, Dimitriadis D, Cui X, Ramabhadran B, Picheny M, Lim L, Roomi B (2017) English conversational telephone speech recognition by humans and machines. In Proceedings of the Interspeech 2017, p 132–136Khatiri S, Di Sorbo A, Zampetti F, Visaggio CA, Di Penta M, Panichella S (2024) Identifying safety-critical concerns in unmanned aerial vehicle software platforms with salient. SoftwareX 27:101748Article Google Scholar Parkhi O, Vedaldi A, Zisserman A (2015) Deep face recognition. In: BMVC 2015-proceedings of the British machine vision conference 2015. British Machine Vision AssociationChen C, Seff A, Kornhauser A, Xiao J (2015) Deepdriving: learning affordance for direct perception in autonomous driving. In: Proceedings of the IEEE international conference on computer vision, p 2722–2730Ma X, Fang G, Wang X (2023) LLM-pruner: on the structural pruning of large language models. Adv Neural Inf Process Syst 36:21702–21720 Google Scholar Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2014) Intriguing properties of neural networks. In: Bengio Y, LeCun Y (eds) 2nd international conference on learning representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, conference track proceedingsAlotaibi A, Rassam MA (2023) Adversarial machine learning attacks against intrusion detection systems: a survey on strategies and defense. Future Internet 15(2):62Article Google Scholar Raiaan MA, Mukta MS, Fatema K, Fahad NM, Sakib S, Mim M, Ahmad J, Ali ME, Azam S (2024) A review on large language models: architectures, applications, taxonomies, open issues and challenges. IEEE Access. https://doi.org/10.1109/ACCESS.2024.3365742Article Google Scholar Boffa M, Drago I, Mellia M, Vassio L, Giordano D, Valentim R, Houidi ZB (2024) Logprécis: unleashing language models for automated malicious log analysis: Précis: a concise summary of essential points, statements, or facts. Comput Secur 141:103805Article Google Scholar Alwahedi F, Aldhaheri A, Ferrag MA, Battah A, Tihanyi N (2024) Machine learning techniques for IoT security: current research and future vision with generative AI and large language models. Internet Things Cyber Phys Syst. https://doi.org/10.1016/j.iotcps.2023.12.003Article Google Scholar Li Z, Fan S, Gu Y, Li X, Duan Z, Dong B, Liu N, Wang J (2024) Flexkbqa: a flexible LLM-powered framework for few-shot knowledge base question answering. In: Proceedings of the AAAI conference on artificial intelligence 38:18608–18616Livne M, Miftahutdinov Z, Tutubalina E, Kuznetsov M, Polykovskiy D, Brundyn A, Jhunjhunwala A, Costa A, Aliper A, Aspuru-Guzik A et al (2024) nach0: multimodal natural and chemical languages foundation model. Chem Sci. https://doi.org/10.1039/D4SC00966EArticle Google Scholar Abe N, Zadrozny B, Langford J (2004) An iterative method for multi-class cost-sensitive learning. In: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, p 3–11Yuan X, He P, Zhu Q, Li X (2019) Adversarial examples: attacks and defenses for deep learning. IEEE Trans Neural Netw Learn Syst 30(9):2805–2824Article MathSciNet Google Scholar Wu C, Fang W, Dai F, Yin H (2023) A model ensemble approach with LLM for Chinese text classification. In: China health information processing conference. Springer, p 214–230Nazir A, Chakravarthy TK, Cecchini DA, Khajuria R, Sharma P, Mirik AT, Kocaman V, Talby D (2024) LangTest: a comprehensive evaluation library for custom LLM and NLP models. Softw Impacts 19:100619Article Google Scholar Sang EF, De Meulder F (2003) Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on natural language learning at HLT-NAACL 2003, p 142–147Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) SQuAD: 100,000+ questions for machine comprehension of text. In: Su J, Duh K, Carreras X (eds) Proceedings of the 2016 conference on empirical methods in natural language processing, Austin, Texas, November 2016. Association for Computational Linguistics, p 2383–2392Wang A, Singh A, Michael J, Hill F, Levy O, Bowman SR (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Linzen T, Chrupała G, Alishahi, A (eds) Proceedings of the 2018 EMNLP workshop BlackboxNLP: analyzing and interpreting neural networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics, p 353–355Wei C, Xie SM, Ma T (2021) Why do pretrained language models help in downstream tasks? An analysis of head and prompt tuning. Adv Neural Inf Process Syst 34:16158–16170 Google Scholar Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-trainingDevlin J, Chang MW, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), Minneapolis, Minnesota, June 2019. Association for Computational Linguistics, p 4171–4186Akbik A, Bergmann T, Blythe D, Rasul K, Schweter S, Vollgraf R (2019) FLAIR: an easy-to-use framework for state-of-the-art NLP. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (demonstrations), p 54–59Chen K, Meng Y, Sun X, Guo S, Zhang T, Li J, Fan C (2022) Badpre: task-agnostic backdoor attacks to pre-trained NLP foundation models. In: International conference on learning representationsFeldman V, Zhang C (2020) What neural networks memorize and why: discovering the long tail via influence estimation. Adv Neural Inf Process Syst 33:2881–2891 Google Scholar Krishna K, Tomar GS, Parikh AP, Papernot N, Iyyer M (2020) Thieves on sesame street! model extraction of BERT-based APIs. In: International conference on learning representationsWang B (2023) Towards trustworthy large language models. PhD thesis, University of Illinois at Urbana-ChampaignLi L, Ma R, Guo Q, Xue X, Qiu X (2020) BERT-ATTACK: adversarial attack against BERT using BERT. In: Webber B, Cohn T, He Y, Liu Y (eds) Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), Online, November 2020. Association for Computational Linguistics, p 6193–6202Yuan L, Zheng X, Zhou Y, Hsieh CJ, Chang KW (2021) On the transferability of adversarial attacks against neural text classifier. In: Moens MF, Huang X, Specia L, Yih SWT (eds) Proceedings of the 2021 conference on empirical methods in natural language processing, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics, p 1612–1625Shen L, Ji S, Zhang X, Li J, Chen J, Shi J, Fang C, Yin J, Wang T (2021) Backdoor pre-trained models can transfer to all. In: Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, CCS ’21, New York, NY, USA, 2021. Association for Computing Machinery, p 3141–3158Rane NL, Tawde A, Choudhary SP, Rane J (2023) Contribution and performance of ChatGPT and other large language models (LLM) for scientific and research advancements: a double-edged sword. Int Res J Mod Eng Technol Sci 5(10):875–899 Google Scholar Yao Y, Duan J, Xu K, Cai Y, Sun Z, Zhang Y (2024) A survey on large language model (LLM) security and privacy: the good, the bad, and the ugly. High Confid Comput 4:100211Article Google Scholar Gupta M, Akiri C, Aryal K, Parker E, Praharaj L (2023) From ChatGPT to ThreatGPT: impact of generative AI in cybersecurity and privacy. IEEE Access. https://doi.org/10.1109/ACCESS.2023.3300381Article Google Scholar Yang J, Jin H, Tang R, Han X, Feng Q, Jiang H, Zhong S, Yin B, Hu X (2024) Harnessing the power of LLMs in practice: a survey on ChatGPT and beyond. ACM Trans Knowl Discov Data 18(6):1–32Article Google Scholar Jia R, Liang P (2017) Adversarial examples for evaluating reading comprehension systems. In: Palmer M, Hwa R, Riedel S (eds) roceedings of the 2017 conference on empirical methods in natural language processing, Copenhagen, Denmark, September 2017. Association for Computational Linguistics, p 2021–2031Omar M, Choi S, Nyang D, Mohaisen D (2022) Robust natural language processing: recent advances, challenges, and future directions. IEEE Access 10:86038–86056Article Google Scholar Akhtar N, Mian A (2018) Threat of adversarial attacks on deep learning in computer vision: a survey. IEEE Access 6:14410–14430Article Google Scholar Wang W, Chen Z, Chen X, Wu J, Zhu X, Zeng G, Luo P, Lu T, Zhou J, Qiao Y et al (2024) Visionllm: large language model is also an open-ended decoder for vision-centric tasks. Adv Neural Inf Process Syst 36:61501–61513 Google Scholar Hu S, Shang X, Qin Z, Li M, Wang Q, Wang C (2019) Adversarial examples for automatic speech recognition: attacks and countermeasures. IEEE Commun Mag 57(10):120–126Article Google Scholar Wang W, Wang R, Wang L, Wang Z, Ye A (2023) Towards a robust deep neural network against adversarial texts: a survey. IEEE Trans Knowl Data Eng 35(3):3159–3179 Google Scholar Das RK, Tian X, Kinnunen T, Li H (2020) The attacker’s perspective on automatic speaker verification: an overview. In Proceedings of the Interspeech 2020, p 4213–4217Abdullah H, Warren K, Bindschaedler V, Papernot N, Traynor P (2021) SoK: the faults in our ASRs: an overview of attacks against automatic speech recognition and speaker identification systems. In: 2021 IEEE symposium on security and privacy (SP). IEEE, p 730–747Chen X, Li S, Huang H (2021) Adversarial attack and defense on deep neural network-based voice processing systems: an overview. Appl Sci 11(18):8450Article Google Scholar Zhang WE, Sheng QZ, Alhazmi A, Li C (2020) Adversarial attacks on deep-learning models in natural language processing: a survey. ACM Trans Intell Syst Technol TIST 11(3):1–41 Google Scholar Xu H, Ma Y, Liu HC, Deb D, Liu H, Tang JL, Jain AK (2020) Adversarial attacks and defenses in images, graphs and text: a review. Int J Autom Comput 17:151–178Article Google Scholar Wang Y, Sun T, Li S, Yuan X, Ni W, Hossain E, Poor HV (2023) Adversarial attacks and defenses in machine learning-empowered communication systems and networks: a contemporary survey. IEEE Commun Surv Tutor. https://doi.org/10.1109/COMST.2023.3319492Article Google Scholar Yuan L, Chen Y, Cui G, Gao H, Zou F, Cheng X, Ji H, Liu Z, Sun M (2024) Revisiting out-of-distribution robustness in NLP: benchmarks, analysis, and LLMs evaluations. Adv Neural Inf Process Syst 36Liu B, Xiao B, Jiang X, Cen S, He X, Dou W (2023) Adversarial attacks on large language model-based system and mitigating strategies: a case study on ChatGPT. Secur Commun Netw 1:8691095 Google Scholar Alsmadi I, Aljaafari N, Nazzal M, Alhamed S, Sawalmeh AH, Vizcarra CP, Khreishah A, Anan M, Algosaibi A, Al-Naeem MA et al (2022) Adversarial machine learning in text processing: a literature survey. IEEE Access 10:17043–17077Article Google Scholar He X, Wang J, Xu Q, Minervini P, Stenetorp P, Rubinstein BI, Cohn T (2024) Transferring troubles: cross-lingual transferability of backdoor attacks in LLMs with instruction tuning. arXiv preprint arXiv:2404.19597Vassilev Apostol, Oprea Alina, Fordyce Alie, Anderson Hyrum (2024) Adversarial machine learning. Gaithersburg, MarylandBook Google Scholar Jin D, Jin Z, Zhou JT, Szolovits P (2020) Is Bert really robust? A strong baseline for natural language attack on text classification and entailment. In: Proceedings of the AAAI conference on artificial intelligence 34(05):8018–8025Ren S, Deng Y, He K, Che W (2019) Generating natural language adversarial examples through probability weighted word saliency. In: Proceedings of the 57th annual meeting of the association for computational linguistics, p 1085–1097Li J, Ji S, Du T, Li B, Wang T (2019) Textbugger: generating adversarial text against real-world applications. In: 26th annual network and distributed system security symposium, NDSS 2019, San Diego, California, USA, 24–27 Feb, 2019. The Internet SocietyGao J, Lanchantin J, Soffa ML, Qi Y. Black-box generation of adversarial text sequences to evade (2018) Black-box generation of adversarial text sequences to evade deep learning classifiers. In: 2018 IEEE security and privacy workshops (SPW). IEEE, p 50–56Alzantot M, Sharma Y, Elgohary A, Ho BJ, Srivastava M, Chang KW (2018) Generating natural language adversarial examples. In: Riloff E, Chiang D, Hockenmaier J, Tsujii J (eds)Proceedings of the 2018 conference on empirical methods in natural language processing, Brussels, Belgium, October-November 2018. Association for Computational Linguistics, p 2890–2896Mrkšić N, Séaghdha DO, Thomson B, Gašić M, Rojas-Barahona LM, Su PH, Vandyke D, Wen T-H, Young S (2016) Counter-fitting word vectors to linguistic constraints. In: Knight K, Nenkova A, Rambow O (eds) Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, San Diego, California, June 2016. Association for Computational Linguistics, p 142–148Liang B, Li H, Su M, Bian P, Li X, Shi W (2018) Deep text classification can be fooled. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence. International joint conferences on artificial intelligence organizationGlockner M, Shwartz V, Goldberg Y (2018) Breaking NLI systems with sentences that require simple lexical inferences. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), Melbourne, Australia, July. Association for Computational Linguistics, p 650–655Bowman SR, Angeli G, Potts C, Manning CD (2015) A large annotated corpus for learning natural language inference. In: Màrquez L, Callison-Burch C, Su J (eds) Proceedings of the 2015 conference on empirical methods in natural language processing, Lisbon, Portugal, September. Association for Computational Linguistics, p 632–642Lei Qi Wu, Lingfei Chen Pin-Yu, Alex Dimakis, Dhillon Inderjit S, Witbrock Michael J (2019) Discrete adversarial attacks and submodular optimization with applications to text classification. Proc Mach Learn Syst 1:146–165 Google Scholar Li H, Guo D, Fan W, Xu M, Huang J, Meng F, Song (2023) Multi-step jailbreaking privacy attacks on ChatGPT. In HBouamor H, Pino J, Bali K (eds) Findings of the association for computational linguistics: EMNLP 2023, Singapore, . Association for Computational Linguistics, p 4138–4153Carlini N (2023) A LLM assisted exploitation of AI-Guardian. arXiv preprint arXiv:2307.15008Liu Y, Deng G, Li Y, Wang K, Zhang T, Liu Y, Wang H, Zheng Y, Liu Y (2023) Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499Chen Y, Arunasalam A, Celik ZB (2023) Can large language models provide security & privacy advice? Measuring the ability of llms to refute misconceptions. In: Proceedings of the 39th annual computer security applications conference, ACSAC ’23, New York, NY, USA, 2023. Association for Computing Machinery, p 366–378Duan H, Dziedzic A, Yaghini M, Papernot N, Boenisch F (2023) On the privacy risk of in-context learning. In: The 61st Annual meeting of the association for computational linguisticsXue J, Zheng M, Hua T, Shen Y, Liu Y, Bölöni L, Lou Q (2024) Trojllm: a black-box trojan prompt attack on large language models. Adv Neural Inf Process Syst 36:65665–65677 Google Scholar Perez F, Ribeiro I (2022) Ignore previous prompt: attack techniques for language models. arXiv preprint arXiv:2211.09527Liu Y, Yao Y, Ton JF, Zhang X, Cheng RG, Klochkov Y, Taufiq MF, Li H (2023) trustworthy llms: a survey and guideline for evaluating large language models’ alignment. In: Socially Responsible Language Modelling ResearchWei A, Haghtalab N, Steinhardt J (2024) Jailbroken: How does llm safety training fail? Adv Neural Inf Process Syst 36:80079–80110 Google Scholar Ebrahimi J, Rao A, Lowd D, Dou D (2018) HotFlip: white-box adversarial examples for text classification. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), Melbourne, Australia, July. Association for Computational Linguistics, p 31–36Chen M, He G, Wu J (2024) ZDDR: a zero-shot defender for adversarial samples detection and restoration. IEEE Access. https://doi.org/10.1109/ACCESS.2024.3356568Article Google Scholar Wallace E, Feng S, Kandpal N, Gardner M, Singh S (2019) Universal adversarial triggers for attacking and analyzing NLP. In: Inui K, Jiang J, Ng V, Wan X (eds) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), Hong Kong, China, November 2019. Association for Computational Linguistics, p 2153–2162Pruthi D, Dhingra B, Lipton ZC (2019) Combating adversarial misspellings with robust word recognition. In: Korhonen A, Traum D, Màrquez L (eds) Proceedings of the 57th annual meeting of the association for computational linguistics, Florence, Italy, July . Association for Computational Linguistics, p 5582–5591Lim S, Schmälzle R (2023) Artificial intelligence for health message generation: an empirical study using a large language model (LLM) and prompt engineering. Front Commun 8:1129082Article Google Scholar Jiang W, Li H, Xu G, Zhang T (2023) Color backdoor: a robust poisoning attack in color space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p 8133–8142Bao R, Wang J, Zhao H (2021) Defending pre-trained language models from adversarial word substitution without performance sacrifice. In: Zong C, Xia F, Li W, Navigli R (eds) Findings of the association for computational linguistics: ACL-IJCNLP 2021, Online, August 2021. Association for Computational Linguistics, p 3248–3258Wang B, Pei H, Pan B, Chen Q, Wang S, Li B (2020) T3: tree-autoencoder constrained adversarial text generation for targeted attack. In: Webber B, Cohn T, He Y, Liu Y (eds) Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), Online, November 2020. Association for Computational Linguistics, p 6134–6150Lin J, Zou J, Ding N (2021) Using adversarial attacks to reveal the statistical bias in machine reading comprehension models. In: Zong C, Xia F, Li W, Navigli R (eds) Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 2: short papers), Online, August 2021. Association for Computational Linguistics, p 333–342Gan WC, Ng HT (2019) Improving the robustness of question answering systems to question paraphrasing. In: Proceedings of the 57th annual meeting of the association for computational linguistics, p 6065–6075Zhang Y, Baldridge J, He L (2019) PAWS: paraphrase adversaries from word scrambling. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), Minneapolis, Minnesota, June . Association for Computational Linguistics, p 1298–1308Kurita K, Michel P, Neubig G (2020) Weight poisoning attacks on pretrained models. In: Jurafsky D, Chai J, Schluter N, Tetreault J (eds) Proceedings of the 58th annual meeting of the association for computational linguistics, Online, July . Association for Computational Linguistics, p 2793–2806Han X, Zhang Z, Ding N, Gu Y, Liu X, Huo Y, Qiu J, Yao Y, Zhang A, Zhang L et al (2021) Pre-trained models: past, present and future. AI Open 2:225–250Article Google Scholar Fursov I, Zaytsev A, Burnyshev P, Dmitrieva E, Klyuchnikov N, Kravchenko A, Artemova E, Komleva E, Burnaev E (2022) A differentiable language model adversarial attack on text classifiers. IEEE Access 10:17966–17976Article Google Scholar Bajaj A, Vishwakarma DK (2023) Evading text based emotion detection mechanism via adversarial attacks. Neurocomputing 558:126787Article Google Scholar Myers D, Mohawesh R, Chellaboina VI, Sathvik AL, Venkatesh P, Ho YH, Henshaw H, Alhawawreh M, Berdik D, Jararweh Y (2024) Foundation and large language models: fundamentals, challenges, opportunities, and social impacts. Clust Comput 27(1):1–26Article Google Scholar Xu X, Kong K, Liu N, Cui L, Wang D, Zhang J, Kankanhalli M (2024) An LLM can fool itself: a prompt-based adversarial attack. In: The twelfth international conference on learning representationsWang T, Wang X, Qin Y, Packer B, Li K, Chen J, Beutel A, Chi E (2020) CAT-gen: improving robustness in NLP models via controlled adversarial text generation. In: Webber B, Cohn T, He Y, Liu Y (eds) Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), Online, November 2020. Association for Computational Linguistics, p 5141–5146Qi F, Chen Y, Li M, Yao Y, Liu Z, Sun M (2021) ONION: a simple and effective defense against textual backdoor attacks. In: Moens MF, Huang X, Specia L, Yih SWT (eds) Proceedings of the 2021 conference on empirical methods in natural language processing, Online and Punta Cana, Dominican Republic, November (2021). Association for Computational Linguistics, p 9558–9566Zhang Z, Xiao G, Li Y, Lv T, Qi F, Liu Z, Wang Y, Jiang X, Sun M (2023) Red alarm for pre-trained models: universal vulnerability to neuron-level backdoor attacks. Mach Intell Res 20(2):180–193Article Google Scholar Li L, Song D, Li X, Zeng J, Ma R, Qiu X (20121) Backdoor attacks on pre-trained models by layerwise weight poisoning. In: Moens MF, Huang X, Specia L, Yih SWT (eds) Proceedings of the 2021 conference on empirical methods in natural language processing, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics, p 3023–3032Cheng M, Yi J, Chen PY, Zhang H, Hsieh CJ (2020) Seq2sick: evaluating the robustness of sequence-to-sequence models with adversarial examples. In: Proceedings of the AAAI conference on artificial intelligence 34:3601–3608Xie X, Wu J, Liu G, Lin Z (2024) SSCNet: learning-based subspace clustering. Vis Intell 2(1):11Article Google Scholar Dong X, Luu AT, Ji R, Liu H (2021) Towards robustness against natural language word substitutions. In: International conference on learning representationsBlum O, Brattoli B, Ommer B (2019) X-GAN: improving generative adversarial networks with convex combinations. In: Pattern Recognition: 40th German conference, GCPR 2018, Stuttgart, Germany, October 9–12, 2018, proceedings 40. Springer, p 199–214Szeghy D, Milacski ZA, Fóthi A, Lorincz A (2021) Adversarial perturbation stability of the layered group basis pursuit. def 1:2Yuan L, Zeng J, Zheng X (2021) Sparsegan: sparse generative adversarial network for text generation. arXiv preprint arXiv:2103.11578Tsiligkaridis T, Roberts J (2022) Understanding and increasing efficiency of Frank–Wolfe adversarial training. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p 50–59La Malfa E (2023) On robustness for natural language processing. PhD thesis, University of OxfordSteinhardt J, Koh PW, Liang PS (2017) Certified defenses for data poisoning attacks. In: I. Guyon and U. Von Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R.Garnett (eds) Curran Associates, Inc. Adv Neural Inf Process Syst 30. https://proceedings.neurips.c/paper_files/paper/2017/file/9d7311ba459f9e45ed746755a32dcd11-Paper.pdfRaghunathan A, Steinhardt J, Liang P (2018) Certified defenses against adversarial examples. In: 6th International conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, conference track proceedings. OpenReview.netWang W, Tang P, Lou J, Xiong L (2021) Certified robustness to word substitution attack with differential privacy. In: Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, p 1102–1112Sato M, Suzuki J, Shindo H, Matsumoto Y (2018) Interpretable adversarial perturbation in input embedding space for text. In: Proceedings of the 27th international joint conference on artificial intelligence, IJCAI’18. AAAI Press, p 4323–4330Gong Z, Wang W, Li B, Song D, Ku WS (2018) Adversarial texts with gradient methods (01)Jia R, Raghunathan A, Göksel K, Liang P (2019) Certified robustness to adversarial word substitutions. In: Inui K, Jiang J, Ng V, Wan X (eds) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), Hong Kong, China, November 2019. Association for Computational Linguistics, p 4129–4142Huang PS, Stanforth R, Welbl J, Dyer C, Yogatama D, Gowal S, Dvijotham K, Kohli P (2019) Achieving verified robustness to symbol substitutions via interval bound propagation. In: Inui K, Jiang J, Ng V, Wan X (eds) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), Hong Kong, China, November 2019. Association for Computational Linguistics, p 4083–4093Dong X (2022) Adversarial attacks and defenses in natural language processingMadry A, Makelov A, Schmidt L, Tsipras D, Vladu A (2018) Towards deep learning models resistant to adversarial attacks. In: 6th International conference on learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, conference track proceedings. OpenReview.netYoo JY, Morris JX, Lifland E, Qi Y (2020) Searching for a search method: Benchmarking search algorithms for generating NLP adversarial examples. In: Alishahi A, Belinkov Y, Chrupała G, Hupkes D, Pinter Y, Sajjad H (eds) Proceedings of the third BlackboxNLP workshop on analyzing and interpreting neural networks for NLP, Online, November 2020. Association for Computational Linguistics, p 323–332Barham S, Feizi S (2019) Interpretable adversarial training for text. arXiv preprint arXiv:1905.12864Papernot N, McDaniel P, Swami A, Harang R (2016) Crafting adversarial input sequences for recurrent neural networks. In: MILCOM 2016–2016 IEEE military communications conference. IEEE, p 49–54Miyato T, Dai AM, Goodfellow I (2016) Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725Guo C, Sablayrolles A, Jégou H, Kiela D (2021) Gradient-based adversarial attacks against text transformers. In: Moens MF, Huang X, Specia L, Yih SWT (eds) Proceedings of the 2021 conference on empirical methods in natural language processing, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics, p 5747–5757Sadrizadeh S , Dolamic L, Frossard P (2022) Block-sparse adversarial attack to fool transformer-based text classifiers. In: ICASSP 2022–2022 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, p 7837–7841Costa JC, Roxo T, Proença H, Inácio PRM. How deep learning sees the world: a survey on adversarial attacks & defenses. IEEE Access (2024)Birbil Şİ, Fang SC, Sheu RL (2004) On the convergence of a population-based global optimization algorithm. J Glob Optim 30:301–318Article MathSciNet Google Scholar Khormali A, Nyang D, Mohaisen D (2020) Generating adversarial examples with an optimized quality. arXiv preprint arXiv:2007.00146Jia R (2020) Building robust natural language processing systems. Stanford University, Stanford Google Scholar Morris JX, Lifland E, Yoo JY, Grigsby J, Jin D, Qi Y (2020) TextAttack: a framework for adversarial attacks, data augmentation, and adversarial training in NLP. In: Liu Q, Schlangen D (eds) Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, Online, October. Association for Computational Linguistics, p 119–126Zang Y, Qi F, Yang C, Liu Z, Zhang M, Liu Q, Sun M (2020) Word-level textual adversarial attacking as combinatorial optimization. In: Jurafsky D, Chai J, Schluter N, Tetreault J (eds) Proceedings of the 58th annual meeting of the association for computational linguistics, Online, July 2020. Association for Computational Linguistics, p 6066–6080Maheshwary R, Maheshwary S, Pudi V (2021) Generating natural language attacks in a hard label black box setting. In: Proceedings of the AAAI conference on artificial intelligence 35:13525–13533Jasser J, Garibay I (2021) Resilience from diversity: population-based approach to harden models against adversarial attacks. arXiv preprint arXiv:2111.10272Garg S, Ramakrishnan G (eds) BAE: BERT-based adversarial examples for text classification. In: Webber B, Cohn T, He Y, Liu Y (eds) Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), Online, November 2020. Association for Computational Linguistics, p 6174–6181Jin D, Jin Z, Zhou JT, Szolovits P (2020) Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In: Proceedings of the AAAI conference on artificial intelligence 34:8018–8025Maheshwary R, Maheshwary S, Pudi V (2021) A strong baseline for query efficient attacks in a black box setting. In: Moens MF, Huang X, Specia L, Yih SWT (eds) Proceedings of the 2021 conference on empirical methods in natural language processing, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics, p 8396–8409Wang B, Xu C, Liu X, Cheng Y, Li B (2022) SemAttack: natural textual attacks via different semantic spaces. In: Carpuat M, De Marneffe MC, Meza Ruiz IV (eds) Findings of the association for computational linguistics: NAACL 2022, Seattle, United States, July 2022. Association for Computational Linguistics, p 176–205Lee D, Moon S, Lee J, Song HO (2022) Query-efficient and scalable black-box adversarial attacks on discrete sequential data via Bayesian optimization. In: International conference on machine learning. PMLR, p 12478–12497Peng H, Wang Z, Zhao D, Wu Y, Han J, Guo S, Ji S, Zhong M (2023) Efficient text-based evolution algorithm to hard-label adversarial attacks on text. J King Saud Univ Comput Inf Sci 35(5):101539 Google Scholar Liu Y, Huang Y, Cai Z (2023) AED: An black-box NLP classifier model attacker. Neurocomputing 550:126489Article Google Scholar Caucheteux C, Gramfort A, King JR (2021) GPT-2’s activations predict the degree of semantic comprehension in the human brain. BioRxiv, p 2021–2004Goldstein A, Zada Z, Buchnik E, Schain M, Price A, Aubrey B, Nastase SA, Feder A, Emanuel D, Cohen A et al (2022) Shared computational principles for language processing in humans and deep language models. Nat Neurosci 25(3):369–380Article Google Scholar Heilbron M, Armeni K, Schoffelen JM, Hagoort P, De Lange FP (2022) A hierarchy of linguistic predictions during natural language comprehension. In: Proceedings of the national academy of sciences, 119(32):e2201968119Kumar S, Sumers TR, Yamakoshi T, Goldstein A, Hasson U, Norman KA, Griffiths TL, Hawkins RD, Nastase SA (2022) Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model. BioRxiv, p 2022–2006Bastings J, Filippova K (2020) The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In: Alishahi A, Belinkov Y, Chrupała G, Hupkes D, Pinter Y, Sajjad H (eds) Proceedings of the third BlackboxNLP workshop on analyzing and interpreting neural networks for NLP, Online, November 2020. Association for Computational Linguistics, p 149–155Ghojogh B,GhodsiA(2020) Attentionmechanism, transformers, BERT, and GPT: OSF Preprints tutorial and survey 12. https://w.researchgate.net/profile/Benyamin-Ghojogh/publication/347623569_Attention_Mechanism_Transformers_BERT_and_GPT_Tutorial_and_Survey/links/640e5b3a1b72772e4eea211/Attention-Mechanism-Transformers-BERT-and-GPT-Tutorial-and-Survey.pdfSundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: International conference on machine learning. PMLR, p 3319–3328Shrikumar A, Greenside P, Kundaje A (2017) Learning important features through propagating activation differences. In: International conference on machine learning. PMLR, p 3145–3153Chen J, Song L, Wainwright M, Jordan M (2018) Learning to explain: an information-theoretic perspective on model interpretation. In: International conference on machine learning. PMLR, p 883–892Zang Y, Hou B, Qi F, Liu Z, Meng X, Sun M (2020) Learning to attack: towards textual adversarial attacking in real-world situations. arXiv preprint arXiv:2009.09192Roth T, Gao Y, Abuadbba A, Nepal S, Liu W (2024) Token-modification adversarial attacks for natural language processing: a survey. AI Commun (04):1–22Cheng R, Jin Y (2014) A competitive swarm optimizer for large scale optimization. IEEE Trans Cybern 45(2):191–204Article Google Scholar Yang Q, Chen WN, Da Deng J, Li Y, Gu T, Zhang J (2017) A level-based learning swarm optimizer for large-scale optimization. IEEE Trans Evolut Comput 22(4):578–594Article Google Scholar Li X, Yao X (2011) Cooperatively coevolving particle swarms for large scale optimization. IEEE Trans Evolut Comput 16(2):210–224MathSciNet Google Scholar Mounsif M, Zehnder K, Motie Y, Adam-Gaxotte Z (2023) SwarMind: harnessing large language models for flock dynamics. In: 2023 10th international conference on soft computing & machine intelligence (ISCMI). IEEE, p 171–177 (2023)Wang Q, Guo P, Sun S, Xie L, Hansen JH (2019) Adversarial regularization for end-to-end robust speaker verification. In: Interspeech, p 4010–4014Abdelali A, Mubarak H, Chowdhury S, Hasanain M, Mousi B, Boughorbel S, Abdaljalil S, Kheir YE, Izham D, Dalvi F, Hawasly M, Nazar N, Elshahawy Y, Ali A, Durrani N, Milic-Frayling N, Alam F (2024) LAraBench: benchmarking Arabic AI with large language models. In: Graham Y, Purver M (eds) Proceedings of the 18th conference of the European chapter of the association for computational linguistics (volume 1: long papers), St. Julian’s, Malta, March 2024. Association for Computational Linguistics, p 487–520Bang Y, Cahyawijaya S, Lee N, Dai W, Su D, Wilie B, Lovenia H, Ji Z, Yu T, Chung W, Do QV (2023) A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In: Park JC, Arase Y, Hu B, Lu W, Wijaya D, Purwarianti A, Krisnadhi A (eds) Proceedings of the 13th international joint conference on natural language processing and the 3rd conference of the Asia–Pacific chapter of the association for computational linguistics (volume 1: long papers), Nusa Dua, Bali, November 2023. Association for Computational Linguistics, p 675–718Chen Y, Wang R, Jiang H, Shi S, Xu R (2023) Exploring the use of large language models for reference-free text quality evaluation: an empirical study (01):361–374Choi M, Pei J, Kumar S, Shu C, Jurgens D (2023) Do LLMs understand social knowledge? Evaluating the sociability of large language models with SocKET benchmark. In: Bouamor H, Pino J, Bali LK (eds) Proceedings of the 2023 conference on empirical methods in natural language processing, Singapore, December 2023. Association for Computational Linguistics, p 11370–11403Chia YK, Hong P, Bing L, Poria S (2024) InstructEval: towards holistic evaluation of instruction-tuned large language models. In: Miceli-Barone AV, Barez F, Cohen S, Voita E, Germann U, Lukasik M (eds) Proceedings of the first edition of the workshop on the scaling behavior of large language models (SCALE-LLM 2024), St. Julian’s, Malta, March 2024. Association for Computational Linguistics, p 35–64Fu Y, Ou L, Chen M, Wan Y, Peng H, Khot T (2023) Chain-of-thought hub: a continuous effort to measure large language models’ reasoning performance. arXiv preprint arXiv:2305.17306Gekhman Z, Herzig J, Aharoni R, Elkind C, Szpektor I (2023) Trueteacher: learning factual consistency evaluation with large language models. In: The 2023 conference on empirical methods in natural language processingHonovich O, Aharoni R, Herzig J, Taitelbaum H, Kukliansy D, Cohen V, Scialom T, Szpektor I, Hassidim A, Matias Y (2022) TRUE: re-evaluating factual consistency evaluation. In: Feng S, Wan H, Yuan C, Yu H (eds) Proceedings of the second DialDoc workshop on document-grounded dialogue and conversational question answering, Dublin, Ireland, May 2022. Association for Computational Linguistics, p 161–175Lai VD, Ngo NT, Veyseh AP, Man H, Dernoncourt F, Bui T, Nguyen TH (2023)ChatGPT beyond English: towards a comprehensive evaluation of large language models in multilingual learning. In: Bouamor H, Pino J, Bali K (eds) Findings of the association for computational linguistics: EMNLP 2023, Singapore, December 2023. Association for Computational Linguistics, p 13171–13189Lopez-Lira A, Tang Y (2023) Can chatgpt forecast stock price movements? return predictability and large language models. In: Return predictability and large language models (April 6, 2023)Durmus E, Nyugen K, Liao TI, Schiefer N, Askell A, Bakhtin A, Chen C, Hatfield-Dodds Z, Hernandez D, Joseph N et al (2023) Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388Lin YT, Chen YN (2023) LLM-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. In: Chen YN, Rastogi A (eds) Proceedings of the 5th workshop on NLP for conversational AI (NLP4ConvAI 2023), Toronto, Canada, July 2023. Association for Computational Linguistics, p 47–58Liu H, Ning R, Teng Z, Liu J, Zhou Q, Zhang Y (2023) Evaluating the logical reasoning ability of ChatGPT and GPT-4. arXiv preprint arXiv:2304.03439Zhang Y, Xiang T, Hospedales TM, Lu H (2018) Deep mutual learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, p 4320–4328Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901 Google Scholar He P, Liu X, Gao J, Chen W (2020) Deberta: decoding-enhanced BERT with disentangled attention. In: International conference on learning representationsYang Y, Lv H, Chen N (2022) A survey on ensemble learning under the era of deep learning. Artif Intell Rev 56(6):5545–5589Article Google Scholar Dong X, Yu Z, Cao W, Shi Y, Ma Q (2020) A survey on ensemble learning. Front Comput Sci 14:241–258Article Google Scholar Zhu X, Gong (2018) Knowledge distillation by on-the-fly native ensemble. Adv Neural Inf Process Syst 31Chen D, Mei JP, Wang C, Feng Y, Chen C (2020) Online knowledge distillation with diverse peers. In: Proceedings of the AAAI conference on artificial intelligence 34:3430–3437Li Z, Huang Y, Chen D, Luo T, Cai N, Pan Z (2020) Online knowledge distillation via multi-branch diversity enhancement. In: Proceedings of the Asian conference on computer visionLiu X, Wang Y, Ji J, Cheng H, Zhu X, Awa E, He P, Chen W, Poon H, Cao G et al (2020) The microsoft toolkit of multi-task deep neural networks for natural language understanding. In: Proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations, p 118–126Luong MT, Le QV, Sutskever I, Vinyals O, Kaiser L (2016) Multi-task sequence to sequence learning. In: Bengio Y, LeCun Y (eds) 4th international conference on learning representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, conference track proceedingsRuder S, Bingel J, Augenstein I, Søgaard A (2019) Latent multi-task architecture learning. In: Proceedings of the AAAI conference on artificial intelligence 33:4822–4829Ramé A, Cord M (2021) Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation. In: ICLR 2021-9th international conference on learning representationsFeng S, Chen H, Ren X, Ding Z, Li K, Sun X (2021) Collaborative group learningWu G, Gong S (2021) Peer collaborative learning for online knowledge distillation. In: Proceedings of the AAAI conference on artificial intelligence 35:10302–10310Gehman S, Gururangan S, Sap M, Choi Y, Smith NA (2020) RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In: Cohn T, He Y, Liu Y (eds) Findings of the association for computational linguistics: EMNLP 2020, Online, November 2020. Association for Computational Linguistics, p 3356–3369Maus N, Chao P, Wong E, Gardner J (2023) Adversarial prompting for black box foundation models. arXiv preprint arXiv:2302.04237Shin T, Razeghi Y, Logan IV RL, Wallace E, Singh S (2020) Autoprompt: eliciting knowledge from language models with automatically generated prompts. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 4222–4235Zou A, Wang Z, Kolter JZ, Fredrikson M (2023) Universal and transferable adversarial attacks on aligned language modelsGoyal S, Doddapaneni S, Khapra M, Ravindran B (2023) A survey of adversarial defenses and robustness in NLP. ACM Comput Surv 55(14s):1–39Article Google Scholar Liu X, Cheng H, He P, Chen W, Wang Y, Poon H, Gao J (2020) Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994Jain N, Schwarzschild A, Wen Y, Somepalli G, Kirchenbauer J, Chiang PY, Goldblum M, Saha A, Geiping J, Goldstein T (2024) Baseline defenses for adversarial attacks against aligned language modelsKumar A, Agarwal C, Srinivas S, Feizi S, Lakkaraju H (2023) Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: I. Guyon and U. Von Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R. Garnett Curran Associates, Inc. Adv Neural Inf Process Syst 30. https://proceedings.neurips.c/paper_files/paper/2017/file/3f5e243547dee91fbd053c1c4a845a-Paper.pdfRadford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9 Google Scholar Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, Grave E, Ott M, Zettlemoyer L, Stoyanov V (2020) Unsupervised cross-lingual representation learning at scale. In: Jurafsky D, Chai J, Schluter N, Tetreault J (eds) Proceedings of the 58th annual meeting of the association for computational linguistics, Online, July 2020. Association for Computational Linguistics, p 8440–8451Carlini N, Liu C, Erlingsson Ú, Kos J, Song D (2019) The secret sharer: evaluating and testing unintended memorization in neural networks. In: 28th USENIX security symposium (USENIX security 19), p 267–284Kawaguchi K, Kaelbling LP, Bengio Y (2022) Generalization in deep learning. In: Grohs P, Kutyniok G (eds) Mathematical aspects of deep learning. Cambridge University Press, Cambridge Google Scholar Brown G, Bun M, Feldman V, Smith A, Talwar K (2021) When is memorization of irrelevant training data necessary for high-accuracy learning? In: Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing, p 123–132Dwork C, McSherry F, Nissim K, Smith A (2006) Calibrating noise to sensitivity in private data analysis. In: Theory of cryptography: third theory of cryptography conference, TCC 2006, New York, NY, USA, March 4–7, 2006. proceedings 3. Springer, p 265–284Song S, Chaudhuri K, Sarwate AD (2013) Stochastic gradient descent with differentially private updates. In: 2013 IEEE global conference on signal and information processing. IEEE, p 245–248Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, Zhang L (2016) Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, p 308–318Li X, Tramer F, Liang P, Hashimoto T (2021) Large language models can be strong differentially private learners. In: International conference on learning representationsMajmudar J, Dupuy C, Peris C, Smaili S, Gupta R, Zemel R (2022) Differentially private decoding in large language modelsDupuy C, Arava R, Gupta R, Rumshisky A (2022) An efficient dp-sgd mechanism for large scale NLU models. In: ICASSP 2022–2022 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, p 4118–4122Dagan Y, Feldman V (2020) Pac learning with stable and private predictions. In: Conference on learning theory. PMLR, p 1389–1410Buckman J, Roy A, Raffel C, Goodfellow I (2018) Thermometer encoding: one hot way to resist adversarial examples. In: International conference on learning representationsGuo C, Rana M, Cisse M, Van Der Maaten L (2018) Countering adversarial images using input transformations. In: International conference on learning representationsDhillon GS, Azizzadenesheli K, Lipton ZC, Bernstein J, Kossaifi J, Khanna A, Anandkumar A (2018) Stochastic activation pruning for robust adversarial defense. In: International conference on learning representationsGrosse K, Manoharan P, Papernot N, Backes M, McDaniel P (2017) On the (statistical) detection of adversarial examples. CoRRGong Z, Wang W (2023) Adversarial and clean data are not twins. In: Proceedings of the sixth international workshop on exploiting artificial intelligence techniques for data management, p 1–5Minh DN, Luu AT (2022) Textual manifold-based defense against natural language adversarial examples. In: Proceedings of the 2022 conference on empirical methods in natural language processing, p 6612–6625Yoo K, Kim J, Jang J, Kwak N (2022) Detection of adversarial examples in text classification: benchmark and baseline via robust density estimation. In: Muresan S, Nakov P, Villavicencio A (eds) Findings of the association for computational linguistics: ACL 2022, Dublin, Ireland, May 2022. Association for Computational Linguistics, p 3656–3672Huber L, Kühn MA, Mosca E, Groh G (2022) Detecting word-level adversarial text attacks via shapley additive explanations. In: Proceedings of the 7th workshop on representation learning for NLP, p 156–166Carlini N, Wagner D (2017) Adversarial examples are not easily detected: bypassing ten detection methods. In: Proceedings of the 10th ACM workshop on artificial intelligence and security, p 3–14Athalye A, Carlini N, Wagner D (2018) Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. In: International conference on machine learning. PMLR, p 274–283Uesato J, O’donoghue B, Kohli P, Oord A (2018) Adversarial risk and the dangers of evaluating against weak attacks. In: International conference on machine learning. PMLR, p 5025–5034Laidlaw C, Feizi S (2019) Functional adversarial attacks. In: H. Wallach and H. Larochelle and A. Beygelzimer and F. d’Alché-Buc and E. Fox and R. Garnett Curran Associates, Inc. Adv Neural Inf Process Syst 32. https://proceedings.neurips.c/paper_files/paper/2019/file/6e923226e43cd6fac7cfe1e13ad000ac-Paper.pdfGowal S, Dvijotham K, Stanforth R, Bunel R, Qin C, Uesato J, Arandjelovic R, Mann T, Kohli P (2018) On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715Dvijotham K, Gowal S, Stanforth R, Arandjelovic R, O’Donoghue B, Uesato J, Kohli P (2018) Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265Mirman M, Gehr T, Vechev M (2018) Differentiable abstract interpretation for provably robust neural networks. In: International conference on machine learning. PMLR, p 3578–3586Wong E, Kolter Z (2018) Provable defenses against adversarial examples via the convex outer adversarial polytope. In: International conference on machine learning. PMLR, p 5286–5295Raghunathan A, Steinhardt J, Liang PS (02018) Semidefinite relaxations for certifying robustness to adversarial examples. In: S. Bengio and H. Wallach and H. Larochelle and K. Grauman and N. Cesa-Bianchi and R. Garnett Curran Associates, Inc. Adv Neural Inf Process Syst 31. https://proceedings.neurips.c/paper_files/paper/2018/file/29c0605a3bab4229e46723f89cf59d83-Paper.pdfSingla S, Feizi S (2020) Second-order provable defenses against adversarial attacks. In: International conference on machine learning. PMLR, p 8981–8991Cohen J, Rosenfeld E, Kolter Z (2019) Certified adversarial robustness via randomized smoothing. In: International conference on machine learning. PMLR, p 1310–1320Lecuyer M, Atlidakis V, Geambasu R, Hsu D, Jana S (2019) Certified robustness to adversarial examples with differential privacy. In: 2019 IEEE symposium on security and privacy (SP). IEEE, p 656–672Li X, Li F (2017) Adversarial examples detection in deep networks with convolutional filter statistics. In: Proceedings of the IEEE international conference on computer vision, p 5764–5772Salman H, Li J, Razenshteyn I, Zhang P, Zhang H, Bubeck S, Yang G (2019) Provably robust deep learning via adversarially trained smoothed classifiers. In: H. Wallach and H. Larochelle and A.Beygelzimer and F. d’Alché-Buc and E. Fox and R. Garnett Curran Associates, Inc. Adv Neural Inf Process Syst 32. https://proceedings.neurips.c/paper_files/paper/2019/file/3a24b25a7b092a252166a1641ae953e7-Paper.pdfYe M, Gong C, Liu Q (2020) SAFER: a structure-free approach for certified robustness to adversarial word substitutions. In: Jurafsky D, Chai J, Schluter N, Tetreault J (eds) Proceedings of the 58th annual meeting of the association for computational linguistics, Online, July. Association for Computational Linguistics,p 3465–3475Zhao H, Ma C, Dong X, Luu AT, Deng ZH, Zhang H (2022) Certified robustness against natural language attacks by causal intervention. In: International conference on machine learning. PMLR, p 26958–26970Zhang Z, Zhang G, Hou B, Fan W, Li Q, Liu S, Zhang Y, Chang S (2023) Certified robustness for large language models with self-denoising. arXiv preprint arXiv:2307.07171Bakhtin A, Gross S, Ott M, Deng Y, Ranzato MA, Szlam A (2019) Real or fake? Learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351Uchendu A, Le T, Shu K, Lee D (2020) Authorship attribution for neural text generation. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), p 8384–8395Antoun W, Mouilleron V, Sagot B, Seddah D (2023) Towards a robust detection of language model-generated text: is ChatGPT that easy to detect? In: Servan C, Vilnat A (eds) Actes de CORIA-TALN 2023. Actes de la 30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN), volume 1 : travaux de recherche originaux—articles longs, Paris, France, 6 2023. ATALA, p 14–27Li Y, Li Q, Cui L, Bi W, Wang L, Yang L, Shi S, Zhang Y (2023) Deepfake text detection in the wild. arXiv preprint arXiv:2305.13242Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2020) Roberta: a robustly optimized BERT pretraining approachFagni T, Falchi F, Gambini M, Martella A, Tesconi M (2021) TweepFake: about detecting deepfake tweets. PLoS ONE 16(5):e0251415Article Google Scholar Wu J, Yang S, Zhan R, Yuan Y, Wong DF, Chao LS (2023) A survey on LLM-gernerated text detection: necessity, methods, and future directions. arXiv preprint arXiv:2310.14724Zuccon G, Koopman B, Shaik R (2023) ChatGPT hallucinates when attributing answers. In: Proceedings of the annual international ACM SIGIR conference on research and development in information retrieval in the Asia Pacific region, SIGIR-AP ’23, New York, NY, USA, 2023. Association for Computing Machinery, page 46–51Liu Y, Zhang Z, Zhang W, Yue S, Zhao X, Cheng X, Zhang Y, Hu H (2023) Argugpt: evaluating, understanding and identifying argumentative essays generated by GPT models. arXiv preprint arXiv:2304.07666Liu Z, Yao Z, Li F, Luo B (2023) Check me if you can: detecting ChatGPT-generated academic writing using checkgpt. arXiv preprint arXiv:2306.05524Chen Y, Kang H, Zhai V, Li L, Singh R, Raj B (2023) GPT-sentinel: distinguishing human and chatgpt generated content. arXiv preprint arXiv:2305.07969Yan Y, Li R, Wang S, Zhang F, Wu W, Xu W (2021) ConSERT: a contrastive framework for self-supervised sentence representation transfer. In: Zong C, Xia F, Li W, Navigli R (eds) Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), Online, August (2021). Association for Computational Linguistics, p 5065–5075Gao T, Yao X, Chen D (2021) SimCSE: simple contrastive learning of sentence embeddings. In: Moens MF, Huang X, Specia L, Yih SWT (eds) Proceedings of the 2021 conference on empirical methods in natural language processing, Online and Punta Cana, Dominican Republic, November (2021). Association for Computational Linguistics, p 6894–6910Chen Q, Zhang R, Zheng Y, Mao Y (2022) Dual contrastive learning: text classification via label-aware data augmentation. arXiv preprint arXiv:2201.08702Liu X, Zhang Z, Wang Y, Pu H, Lan Y, Shen C (2023) Coco: coherence-enhanced machine-generated text detection under low resource with contrastive learning (01):16167–16188Zhong W, Tang D, Xu Z, Wang R, Duan N, Zhou M, Wang J, Yin J (2020) Neural deepfake detection with factual structure of text. In: Webber B, Cohn T, He Y, Liu Y (eds) Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), Online, November 2020. Association for Computational Linguistics, p 2461–2470Bhattacharjee A, Kumarage T, Moraffah R, Liu H. (2023) ConDA: contrastive domain adaptation for AI-generated text detection. In: Park JC, Arase Y, Hu B, Lu W, Wijaya D, Purwarianti A, Krisnadhi A (eds) Proceedings of the 13th international joint conference on natural language processing and the 3rd conference of the Asia–Pacific chapter of the association for computational linguistics (volume 1: long Papers), Nusa Dua, Bali, November 2023. Association for Computational Linguistics, p 598–610Ullah S, Han M, Pujar S, Pearce H, Coskun A, Stringhini G (2024) LLMS cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In: IEEE symposium on security and privacyRoshan K, Zafar A (2024) Black-box adversarial transferability: an empirical study in cybersecurity perspective. Comput Secur 141:103853Article Google Scholar Zhao Y, Pang T, Du C, Yang X, Li C, Cheung NM, Lin M (2014) On evaluating adversarial robustness of large vision-language models. Adv Neural Inf Process Syst 36:54111–54138Akhtar N, Mian A, Kardan N, Shah M (2021) Advances in adversarial attacks and defenses in computer vision: a survey. IEEE Access 9:155161–155196Article Google Scholar Demontis A, Melis M, Pintor M, Jagielski M, Biggio B, Oprea A, Nita-Rotaru C, Roli F (2019) Why do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks. In: 28th USENIX security symposium (USENIX security 19), p 321–338Le T, Wang S, Lee D (2020) Malcom: generating malicious comments to attack neural fake news detection models. In: 2020 IEEE international conference on data mining (ICDM). IEEE, p 282–291Zhang D, Zhang T, Lu Y, Zhu Z, Dong (2019) You only propagate once: accelerating adversarial training viamaximal principle. In: H. Wallach and H. Larochelle and A. Beygelzimer and F. d’Alché-Buc and E. Fox and R. Garnett Curran Associates, Inc. Adv Neural Inf Process Syst 32. https://proceedings.neurips.c/paper_files/paper/2019/file/812b4ba287f5e0bc9d43bbf5bbe87fb-Paper.pdfShafahi A, Najibi M, Ghiasi MA, Xu Z, Dickerson J, Studer C, Davis LS, Taylor G, Goldstein T (2019) Adversarial training for free! In: H. Wallach and H. Larochelle and A. Beygelzimer and F. d’Alché-Buc and E. Fox and R. Garnett Curran Associates, Inc. Adv Neural Inf Process Syst 32Peris C, Dupuy C, Majmudar J, Parikh R, Smaili S, Zemel R, Gupta R (2023) Privacy in the time of language models. In: Proceedings of the sixteenth ACM international conference on web search and data mining, p 1291–1292Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A (2018) Towards deep learning models resistant to adversarial attacks. In: International conference on learning representationsWong E, Rice L, Kolter JZ (2020) Fast is better than free: revisiting adversarial training. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.netYang H, Liang L, Carlone L, Toh KC (2023) An inexact projected gradient method with rounding and lifting by nonlinear programming for solving rank-one semidefinite relaxation of polynomial optimization. Math Progr 201(1):409–472Article MathSciNet Google Scholar Narang S, Diamos G, Elsen E, Micikevicius P, Alben J, Garcia D, Ginsburg B, Houston M, Kuchaiev O, Venkatesh G, Wu H (2018) Mixed precision training. In: International conference on learning representationsSmith LN (2017) Cyclical learning rates for training neural networks. In: 2017 IEEE winter conference on applications of computer vision (WACV). IEEE, p 464–472Coleman C, Narayanan D, Kang D, Zhao T, Zhang J, Nardi L, Bailis P, Olukotun K, Ré C, Zaharia M (2017) Dawnbench: an end-to-end deep learning benchmark and competition. Training 100(101):102 Google Scholar Chen Y, Wang Q, Wu S, Gao Y, Xu T, Hu Y (2024) TOMGPT: reliable text-only training approach for cost-effective multi-modal large language model. ACM Trans Knowl Discov Data. https://doi.org/10.1145/3654674Article Google Scholar Keraghel I, Morbieu S, Nadif M (2024) Beyond words: a comparative analysis of LLM embeddings for effective clustering. In: International symposium on intelligent data analysis. Springer, p 205–216Mewada A, Dewang RK (2023) SA-ASBA: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language BERT model with extreme gradient boosting. J Supercomput 79(5):5516–5551Article Google Scholar Wang Y, Pan Y, Yan M, Su Z, Luan TH (2023) A survey on ChatGPT: AI-generated contents, challenges, and solutions. IEEE Open J Comput Soc. https://doi.org/10.1109/OJCS.2023.3300321Article Google Scholar Ribeiro MT, Singh S, Guestrin C (2018) Semantically equivalent adversarial rules for debugging NLP models. In: Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), p 856–865Liang B, Li H, Su M, Bian P, Li X, Shi W (2018) Deep text classification can be fooled. In: Proceedings of the 27th international joint conference on artificial intelligence, IJCAI’18. AAAI Press, p 4208–4215Qi X, Huang K, Panda A, Henderson P, Wang M, Mittal P (2024) Visual adversarial examples jailbreak aligned large language models. In: Proceedings of the AAAI conference on artificial intelligence, vol 38, p 21527–21536Zhang Y, Ye L, Tian Z, Chen Z, Zhang H, Li B, Fang B (2024) UCTT: universal and low-cost adversarial example generation for tendency classification. Neural Comput Appl. https://doi.org/10.1007/s00521-024-09760-5Mnassri K, Farahbakhsh R, Crespi N (2024) Multilingual hate speech detection: a semi-supervised generative adversarial approach. Entropy 26(4):344Article Google Scholar Wu X, Zhao H, Zhu Y, Shi Y, Yang F, Liu T, Zhai X, Yao W, Li J, Du M et al (2024) Usable XAI: 10 strategies towards exploiting explainability in the LLM era. arXiv preprint arXiv:2403.08946Download referencesAcknowledgementsNot applicableAuthor informationAuthors and AffiliationsDepartment of Intelligent Systems, School of Computer Science and Engineering, Lovely Professional University, Phagwara, Punjab, 144411, IndiaPranjal KumarAuthorsPranjal KumarView author publicationsSearch author on:PubMed Google ScholarContributionsPranjal Kumar conceived the idea and drafted the manuscript.Corresponding authorCorrespondence to Pranjal Kumar.Ethics declarations Conflict of interest On behalf of all authors, the corresponding author states that there is no Conflict of interest. Consent for publication Yes. Additional informationPublisher's NoteSpringer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Rights and permissionsSpringer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.Reprints and permissionsAbout this articleCite this articleKumar, P. Adversarial attacks and defenses for large language models (LLMs): methods, frameworks & challenges. Int J Multimed Info Retr 13, 26 (2024). https://doi.org/10.1007/s13735-024-00334-8Download citationReceived: 22 December 2023Revised: 13 May 2024Accepted: 22 May 2024Published: 25 June 2024Version of record: 25 June 2024DOI: https://doi.org/10.1007/s13735-024-00334-8Share this articleAnyone you share the following link with will be able to read this content:Get shareable linkSorry, a shareable link is not currently available for this article.Copy shareable link to clipboard Provided by the Springer Nature SharedIt content-sharing initiative KeywordsAdversarial attacksArtificial intelligenceNatural language processingMachine learningNeural networksLarge language modelsChatGPTGPT Access this article Log in via an institution Subscribe and save Springer+ from $39.99 /Month Starting from 10 chapters or articles per month Access and download chapters and articles from more than 300k books and 2,500 journals Cancel anytime View plans Buy Now Buy article PDF USD 39.95 Price excludes VAT (USA) Tax calculation will be finalised during checkout. Instant access to the full article PDF. Institutional subscriptions Advertisement Adversarial Attacks on Large Language Models Chapter © 2024 Adversarial Attacks on Large Language Models: A Survey Chapter © 2026 Exploring Advanced Methodologies in Security Evaluation for Large Language Models Chapter © 2024