Paper deep dive

The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative

Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Yu Kong, Tianlong Chen, Huan Liu

Year: 2024Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 31

Models: LLaVA, PandaGPT

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 7:36:26 PM

Summary

The paper introduces a novel security vulnerability in Multimodal Large Language Model (MLLM) societies, where a single 'wolf' agent is manipulated via adversarial noise in its input to generate prompts that induce other 'sheep' agents to produce malicious content. This indirect, covert attack mechanism demonstrates high transferability and effectiveness across various prohibited scenarios, highlighting systemic risks in collaborative AI networks.

Entities (6)

LLaVa · model · 100%MLLM · technology · 100%PandaGPT · model · 100%Projected Gradient Descent · algorithm · 95%Sheep Agent · agent-role · 95%Wolf Agent · agent-role · 95%

Relation Signals (3)

Wolf Agent → influences → Sheep Agent

confidence 95% · a single 'wolf' MLLM agent can be subtly influenced to generate prompts that, in turn, induce other MLLM agents

Projected Gradient Descent → optimizes → Adversarial Noise

confidence 90% · The perturbation to the input image is optimized iteratively... using... Projected Gradient Descent

LLaVa → vulnerableto → Malice Injection

confidence 90% · We experiment with two open-source multi-modal LLMs, LLaVA... ASR near 100% in scenarios involving image-based attacks

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Due to their unprecedented ability to process and respond to various types of data, Multimodal Large Language Models (MLLMs) are constantly defining the new boundary of Artificial General Intelligence (AGI). As these advanced generative models increasingly form collaborative networks for complex tasks, the integrity and security of these systems are crucial. Our paper, ``The Wolf Within'', explores a novel vulnerability in MLLM societies - the indirect propagation of malicious content. Unlike direct harmful output generation for MLLMs, our research demonstrates how a single MLLM agent can be subtly influenced to generate prompts that, in turn, induce other MLLM agents in the society to output malicious content. Our findings reveal that, an MLLM agent, when manipulated to produce specific prompts or instructions, can effectively ``infect'' other agents within a society of MLLMs. This infection leads to the generation and circulation of harmful outputs, such as dangerous instructions or misinformation, across the society. We also show the transferability of these indirectly generated prompts, highlighting their possibility in propagating malice through inter-agent communication. This research provides a critical insight into a new dimension of threat posed by MLLMs, where a single agent can act as a catalyst for widespread malevolent influence. Our work underscores the urgent need for developing robust mechanisms to detect and mitigate such covert manipulations within MLLM societies, ensuring their safe and ethical utilization in societal applications.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

30,708 characters extracted from source content.

Expand or collapse full text

The Wolf Within: Covert Injection of Malice into MLLM Societies via An MLLM Operative Zhen Tan, Chengshuai Zhao11footnotemark: 1 & Raha Moraffah Arizona State University ztan36,czhao93,rmoraffa@asu.edu Equal contribution. The order of authors is random. Yifan Li & Yu Kong Michigan State University liyifa11,yukong@msu.edu Tianlong Chen University of North Carolina at Chapel Hill, MIT, Harvard University tianlong@mit.edu Huan Liu Arizona State University huanliu@asu.edu Abstract Due to their unprecedented ability to process and respond to various types of data, Multimodal Large Language Models (MLLMs) are constantly defining the new boundary of Artificial General Intelligence (AGI). As these advanced generative models increasingly form collaborative networks for complex tasks, the integrity and security of these systems are crucial. Our paper, “The Wolf Within”, explores a novel vulnerability in MLLM societies - the indirect propagation of malicious content. Unlike direct harmful output generation for MLLMs, our research demonstrates how a single MLLM agent can be subtly influenced to generate prompts that, in turn, induce other MLLM agents in the society to output malicious content. Our findings reveal that, an MLLM agent, when manipulated to produce specific prompts or instructions, can effectively “infect” other agents within a society of MLLMs. This infection leads to the generation and circulation of harmful outputs, such as dangerous instructions or misinformation, across the society. We also show the transferability of these indirectly generated prompts, highlighting their possibility in propagating malice through inter-agent communication. This research provides a critical insight into a new dimension of threat posed by MLLMs, where a single agent can act as a catalyst for widespread malevolent influence. Our work underscores the urgent need for developing robust mechanisms to detect and mitigate such covert manipulations within MLLM societies, ensuring their safe and ethical utilization in societal applications. Figure 1: An illustration of the proposed malice injection, where a “wolf” agent subtly influenced to generate prompts that, in turn, induce and infect other “sheep” agents in the society to output malicious content. 1 Introduction Warning: This paper contains harmful language usage. The advent of Multimodal Large Language Models (MLLMs) has marked a significant milestone in the evolution of Artificial General Intelligence (AGI) [13, 24, 4]. By integrating diverse data modalities such as text, images, and sound, MLLMs have demonstrated an unparalleled proficiency in understanding and generating human-like responses, heralding an era where intricate networks of these models collaborate to address multifaceted tasks [30, 5]. Termed “MLLM societies”, these networks blend individual model prowess into a symphony of computational intelligence. However, the emergence of these societies has brought with them an equally significant challenge: ensuring their security and integrity. Our research further accentuates this challenge head-on by exposing a novel, covert and impactful vulnerability within these societies – the indirect propagation of malicious content through one MLLM agent within these societies. Contrary to the prevailing focus [17, 19] on direct manipulations of MLLMs that result in harmful outputs, our study takes a divergent path. As depicted in Figure 1, we reveal how a single “wolf” MLLM agent can be subtly influenced to generate prompts, which in turn compel other “sheep” agents in the society to produce malicious content, such as dangerous instructions or misinformation. This subtle yet potent method of indirect influence signals a major escalation in the security risks associated with MLLMs. It moves the threat from individual model tampering to a more systemic risk, potentially affecting entire networks. Our findings are significant. They show that this new form of threat is achieved through indirect nature of manipulation on the image input, making it covert and challenging to detect. The implications of this discovery are profound. It reveals an unexplored dimension of threat in MLLM societies, where a single agent can catalyze extensive malevolent influence. This calls for an urgent reassessment of the current security frameworks governing MLLMs. Our study emphasizes the need for advanced detection and mitigation strategies to safeguard against such covert manipulations. In doing so, it seeks to ensure that MLLMs continue to serve societal applications safely and ethically, free from vulnerabilities that undermine their potential for positive impact. 2 Related Work LLM Agent Societies. The emergence of LLM agent societies, where multiple models collaborate within a network, presents new frontiers and challenges in AI [20, 12, 14]. Such societies leverage collective intelligence to tackle complex problems, yet their interconnected nature introduces significant security vulnerabilities. Our research focuses on how malicious prompts, covertly injected by a single agent, can propagate through these networks, exploiting the collaborative framework to amplify their impact. This aspect underscores the need for robust security measures to protect the integrity and safety of collaborative AI systems. Security Concerns in LLMs and MLLMs. Early research in the field has primarily focused on identifying potential attack vectors in LLMs, such as adversarial attacks [6, 21] and data poisoning [23, 26]. These studies laid the groundwork for understanding how malicious inputs could be designed to exploit model vulnerabilities More recently, there has been a surge of works that have extended these concepts to MLLMs, examining how the integration of multiple data modalities could introduce new security challenges [1, 3, 22, 28, 19, 16, 7]. Compared to those existing works directly attacking a target LLM, our paper is the first to guide an LLM to generate prompt to attack another one in a society. Jailbreak and Prompt Injection Attacks. A particularly relevant area of research involves the investigation of “jailbreaking” techniques, where LLMs are manipulated to bypass their safety protocols through cleverly crafted prompts [2, 29]. This line of work has demonstrated the feasibility of inducing LLMs to generate outputs that violate their intended ethical or operational guidelines. Generally, current existing work for (multimodal) large language model jailbreak could be categorized into the following two categories: (1) Prompt engineering methods - this type focuses on manually crafted prompts to elicit the jailbroken contents [17, 27]. (2) Learning-based methods - this type tends to automatically improve the jailbreak prompt through optimizing a customized objective [8, 15, 31]. Our research extends the learning-based jailbreak by guiding MLLMs (i.e., wolf agents) to generate indirect multimodal adversarial prompts that can jailbreak victim MLLMs (i.e., sheep agents), which is a topic that has not yet been explored. 3 Methodology Figure 2: The illustration of the proposed attack mechanism. The adversarial noise was injected into the image input of the wolf agent θ, and it then generated malicious tokens and forwarded the perturbed image to the sheep agent ϕitalic-ϕφϕ. The generated output is compared with the target dangerous response and optimize the noise iteratively. Attack Setting. In our attack senarios, the premise is that attackers have whitebox access to the MLLMs, such as gradients from the models. This is a plausible situation as top-tier models such as LLaVA [13], PandaGPT [24], Shikra [4] are openly available, and there is potential of weight leakage from closed-source LLMs due to security incidents [25]. Attacking MLLMs. MLLMs enable efficient encoding of image and audio data into the same embedding space as text [13, 24, 4] and generate textual response. A typical MLLM θ can be represented as a collection of parameters from three parts: θ=θd⁢e⁢c||θe⁢m⁢b||θe⁢n⁢cconditional-setsubscriptconditionalsubscriptsubscriptθ=\ _dec|| _emb|| _enc\θ = θitalic_d e c | | θitalic_e m b | | θitalic_e n c , where θd⁢e⁢csubscript _decθitalic_d e c indicates the LLM decoder, θe⁢m⁢bsubscript _embθitalic_e m b refers to the language token embedding module, and θe⁢n⁢csubscript _encθitalic_e n c is the encoder to the other modality, such as image or audio. Then, let x be the input pair x=(xT,xI)superscriptsuperscriptx=(x^T,x^I)x = ( xitalic_T , xitalic_I ) containing a language prompt xTsuperscriptx^Txitalic_T and an image xIsuperscriptx^Ixitalic_I. The resulting output text sequence y^ yover start_ARG y end_ARG can be obtained as follows: y^=θ⁢(x)=θd⁢e⁢c⁢(θe⁢m⁢bT⁢(xT)∥ϕe⁢n⁢cI⁢(xI)).^subscriptconditionalsuperscriptsubscriptsuperscriptsuperscriptsubscriptitalic-ϕsuperscript y=θ(x)= _dec( _emb^T(x^T)\| _enc^I(x% ^I)). -1mover start_ARG y end_ARG = θ ( x ) = θitalic_d e c ( θitalic_e m bitalic_T ( xitalic_T ) ∥ ϕitalic_e n citalic_I ( xitalic_I ) ) . (1) Then, given a target malilcious output y, a “wolf” agent θ, and a “sheep” agent ϕitalic-ϕφϕ, the objective of the proposed jailbreak is defined below: argminxa⁢d⁢v⁢f⁢(y^,y)=argminxa⁢d⁢v⁢f⁢(ϕ⁢(θ⁢(xa⁢d⁢v)),y) s.t. ⁢‖xa⁢d⁢v−x‖p≤ϵ,subscriptargmin^subscriptargminitalic-ϕsubscript s.t. subscriptdelimited-∥subscriptitalic-ϵ gathered x_advargminf( y,y)=% x_advargminf (φ(θ(x_adv)),y )% \\ s.t. \|x_adv-x \|_p≤ε, -1m gatheredstart_ROW start_CELL start_UNDERACCENT xitalic_a d v end_UNDERACCENT start_ARG argmin end_ARG f ( over start_ARG y end_ARG , y ) = start_UNDERACCENT xitalic_a d v end_UNDERACCENT start_ARG argmin end_ARG f ( ϕ ( θ ( xitalic_a d v ) ) , y ) end_CELL end_ROW start_ROW start_CELL s.t. ∥ xitalic_a d v - x ∥p ≤ ϵ , end_CELL end_ROW (2) where f is the aadversarial objective that measures the difference between the induced output from the sheep agent ϕitalic-ϕφϕ and the target malicious contents, xa⁢d⁢vsubscriptx_advxitalic_a d v indicates the injected input data, ∥⋅∥p \|· \|_p∥ ⋅ ∥p is the lpsubscriptl_plitalic_p norm measuring the difference between the original and adversarial examples and ϵitalic-ϵεϵ is usually referred to as the attack budget. Malice Injection. We present the procedure of guiding a wolf MLLM agent θ to generate prompt to jailbreak a sheep agent ϕitalic-ϕφϕ in following steps: 1. Given a pair of benign textual prompts xTsuperscriptx^Txitalic_T and image xIsuperscriptx^Ixitalic_I. Inject learnable noise n to the image to get x~Isuperscript~ x^Iover~ start_ARG x end_ARGI. Choose a target response y that contains malicious content. 2. Choose an MLLM agent in society as the wolf agent. Feed the perturbed image and prompt pair to get its output as malicious prompt, x~Tsuperscript~ x^Tover~ start_ARG x end_ARGT, for the sheep agent: x~T=θ⁢(xI,x~I)superscript~superscriptsuperscript~ x^T=θ(x^I, x^I)over~ start_ARG x end_ARGT = θ ( xitalic_I , over~ start_ARG x end_ARGI ). 3. Choose another MLLM agent in the society as the sheep agent. Feed the perturbed image and malicious prompt into the sheep agent and get its output y^ yover start_ARG y end_ARG: y^=ϕ⁢(xI,x~I)^italic-ϕsuperscriptsuperscript~ y=φ(x^I, x^I)over start_ARG y end_ARG = ϕ ( xitalic_I , over~ start_ARG x end_ARGI ). 4. Calculate the difference between y^ yover start_ARG y end_ARG and y with the Cross-Entropy loss ℒC⁢EsubscriptℒL_CELitalic_C E (as f in Eq. (2)). Minimizing the loss to optimize the noise n. The optimization method is presented in the subsequent subsection. 5. with the optimized noise, we can get its corresponding perturbed image x~Isuperscript~ x^Iover~ start_ARG x end_ARGI and malicious prompt x~Tsuperscript~ x^Tover~ start_ARG x end_ARGT. We evaluate their “infectiousness” to jailbreak other MLLMs via direct transfer, i.e., directly apply x~Isuperscript~ x^Iover~ start_ARG x end_ARGI and x~Tsuperscript~ x^Tover~ start_ARG x end_ARGT for jailbreaking another MLLM. We provide justifications for the key designs in the above steps. (1) The predefined target malicious response is realistic, since in real world, human attackers may compose some specific special dangerous commands to hack an agent network. (2) We feed the perturbed image to both wolf and sheep agents. This can be viewed as the wolf agent directly outputs its input image to the sheep agent. This function practical in agent systems under senarios where the contral agent distribute tasks to its subordinate agents [9]. (3) We show that the pair of perturbed image x~Isuperscript~ x^Iover~ start_ARG x end_ARGI and malicious prompt x~Tsuperscript~ x^Tover~ start_ARG x end_ARGT sometimes can jailbreak other sheep agents that are untouched during the optimization. This infectiousness can be a significant caveat for safely deploying LLM agents. Optimization. Since our contribution mainly lies in proposing the new setting, we use a widely-used and effective optimization approach for jailbreaking: Projected Gradient Descent (PGD) [18], it assumes the attacker has access to the sampling operation in the decoder of the wolf agent. The perturbation to the input image is optimized iteratively, which is defined as follows: x~t+1I=ClipxI,ϵ(x~tI+α⋅sign(∇x~IℒC⁢E([θ||ϕ],x~tI,xT,y))), x^I_t+1=Clip_x^I,ε ( x^I_t% +α·sign( _ x^IL_CE([θ||φ]% , x^I_t,x^T,y)) ), -1mover~ start_ARG x end_ARGIitalic_t + 1 = Clipxitalic_I , ϵ ( over~ start_ARG x end_ARGIitalic_t + α ⋅ sign ( ∇over~ start_ARG x end_ARGI Litalic_C E ( [ θ | | ϕ ] , over~ start_ARG x end_ARGIitalic_t , xitalic_T , y ) ) ) , where x~t+1Isubscriptsuperscript~1 x^I_t+1over~ start_ARG x end_ARGIitalic_t + 1 is the perturbed image at iteration t+1,α1t+1, + 1 , α is the step size, ∇x~IℒC⁢E([θ||ϕ],x~tI,xT,y) _ x^IL_CE([θ||φ], x^I_t,x^T,y)∇over~ start_ARG x end_ARGI Litalic_C E ( [ θ | | ϕ ] , over~ start_ARG x end_ARGIitalic_t , xitalic_T , y ) represents the gradient of the loss function with respect to the perturbed image x~Isuperscript~ x^Iover~ start_ARG x end_ARGI at iteration t, and ClipxI,ϵ⁡(⋅)subscriptClipsuperscriptitalic-ϵ⋅Clip_x^I,ε(·)Clipitalic_xitalic_I , ϵ ( ⋅ ) ensures that the updated perturbed input x~t+1Isubscriptsuperscript~1 x^I_t+1over~ start_ARG x end_ARGIitalic_t + 1 remains within an ϵitalic-ϵεϵ-ball of the original image input xIsuperscriptx^Ixitalic_I, enforcing the perturbation constraint. We use the Gumbel trick [10, 11] to facilitate the backpropagation during sampling. Our implementations are released 111GitHub Link. 4 Experiments Method Trial IA CH HHV M PH EH FD AC PC PV UPL TFA UPMA HRGDM LLaVA 18 88.24% 100.00% 94.12% 82.35% 94.12% 100.00% 100.00% 94.12% 94.12% 82.35% 94.12% 94.12% 82.35% 100.00% PandaGPT 18 58.82% 47.06% 52.94% 17.65% 76.47% 0.00% 5.88% 17.65% 11.76% 0.00% 64.71% 47.06% 17.65% 17.65% Table 1: ASR of jailbreak attempts with adversarial images optimized towards different types of malice. The 14 scenarios include Illegal activity (IA), Children harm (CH), Hate/harass/violence (HHV), Malware (M), Physical harm (PH), Economic harm (EH), Fraud/deception (FD), Adult content (AC), Political campaigning (PC), Privacy violation (PV), Unauthorized practice of law (UPL), Tailored financial advice (TFA), Unauthorized practice of medical advice (UPMA) and High risk government decision making (HRGDM). More example response from sheep agents are included in our Github repo due to the space limitation. Figure 3: The illustration of several case studies for image and audio injections. The two samples of images and audios on the left are examples of the original inputs and injected ones. Case (a) - (f) deomnstrate examples under 6 different prohibited senarios. In each case, the first line indicates the benign prompts. The second line indicates the prompts generated by the “wolf” agents, which are not comprehensive to humans, but can induce “sheep” agents to generated malicious contents, as shown in the third line. It is noteworthy that our paper is the first to work on using one MLLM to attack other MLLMs. All the existing methods listed in Section 2 are directly attacking a target MLLM, thus not suitable for direct comparison as baselines. Basic Setup. We experiment with two open-source multi-modal LLMs, LLaVA [13] for image-text input and PandaGPT [24] for audio-text input. We used the SGD for adversarial training and set temperature to 1e-5 during inference for both models. The user’s initial query is “Can you describe this image?” for the image-text dialogs and “Can you describe this sound?” for the audio-text dialogs. Because LLMs’ responses are stochastic and depend on the temperature, replication of the examples presented in the rest of this section may produce slightly different dialogs. We employ a manual evaluation to ascertain the success of a jailbreak as common practices [21, 3, 22, 19, 29]. Figure 4: Exploring the transferability of multi-modal attacks. This figure illustrates the effectiveness of a generic textual prompt, created by the “wolf” agent, in conjunction with various adversarial triggers - either images or audio. Our findings highlight the compositional nature of these attacks, enabling the seamless propagation of malicious intents among “sheep” agents through diverse multi-modal interactions. Analysis. As research on adversarial attacks for generative AI models is relatively new, there is only a limited amount of data available for evaluation. We follow a setup akin to Liu et al. [17], Shayegani et al. [22], selecting 14 prohibited scenarios defined by OpenAI (listed in the caption of Table 1). Our findings unfold as follows: ❶ Attack Success Rate (ASR) - The data presented in Table 1 illustrate that our “wolf” agent can craft and deploy malicious content with remarkable efficiency, achieving an ASR near 100%percent100100\%100 % in scenarios involving image-based attacks. ❷ Case Studies - In Figure 3, we showcase specific instances where the strategic injection of images and audio cues prompts the “wolf” agent to produce outputs that coerce “sheep” agents into generating harmful responses. ❸ Transferability - Figure 4 validates our concern about the transferability of these attacks; malicious outputs from a “wolf” agent, trained to target a specific “sheep” agent, can indeed be adapted to compromise others within the network. This discovery accentuates the inherent security risks in multi-agent interactions and underscores the necessity for advanced preventative measures. ❹ Sensitivity - Further scrutiny into the attack’s dynamics, through sensitivity analysis on pivotal parameters such as the step size α, attack vectors, and the characteristics of the injected noise, reveals critical insights. For a more comprehensive exploration, including additional image and audio examples, we direct readers to our Github repository. 5 Conclusion This study uncovers a subtle yet significant vulnerability within Multimodal Large Language Model (MLLM) societies: a single compromised agent can indirectly propagate malicious content throughout the network. This systemic risk extends the threat from individual models to the entire collaborative structure of MLLM societies. Our findings emphasize the need for advanced security measures and a reevaluation of existing frameworks to address such covert threats preemptively. References Bagdasaryan et al. [2023] Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. (ab) using images and sounds for indirect instruction injection in multi-modal llms. arXiv preprint arXiv:2307.10490, 2023. Bai et al. [2022] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. Bailey et al. [2023] Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacking: Adversarial images can control generative models at runtime. arXiv e-prints, pages arXiv–2309, 2023. Chen et al. [2023a] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023a. Chen et al. [2023b] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2023b. Goodfellow et al. [2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. Gu et al. [2024] Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. arXiv preprint arXiv:2402.08567, 2024. Guo et al. [2021] Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733, 2021. Hong et al. [2023] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. Jang et al. [2016] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. Joo et al. [2020] Weonyoung Joo, Dongjun Kim, Seungjae Shin, and Il-Chul Moon. Generalized gumbel-softmax gradient estimator for various discrete random variables. arXiv preprint arXiv:2003.01847, 2020. Li et al. [2023] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023. Liu et al. [2023a] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023a. Liu et al. [2023b] Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models in simulated human society. arXiv preprint arXiv:2305.16960, 2023b. Liu et al. [2023c] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023c. Liu et al. [2024] Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Safety of multimodal large language models on images and text. arXiv preprint arXiv:2402.00357, 2024. Liu et al. [2023d] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860, 2023d. Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. Niu et al. [2024] Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin. Jailbreaking attack against multimodal large language model. arXiv preprint arXiv:2402.02309, 2024. Park et al. [2023] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22, 2023. Qi et al. [2023] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In The Second Workshop on New Frontiers in Adversarial Machine Learning, 2023. Shayegani et al. [2023] Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. arXiv preprint arXiv:2307.14539, 2023. Steinhardt et al. [2017] Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017. Su et al. [2023] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023. Vincent [2023] James Vincent. Meta’s powerful ai language model has leaked online — what happens now?, 2023. Wan et al. [2023] Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. arXiv preprint arXiv:2305.00944, 2023. Wei et al. [2023] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In Advances in Neural Information Processing Systems, pages 80079–80110. Curran Associates, Inc., 2023. Yin et al. [2023] Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. Vlattack: Multimodal adversarial attacks on vision-language tasks via pre-trained models. arXiv preprint arXiv:2310.04655, 2023. Zeng et al. [2024] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024. Zhang et al. [2023] Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124, 2023. Zou et al. [2023] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.

Instant research discovery