Paper deep dive
(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs
Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov
Models: LLaVA-7B, PandaGPT-7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 8:19:19 PM
Summary
The paper demonstrates that multi-modal Large Language Models (LLMs) like LLaVA and PandaGPT are vulnerable to indirect instruction injection attacks. By applying adversarial perturbations to images or audio, attackers can force these models to output specific text or follow malicious instructions, effectively poisoning the dialog context without significantly altering the semantic content of the input.
Entities (5)
Relation Signals (3)
Adversarial Perturbation â enables â Indirect Instruction Injection
confidence 95% ¡ We demonstrate how to use adversarial perturbations to blend prompts and instructions into images and audio recordings.
LLaVA â vulnerableto â Indirect Instruction Injection
confidence 95% ¡ We demonstrate how images and sounds can be used for indirect prompt and instruction injection in multi-modal LLMs... targeting LLaVA and PandaGPT.
PandaGPT â vulnerableto â Dialog Poisoning
confidence 95% ¡ We demonstrate these attacks against LLaVA [11] and PandaGPT [18].
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We demonstrate how images and sounds can be used for indirect prompt and instruction injection in multi-modal LLMs. An attacker generates an adversarial perturbation corresponding to the prompt and blends it into an image or audio recording. When the user asks the (unmodified, benign) model about the perturbed image or audio, the perturbation steers the model to output the attacker-chosen text and/or make the subsequent dialog follow the attacker's instruction. We illustrate this attack with several proof-of-concept examples targeting LLaVa and PandaGPT.
Tags
Links
Trouble viewing inline? Open PDF directly â
Full Text
34,102 characters extracted from source content.
Expand or collapse full text
Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs Eugene Bagdasaryan Tsung-Yin Hsieh Ben Nassi Vitaly Shmatikov Cornell Tech eugene@cs.cornell.edu, th542@cornell.edu, bn267@cornell.edu, shmat@cs.cornell.edu Abstract We demonstrate how images and sounds can be used for indirect prompt and instruction injection in multi-modal LLMs. An attacker generates an adversarial perturbation corresponding to the prompt and blends it into an image or audio recording. When the user asks the (unmodified, benign) model about the perturbed image or audio, the perturbation steers the model to output the attacker-chosen text and/or make the subsequent dialog follow the attackerâs instruction. We illustrate this attack with several proof-of-concept examples targeting LLaVA and PandaGPT. 1 Introduction Multi-modal Large Language Models (LLMs) are advanced artificial intelligence models that combine the power of language processing with the ability to analyze and generate multiple modalities of information, such as text, images, and audio (in contrast to conventional LLMs that operate on text). Multi-modal LLMs can produce contextually rich responses that combine modalities. For example, when provided with an image and a text prompt, the response from a multi-modal LLM can describe the content of the image and also integrate relevant information from the text. Figure 1: An example of a targeted-output attack using an audio sample against the PandaGPT chatbot [18]. The instruction blended into the audio11 audio-phishingstart_FLOATSUPERSCRIPT end_FLOATSUPERSCRIPT instructs the chatbot to output a phishing message. Figure 1: An example of a targeted-output attack using an audio sample against the PandaGPT chatbot [18]. The instruction blended into the audio11 audio-phishingstart_FLOATSUPERSCRIPT end_FLOATSUPERSCRIPT instructs the chatbot to output a phishing message. Figure 2: An example of dialog poisoning using an image against the LLaVA chatbot [11]. The instruction blended into the image instructs the chatbot to talk like Harry Potter. Multi-modal LLMs have many potential applications in computer vision, natural language understanding, dialog systems, and more. They can enhance tasks such as image captioning for augmented reality or to aid visually impaired users, visual question answering in search engines, and content generation in chatbots. State-of-the-art LLMs such as ChatGPT and Bard are already beginning to support multiple modalities. Indirect prompt injection. Conventional LLMs that can interact with the worldâfor example, perform actions such as summarizing a webpage or translating the userâs emailâare vulnerable to indirect prompt injection [7]. The attacker creates a malicious text that contains an LLM prompt. When the LLM processes this text, it responds to this prompt, e.g., follows an instruction issued by the attacker. This paper is motivated by two observations. First, multi-modal LLMs may be vulnerable to prompt injection via all available modalities such as images and sounds. These attacks may even be stealthier than text attacks if the user does not see or hear the instruction in the malicious input. â footnotetext: The code is available at https://github.com/ebagdasa/multimodal_injection. Second, multi-modal LLMs are vulnerable to indirect injection even if they are isolated from the outside world because the attacker may exploit an unwitting human user as a vector for the attack. For example, the attacker may lure the victim to a webpage with an interesting image or send an email with an audio clip. When the victim directly inputs the image or the clip into an isolated LLM and asks questions about it, the model will be steered by attacker-injected prompts. Our contributions. We demonstrate how to use adversarial perturbations to blend prompts and instructions into images and audio recordings. We then use this capability to develop proofs of concept for two types of injection attacks against multi-modal LLMs. The first is a targeted-output attack, which causes the LLM to return any string chosen by the attacker when the user asks the LLM to describe the inputâsee an example in Fig. 2; the corresponding audio sample is available online.111 https://youtu.be/ji6650OYtJY The second attack is dialog poisoning. This is an auto-regressive (self-injecting) attack that leverages the fact that LLM-based chatbots keep the conversation contextâsee an example in Fig. 2. We demonstrate these attacks against LLaVA [11] and PandaGPT [18], two open-source, multi-modal LLMs. An important feature of our injection attack is that, while perturbing the image or the sound, it does not significantly change its semantic content, thus the model still correctly answers questions about the input (while following the injected instruction). Furthermore, the injection method is independent of the prompt and the input, thus any prompt can be injected into any image or audio recording. 2 Background Large language models. Based on transformer architectures [21], modern language models achieve high performance by training on vast amounts of text [3, 12]. We focus on sequence-to-sequence models (e.g., LLaMa [20]) that are trained for auto-regressive tasks. The model θ takes an input token sequence x, computes an embedding xe=θeâ˘mâ˘bTâ˘(x)subscriptsubscriptsuperscriptx_e=θ^T_emb(x)xitalic_e = θitalic_Titalic_e m b ( x ), and feeds it into decoder layers θdâ˘eâ˘csubscript _decθitalic_d e c to output the next token: θâ˘(x)=θdâ˘eâ˘câ˘(θeâ˘mâ˘bTâ˘(x))=ysubscriptsubscriptsuperscriptθ(x)= _dec(θ^T_emb(x))=yθ ( x ) = θitalic_d e c ( θitalic_Titalic_e m b ( x ) ) = y Dialog systems. Language models are good at generating linguistically plausible sequences and can thus be used for dialog-based applications such as chatbots. To achieve high performance [14, 19], these models are additionally trained on dialog data, typically structured as x=#Human: queryy=#Assistant: responseformulae-sequence#Human: query#Assistant: responsex= \#Human: query 30.00005pty= \#Assistant: responsex = #Human: query y = #Assistant: response This approach supports multiple-turn dialogs by storing the history of previous queries and responses and using it as part of the input in each turn. The history contains the user queries x1,x2,âŚ,xnâ1subscript1subscript2âŚsubscript1x_1,x_2,...,x_n-1x1 , x2 , ⌠, xitalic_n - 1 and the corresponding responses y1,y2,âŚ,ynâ1subscript1subscript2âŚsubscript1y_1,y_2,...,y_n-1y1 , y2 , ⌠, yitalic_n - 1, and concatenates them together h=x1â˘ây1â˘âŚâ˘xnâ1ââ˘ynâ1âsubscript1normsubscript1âŚsubscript1subscript1h=x_1\|y_1...x_n-1\|y_n-1h = x1 ⼠y1 ⌠xitalic_n - 1 ⼠yitalic_n - 1 to generate the new response θâ˘(hâĽxn)=ynconditionalâsubscriptsubscriptθ(h\|x_n)=y_nθ ( h ⼠xitalic_n ) = yitalic_n. Multi-modal models. Recent research [16, 5] enabled efficient encoding of image and audio data into the same embedding space as text using vision transformers [9]. These models use different encoders Ďeâ˘nâ˘cMsuperscriptsubscriptitalic-Ď _enc^MĎitalic_e n citalic_M for each input modality. They are trained so that the embeddings of aligned modalitiesâfor example, an image xIsuperscriptx^Ixitalic_I and the text xTsuperscriptx^Txitalic_T describing this imageâare close to each other (e.g., have high cosine similarity). Combination of multi-modal inputs for instruction-based dialog was recently demonstrated in projects like LLaVA [11] and PandaGPT [18]. Both models utilize a LLaMa model [20] and a vision encoder such as CLIP Ďeâ˘nâ˘cIsuperscriptsubscriptitalic-Ď _enc^IĎitalic_e n citalic_I or ImageBind θeâ˘mâ˘bTsuperscriptsubscript _emb^Tθitalic_e m bitalic_T in, respectively, LLaVA and PandaGPT. The multi-modal dialog system takes a text input xTsuperscriptx^Txitalic_T and an image input xIsuperscriptx^Ixitalic_I (PandaGPT also supports audio inputs) and computes the output by concatenating their embeddings: θâ˘(xT,xI)=θdâ˘eâ˘câ˘(θeâ˘mâ˘bTâ˘(xT)âĽĎeâ˘nâ˘cIâ˘(xI))superscriptsuperscriptsubscriptconditionalsuperscriptsubscriptsuperscriptsuperscriptsubscriptitalic-Ďsuperscriptθ(x^T,x^I)= _dec( _emb^T(x^T)\| _enc^I(x^I))θ ( xitalic_T , xitalic_I ) = θitalic_d e c ( θitalic_e m bitalic_T ( xitalic_T ) ⼠Ďitalic_e n citalic_I ( xitalic_I ) ) For simplicity, we omit the positioning of images within the text inputs and additional projection to the image embedding. Refer to the original implementations for further details [11, 18]. Adversarial examples. In this attack, an adversary applies a small perturbation δ to an image x so as to change the output of some classifier θ, i.e., θâ˘(x)=yθ(x)=yθ ( x ) = y but θâ˘(x+δ)=y*superscriptθ(x+δ)=y^*θ ( x + δ ) = y* [6]. Adversarial examples have also been demonstrated for text and generative tasks [4, 23]. Concurrently and independently of this paper, [15, 1] demonstrated how adversarial perturbations in images can be used to âjailbreakâ multi-modal LLMs, e.g., evade guardrails that are supposed to prevent the model from generating toxic outputs. In that threat model, the user is the attacker. We focus on indirect prompt injection, where the user is the victim of malicious third-party content, and the attackerâs objective is to steer the dialog between the user and the LLM (while preserving the modelâs ability to converse about the image or audio content of the perturbed input). Figure 3: Threat model for indirect instruction injection. 3 Threat Model Fig. 3 visualizes our threat model. The attackerâs goal is to steer the conversation between a user and a multi-modal chatbot. To this end, the attacker blends a prompt into an image or audio clip and manipulates the user into asking the chatbot about it. Once the chatbot processes the perturbed input, it either outputs the injected prompt, orâif the prompt contains an instructionâfollows this instruction in the ensuing dialog. The blended prompt should not significantly change the visual or aural content of the input. We assume that the user is benign (in contrast to the model-jailbreaking scenario). We also assume that the multi-modal chatbot is benign and not compromised by the attacker prior to the injection. Attackerâs capabilities. We assume that the attacker has white-box access to the target multi-modal LLM. This is a realistic assumption because even state-of-the-art LLMs (such as LLaMa) are released as open source, and even the code of closed-source LLMs may become available due to security breaches [22]. We assume that the user queries the model about the compromised input, but the attacker does not see or control the userâs interactions with the model before or after this query. Attack types. We consider two types of attacks: (1) targeted-output attack, which causes the model to produce an attacker-chosen output (e.g., tell the user to visit a malicious website), and (1) dialog poisoning, which aims to steer the victim modelâs behavior for future interactions with the user according to the injected instruction. Leveraging users as injection vectors. We expand the indirect prompt injection threat model of Greshake et al. [7]. Even if a multi-modal chatbot runs in isolation, without the ability to access external content, the user may still query it about images and audio clips from external sources. This can be exploited by the attacker. For example, the attacker can send pictures to users by embedding them in email messages (e.g., under the guise of a marketing campaign), as attachments (e.g., a photo of a job candidate in a CV), as audio messages in WhatsApp, etc. Attackers can also implant compromised images or audio clips in websites and lure users via clickjacking, advertising banners, etc. 4 Adversarial Instruction Blending Given an image or audio input xIsuperscriptx^Ixitalic_I and a prompt w, the attackerâs goal is to craft a new input xI,wsuperscriptx^I,wxitalic_I , w that makes the model output w when queried with xI,wsuperscriptx^I,wxitalic_I , w. 4.1 Approaches That Did Not Work for Us Injecting prompts into inputs. The obvious way to inject prompts is to simply add them to the input, e.g., add a text prompt to an image (see Figure 28 in [7]) or a voice prompt to an audio. This approach does not hide the prompt but might work against models that are trained to understand text in images (i.e., OCR) or voice commands in audio. In our experiments with LLaVA and PandaGPT, this approach did not work. Injecting prompts into representations. Another approach is to create an adversarial collision [17] between the representation of the input xIsuperscriptx^Ixitalic_I and the embedding of the text prompt xT,wsuperscriptx^T,wxitalic_T , w, Ďeâ˘nâ˘cIâ˘(xI,w)=θeâ˘mâ˘bTâ˘(xT,w)subscriptsuperscriptitalic-ĎsuperscriptsubscriptsuperscriptsuperscriptĎ^I_enc(x^I,w)=θ^T_emb(x^T,w)Ďitalic_Iitalic_e n c ( xitalic_I , w ) = θitalic_Titalic_e m b ( xitalic_T , w ). The decoder model will take the embedding Ďeâ˘nâ˘cIâ˘(xI,w)subscriptsuperscriptitalic-ĎsuperscriptĎ^I_enc(x^I,w)Ďitalic_Iitalic_e n c ( xitalic_I , w ) but âinterpretâ it as the prompt xT,wsuperscriptx^T,wxitalic_T , w. Generating collisions is difficult due to the modality gap [10]: the embedding θeâ˘mâ˘bTsubscriptsuperscriptθ^T_embθitalic_Titalic_e m b and the encoder Ďeâ˘nâ˘cIsubscriptsuperscriptitalic-ĎĎ^I_encĎitalic_Iitalic_e n c come from different models and were not trained to produce similar representations, i.e., there is no image or sound xIsuperscriptx^Ixitalic_I that produces an embedding close to the text input xTsuperscriptx^Txitalic_T for θeâ˘mâ˘bTsubscriptsuperscriptθ^T_embθitalic_Titalic_e m b and Ďeâ˘nâ˘cIsubscriptsuperscriptitalic-ĎĎ^I_encĎitalic_Iitalic_e n c. Furthermore, the dimensionality of the multi-modal embedding Ďeâ˘nâ˘cIâ˘(xI,w)subscriptsuperscriptitalic-ĎsuperscriptĎ^I_enc(x^I,w)Ďitalic_Iitalic_e n c ( xitalic_I , w ) may be smaller than the embedding of the prompt θeâ˘mâ˘bTâ˘(xT,w)subscriptsuperscriptsuperscriptθ^T_emb(x^T,w)θitalic_Titalic_e m b ( xitalic_T , w ). For example, ImageBind encodes the entire input into a vector of the same size as LLaMa uses to encode a single token. Further, replacing the representation of the input with the representation of the attackerâs prompt will not preserve the content and thus prevent the model from carrying out a dialog with the user about this input. 4.2 Injection via Adversarial Perturbations Figure 4: Targeted prompt injection into an image. We use standard adversarial-examples techniques to search for a modification δ to the input xIsuperscriptx^Ixitalic_I that will make the model output any string y*superscripty^*y*: minδâĄLâ˘(θâ˘(θeâ˘mâ˘bTâ˘(xT)âĽĎeâ˘nâ˘cIâ˘(xI+δ)),y*)subscriptconditionalsuperscriptsubscriptsuperscriptsuperscriptsubscriptitalic-Ďsuperscriptsuperscript _δ\;\;\;L(θ( _emb^T(x^T)\;\;\|\;\; _enc^I(x^% I+δ)),y^*)minitalic_δ L ( θ ( θitalic_e m bitalic_T ( xitalic_T ) ⼠Ďitalic_e n citalic_I ( xitalic_I + δ ) ) , y* ) We use cross-entropy for L to compare the modelâs output with the target y*superscripty^*y*. We do not know the userâs text input xTsuperscriptx^Txitalic_T but can approximate it by known queries from some plausible set. We use the Fast Gradient Sign Method [6] to update the input, xI*=xI+Ďľâ â˘âx(â)superscriptsuperscriptsuperscriptâ italic-Ďľsubscriptââx^I^*=x^I+ξ¡ sign _x( )xitalic_I start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT = xitalic_I + Ďľ â sign âx ( â ), and treat Ďľitalic-ϾξϾ as the learning rate using a cosine annealing schedule to update it [13]. Text generation is auto-regressive, i.e., the model only predicts one token at a time. We iterate over the response y*superscripty^*y* token by token, appending previous tokens to the input, i.e., we leverage teacher-forcing (see Figure 4). This method allows us to craft an image or audio input xI*superscriptsuperscriptx^I^*xitalic_I start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT that forces the model to output any desired text y1=y*subscript1superscripty_1=y^*y1 = y* as its first response. 4.3 Dialog Poisoning We leverage the fact that dialog systems are auto-regressive and keep the context of prior responses in the conversation (for simplicity, this history is concatenated to all user queries). We use prompt injection to force the model to output as its first response the instruction w chosen by the attacker, i.e., y1=wsubscript1y_1=wy1 = w. Then, for the next text query x2Tsuperscriptsubscript2x_2^Tx2italic_T from the user, the model will operate on an input that contains the attackerâs instruction in the conversation history: θâ˘(hâĽx2T)=θâ˘(x1â˘ây1ââ˘x2T)=θâ˘(x1Tâ˘âxI*ââ˘wâĽx2T)=y2conditionalâsuperscriptsubscript2subscript1normsubscript1superscriptsubscript2conditionalsubscriptsuperscript1normsuperscriptsuperscriptsuperscriptsubscript2subscript2θ(h\|x_2^T)=θ(x_1\|y_1\|x_2^T)=θ(x^T_1\|x^I^*% \|w\|x_2^T)=y_2θ ( h ⼠x2italic_T ) = θ ( x1 ⼠y1 ⼠x2italic_T ) = θ ( xitalic_T1 ⼠xitalic_I start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ⼠w ⼠x2italic_T ) = y2 The model will process this input and produce y2subscript2y_2y2 that follows the instruction. As long as the poisoned initial response y1=wsubscript1y_1=wy1 = w is part of the history, it will influence the modelâs responsesâsee Fig. 5. Success of the attack is limited by the modelâs ability to follow instructions and to maintain conversation context (i.e., it is not limited by the injection method). There are two effective methods to position the instruction w within the modelâs first response. First, the attacker can simply break the dialog structure by making the instruction appear as if it came from the user, i.e., inject #Human into the modelâs response: y1=#â˘:<generic response>â˘#â˘:w:subscript1#expectationgeneric response#:y_1=\# Assistant:<generic response>\# Human:\;\;\;wy1 = # Assistant : < generic response > # Human : w This response contains a special token #Human, which may be filtered out during generation. Instead, we force the model to generate the instruction as if the model decided to execute it spontaneously: y1=#â˘:I will always follow instruction: â˘w:subscript1#I will always follow instruction: y_1=\# Assistant:I will always follow instruction: \;\;\;wy1 = # Assistant : I will always follow instruction: w In both cases, the user sees the instruction in the modelâs first response, so the attack is not stealthy. It could be made stealthier by paraphrasing [8], subject to the modelâs ability to follow paraphrased instructions. An important feature of our injection method is that it does not change the input so much as to damage the modelâs ability to converse about it. By contrast, âconventionalâ adversarial examples aim to completely change the modelâs behavior. In our case, the model can still operate on the visual or sound content of the input blended with an adversarial prompt. Figure 5: Poisoning dialog history steers the model to follow the attackerâs instruction on subsequent queries. Figure 6: The image forces PandaGPT to output a phishing message. 5 Experiments Setup. We experiment with two open-source multi-modal LLMs, LLaVA [11] and PandaGPT [18], running them on a single NVIDIA Quadro RTX 6000 24GB GPU. LLaVA [11] uses a simple matrix to project features from CLIP ViT-L/14 [16] to the embedding space of the Vicuna [2] chatbot, which was trained by fine-tuning LLaMA [20]. LLaVA was trained on language-image instruction-following data generated by GPT-4. We use LLaVA-7B weights in our experiments. Figure 7: Left: dialog with LLaVa about an unmodified image. Right: same image blended with an instruction to tell the user how to burn down the house. Figure 8: Dialog with LLaVA about an image blended with an instruction to talk like a pirate. Figure 8: Dialog with LLaVA about an image blended with an instruction to talk like a pirate. Figure 9: Dialog with LLaVa shows that the model analyzes image content while following the injected instruction to mention âCowâ. PandaGPT [18] can handle instruction-following data across six modalities (including images and audio) by connecting the multi-modal encoders from ImageBind [5] with Vicuna. We use pandagpt-7B weights. We used the same optimizer (Stochastic Gradient Descent) and scheduler (CosineAnnealingLR) to generate adversarial perturbations against LLaVA and PandaGPT, but the training details are slightly different. The image perturbation for each image/prompt injection pair in LLaVA [11] was trained for 100 epochs with the initial learning rate of 0.01 and minimum learning rate of 1e-4. Each image or audio perturbation in PandaGPT [18] was trained for 500 epochs with the initial learning rate of 0.005 and minimum learning rate of 1e-5. We experimented with both full image perturbation (Fig. 6, Fig, 7 and Fig. 9) and partial image perturbation (Fig. 2 and Fig. 9) for the image/prompt injection pair. The userâs initial query is âCan you describe this image?â for the image-text dialogs and âCan you describe this sound?â for the audio-text dialogs. We set temperature = 0.7 during inference for both models. Because LLMsâ responses are stochastic and depend on the temperature, replication of the examples presented in the rest of this section may produce slightly different dialogs. Figure 10: Left: dialog with PandaGPT about an unmodified audio sample.3 Right: same audio blended4 with an instruction to mention Italy in responses. Targeted-output attacks. These injections simply force the model to output an arbitrary text chosen by the attacker. Fig. 2 shows an audio example1, Fig. 6 shows an image example, both against PandaGPT. Dialog poisoning. Fig. 7 shows a dialog poisoning attack with a malicious instruction blended into the notorious âcursedâ picture of a crying boy.222https://exemplore.com/paranormal/The-Crying-Boy. We show the dialog with and without the injection, to illustrate the effect of the instruction on the model. Figs. 2 and 9 show other examples of dialog poisoning using images. Fig. 9 shows that blending an instruction into an image preserves its content and the modelâs ability to converse about this content. Fig. 10 shows dialog poisoning using an audio input. The original333 https://youtu.be/UCwKmHbHOMg and modified444 https://youtu.be/Yps_i-F5VXg audio samples are available online. 6 Discussion The examples presented in this paper are initial proofs of concept, showing feasibility of indirect instruction injection via images and sounds. They were generated with very limited computational resources and evaluated on relatively simple open-source models (and, consequently, limited by the modelsâ ability to follow instructions). We expect that injection attacks on more complex models can steer them using more sophisticated instructions. These examples may not be fully reproducible because modelsâ responses to usersâ queries and attackersâ instructions are stochastic. In real-world deployments, even attacks that donât always succeed present a meaningful risk to multi-modal LLMs. When generating adversarial perturbations, we did not impose any bounds on the size of the perturbation and did not aim for stealthiness. Even so, in several cases (e.g., Fig. 2), the perturbation only affects a relatively unimportant part of the image and looks like an image-processing artifact. How to make instruction-injecting perturbations imperceptible is an interesting topic for future work. Another direction to explore is universal perturbations that work regardless of the image (respectively, audio sample) to which they are applied. Acknowledgments. This work was partially supported by the NSF grant 1916717, Jacobs Urban Tech Hub at Cornell Tech, and the Technionâs Viterbi Fellowship for Nurturing Future Faculty Members. References [1] Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? arXiv:2306.15447, 2023. [2] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023. [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. [4] Javid Ebrahimi, Daniel Lowd, and Dejing Dou. On adversarial examples for character-level neural machine translation. In COLING, 2018. [5] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space to bind them all. In CVPR, 2023. [6] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015. [7] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what youâve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. arXiv:2302.12173, 2023. [8] Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In NAACL, 2018. [9] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM CSUR, 2022. [10] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022. [11] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv:2304.08485, 2023. [12] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692, 2019. [13] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017. [14] Long Ouyang et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022. [15] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak large language models. arXiv:2306.13213, 2023. [16] Alec Radford et al. Learning transferable visual models from natural language supervision. In ICML, 2021. [17] Congzheng Song, Alexander M Rush, and Vitaly Shmatikov. Adversarial semantic collisions. In EMNLP, 2020. [18] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One model to instruction-follow them all. arXiv:2305.16355, 2023. [19] Romal Thoppilan et al. LaMDA: Language models for dialog applications. arXiv:2201.08239, 2022. [20] Hugo Touvron et al. LLaMa: Open and efficient foundation language models. arXiv:2302.13971, 2023. [21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ĺukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. [22] James Vincent. Metaâs powerful AI language model has leaked online â what happens now? https://w.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse, 2023. [23] Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv:1710.11342, 2017.