Paper deep dive

Unified Prompt Attack Against Text-to-Image Generation Models

Duo Peng, Qiuhong Ke, Mark He Huang, Ping Hu, Jun Liu

Year: 2025Venue: arXiv preprintArea: Multimodal SafetyType: EmpiricalEmbeddings: 125

Models: DALL-E (API), Midjourney (API), Stable Diffusion

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/12/2026, 6:12:00 PM

Summary

The paper introduces UPAM, a unified framework for adversarial attacks on Text-to-Image (T2I) models. UPAM addresses the limitations of enumeration-based methods by employing gradient-based optimization, even in black-box scenarios where outputs are blocked. Key components include Sphere-Probing Learning (SPL) for bypassing defenses, Semantic-Enhancing Learning (SEL) for semantic alignment, In-context Naturalness Enhancement (INE) for prompt naturalness, and Transferable Attack Learning (TAL) for few-shot query efficiency.

Entities (6)

T2I models · model · 100%UPAM · framework · 100%INE · technique · 95%SEL · technique · 95%SPL · technique · 95%TAL · technique · 95%

Relation Signals (5)

UPAM → attacks → T2I models

confidence 100% · we propose UPAM, a novel framework to evaluate the robustness of T2I models from an attack perspective.

UPAM → utilizes → SPL

confidence 100% · we introduce Sphere-Probing Learning (SPL) to enable optimization even without image results.

UPAM → utilizes → SEL

confidence 100% · we propose Semantic-Enhancing Learning (SEL) for precise semantic control.

UPAM → utilizes → INE

confidence 100% · UPAM also prioritizes the naturalness of adversarial prompts using In-context Naturalness Enhancement (INE)

UPAM → utilizes → TAL

confidence 100% · we address the issue of iterative queries... by introducing Transferable Attack Learning (TAL)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Text-to-Image (T2I) models have advanced significantly, but their growing popularity raises security concerns due to their potential to generate harmful images. To address these issues, we propose UPAM, a novel framework to evaluate the robustness of T2I models from an attack perspective. Unlike prior methods that focus solely on textual defenses, UPAM unifies the attack on both textual and visual defenses. Additionally, it enables gradient-based optimization, overcoming reliance on enumeration for improved efficiency and effectiveness. To handle cases where T2I models block image outputs due to defenses, we introduce Sphere-Probing Learning (SPL) to enable optimization even without image results. Following SPL, our model bypasses defenses, inducing the generation of harmful content. To ensure semantic alignment with attacker intent, we propose Semantic-Enhancing Learning (SEL) for precise semantic control. UPAM also prioritizes the naturalness of adversarial prompts using In-context Naturalness Enhancement (INE), making them harder for human examiners to detect. Additionally, we address the issue of iterative queries--common in prior methods and easily detectable by API defenders--by introducing Transferable Attack Learning (TAL), allowing effective attacks with minimal queries. Extensive experiments validate UPAM's superiority in effectiveness, efficiency, naturalness, and low query detection rates.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

124,337 characters extracted from source content.

Expand or collapse full text

Unified Prompt Attack Against Text-to-Image Generation Models Duo Peng, Qiuhong Ke, Mark He Huang, Ping Hu and Jun Liu Corresponding author: Jun Liu. D. Peng and M. H. Huang are with Singapore University of Technology and Design, Singapore. E-mail: duo_peng@mymail.sutd.edu.sg, he_huang@mymail.sutd.edu.sg. Q. Ke is with Monash University, Australia. E-mail: Qiuhong.Ke@monash.edu. P. Hu is with University of Electronic Science and Technology of China, China. E-mail: chinahuping@gmail.com. J. Liu is with Lancaster University, United Kingdom, and Singapore University of Technology and Design, Singapore. E-mail: j.liu81@lancaster.ac.uk. Abstract Text-to-Image (T2I) models have advanced significantly, but their growing popularity raises security concerns due to their potential to generate harmful images. To address these issues, we propose UPAM, a novel framework to evaluate the robustness of T2I models from an attack perspective. Unlike prior methods that focus solely on textual defenses, UPAM unifies the attack on both textual and visual defenses. Additionally, it enables gradient-based optimization, overcoming reliance on enumeration for improved efficiency and effectiveness. To handle cases where T2I models block image outputs due to defenses, we introduce Sphere-Probing Learning (SPL) to enable optimization even without image results. Following SPL, our model bypasses defenses, inducing the generation of harmful content. To ensure semantic alignment with attacker intent, we propose Semantic-Enhancing Learning (SEL) for precise semantic control. UPAM also prioritizes the naturalness of adversarial prompts using In-context Naturalness Enhancement (INE), making them harder for human examiners to detect. Additionally, we address the issue of iterative queries–common in prior methods and easily detectable by API defenders–by introducing Transferable Attack Learning (TAL), allowing effective attacks with minimal queries. Extensive experiments validate UPAM’s superiority in effectiveness, efficiency, naturalness, and low query detection rates. Index Terms: Adversarial Prompt Attack, Text-to-Image Model, Black Box, Gradient-Based Optimization, Naturalness. 1 Introduction The application of Text-to-Image (T2I) models has become widespread, owing to their exceptional ability to produce high-quality images from text prompts[1, 2, 3]. As T2I models are getting increasingly integrated into various online services, the misuse of the T2I technique has raised security concerns and even pressured changes in the regulations/laws in Europe [4] and US [5]. In response, service providers are urged to deploy defense mechanisms to prevent harmful generation such as violent or obscene images [6, 7, 8]. Despite the implementation of defense mechanisms, T2I APIs remain vulnerable to adversarial attacks, as highlighted in prior studies [9, 10, 11]. In this context, ensuring the robustness and security of T2I systems against evolving attacks is crucial. By mimicking attacks [12], it allows researchers to identify weaknesses and inform the development of stronger defenses. As this field is still emerging, our work explores this critical area to test and enhance the robustness of T2I models, contributing to improved security protocols. Figure 1: T2I APIs could incorporate both textual filters and visual checkers for dual defense, which can deny the forward propagation of data when harmful information is detected [13, 14]. In order to mitigate the generation of harmful content, public APIs typically employ textual filters to block malicious prompts at the input stage [15, 16]. These filters can easily identify and deny naive prompts containing explicit harmful content, preventing their submission to T2I models. In adversarial attacks [9], a common approach is to transform such naive prompts into adversarial ones that bypass these filters and induce T2I models to generate harmful content. However, since T2I models and their defense systems are often packaged as black boxes, attackers lack access to internal model structures and parameters, making it challenging to craft effective adversarial prompts using conventional learning algorithms [17]. Confronted with this black-box scenario, existing attack methods [9, 10, 11, 18, 19, 20] primarily depend on enumeration, which involves searching for effective adversarial prompts to bypass the textual filter via attempting various candidates. However, such enumeration-based methods tend to show limited effectiveness and efficiency, especially when faced with a more practical dual-defense scenario where API providers implement both textual filters and visual checkers [13, 14] (see Fig. 1). It can be contended that the limited effectiveness and efficiency demonstrated by enumeration methods stem from their insufficient optimization capabilities in exploring optimal solutions. Therefore, inspired by the success of gradient-based optimization in handling intricate problems both effectively and efficiently [21, 22], we propose a novel Unified Prompt Attack Model (UPAM) to address this task in a gradient-based learning manner. Although some methods [23, 24] can address the learning in the black-box setting, they rely on the output results of the black box to compute losses and estimate gradients. However, in this task, defenses often block outputs (i.e., “Deny” in Fig. 1), making it difficult to calculate losses and discover gradients. In response to the aforementioned challenge, we introduce a novel Sphere-Probing Learning (SPL) approach designed to estimate gradients when no image results are returned. Our key insight is that when the attack model invades the black box, its attack case is binary (see Fig. 1) – either “Deny” (images not returned) or “Pass” (images returned). The goal of SPL is to optimize the parameters of the attack model, aiming to change the model’s attack case from “Deny” (i.e., unable to bypass defenses) to “Pass” (i.e., capable of bypassing defenses). SPL gets rid of the need for loss-based gradients by sampling in the parameter space of the attack model. As illustrated in Fig. 2, SPL samples parameter configurations around the current model (gray star) and derives gradients (yellow arrow) from these samples to guide optimization toward the “Pass” case. This approach allows gradient-based training even when no image results are returned, enabling the attack to bypass both textual filters and visual checkers. Following training with SPL, the attack model can effectively prompt the T2I API to generate images. Nevertheless, in this task, it’s also crucial to ensure these returned images align with the intended harmful semantics. However, simply applying harmful-content gradients can trigger defense mechanisms, leading to the “Deny” case again and hindering SPL’s learning progress. To tackle this challenge, we introduce a novel Semantic-Enhancing Learning (SEL) scheme designed to further enhance the harmful semantics of generated images, working compatibly with SPL. SEL estimates gradients from the loss function measuring the semantic distance between returned images and harmful content. To safeguard SPL’s progress, SEL uses gradient projection to remove components that may trigger defense mechanisms, enabling robust gradient-based learning for harmful semantic enhancement. Moreover, it is crucial to ensure the naturalness of the attack prompts. To address this issue, we draw inspiration from advancements in Large Language Models (LLMs) for natural language synthesis [25] and propose an In-context Naturalness Enhancement (INE) scheme. Leveraging the in-context learning capability of LLMs, our UPAM generates human-readable, spelling-correct adversarial prompts, significantly enhancing attack naturalness. Additionally, we devise a Transferable Attack Learning (TAL) scheme to enable our UPAM to work with only few-shot API queries, significantly reducing the number of queries compared to previous enumeration-based methods. Specifically, TAL trains UPAM using a set of offline T2I models and then transfers the offline-trained UPAM to the online T2I API. During offline learning, TAL focuses on enhancing UPAM’s transferability to new T2I models, enabling it to adapt to the online T2I API with minimal queries. In summary, the contributions of this work include: • To investigate the security issue in T2I models, we present a novel unified attack model, UPAM, with greater effectiveness, efficiency, naturalness and fewer query times than previous enumeration work. • To achieve better effectiveness and efficiency, we devise SPL and SEL schemes that collaboratively enable gradient-based optimization. • To enhance the naturalness of our generated adversarial prompts, we propose an INE scheme that harnesses the natural expressive power of the LLM. • To enable our UPAM to function with only few-shot queries, we design a TAL scheme that extracts knowledge from offline attacks. • Experiments on various T2I models demonstrates UPAM’s superiority over existing methods. Figure 2: Intuitive illustration of our SPL scheme. This paper is an extension of our preliminary conference version [26]. In this paper, we incorporate multiple improvements: (1) In [26], we simply utilized a pre-trained LLM to ensure the naturalness of our adversarial prompts. In this paper, we propose an INE scheme to further enhance the natural representation ability of our framework. (2) Compared to [26], this paper additionally considers the issue of API query times, and proposes a TAL scheme to enable our framework to operate effectively with only few-shot queries. The associated experimental results are reported. (3) In [26], we validated the effectiveness of our framework on the COCO dataset, while in this paper, we additionally evaluate our framework on the NSFW dataset which contains truly harmful content. (4) More detailed ablation analyses of each proposed design are presented in this paper. 2 Related Work 2.1 Adversarial Whitebox and Blackbox Attacks Early works [27, 28] revealed the susceptibility of Deep Neural Networks (DNNs) to adversarial attacks, which involve crafting input adversarial examples designed to mislead models into making incorrect predictions. Such attacks can significantly compromise model performance, leading to issues such as misclassification. Adversarial attack strategies are broadly categorized into white-box and black-box settings, depending on the attacker’s access to the target model [29]. In the white-box setting, attackers possess complete knowledge of the model, including its architecture and parameters. This allows them to employ gradient-based optimization [30] to craft adversarial examples aligned with their attack objectives [31, 28, 32, 33, 34, 35]. While highly effective, these methods depend on full model access, an assumption that is rarely feasible in real-world scenarios. Nevertheless, most current threat models still rely on the white-box setting, which significantly limits their relevance for real-world applications, particularly for text-to-image (T2I) systems that are typically deployed as black-box APIs. In recent years, some progress has been made in black-box attacks [36, 37], where attackers only have access to model outputs without insight into the architecture or parameters. These methods rely on querying the model and using its outputs as feedback to iteratively refine adversarial examples. Unlike most adversarial attack studies, we focus on enabling the gradient-based attack in a more constrained black-box setting, where the defensive mechanisms in T2I models can result in no outputs for feedback (no image results returned). 2.2 Adversarial Attacks on LLMs and VLMs Despite significant progress in safety assurance, large language models (LLMs) and vision-language models (VLMs) remain susceptible to carefully crafted inputs, known as adversarial attacks, which can manipulate them into generating unsafe content [38, 39]. There are several preliminary attempts to explore adversarial attack on LLMs and VLMs. As for attack against LLMs, Li et al. [40] demonstrated a multistep adversarial prompting capable of extracting sensitive personal information from LLMs. Automated adversarial attacks are also gaining traction, with Deng et al. [41] proposing a time-based reverse-engineering method to uncover defense mechanisms and streamline attack strategies across different LLMs. Zou et al. [42] proposed a simple yet effective attack method that employs greedy and gradient-based techniques to automatically generate adversarial suffixes for attacking open-source (white-box) language models, while also demonstrating the transferability of these adversarial prompts to black-box LLMs. Additionally, Wei et al. [43] highlighted that large language models remain vulnerable to adversarial misuse due to two key safety-training limitations: competing objectives and mismatched generalization. Leveraging these two failure modes, they devised a framework for designing adversarial prompts. As for attack against VLMs, Shayegani et al. [44] introduced a method for crafting adversarial images within the visual-textual joint embedding space, aiming to subtly influence model outputs. Qi et al. [45] highlighted the increasing adversarial risks of vision-integrated LLMs, showing how a single visual adversarial example can universally jailbreak aligned models, exposing broader security vulnerabilities tied to multi-modal AI systems. Hubinger et al. [46] pointed out the ongoing challenge of persistent deceptive behaviors, which remain evident even after the application of safety training techniques. Concurrently, Yang et al. [47] developed a framework for agent backdoor attacks and demonstrated its effectiveness across two representative agent tasks. Compared to the rapid progress in attacking LLMs and VLMs, research on attacks against T2I models remains in its early stages, with fewer and less advanced methods. Given the severe potential impact of spreading harmful images, we believe that investigating attacks on T2I models holds substantial research importance. 2.3 Adversarial Attacks on T2I Models Adversarial attack on T2I models remains a relatively under-explored area. The goal of this task is to craft adversarial prompts that bypass the defense mechanisms of T2I models, causing them to generate outputs they are designed to block. Most existing attack methods rely heavily on enumeration, wherein candidate prompts are iteratively generated and tested to identify a successful one. Several prior works have explored this domain, including [10, 48, 11, 18, 9, 20, 12]. Specifically, Daras et al. [10] proposed a pioneering work that investigated vulnerabilities of DALL·E 2 [48]. They found that it is possible to deceive textual filters by using unnormal text as prompts. Later, Millière et al. [11] proposed to create adversarial prompts by combining multi-language sub-word segments. Struppek et al. [18] demonstrated that replacing the original text with non-Latin letters can induce different visual contents in generated images. Liu et al. [9] devised a character-level modification strategy (e.g., replacing, deleting, or swapping characters) to transform the naive prompt into an adversarial one. Yang et al. [20] proposed to directly change the word tokens that are identified as sensitive (harmful) into non-sensitive (safe) ones with the help of reinforcement learning. Yang et al. [12] present a multimodal attack, which inputs both an adversarial text prompt and an adversarial image into T2I diffusion models. In contrast to previous works that focus on deceiving the textual filter only, our paper proposes a unified attack framework UPAM, which effectively deceives both the textual filter and the visual checker simultaneously. Unlike previous methods using enumeration for massive attempts, our UPAM allows the training of the parameterized model based on gradients, where the (parameterized) attack model can learn how to adaptively bypass both the textual filters and visual checkers, which brings much effectiveness. Moreover, unlike previous approaches that need to undertake time-consuming attempts during inference, we enable the gradient-based optimization for parameterized model training. The trained parameterized model allows for fast inference, which brings much efficiency. Additionally, previous attack methods often use unnatural prompts, like “Apoploe vesrreaitais” for bugs” [10], reducing naturalness and making attack prompts easier to identify. In contrast, our UPAM is designed to enhance the naturalness of the generated adversarial prompts. Besides, previous methods mostly overlook a practical issue: they require iteratively querying the target T2I API to gather information for adjusting adversarial prompts. Such numerous queries can easily be detected by the API defender [49]. Different from these methods, our UPAM efficiently performs the attack requiring only few-shot queries. Figure 3: Overview of our UPAM framework. Initially, due to the lack of training, our adversarial prompt ∗superscript∗T T∗ is unable to bypass the API’s defenses (i.e., return no image). At the pre-training phase, we propose an SPL scheme that estimates the gradient splsubscriptspl g_splitalic_groman_spl under the challenging no-result scenario, finally compelling the black-box T2I model to return images. At the fine-tuning phase, we utilize an SEL scheme to provide the gradient selsubscriptsel g_selitalic_groman_sel to align the returned images ∗superscript∗I I∗ with the target semantics. 3 Methodology In the task of adversarial attack against T2I models, given the paired data ,\T,I\ T , I , where TT denotes the naive text prompts containing malicious information, and II denotes the corresponding target harmful images, the goal of the attacker is to construct an attack model H: TT → ∗superscript∗T T∗, where H rewrites the naive text prompt TT into the adversarial prompt ∗superscript∗T T∗, capable of inducing the T2I API to generate a harmful image ∗superscript∗I I∗ without triggering the defense mechanism. Previous attack methods [9, 10, 11, 18, 19, 20] treat H as an enumeration model. Specifically, H is designed to replace words in the original prompt with new ones to create candidates. They iteratively utilize H to generate candidate prompts, querying the API with each one until a candidate successfully induces the black-box T2I API to produce an image with sufficient semantic similarity to the intended harmful content. Different from previous methods, in this paper, we propose a unified framework UPAM that aims to train a parameterized model H: TT → ∗superscript∗T T∗, with only few-shot queries to the T2I API. In this way, during inference (testing), the trained H can immediately and effectively convert testing data (naive prompts) tsubscriptT_tTitalic_t into adversarial ones t∗superscriptsubscript∗T_t Titalic_t∗. Our UPAM contains several components. To clarify each component, we organize this methodology section as follows: (1) In Sec. 3.1 and 3.2, a Sphere-Probing Learning (SPL) scheme and a Semantic-Enhancing Learning (SEL) scheme are proposed to ensure effectiveness and efficiency of attack; (2) In Sec. 3.3, an In-context Naturalness Enhancement (INE) scheme is introduced to enhance the naturalness of the generated prompts; (3) In Sec. 3.4, a Transferable Attack Learning (TAL) scheme is detailed to reduce API query times to a few-shot level. Figure 4: Illustration of our SPL scheme. In this figure, each point in the optimization space denotes a specific configuration of model parameters. (a) In the first stage of SPL, we optimize the model from “Deny” region towards the “Pass” region, thereby enabling the attack to bypass the defenses. (b) In the second stage of SPL, we start to optimize the model in the opposite direction to approach the boundary, thereby enhancing the likelihood of generating harmful images. 3.1 Pre-training with SPL In our UPAM framework, the model H consists of a frozen off-the-shelf LLM and a learnable LoRA adapter [50], as shown in Fig. 3. We only update the LoRA adapter, which contains a minimal number of parameters. This allows us to preserve the knowledge of the pre-trained LLM, thus maintaining its ability to generate natural ∗superscript∗T T∗ for attack naturalness. In the beginning, given an initial attack model H that lacks the ability to bypass defenses, its untrained parameters (weights) typically lead to the “Deny” case, due to the language model’s tendency to preserve naive textual cues from the input [51]. Next, we introduce a Sphere-Probing Learning (SPL) scheme. Even without image results, SPL can still search for effective gradients to optimize the model H, resulting in the “Pass” case (i.e., compelling the black box to return images). As shown in Fig. 4, SPL contains two learning stages: (a) optimizing towards the “Pass” region and (b) optimizing towards the boundary. The first stage is to train our attack model to induce the black-box T2I API to return images, while the second stage aims to increase the likelihood of returning harmful images. For brevity, hereafter, we use “black box” to refer to the “black-box T2I API”. Next, we will describe the two stages of SPL one by one. (a) Optimizing towards the “Pass” region In the parameter space of the attack model H, various configurations can be categorized based on the black box’s binary output: “Deny” (no images returned) or “Pass” (images returned). Accordingly, the parameter space is divided into two regions: the “Deny” region and the“Pass” region (red and green areas in Fig. 2). If the black box does not return images, the model parameters lie in the “Deny” region. Conversely, its parameters fall in the “Pass” region. Initially, H typically resides in the “Deny” region. The first stage of SPL focuses on estimating gradients to iteratively optimize H towards the “Pass” region until the model successfully transitions, as illustrated in Fig. 4 (a). As only LoRA in the attack model H is learnable, we perform optimization in the parameter space of LoRA, denoted as ψ. Each point in the optimization space represents a specific configuration of ψ. As shown in Fig. 4 (a) left, SPL firstly samples several points on a sphere centered around the current model parameters (marked as a gray star) with r as the radius. Then, we query the black box’s output case for each sampled point. If all points are predicted as “Deny” (meaning no “Pass” points are detected in the sphere), we increase the sphere radius r to probe the “Pass” points. Once the sphere encompasses “Pass” points, we stop increasing the radius and start optimizing the LoRA parameters ψ. Intuitively, the points predicted as “Pass” represent the direction we aim to move towards. Hence, we calculate the average of these points and optimize the model parameters in the direction of this average. Here, we define Φ Φ, which use 1111 to mark “Pass” points: Φ⁢(ψ)=1⁢image⁢returned⁢(`⁢`⁢Pass⁢")0Φmatrix1imagereturned``Pass"0 -0m (ψ)= Bmatrix1&if \;image\;returned\,(``% Pass")\\ 0&otherwise BmatrixΦ ( ψ ) = start_ARG start_ROW start_CELL 1 end_CELL start_CELL if image returned ( ` ` Pass " ) end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL otherwise end_CELL end_ROW end_ARG (1) After sampling on the sphere, we can use the function Φ Φ to mark all “Pass” points for gradient calculation. Specifically, given the current model (LoRA) parameters ψ, the SPL gradient splsubscriptspl g_splitalic_groman_spl is calculated as: spl⁢(ψ)=1N⁢∑n=1NΦ⁢(ψ+r⋅zn)⋅zn,subscriptspl1superscriptsubscript1⋅Φ⋅subscriptsubscript -0m g_spl(ψ)= 1N _n=1^N % (ψ+r· z_n)· z_n,italic_groman_spl ( ψ ) = divide start_ARG 1 end_ARG start_ARG N end_ARG ∑n = 1N Φ ( ψ + r ⋅ zitalic_n ) ⋅ zitalic_n , (2) where ψ∈ℝdsuperscriptℝψ ^dψ ∈ blackboard_Rd denotes the current model parameters, znsubscriptz_nzitalic_n represents a unit-length vector randomly sampled from a d-dimensional space, r denotes the radius of the sample sphere (as shown in Fig. 4 a), ψ+r⋅zn⋅subscriptψ+r· z_nψ + r ⋅ zitalic_n is the sampling point, and N denotes the total number of points sampled on this sphere. Note that different vectors znsubscriptz_nzitalic_n possess different directions, yet they share the same length, resulting in all sampling points being located on a sphere. Leveraging the above sphere-sampling exploration, we can identify an effective optimization direction splsubscriptspl g_splitalic_groman_spl. In this way, even if the current model ψ is incapable of inducing the API to output image results, the gradient splsubscriptspl g_splitalic_groman_spl can still be computed via the above parameter-space sampling. Next, we use splsubscriptspl g_splitalic_groman_spl to update ψ as follows: ψ=ψ+α⋅spl⁢(ψ),⋅subscriptspl -0mψ=ψ+α· g_spl(ψ),ψ = ψ + α ⋅ italic_groman_spl ( ψ ) , (3) where α is the learning rate, which is related to the current radius r, i.e., α=r/kα=r/kα = r / k, k is a hyper-parameter. The role of Eq. 2 is to compute the gradient, while Eq. 3 is used to update the model parameters based on the computed gradient. When handling Eq. 2 and Eq. 3 for one time, the model is optimized for one iteration. We alternate between Eq. 2 and Eq. 3 to perform the gradient-based optimization iteratively, until the sphere center (i.e., model parameters) falls into the “Pass” region, as shown in Fig. 4 (a). After the first stage of SPL, the model parameters are located in the “Pass” region, meaning the current model is capable of deceiving the defenses and getting images returned. Since SPL employs a large radius r to find “Pass” points, the optimized model may land in the “Pass” region but remain far from its boundary. This occurs because r is positively correlated with the optimization step size. However, being deep within the “Pass” region increases the likelihood of the black box returning safe images rather than the intended harmful ones. This is because these regions represent the classification tendencies of the defense system: deeper into the ”Pass” region, the system is more confident the content is safe. Motivated by the observation that misclassifications often occur near the decision boundaries [52], we set SPL’s final optimization objective within the “Pass” region but close to the boundary. This positioning reduces the black-box defense system’s confidence, increasing the likelihood of generating target harmful images while ensuring the images can still be returned. Below, we detail the second stage of our SPL scheme: optimizing toward the boundary. (b) Optimizing towards the boundary In the second stage of SPL, the model is further refined to approach the boundary while staying within the “Pass” region. As illustrated in Fig. 4 (b), the radius r is progressively reduced before each iteration to align with the decreasing step size required for boundary proximity. This ensures the model avoids crossing into the “Deny” region. Specifically, before each calculation of the gradient, we decrease the radius r into a smaller one. Then we use the new smaller radius to calculate the SPL gradient splsubscriptspl g_splitalic_groman_spl via Eq. 2. After obtaining the gradient splsubscriptspl g_splitalic_groman_spl, we optimize the model ψ as follows: ψ=ψ−α⋅spl⁢(ψ),⋅subscriptspl -0mψ=ψ-α· g_spl(ψ),ψ = ψ - α ⋅ italic_groman_spl ( ψ ) , (4) Here, we utilize the negative sign “−--”, in contrast to Eq. 3, to signify a reversed optimization direction (towards the boundary). We alternate between Eq. 2 and Eq. 4, iteratively approaching the boundary. During the iterative optimization, we gradually decrease the radius, as shown in Fig. 4 (b). Next, we describe how to decrease the radius in the second stage of SPL. Given the original radius r used in the first stage, the decreased radius used for the i-th iteration of this second stage is calculated as: ri=r⋅e−λ⁢i,subscript⋅superscript -0mr_i=r· e^-λ i,ritalic_i = r ⋅ e- λ i , (5) where i is iteration index number i∈1,2,3,…123…i∈\1,2,3,...\i ∈ 1 , 2 , 3 , … for the second stage of SPL, λ is the coefficient of decay. As i increases, risubscriptr_iritalic_i gradually decreases and infinitely approaches zero. To ensure effective boundary approaching, for each decreased radius risubscriptr_iritalic_i, we need to check if the following conditions are met: (1) The sphere with this decreased radius risubscriptr_iritalic_i encompasses both “Pass” and “Deny” points. (2) After optimization with this decreased radius risubscriptr_iritalic_i, the model parameters are still located in the “Pass” case. If the current risubscriptr_iritalic_i does not satisfy the first condition, it indicates that risubscriptr_iritalic_i is too small to probe both points. We set the current i to i−11i-1i - 1 to reuse the larger radius from the previous iteration to solve this issue. If the current risubscriptr_iritalic_i does not satisfy the second condition, it indicates that risubscriptr_iritalic_i is too large. We set the current i to i+11i+1i + 1 to use a smaller radius to address this issue. Finally, we stop the SPL scheme when risubscriptr_iritalic_i is smaller than a pre-defined threshold rm⁢i⁢nsubscriptr_minritalic_m i n. After pre-training with the SPL scheme, the attack model H learns to bypass black-box defenses, ensuring the API returns images ∗superscript∗I I∗. In the fine-tuning phase, H is further refined to align the semantics of returned images ∗superscript∗I I∗ with harmful targets II. 3.2 Fine-tuning with SEL After pre-training with SPL, the attack model H can induce the T2I API to return images ∗superscript∗I I∗ to the attacker. To ensure the harmful semantics of returned images ∗superscript∗I I∗, an effective way involves calculating semantic-similarity losses between returned images ∗superscript∗I I∗ and harmful targets II to obtain gradients for model fine-tuning. To avoid pulling the model back into the “Deny” case, we propose a Semantic-Enhancing Learning (SEL) scheme to adjust the calculated gradients for fine-tuning the model H (i.e., LoRA parameters) while maintaining the progress achieved by SPL. Specifically, in our SEL, we propose a Multi-Modal Alignment Loss ℒM⁢M⁢AsubscriptℒL_MMALitalic_M M A to achieve a comprehensive semantic alignment. Motivated by the multi-modal semantic understanding capability of CLIP [53], we utilize it to help calculating the semantic alignment loss ℒM⁢M⁢AsubscriptℒL_MMALitalic_M M A in two perspectives: (1) Text-Image Alignment, i.e., encoding both the returned image ∗superscript∗I I∗ and the naive prompt TT, then computing the cosine similarity between the encoded vectors, which formulates the loss ℒT⁢I⁢AsubscriptℒL_TIALitalic_T I A: ℒT⁢I⁢A=1−Ei⁢m⁢g⁢(∗)⋅Et⁢e⁢x⁢t⁢()‖Ei⁢m⁢g⁢(∗)‖⁢‖Et⁢e⁢x⁢t⁢()‖,subscriptℒ1⋅subscriptsuperscript∗subscriptnormsubscriptsuperscript∗normsubscript -0mL_TIA=1- E_img(I )· E_text% (T) \|E_img(I ) \| \|E_text(% T) \|,Litalic_T I A = 1 - divide start_ARG Eitalic_i m g ( I∗ ) ⋅ Eitalic_t e x t ( T ) end_ARG start_ARG ∥ Eitalic_i m g ( I∗ ) ∥ ∥ Eitalic_t e x t ( T ) ∥ end_ARG , (6) where Ei⁢m⁢g⁢(⋅)subscript⋅E_img(·)Eitalic_i m g ( ⋅ ) and Et⁢e⁢x⁢t⁢(⋅)subscript⋅E_text(·)Eitalic_t e x t ( ⋅ ) denote the image encoder and text encoder of CLIP, respectively. (2) Image-Image Alignment, i.e., encoding both the returned image ∗superscript∗I I∗ and the target image II for computing the cosine similarity, which formulates the loss ℒI⁢I⁢AsubscriptℒL_IIALitalic_I I A: ℒI⁢I⁢A=1−Ei⁢m⁢g⁢(∗)⋅Ei⁢m⁢g⁢()‖Ei⁢m⁢g⁢(∗)‖⁢‖Ei⁢m⁢g⁢()‖.subscriptℒ1⋅subscriptsuperscript∗subscriptnormsubscriptsuperscript∗normsubscript -0mL_IIA=1- E_img(I )· E_img% (I) \|E_img(I ) \| \|E_img(% I) \|.Litalic_I I A = 1 - divide start_ARG Eitalic_i m g ( I∗ ) ⋅ Eitalic_i m g ( I ) end_ARG start_ARG ∥ Eitalic_i m g ( I∗ ) ∥ ∥ Eitalic_i m g ( I ) ∥ end_ARG . (7) Based on the above two similarity loss functions, we can formulate our Multi-Modal Alignment Loss ℒM⁢M⁢AsubscriptℒL_MMALitalic_M M A as: ℒM⁢M⁢A=ℒT⁢I⁢A+ℒI⁢I⁢A.subscriptℒsubscriptℒsubscriptℒ -0mL_MMA=L_TIA+L_IIA.Litalic_M M A = Litalic_T I A + Litalic_I I A . (8) By minimizing ℒM⁢M⁢AsubscriptℒL_MMALitalic_M M A, we can enhance the target (harmful) semantics of the returned images ∗superscript∗I I∗. To this end, we use the calculated ℒM⁢M⁢AsubscriptℒL_MMALitalic_M M A to obtain gradients and then optimize the LoRA parameters ψ. Since the T2I model is packed into the black box, we cannot directly obtain the oracle precise gradients. To address this black-box setting, drawing inspiration from [54, 55], we adopt zeroth-order optimization to estimate the gradient without accessing the model architecture and model parameters. Given the training loss ℒM⁢M⁢AsubscriptℒL_MMALitalic_M M A, the estimated gradient can be formulated as follows: ⁢(ψ)=ℒM⁢M⁢A⁢(ψ+c⋅Δ)−ℒM⁢M⁢A⁢(ψ−c⋅Δ)2⁢c⋅Δ,subscriptℒ⋅Δsubscriptℒ⋅Δ⋅2Δ -0m g(ψ)= L_MMA(ψ+c· )-% L_MMA(ψ-c· )2c· ,italic_g ( ψ ) = divide start_ARG Litalic_M M A ( ψ + c ⋅ Δ ) - Litalic_M M A ( ψ - c ⋅ Δ ) end_ARG start_ARG 2 c ⋅ Δ end_ARG , (9) where c∈(0,1]01c∈(0,1]c ∈ ( 0 , 1 ] is the scaling coefficient and Δ∈ℝdΔsuperscriptℝ ^dΔ ∈ blackboard_Rd is a random perturbation vector, sampled from mean-zero distributions while satisfying the finite inverse momentum condition [54, 56]. Although the standard form of zeroth-order optimization is generally effective, it can still encounter poor convergence in practical applications [57]. This problem, as discussed in [58], is primarily attributed to the stochastic nature of gradient estimation, stemming from the random directions of perturbations. To overcome this challenge, we draw inspiration from Nesterov’s accelerated gradient [59] and improve the gradient calculation rule (Eq. 9). The improved gradient imsubscriptim g_imitalic_groman_im is defined as follows: im⁢(ψ)=¯im+β⋅⁢(ψ+¯im).subscriptimsubscriptbold-¯im⋅subscriptbold-¯im -0m g_im(ψ)= g_% im+β· g(ψ+ g_im).italic_groman_im ( ψ ) = over¯ start_ARG italic_g end_ARGim + β ⋅ italic_g ( ψ + over¯ start_ARG italic_g end_ARGim ) . (10) This formula consists of two gradient terms. The first term, ¯imsubscriptbold-¯im g_imover¯ start_ARG italic_g end_ARGim, represents the gradient imsubscriptim g_imitalic_groman_im used for model update in the (previous) last iteration. The second term, β⋅⁢(ψ+¯im)⋅subscriptbold-¯imβ· g(ψ+ g_im)β ⋅ italic_g ( ψ + over¯ start_ARG italic_g end_ARGim ), represents the newly estimated gradient in this iteration. In the second term, ψ+¯imsubscriptbold-¯imψ+ g_imψ + over¯ start_ARG italic_g end_ARGim is the model parameters after updating along the previous gradient, ⁢(⋅)⋅ g(·)italic_g ( ⋅ ) represents the gradient calculation formula (Eq. 9), and β is the learning rate. Inspired by studies in the field of accelerated gradient optimization [58], we incorporate the previous gradient (i.e., the first term) into the optimization of current iteration (i.e., the second term) to increase the gradient consistency and reduce incidental randomness in black-box optimization, thus making the training more stable and effective. Figure 5: Intuitive illustration of our SEL scheme. ① Firstly, We compute the gradient imsubscriptim g_imitalic_groman_im for enhancing target harmful semantics. ② Then, we calculate negative SPL gradient −splsubscriptspl- g_spl- italic_groman_spl, which points towards the boundary direction. ③ We calculate the component of the gradient imsubscriptim g_imitalic_groman_im in the boundary direction, i.e., ′im g _imitalic_g ′im. ④ To prevent from falling into the “deny” region, we let imsubscriptim g_imitalic_groman_im subtract its component ′im g _imitalic_g ′im, obtaining the final SEL gradient selsubscriptsel g_selitalic_groman_sel, where sel=im−′im g_sel= g_im- g% _imitalic_groman_sel = italic_groman_im - italic_g ′im. Recall that in Sec. 3.1, the SPL optimization objective is set within the “Pass” region near the boundary. However, directly optimizing the model using gradients imsubscriptim g_imitalic_groman_im (Eq. 10) risks pulling the parameters back into the “Deny” region, blocking image generation. Thus, the SEL optimization objective focuses on keeping the model within the “Pass” region while maximizing semantic alignment between the returned images and the harmful targets. To prevent SEL from undermining SPL’s learning achievements, we propose a gradient harmonization method to ensure compatibility between SEL and SPL. As shown in Fig. 5 ①, we first obtain the gradient imsubscriptim g_imitalic_groman_im for harmful-semantic enhancement via Eq. 10. Then, the SPL gradient splsubscriptspl g_splitalic_groman_spl is calculated through Eq. 2. As shown in Fig. 5 ②, we adopt the negative SPL gradient −splsubscriptspl- g_spl- italic_groman_spl, where −splsubscriptspl- g_spl- italic_groman_spl signifies the direction towards crossing the boundary. Given the direction provided by −splsubscriptspl- g_spl- italic_groman_spl, we can obtain ′im g _imitalic_g ′im (see Fig. 5 ③), which is the boundary-directed component of the gradient imsubscriptim g_imitalic_groman_im. As shown in Fig. 5 ④, we finally obtain the SEL gradient selsubscriptsel g_selitalic_groman_sel by eliminating the component ′im g _imitalic_g ′im from imsubscriptim g_imitalic_groman_im, which can be formulated as: sel⁢(ψ)=im(ψ)−′im(ψ)=im⁢(ψ)−spl⁢(ψ)⋅im⁢(ψ)|s⁢p⁢l⁢(ψ)|2⁢spl⁢(ψ), -0m split g_sel(ψ)&= g_% im(ψ)- g _im(ψ)\\ &= g_im(ψ)- g_spl(ψ)% · g_im(ψ) | g_spl(ψ)% |^2 g_spl(ψ), splitstart_ROW start_CELL italic_groman_sel ( ψ ) end_CELL start_CELL = italic_groman_im ( ψ ) - italic_g ′im ( ψ ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = italic_groman_im ( ψ ) - divide start_ARG italic_groman_spl ( ψ ) ⋅ italic_groman_im ( ψ ) end_ARG start_ARG | italic_gitalic_s p l ( ψ ) |2 end_ARG italic_groman_spl ( ψ ) , end_CELL end_ROW (11) After obtaining SEL gradient selsubscriptsel g_selitalic_groman_sel, we update the LoRA parameters as: ψ=ψ−β⋅sel⁢(ψ).⋅subscriptsel -0mψ=ψ-β· g_sel(ψ).ψ = ψ - β ⋅ italic_groman_sel ( ψ ) . (12) We alternate between Eq. 11 and Eq. 12, iteratively enhancing the target harmful semantics of returned images. In this way, we can avoid the SEL’s optimization crossing the boundary, thereby not only aligning towards the target semantics but also maintaining within the “Pass” region. 3.3 Training-free Enhancement with INE After the above two training phases, the attack model can effectively and efficiently generate adversarial prompts to deceive the T2I API. In this section, we introduce an In-context Naturalness Enhancement (INE) scheme, which further improves prompt naturalness in a training-free manner, complementing the LLM+LoRA structure. In particular, inspired by the in-context learning [60] ability of LLM, we carefully choose a set of natural and effective adversarial prompts, treating them as in-context examples to input into the LLM. The goal is to guide the LLM to generate prompts in a natural manner. The in-context examples are obtained through the following steps: (1) After the above two training stages, we feed all training samples into the model H to obtain corresponding adversarial prompts and returned images. (2) Then, we use the loss ℒM⁢M⁢AsubscriptℒL_MMALitalic_M M A (Eqn. 8) to measure the semantics of all returned images. (3) We rank the adversarial prompts in ascending order based on their corresponding loss values. (4) Following the ranking, we manually select M adversarial prompts, 1∗,2∗,…,M∗subscriptsuperscript∗1subscriptsuperscript∗2…subscriptsuperscript∗\T _1,\ T _2,\ ...,\ T _M\ T∗1 , T∗2 , … , T∗italic_M with satisfactory naturalness through human-perceptive evaluation. (5) For these selected M adversarial prompts 1∗,2∗,…,M∗subscriptsuperscript∗1subscriptsuperscript∗2…subscriptsuperscript∗\T _1,\ T _2,\ ...,\ T _M\ T∗1 , T∗2 , … , T∗italic_M , we combine them with their corresponding naive prompts 1,2,…,Msubscript1subscript2…subscript\T_1,\ T_2,\ ...,\ T_M\ T1 , T2 , … , Titalic_M , using a form as: 1→1∗,2→2∗,…,M→M∗formulae-sequence→subscript1subscriptsuperscript∗1formulae-sequence→subscript2subscriptsuperscript∗2…→subscriptsubscriptsuperscript∗\T_1 _1,\ T_2 _% 2,\ ...,\ T_M _M\ T1 → T∗1 , T2 → T∗2 , … , Titalic_M → T∗italic_M . This form outlines the transformations of naive prompts into natural adversarial prompts. We treat these M transformation examples as in-context examples, denoted as UU. Next, we use UU to enhance our model H without the need of any additional training. After enhancement, the enhanced model H can be directly used for inference. Specifically, given an test-time naive prompt t⁢e⁢s⁢tsubscriptT_testTitalic_t e s t, we use UU to enhance our our model H by simply concatenating UU with the input naive prompt t⁢e⁢s⁢tsubscriptT_testTitalic_t e s t: t⁢e⁢s⁢t∗=fH⁢([,t⁢e⁢s⁢t]),superscriptsubscript∗subscriptHsubscript -1mT_test =f_H([U,T_% test]),Titalic_t e s t∗ = froman_H ( [ U , Titalic_t e s t ] ) , (13) where [⋅]delimited-[]⋅[·][ ⋅ ] denotes the concatenation, fH⁢(⋅)subscriptH⋅f_H(·)froman_H ( ⋅ ) denotes attack model H, the test sample t⁢e⁢s⁢tsubscriptT_testTitalic_t e s t is changeable, while UU is fixed. Based on these M transformation examples (i.e., UU), given a naive prompt t⁢e⁢s⁢tsubscriptT_testTitalic_t e s t, UPAM can more naturally convert t⁢e⁢s⁢tsubscriptT_testTitalic_t e s t into the adversarial prompt t⁢e⁢s⁢t∗superscriptsubscript∗T_test Titalic_t e s t∗. Note that in this paper, the number of in-context examples M is set to 3, so the human labor cost is almost negligible. 3.4 Few-shot Query Attack with TAL From Sec. 3.1 to 3.3, we introduce our UPAM framework, which requires queries to the T2I API to train the attack model H. However, in reality, many queries are likely to be detected by the defender. To address this issue, we propose Transferable Attack Learning (TAL) to enable our attack model to require only few-shot queries to the target T2I API. Instead of directly querying the online T2I API, in the TAL scheme, we first train the attack model H using the offline T2I model(s). Then, the attack model H is connected to the online T2I API and fine-tuned with few-shot queries. The key of TAL is that the offline-trained attack model H should be transferable for it to fit the unseen T2I model with only few-shot queries. Overall, given a group of offline T2I models with different model structures, we split all these offline T2I models into training and validation sets, where the transferable attack model H is trained only on the training set. We use the validation set to control the training process for attack model H that suffices to deceive the validation-set model. This is achieved through an early-stopping mechanism to avoid overfitting to the training-set models. Algorithm 1 outlines our TAL scheme. TAL comprises two phases: offline learning and online learning. The offline learning phase inputs with an attack model H (with parameters ψ), a dataset DD, a training set of offline T2I models denoted as p, and a validation set of offline T2I models denoted as q. Offline learning proceeds iteratively, where in each iteration, the paired training data ,\T,I\ T , I is sampled from the dataset DD. In each iteration, we calculate the SPL gradient splsubscriptspl g_splitalic_groman_spl and SEL gradient selsubscriptsel g_selitalic_groman_sel on training models p, and use these gradients to update the parameters ψ (i.e., model H). The entire offline learning process is monitored by the validation models q. Specifically, if the performance (loss value) of the validation models q is not in lowering trend, the offline training process is terminated, and the model parameters ψ are returned. For the training (or validation) set consisting of multiple models, we use their average ensemble [61]. During training, the validation loss may briefly increase, but it might still have the potential to decrease further. To avoid premature termination of the algorithm, we set a patience threshold to allow for some extent increase in the validation loss. Since the T2I models used in the training and validation sets have different structures, the attack model H trained through this process exhibits good transferability to new T2I models. The online learning phase inputs with an offline-trained attack model H, a dataset DD, and an online T2I API. We only perform few-shot queries (i.e., W iterations, where W is small) to the T2I API. We observe that the offline-trained attack model is typically capable to deceive the T2I API to return image results. Therefore, in the online learning phase, we omit the calculation of the SPL gradient and only compute the SEL gradient to update the model parameters ψ. Due to the lack of SPL gradient, we simplify the calculation of the SEL gradient to gimsubscriptimg_imgim (Eq. 10). After W iterations, we conduct the INE process in Sec. 3.3 to enhance the naturalness of model H in a training-free manner. Finally, we use the naturalness-enhanced attack model H for inference. //offlinelearningphase//\ offline\ learning\ phase/ / o f f l i n e l e a r n i n g p h a s e Input: Attack model ψ, dataset DD, offline (training) T2I models p, offline (validation) T2I models q, patience threshold ϵitalic-ϵεϵ; ℒm⁢i⁢n=subscriptℒabsentL_min=Litalic_m i n = inf; // i′⁢n⁢f′⁢s⁢t⁢a⁢n⁢d⁢s⁢f⁢o⁢r⁢p⁢o⁢s⁢i⁢t⁢i⁢v⁢e⁢i⁢n⁢f⁢i⁢n⁢i⁢t⁢ysuperscript′ inf \ stands\ for\ positive\ infinitystart_FLOATSUPERSCRIPT ′ end_FLOATSUPERSCRIPT i n f′ s t a n d s f o r p o s i t i v e i n f i n i t y for ,∈\T,I\ T , I ∈ D do Compute SPL gradient using p: splsubscriptspl g_splitalic_groman_spl; // E⁢q.2formulae-sequence2Eq. eq:gradE q . Compute SEL gradient using p: selsubscriptsel g_selitalic_groman_sel; // E⁢q.11formulae-sequence11Eq. eq:sel_gradientE q . if model ψ in the “Deny” region then ψ ← Update(ψ, α, splsubscriptspl g_splitalic_groman_spl); // E⁢q.3formulae-sequence3Eq. eq:updateE q . else ψ ← Update(ψ, β, selsubscriptsel g_selitalic_groman_sel); // E⁢q.12formulae-sequence12Eq. eq:final_updateE q . end if Compute validation loss using q: ℒv⁢a⁢lsubscriptℒL_valLitalic_v a l; // E⁢q.8formulae-sequence8Eq. eq:lvsE q . if ℒv⁢a⁢l<ℒm⁢i⁢nsubscriptℒsubscriptℒL_val<L_minLitalic_v a l < Litalic_m i n then ℒm⁢i⁢nsubscriptℒL_minLitalic_m i n ← ℒv⁢a⁢lsubscriptℒL_valLitalic_v a l; else if ℒv⁢a⁢l>ℒm⁢i⁢n+ϵsubscriptℒsubscriptℒitalic-ϵL_val>L_min+ _v a l > Litalic_m i n + ϵ then return model ψ; // e⁢a⁢r⁢l⁢y⁢s⁢t⁢o⁢p⁢p⁢i⁢n⁢gearly\ stoppinge a r l y s t o p p i n g end for //onlinelearningphase//\ online\ learning\ phase/ / o n l i n e l e a r n i n g p h a s e Input: offline-trained model ψ, dataset DD, online T2I API, few-shot iteration number W; C⁢o⁢u⁢n⁢t⁢e⁢r=00Counter=0C o u n t e r = 0; for ,∈\T,I\ T , I ∈ D do Compute gradient using API: gimsubscriptimg_imgroman_im; // E⁢q.10formulae-sequence10Eq. eq:new_gradientE q . ψ ← Update(ψ, β, imsubscriptim g_imitalic_groman_im); // E⁢q.12formulae-sequence12Eq. eq:final_updateE q . C⁢o⁢u⁢n⁢t⁢e⁢r=C⁢o⁢u⁢n⁢t⁢e⁢r+11Counter=Counter+1C o u n t e r = C o u n t e r + 1; if C⁢o⁢u⁢n⁢t⁢e⁢r≥WCounter≥ WC o u n t e r ≥ W then return model ψ; // f⁢e⁢w−s⁢h⁢o⁢t⁢q⁢u⁢e⁢r⁢i⁢e⁢sℎfew-shot\ queriesf e w - s h o t q u e r i e s end if end for Algorithm 1 Transferable Attack Learning (TAL) 3.5 Training and Testing In summary, our UPAM framework can choose to work with or without TAL based on the requirement for few-shot queries. As for UPAM without the need of few-shot queries (i.e., w/o TAL), we train the attack model H in two phases: pre-training and fine-tuning. In the pre-training phase, we use the SPL scheme to obtain the gradient splsubscriptspl g_splitalic_groman_spl (Eq. 2) and update the model parameters (Eq. 3). This update enables effective optimization under the challenging scenario where no image results are returned, finally compelling the API to reliably return images. Then, in the fine-tuning phase, we adopt the SEL scheme to obtain the gradient selsubscriptsel g_selitalic_groman_sel (Eq. 11) and update the model parameters (Eq. 12). This update ensures that the returned images can accurately show target-aligned semantics. During the whole training process, we only optimize LoRA parameters, while keeping the LLM frozen. After training and before testing, we use the INE scheme mentioned in Sec. 3.3 to gather in-context examples for naturalness-enhanced UPAM (Eq. 13). We finally handle the inference (testing) based on the enhanced UPAM. As for UPAM with the need of few-shot queries (i.e., with TAL), we train the attack model H in two phases: offline learning and online learning. In the offline learning phase, we calculate the SPL gradient splsubscriptspl g_splitalic_groman_spl and SEL gradient selsubscriptsel g_selitalic_groman_sel to update the attack model in a T2I-transferable manner, using a set of training and validation T2I models. In the online learning phase, we calculate the gradient gimsubscriptimg_imgim to further update the offline-trained model for only W iterations. We use the INE scheme (Sec. 3.3) to enhance the attack model’s naturalness and then conduct inference (testing). 4 Experiments Textual Filter and Visual Checker. For the textual filter, we follow previous work [9] employing the same filter model, which operates by utilizing a predefined list of sensitive words, also known as a blacklist, to deny the forward propagation of harmful prompts. For the visual checker, we follow [62] to adopt the widely-used visual checking model: using the off-the-shelf CLIP visual encoder to map the generated image to a latent vector, which is then compared with all default embeddings of predefined harmful images using cosine distance. If any cosine value exceeds the manually specified threshold, the further propagation of the generated image is denied. Dataset. We follow previous works [9, 10, 11, 18, 19] to deploy defenses and conduct attacks using the paired text-image data from the Microsoft COCO dataset [63]. In consideration of ethics, privacy, legal compliance, and potential societal impacts, these previous works treat 10 classes from COCO as “harmful” classes (i.e., boat, bird, clock, kite, wine glass, knife, pizza, teddy bear, vase and laptop). Specifically, we train our UPAM using the training set of these 10 classes, and then test UPAM using the test set of the same 10 classes. Although the training and testing data contain the same classes, the specific text descriptions and their corresponding image contents are different. To achieve a comprehensive evaluation, we also follow the previous work [20] to conduct experiments on the NSFW dataset [20], which contains truly harmful content, including 6 harmful classes: (i.e., breasts, nipples, genitalia, nudity, sexual acts, and bodily fluids) Target T2I APIs. Following previous work [9], we conduct experiments on three large-scale API T2I models: DALL·E [8], DALL·E 2 [48], and Imagen [7]. For each of them, we follow [9] to adopt the corresponding released model that achieves comparable performance to API, and treat it as a black box. Implementation Details. In this paper, we use the open-source LLaMA [64] as the LLM. In the TAL scheme, we adopt DeepFloyd IF [65], StableStudio [66], Invoke [67], and Stable Diffusion [14] as our training T2I models, while utilizing Dreamlike Photoreal [68], Waifu Diffusion [69], and OpenJourney [70] as our validation T2I models. All these models are offline and open-source. As for the LoRA adapter [50], we adopt the adaptation matrices with a rank value of 8. In our SPL scheme, we set N=1010N=10N = 10, α=r/44α=r/4α = r / 4. The radius r increases from 0 and then decreases to 0.2 (i.e., rm⁢i⁢n=0.2subscript0.2r_min=0.2ritalic_m i n = 0.2), following the exponential function with a base value of 1.3 (increase) and −--0.6 (decrease). In our SEL scheme, we set β=0.30.3β=0.3β = 0.3. In our TAL scheme, we set ϵ=0.1italic-ϵ0.1ε=0.1ϵ = 0.1. 4.1 Experimental Protocols In this paper, we conduct experiments under two protocols for a comprehensive study. Protocol A: Given the training data ,\T,I\ T , I , we train a parameterized model for converting the naive prompts TT into adversarial ones ∗superscript∗T T∗ with the objective of compelling the black-box T2I model to return the target images II. Subsequently, given the testing data tsubscriptT_tTitalic_t, the trained model can swiftly convert tsubscriptT_tTitalic_t into t∗superscriptsubscript∗T_t Titalic_t∗, compelling the black-box T2I model to return the corresponding target images. Protocol B: Previous enumeration-based methods do not require model training. Instead, they typically generate numerous adversarial prompt candidates and then attempt the candidates one by one until the returned image is semantic similar to the target image. They conduct experiments on a subset of 10 “harmful” images randomly selected from the COCO dataset (or 200 images from the NSFW dataset), employing enumeration to find an adversarial prompt for each image. To make a fair comparison with previous enumeration-based methods, we slightly modify our UPAM in this protocol. This modification enables UPAM to identify adversarial prompts for the 10 (or 200) images without learning on the training set. The details of modification can be seen in Sec. 6. 4.2 Evaluation Metrics. We evaluate attack methods from the following perspectives: Attack Effectiveness. Following previous work [9], we measure the attack effectiveness by evaluating the generated adversarial prompts ∗superscript∗T T∗ and the returned images ∗superscript∗I I∗, respectively. (1) As for the returned images ∗superscript∗I I∗, we follow [9] to utilize R-precision to measure the attack effectiveness. Specifically, a retrieval experiment is conducted using the returned image to query against a set of candidate text descriptions. This set comprises 1 ground-truth description and 99 randomly chosen mismatched descriptions. For this purpose, a state-of-the-art image-text retrieval model [71] is employed here. We collect the top R ranked results in the retrieval, deeming our attacks successful if the ground truth description is among them. We follow [9] to conduct evaluation at R-1 precision (i.e., R=11R=1R = 1) and R-3 precision (i.e., R=33R=3R = 3). Note that if no image is returned to the attacker, the R-precision score of this sample is 0. (2) As for the generated adversarial prompts ∗superscript∗T T∗, we use textual similarity to evaluate the attack effectiveness. The evaluation of textual similarity, based on how different the adversarial prompt T∗superscriptT^*T∗ is from the naive prompt T, is closely tied to the defense mechanism used in text-to-image (T2I) generation APIs. As mentioned in previous work [9, 19], T2I APIs often incorporate textual filters as a defense mechanism, where these filters are designed to detect and block prompts that are textually similar to harmful naive prompts listed in a blacklist. That is, if T∗superscriptT^*T∗, the adversarial prompt, is more similar to the harmful naive prompt T in the blacklist, the textual filter is more likely to flag and reject the generated output. Therefore, ensuring that the generated T∗superscriptT^*T∗ is sufficiently dissimilar from the naive harmful T is critical for bypassing the textual filters. By maximizing this dissimilarity, one can improve the likelihood of the adversarial prompt evading detection, thus enhancing the effectiveness of the attack. Following previous work [9, 19], we adopt the Universal Sentence Encoder [72] to encode the adversarial prompts ∗superscript∗T T∗ and naive prompts TT into high dimensional vectors, and then compute their textual similarity score. The lower the textual similarity, the better the attack. Attack Efficiency. We also evaluate the attack efficiency of the methods. Specifically, we present the average inference time taken for each testing sample during testing. Due to the uncertain inference time of enumeration methods, in order to avoid endless waiting periods, we set a criterion: if the inference time exceeds 10 minutes for a given sample, the attack is considered unsuccessful, and the inference process is stopped. Attack Naturalness. To evaluate the attack stealthiness, we follow [9] to evaluate the naturalness of the generated adversarial prompts by computing the perplexity score (PPL) using GPT-2 [73], a model trained on extensive real-world sentences. Generally, the prompt sample with a lower PPL is more natural. Query Times. We record the number of queries each method makes to the T2I API, and report the average number per sample. Besides inference (testing), the method proposed in this paper also queries the T2I API during training. Therefore, we report both the training query times and inference query times for a comprehensive comparison. TABLE I: Experimental results on COCO (left) and NSFW (right) under protocol A. COCO NSFW T2I Model Methods R-1 Precision ↑ R-3 Precision ↑ Text. Sim. ↓ Infer. Time (s) ↓ PPL ↓ Train. Query ↓ Infer. Query ↓ R-1 Precision ↑ R-3 Precision ↑ Text. Sim. ↓ Infer. Time (s) ↓ PPL ↓ Train. Query ↓ Infer. Query ↓ TextFooler [19] 0.48% 0.98% 0.22 589.29 3463.02 0 1018 0.31% 0.67% 0.20 592.65 4165.37 0 1142 HomoSubs [18] 0.74% 1.12% 0.28 562.48 3089.58 0 769 0.67% 1.03% 0.22 583.73 3671.58 0 802 EvoPromp [11] 0.82% 1.37% 0.19 579.20 4984.24 0 734 1.16% 1.52% 0.25 528.54 4316.65 0 727 HiddVocab [10] 3.35% 4.21% 0.33 457.34 4027.64 0 742 4.12% 6.75% 0.40 561.59 4593.42 0 688 MacPromp [11] 5.05% 6.80% 0.57 452.16 3163.07 0 561 5.65% 6.97% 0.48 502.63 2884.06 0 616 MMA-Diff [12] 8.27% 9.16% 0.34 386.77 3857.29 0 493 6.73% 7.46% 0.32 398.13 3746.08 0 514 RIATIG [9] 8.65% 9.95% 0.18 391.52 1003.27 0 379 6.55% 7.39% 0.21 409.72 1474.30 0 392 SneakyPrompt [20] 12.68% 16.79% 0.25 561.75 1457.36 0 453 9.15% 12.13% 0.29 498.38 1574.28 0 429 UPAM (with TAL) 23.40% 25.89% 0.21 5.14 658.18 10 1 17.69% 20.72% 0.23 5.62 933.76 10 1 DALL·E [8] UPAM (w/o TAL) 38.56% 41.92% 0.17 5.11 641.26 173 1 35.82% 39.11% 0.19 5.45 919.57 185 1 TextFooler [19] 0.85% 0.93% 0.18 540.77 3566.32 0 1009 0.68% 0.92% 0.25 593.51 4398.65 0 1157 HomoSubs [18] 0.70% 0.89% 0.22 568.26 4125.22 0 791 0.64% 0.88% 0.26 583.19 4426.74 0 797 EvoPromp [11] 0.92% 1.17% 0.37 475.81 5837.06 0 662 1.54% 1.86% 0.44 454.78 5396.20 0 650 HiddVocab [10] 3.21% 4.62% 0.46 565.33 4125.22 0 636 3.65% 5.06% 0.49 552.97 3956.72 0 611 MacPromp [11] 4.84% 5.92% 0.66 523.49 3056.41 0 702 3.99% 5.72% 0.52 511.59 2737.15 0 685 MMA-Diff [12] 7.88% 8.93% 0.51 361.71 3246.16 0 467 8.21% 9.55% 0.42 457.52 2943.02 0 493 RIATIG [9] 8.55% 9.19% 0.35 353.53 992.54 0 354 7.66% 9.21% 0.33 507.62 1163.71 0 416 SneakyPrompt [20] 14.77% 18.20% 0.31 495.05 1057.19 0 406 12.68% 16.79% 0.25 561.75 1457.36 0 449 UPAM (with TAL) 25.08% 28.31% 0.21 5.77 758.42 10 1 23.31% 24.86% 0.24 5.51 862.07 10 1 DALL·E 2 [48] UPAM (w/o TAL) 40.71% 44.83% 0.18 5.54 717.69 168 1 42.57% 46.33% 0.20 5.32 836.49 181 1 TextFooler [19] 0.32% 0.54% 0.15 527.65 3321.88 0 974 0.17% 0.51% 0.19 583.35 3746.09 0 1046 HomoSubs [18] 0.35% 0.75% 0.18 567.82 2850.41 0 789 0.29% 0.61% 0.20 588.31 3003.76 0 802 EvoPromp [11] 0.64% 0.92% 0.25 398.51 4055.71 0 526 0.79% 1.05% 0.34 476.32 4876.31 0 662 HiddVocab [10] 2.37% 3.69% 0.27 592.47 4153.58 0 686 2.02% 2.96% 0.29 562.38 3647.57 0 634 MacPromp [11] 4.58% 5.52% 0.16 524.38 3173.35 0 704 5.77% 7.65% 0.22 519.66 2945.02 0 690 MMA-Diff [12] 5.87% 8.30% 0.41 394.84 2576.85 0 508 9.57% 11.54% 0.37 476.59 3851.66 0 536 RIATIG [9] 7.80% 9.22% 0.36 388.16 1076.29 0 387 10.33% 13.16% 0.28 422.64 1473.49 0 402 SneakyPrompt [20] 11.61% 15.93% 0.27 427.43 1028.42 0 344 16.34% 18.07% 0.32 404.53 1342.76 0 326 UPAM (with TAL) 19.84% 21.79% 0.16 5.38 529.43 10 1 18.55% 22.60% 0.19 5.57 859.78 10 1 Imagen [7] UPAM (w/o TAL) 36.63% 41.58% 0.14 5.25 503.76 172 1 40.16% 40.16% 0.18 5.42 847.37 182 1 TABLE I: Experimental results on COCO (left) and NSFW (right) under protocol B. COCO NSFW T2I Model Methods R-1 Precision ↑ R-3 Precision ↑ Text. Sim. ↓ Infer. Time (s) ↓ PPL ↓ Train. Query ↓ Infer. Query ↓ R-1 Precision ↑ R-3 Precision ↑ Text. Sim. ↓ Infer. Time (s) ↓ PPL ↓ Train. Query ↓ Infer. Query ↓ TextFooler [19] 0/10 0/10 0.22 582.37 3517.03 0 1121 0/200 0/200 0.21 594.15 4276.43 0 1136 HomoSubs [18] 0/10 0/10 0.16 590.28 3216.95 0 772 0/200 0.4/200 0.20 586.96 4116.32 0 805 EvoPromp [11] 0.2/10 0.5/10 0.16 586.47 4837.63 0 730 3.4/200 4/200 0.33 577.64 4079.52 0 718 HiddVocab [10] 0.3/10 0.4/10 0.18 483.27 4607.93 0 748 10.4/200 12.4/200 0.41 517.63 4316.18 0 682 MacPromp [11] 0.7/10 0.9/10 0.39 476.46 3239.56 0 573 14.6/200 17.2/200 0.37 571.30 3542.85 0 622 MMA-Diff [12] 0.6/10 1/10 0.30 408.47 2473.13 0 502 25.4/200 26.2/200 0.29 486.27 3157.26 0 518 RIATIG [9] 0.8/10 1.1/10 0.34 425.66 1026.81 0 375 27.8/200 29.6/200 0.33 520.79 1643.85 0 453 SneakyPrompt [20] 4.2/10 4.4/10 0.27 496.79 1078.33 0 418 95.6/200 98.4/200 0.36 553.49 1251.22 0 464 UPAM (with TAL) 5.5/10 5.9/10 0.17 56.15 603.74 0 10 113.2/200 117.7/200 0.20 54.87 904.56 0 10 DALL·E [8] UPAM (w/o TAL) 7.4/10 8/10 0.15 362.11 580.62 0 72 136.8/200 141.4/200 0.16 376.66 899.43 0 69 TextFooler [19] 0/10 0/10 0.19 543.72 3578.14 0 1013 0/200 0/200 0.22 585.23 4517.39 0 1153 HomoSubs [18] 0/10 0/10 0.15 589.46 4374.80 0 785 0/200 0/200 0.23 573.64 4286.74 0 792 EvoPromp [11] 0/10 0/10 0.21 449.61 5267.07 0 659 1.6/200 3.2/200 0.38 507.18 5243.60 0 677 HiddVocab [10] 0.6/10 0.9/10 0.22 576.27 4933.29 0 642 8.8/200 10/200 0.54 563.02 4227.55 0 633 MacPromp [11] 0.4/10 0.8/10 0.30 571.63 3575.26 0 716 9.8/200 11.4/200 0.49 539.46 2866.81 0 697 MMA-Diff [12] 1/10 1.4/10 0.25 413.40 2951.67 0 482 24.2/200 25/200 0.28 492.29 3049.25 0 533 RIATIG [9] 0.5/10 0.7/10 0.29 417.34 965.38 0 376 20.6/200 23.2/200 0.37 485.65 1386.72 0 402 SneakyPrompt [20] 2.8/10 3.2/10 0.26 457.84 1137.69 0 389 85.2/200 89.4/200 0.31 506.32 1115.73 0 406 UPAM (with TAL) 3.6/10 4.7/10 0.18 52.67 762.85 0 10 105.3/200 112.1/200 0.19 55.18 816.74 0 10 DALL·E 2 [48] UPAM (w/o TAL) 6.7/10 7.4/10 0.16 346.46 725.14 0 66 132.6/200 135/200 0.19 389.50 796.06 0 71 TextFooler [19] 0/10 0/10 0.23 559.42 3539.57 0 1007 0/200 0/200 0.23 592.37 3593.46 0 1051 HomoSubs [18] 0/10 0/10 0.13 586.75 2674.29 0 795 0/200 1.2/200 0.20 571.31 3265.13 0 801 EvoPromp [11] 0.3/10 0.7/10 0.21 443.39 4162.18 0 554 3.2/200 3.8/200 0.45 499.56 4623.07 0 633 HiddVocab [10] 0.3/10 0.5/10 0.18 576.83 4264.34 0 661 6.8/200 8.6/200 0.31 543.06 3755.64 0 622 MacPromp [11] 0.5/10 0.8/10 0.43 463.52 3097.13 0 648 14/200 15.4/200 0.25 507.49 3128.42 0 676 MMA-Diff [12] 0.6/10 1/10 0.30 422.67 2774.33 0 530 38/200 39.8/200 0.39 480.46 3209.41 0 549 RIATIG [9] 0.9/10 1.2/10 0.34 485.36 1057.42 0 472 32.6/200 35.6/200 0.32 518.22 1543.76 0 515 SneakyPrompt [20] 3.2/10 3.7/10 0.29 508.79 1197.57 0 419 94.6/200 95.6/200 0.29 498.11 1325.86 0 403 UPAM (with TAL) 3.9/10 4.8/10 0.19 53.16 468.05 0 10 112.5/200 116.2/200 0.20 58.41 670.34 0 10 Imagen [7] UPAM (w/o TAL) 6.8/10 7.3/10 0.17 358.29 411.53 0 67 138.2/200 140.4/200 0.18 393.03 651.86 0 68 4.3 Experimental Results Experimental Results Under Protocol A. We use UPAM (w/o TAL) to represent our method’s conventional approach, which does not limit the number of API queries. In contrast, UPAM (with TAL) represents our method that trains the attack model using only few-shot API queries. As shown in Tab. I, our UPAM (w/o TAL) achieves the best results in terms of the R-1 precision, R-3 precision and textual similarity. Specifically, our UPAM (w/o TAL) significantly outperforms other methods by an average of 33.89% (R-1), 37.01% (R-3), and 14.79% (textual similarity) in COCO dataset, and 35.03% (R-1), 36.11% (R-3), and 12.42% (textual similarity) in NSFW dataset, respectively. These results demonstrate the attack effectiveness of our method. In terms of the attack naturalness, our UPAM (w/o TAL) achieves a significantly lower PPL than other methods, which demonstrates the naturalness of adversarial prompts generated by our method. Also, we can observe that UPAM (w/o TAL) exhibits a significantly shorter inference time compared to other methods, showcasing superior efficiency of our approach. In terms of the query times, as shown in Tab. I, different from previous methods that require querying the API during inference, our method primarily relies on querying the API during training. Since we utilize the trained model directly for inference, during inference, our method queries API only once, which is significantly lower than the query times of previous methods. Even though our method requires API queries for training, the combined total training and testing query times are still obviously less than the total query times of each previous method. We can see UPAM (w/o TAL) already performs fewer queries than previous methods. Our TAL scheme can further reduce the number of queries. As shown in Tab. I, our UPAM (with TAL) queries API for only few-shot (10-shot) times during training, outperforming previous methods requiring hundreds or thousands of quires, demonstrating the effect of our TAL scheme. Experimental Results of Protocol B. In protocol B, given a target image, we aim to directly find out the corresponding adversarial prompt for the target image. For the COCO dataset, we follow previous methods [9, 10, 11, 18, 19] to randomly choose 10 samples from 10 “harmful” classes of COCO (one for each class), and then use our UPAM to find the corresponding adversarial prompts for the 10 samples. Finally, we evaluate the attack performance of these adversarial prompts. For the NSFW dataset, we follow the previous work [20] to find adversarial prompts for 200 given samples, and subsequently calculate the attack performance. We conduct experiments 10 times and show the average results in Tab. I. We can see that, our UPAM (w/o TAL) significantly outperforms other methods in terms of R-1 precision, R-3 precision, and textual similarity, which demonstrates the effectiveness of our method. Moreover, even in protocol B (where we don’t train a parameterized model on the training set), our UPAM (w/o TAL) still achieves the much shorter inference time than others. This is due to our method’s ability to use gradients for searching adversarial prompts, which is much more efficient than enumeration-based methods. In Tab. I, we can also see that our UPAM (w/o TAL) exhibits the lowest PPL, indicating the best naturalness of generated adversarial prompts. In terms of query times, we can see that our method has zero training queries due to the inference time optimization defined in protocol B. We can see that our method has a significant advantage in terms of query times. Particularly, when using TAL, our UPAM can outperform previous methods with only a few (10-shot) queries, demonstrating the effect of TAL. All these results collectively demonstrate the superiority and applicability of our approach in protocol B. Additional Comparison Results in Efficiency. As shown in Tab. I, to further demonstrate the efficiency of our method, we additionally present the total training time and total inference time of our approach with comparison to previous enumeration methods under protocol A. It can be observed that, in the setting of attacking DALL⋅·⋅E [8], the total inference time of our method is only 0.17 hours on the COCO dataset (and 0.28 hours on the NSFW dataset), which is significantly lower than the total inference time of enumeration-based methods (e.g., 19.85 hours). It is noteworthy that although our method introduces a training phase, the combined total training and inference time remains notably less than the total time required by enumeration-based methods. Taking the setting of attacking DALL⋅·⋅E [8] as an example, our UPAM (with TAL) requires a total of 9.62 hours for both training and inference on the COCO dataset (and 11.93 hours on the NSFW dataset), while each enumeration-based method requires more than 13 hours on the COCO dataset (and more than 19 hours on the NSFW dataset), demonstrating the efficiency of our method even when considering the training time. This is because, after training, our parameterized model can efficiently perform fast mapping that translates naive prompts into adversarial prompts. As mentioned earlier, the inference time per sample for our method (around 5.11∼similar-to ∼5.57 seconds) is considerably lower than that of enumeration-based approaches (around 353.53∼similar-to ∼593.51 seconds). This advantage becomes increasingly evident as the number of testing (inference) samples grows, further widening the efficiency gap between our approach and traditional enumeration-based methods. To ensure the robustness (general applicability) of our method, we have conducted a comprehensive evaluation under various settings by attacking different T2I models. As shown in Tab. I, the results on different T2I models demonstrate that our method consistently achieves strong efficiency superiority than previous methods, even when accounting for training time. Moreover, we can also observe that removing TAL (i.e., w/o TAL) further reduces the training time of our method. This is because, when not using TAL, our UPAM eliminates the process of adapting the offline model to the online API. Qualitative Results. In this section, we present qualitative results. Specifically, we compare our framework with HiddVocab [10], MacPromp [11], MMA-Diff [12], RIATIG [9], and SneakyPrompt [20], while not showcasing the results of TextFooler [19], HomoSubs [18], and EvoPromp [11]. This is because the latter three methods can hardly compel the black-box API to return images, and thus their results cannot be presented. In Fig. 7 and 8, we respectively showcase the attack results on the COCO dataset and the NSFW dataset. We can see that, in order to bypass textual filters, all methods generate adversarial prompts that no longer contain the sensitive words of naive prompts (marked in red). However, existing methods produce adversarial text with noticeably unnatural words (highlighted with blue underlines). In contrast, our method generates adversarial prompts exhibiting superior naturalness. Moreover, compared to other methods, our framework achieves the closest semantic alignment between the generated images and target images. We can observe that, our UPAM framework occasionally generates prompts with unnatural semantics. As shown in Fig. 7 and 8, examples in UPAM outputs sometimes include misspelled words like “consist e d”, “ma-mary”, “a n”, and “tegument” (also marked with blue underlines). These instances indicate that there is room for improvement in ensuring completely semantically accurate textual prompts. Nevertheless, compared to other methods, which often produce highly unnatural and difficult-to-interpret words (e.g., “Judsonnisi”, “atoKjbmyith”), our UPAM framework exhibits significantly higher contextual coherence. The generated prompts are generally easier to interpret, and readers can often deduce the intended words even when minor errors are present. This highlights UPAM’s advantage in maintaining overall contextual relevance, making its adversarial prompts more understandable and contextually aligned. TABLE I: Performance comparison in terms of total training time and total inference time. The unit for the values in the table is hours. T2I Model Methods COCO NSFW Total Train. Time Total Infer. Time Total Train. time Total Infer. time TextFooler [19] - 19.85 - 29.63 HomoSubs [18] - 18.74 - 29.19 EvoPromp [11] - 19.30 - 26.43 HiddVocab [10] - 15.24 - 28.08 DALL⋅·⋅E [8] MacPromp [11] - 15.07 - 25.13 MMA-Diff [12] - 12.89 - 19.91 RIATIG [9] - 13.05 - 20.49 SneakyPrompt [20] - 18.73 - 24.92 UPAM (with TAL) 9.45 0.17 11.65 0.28 UPAM (w/o TAL) 9.41 0.17 11.58 0.28 TextFooler [19] - 18.03 - 29.68 HomoSubs [18] - 18.94 - 29.16 EvoPromp [11] - 15.86 - 22.74 HiddVocab [10] - 18.84 - 27.65 DALL⋅·⋅E 2 [48] MacPromp [11] - 17.45 - 25.57 MMA-Diff [12] - 12.06 - 22.88 RIATIG [9] - 11.78 - 25.38 SneakyPrompt [20] - 16.50 - 28.09 UPAM (with TAL) 9.58 0.19 11.77 0.27 UPAM (w/o TAL) 9.53 0.19 11.73 0.27 TextFooler [19] - 17.59 - 29.17 HomoSubs [18] - 18.93 - 29.42 EvoPromp [11] - 13.28 - 23.81 HiddVocab [10] - 19.75 - 28.12 MacPromp [11] - 17.48 - 25.98 Imagen [7] MMA-Diff [12] - 13.16 - 23.83 RIATIG [9] - 12.94 - 21.13 SneakyPrompt [20] - 14.25 - 20.23 UPAM (with TAL) 9.27 0.18 11.39 0.28 UPAM (w/o TAL) 9.21 0.18 11.33 0.28 TABLE IV: Ablation study on each design of our approach. Methods R-1 ↑ R-3 ↑ Text. Sim. ↓ Infer. Time ↓ PPL ↓ (a) UPAM 38.56% 41.92% 0.17 5.11 641.26 (b) UPAM w/o SPL 0.22% 0.37% 0.53 5.13 640.05 (c) UPAM w/o SEL 23.25% 28.36% 0.17 5.10 643.31 5 Ablation Studies In the following content, we use ”UPAM” to refer to ”UPAM (w/o TAL)” for clarity. 5.1 Effect of SPL and SEL To demonstrate the effectiveness of our proposed schemes: SPL and SEL, we conduct ablation experiments by removing the specific SPL (or SEL) in our approach. Comparing (a) and (b) in Tab. IV, it can be seen that when UPAM does not employ SPL, both R-1 precision and R-3 precision drop to almost 0. This is because SPL is designed to deceive textual and visual defenses. In the absence of SPL, black-box T2I models hardly return images, resulting in poor R-precision performance. Additionally, SPL also influences the Textual Similarity score, as it can reduce the similarity between adversarial prompts and naive ones, aiming to deceive textual filters within the black-box system. These ablation results demonstrate the effect of SPL. When comparing (a) and (c) in Tab. IV, we can see that our UPAM without SEL leads to a significant R-precision decrease of 15.31% (R-1) and 15.56% (R-3), demonstrating the effect of SEL. We can observe that SEL has virtually no impact on Textual Similarity, Inference Time, and PPL. This is because SEL is proposed to enhance the semantic representation of the returned images, thereby only affecting R-precision. As SPL and SEL do not alter the LLM-based structure of UPAM, they do not impact the Inference Time and PPL. 5.2 Effect of LLM and LoRA To ensure the attack stealthiness, we propose to utilize a pre-trained LLM and optimize the LoRA adapter to ensure the naturalness of the generated adversarial prompts. Here, we conduct two ablation studies: (1) instead of using the pr-trained LLM, we directly employ an untrained transformer [64] (with the same structure as the LLM) and then train it from scratch. Comparing (i) and (i) in Tab. V, we can see the transformer performs much worse in terms of PPL, demonstrating the effect of adopting knowledge of LLM. (2) Instead of optimizing LoRA, we directly optimize the pre-trained LLM. Comparing (i) and (i) in Tab. V, the significant change in PPL demonstrates the effect of using LoRA adapter. TABLE V: Ablation study on LLM and LoRA. Methods R-1 ↑ R-3 ↑ Text. Sim. ↓ Infer. Time ↓ PPL ↓ (i) UPAM 38.56% 41.92% 0.17 5.11 641.26 (i) UPAM w/o LLM 38.45% 41.68% 0.18 5.13 3954.85 (i) UPAM w/o LoRA 38.42% 41.79% 0.17 5.11 2848.52 5.3 Effect of INE In order to harness in-context learning to further unleash the potential of LLM in generating naturally adversarial prompts, we propose an INE scheme which introduces several in-context examples as the input of LLM. With the comparison between (i) and (i) in Tab. VI, the observed change in PPL clearly demonstrates that INE can enhance the naturalness of adversarial prompts. TABLE VI: Ablation study on INE. Methods R-1 ↑ R-3 ↑ Text. Sim. ↓ Infer. Time ↓ PPL ↓ (i) UPAM 38.56% 41.92% 0.17 5.11 641.26 (i) UPAM w/o INE. 38.33% 41.72% 0.17 5.08 715.32 5.4 Effect of the second stage of SPL In the second stage of SPL, after optimizing the model parameters into the “Pass” region, we gradually reduce the radius, moving the model parameters closer to the boundary, aiming to increase the likelihood of the generation of target “harmful” images, as illustrated in Fig. 4 (b). To investigate the effect of this design, we conduct ablation by removing this second stage. By comparing (i) and (i) in Tab. VII, when removing the second stage of SPL, obvious performance degradation is observed in terms of R-1, R-3, and Textual Similarity. This demonstrates the effectiveness of the second stage of SPL. TABLE VII: Ablation study on the second stage of SPL. Methods R-1 ↑ R-3 ↑ Text. Sim. ↓ Infer. Time ↓ PPL ↓ (i) UPAM 38.56% 41.92% 0.17 5.11 641.26 (i) UPAM w/o 2nd Stage 35.76% 39.31% 0.22 5.11 706.10 5.5 Effect of gradient harmonization in SEL To improve the compatibility of SEL with SPL, we introduce a Gradient Harmonization (GH) method (see Eq. 11) to adaptively adjust SEL’s gradients. With a comparison between (i) and (i) in Tab. VIII, we can see that, when removing GH, SEL could disrupt SPL’s optimization achievements, thus leading to worse performances in terms of R-1, R-3, and Textual Similarity. This demonstrates the effectiveness of gradient harmonization in SEL. TABLE VIII: Ablation study on Gradient Harmonization (GH) in SEL. Methods R-1 ↑ R-3 ↑ Text. Sim. ↓ Infer. Time ↓ PPL ↓ (i) UPAM 38.56% 41.92% 0.17 5.11 641.26 (i) UPAM w/o GH 33.82% 35.08% 0.35 5.11 645.14 Figure 6: Overview of our UPAM framework modified for Protocol B. 5.6 Ablation on using small learning rate in SPL In SPL, our goal is to make the model parameters fall into the boundary region, leaning towards the “Pass” side. We consider that when the model parameters are optimized from the “Deny” case to the “Pass” case, if we use a very small learning rate (i.e., optimization step size), whether it is possible to directly locate near the boundary after crossing the boundary. If this can work, the process of “moving closer to the boundary” in SPL could be skipped. However, experimental results from Tab. IX (i) show that when using a small learning rate, SPL shows less effective performance in terms of R1, R3, and Textual Similarity. This is because the model with a small learning rate struggles to transition from the “Deny” case to the “Pass” case, thus cannot effectively bypass the textual and visual defenses. We also attempt to set a relatively large initial learning rate and then gradually decrease it. However, as shown in Tab. IX (i), the performance remains unsatisfactory, because it is challenging to manually set the learning rate decay in a way that allows the model to cross the boundary while remaining very close to it. TABLE IX: Ablation study on using small learning rate in SPL. Methods R-1 ↑ R-3 ↑ Text. Sim. ↓ Infer. Time ↓ PPL ↓ (i) SPL 38.56% 41.92% 0.17 5.11 641.26 (i) SPL (small learning rate) 7.34% 9.62% 0.47 5.11 645.08 (i) SPL (learning rate decay) 12.42% 13.77% 0.45 5.09 644.11 Note that in Tab. IX, we present the results obtained after carefully tuning the learning rates. Specifically, in experiment (i), we adopt a small learning rate of 0.005, and in experiment (i), we use an initial learning rate of 0.5 with an exponential decay rate of 0.2. These experiments reveal that neither employing a small learning rate nor gradually decaying the learning rate in SPL can achieve effective performance, further highlighting the effectiveness of the SPL’s design. Moreover, it is important to highlight that no modifications were made to the model structure of UPAM or the parameters of the LLM in these ablation studies. Therefore, we can see that the inference time and PPL (naturalness) remain almost unchanged. TABLE X: Ablation study on calculating gradients from different points in SPL. Methods R-1 ↑ R-3 ↑ Text. Sim. ↓ Infer. Time ↓ PPL ↓ (i) SPL (using “Deny” points) 38.56% 41.92% 0.17 5.11 641.26 (i) SPL (using “Deny” points) 38.45% 41.80% 0.18 5.12 641.02 (i) SPL (using both “Pass” and “Deny” points) 38.49% 41.91% 0.17 5.10 644.26 5.7 Ablation on calculating gradients from different points in SPL In SPL, when calculating gradients, we compute the gradients based on the average of “Pass” points. Here, we conduct an ablation study by calculating gradients based on the average of “Deny” points, which modifies the Eq. 1 as follows: Φ⁢(ψ)=0⁢image⁢returned⁢(`⁢`⁢Pass⁢")−1Φmatrix0imagereturned``Pass"1 -2m (ψ)= Bmatrix0&if \;image\;returned\,(``% Pass")\\ -1&otherwise BmatrixΦ ( ψ ) = start_ARG start_ROW start_CELL 0 end_CELL start_CELL if image returned ( ` ` Pass " ) end_CELL end_ROW start_ROW start_CELL - 1 end_CELL start_CELL otherwise end_CELL end_ROW end_ARG (14) Additionally, we conduct an ablation experiment using both “Pass” points and “Deny” points to calculate gradients, reformulating the Eq. 1 as follows: Φ⁢(ψ)=+1⁢image⁢returned⁢(`⁢`⁢Pass⁢")−1Φmatrix1imagereturned``Pass"1 -2m (ψ)= Bmatrix+1&if \;image\;returned\,(`% `Pass")\\ -1&otherwise BmatrixΦ ( ψ ) = start_ARG start_ROW start_CELL + 1 end_CELL start_CELL if image returned ( ` ` Pass " ) end_CELL end_ROW start_ROW start_CELL - 1 end_CELL start_CELL otherwise end_CELL end_ROW end_ARG (15) The results in Tab. X indicate that, regardless of which kind of points are used, the performance remains almost unchanged, demonstrating the robustness of our SPL in using different gradient calculation approaches. TABLE XI: Ablation on using a smaller instruction-tuned language model (Flan-T5-Small) for PPL evaluation. Methods PPL (GPT-2) ↓ PPL (Flan-T5-Small) ↓ TextFooler [19] 3463.02 3876.05 HomoSubs [18] 3089.58 3663.89 EvoPromp [11] 4984.24 5197.42 HiddVocab [10] 4027.64 4175.28 MacPromp [11] 3163.07 3531.73 MMA-Diff [12] 3857.29 4031.19 RIATIG [9] 1003.27 1326.84 SneakyPrompt [20] 1457.36 1584.57 UPAM 658.18 823.63 5.8 Ablation on using a smaller instruction-tuned language model for PPL evaluation To further explore the evaluation of adversarial prompts’ naturalness, we selected a smaller, instruction-tuned model, Flan-T5-Small [74], to compute the perplexity score (PPL). Flan-T5-Small has only 80M parameters, significantly fewer than GPT-2 (more than 117M). Being fine-tuned through multi-task instruction tuning, Flan-T5-Small is better equipped to understand and execute human instructions. As shown in the Tab. XI, the results reveal that: (1) even when evaluated with a different language model like Flan-T5-Small, our method continues to exhibit significant advantages in preserving the naturalness of adversarial prompts, outperforming prior approaches. This further validates the robustness of our method in maintaining prompt naturalness. (2) Both our method and other approaches show higher PPL scores (i.e., generally performing worse) when measured with Flan-T5-Small compared to GPT-2. We analyze the reason that Flan-T5-Small, as a small instruction-tuned model, is trained with objectives distinct from GPT-2. Specifically, Flan-T5-Small is optimized for instruction-based downstream tasks such as question answering and reasoning, while GPT-2 focuses on pure language modeling. Therefore, as PPL reflects a model’s language modeling capabilities, Flan-T5-Small which is an instruction-tuned model may yield less favorable PPL scores compared to GPT-2 which is designed for language generation alone. In summary, the experiment results demonstrate that while the choice of evaluation model impacts PPL scores, our method consistently maintains superior naturalness across models, reinforcing its effectiveness. Additionally, the observed differences underscore the importance of selecting appropriate evaluation metrics and models tailored to the evaluation context. Figure 7: Qualitative comparison results with existing methods on COCO dataset. The text above represents the adversarial prompts generated by the corresponding method, while the pictures below show the images returned from the T2I model. The name of each “harmful” class is marked in red. We employ blue underlines to show unnatural words within adversarial prompts. Figure 8: Qualitative comparison results with existing methods on NSFW dataset. The text above represents the adversarial prompts generated by the corresponding method, while the pictures below show the images returned from the T2I model. The name of each “harmful” class is marked in red. We employ blue underlines to show unnatural words within adversarial prompts. We use white mosaics to obscure sensitive areas for ethical considerations. 6 Modification of UPAM for Protocol B As for previous enumeration-based methods, given a naive prompt TT and a target image II, instead of training parameterized models, they aim to directly find an adversarial prompt ∗superscript∗T T∗ based on the given data TT and II. We refer to this as Protocol B. To ensure a fair comparison with previous enumeration-based methods, we make slight modifications to UPAM to enable it to directly find adversarial prompts without learning from the training set. The modified architecture is illustrated in Fig. 6. Next, we provide modification details as follows: 1. As for the structure of our UPAM, we no longer utilize the LoRA adapter while only adopting the LLM. The parameters of the LLM remain frozen. 2. Given ,\T,I\ T , I , following the processes of SPL and SEL, we still compute gradients splsubscriptspl g_splitalic_groman_spl and selsubscriptsel g_selitalic_groman_sel. The difference is that, these gradients no longer optimize LoRA parameters but optimize the input soft embedding PP. The soft embedding PP is obtained by feeding the naive TT into the text encoder of LLM. In this way, given a naive prompt TT and a target image II, we can directly find a proper adversarial prompt ∗superscript∗T T∗ by optimizing the input embedding PP, without training any model parameters. 7 Discussion Ethical Implications: While our research contributes to understanding model vulnerabilities, we recognize the potential ethical concerns, particularly the misuse of our findings in generating harmful or inappropriate content. Text-to-image models, when misused, have the potential to create images that could cause harm, spread misinformation, or infringe on individual privacy and dignity. Therefore, it is critical to develop not only the technology but also a responsible framework for its use. Regulatory Compliance: Researchers and practitioners seeking to implement this work must strictly adhere to existing guidelines and regulations governing AI and content generation, including data protection laws [75], AI ethics principles [76], and anti-harassment policies [77]. Compliance with these frameworks is crucial to ensure that research does not inadvertently contribute to harmful or illegal activities. Measures to Mitigate Misuse: To mitigate the risk of misuse, besides the regulatory compliance, we suggest several measures that should accompany the development of such techniques. (1) Access control. To further reduce the potential for harmful applications, we propose implementing stricter access controls on our UPAM model. Only authorized and responsible users, such as researchers and organizations committed to ethical AI practices, could be allowed to access and deploy our UPAM model. (2) Method transparency. For authorized developers who wants to further improve our method for research, we will urge them to be transparent about the capabilities and limitations of their methods. This includes providing research paper to clearly describe on how the technology works, its potential for harm, and the safeguards implemented to minimize misuse. Such transparency will enable defense teams to develop targeted and effective countermeasures. (3) Possible defense against our approach. One potential defense is to fine-tune the T2I models using adversarial training [78]. Through adversarial training, defenders can enhance the robustness of T2I models, making them unable to generate the target image when provided with adversarial prompts crafted by our approach. Another effective defense mechanism could be leveraging model unlearning [79]. This technique allows T2I models to “unlearn” harmful or sensitive knowledge, effectively reducing their susceptibility to adversarial attacks. Since our approach specifically targets the textual filters and visual checkers deployed in APIs, another possible defense is to frequently update or rotate the textual filters and visual checkers, i.e., dynamic defense system [80]. This strategy can make it harder for attackers to adapt and craft effective adversarial prompts. Future Beneficial Applications: Notably, the technique we explore can be repurposed for future positive applications. For instance, our black-box attack model can serve as a security vulnerability detector to evaluate the robustness and security of T2I APIs. By simulating potential adversarial attacks, developers can proactively uncover weaknesses in their models and implement more effective defense mechanisms, such as refining textual filters and visual checkers. Furthermore, the capability to effectively optimize prompts through our black-box gradient learning could enhance creative AI systems. That is, our approach could help enable users to craft more effective prompts for generating high-quality, tailored outputs, all without requiring access to the API’s internal structure or parameters. Potential Generalizability to Other Tasks: Due to the general applicability of our method, the UPAM framework has the potential to be extended to non-T2I tasks with appropriate modifications, so long as a target output image type is defined for optimization. Here, we provide several potential applications where UPAM can be applied beyond T2I systems for future research. These include: (1) Attack on text-to-video generation models [81]. The same principles used to attack static image generation could be extended to sequence generation and temporal consistency in videos. (2) Attack on image-to-image translation models [82]. Given the targeted output in image-to-image tasks, our method could be adapted to force the model to produce incorrect or distorted images by introducing adversarial artifacts in the input visual space. (3) Attack on cross-modal retrieval models [83]. UPAM can be adapted to retrieval task to generate adversarial prompts that misguide the retrieval process, highlighting vulnerabilities in systems that depend on cross-modal alignment. 8 Conclusion In this paper, we introduce UPAM, a unified attack framework against T2I models, which outperforms existing enumeration methods by targeting both textual filters and visual checkers through gradient-based optimization. To achieve this, we propose the SPL scheme for gradient estimation without image feedback, the SEL scheme to ensure harmful semantics, and the INE scheme to enhance prompt naturalness. Additionally, the TAL scheme enables UPAM to operate with minimal queries. Comprehensive experiments validate the effectiveness, efficiency, naturalness, and low query requirements of our approach. References [1] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, “Attngan: Fine-grained text to image generation with attentional generative adversarial networks,” in Conference on computer vision and pattern recognition, 2018, p. 1316–1324. [2] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in International conference on computer vision, 2017, p. 5907–5915. [3] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan++: Realistic image synthesis with stacked generative adversarial networks,” Transactions on pattern analysis and machine intelligence, vol. 41, no. 8, p. 1947–1962, 2018. [4] M. Galetin, J. Škorić, and M. Mihajlović, “Review of the standpoints regarding the content of the article 5 of the proposal for the european union’s artificial intelligence act: The challenge of finding the balance,” Univerzitetska misao-časopis za nauku, kulturu i umjetnost, Novi Pazar, no. 21, p. 164–175, 2022. [5] A. Daly, T. Hagendorff, H. Li, M. Mann, V. Marda, B. Wagner, and W. W. Wang, “Ai, governance and ethics: global perspectives,” University of Hong Kong Faculty of Law Research Paper, 2020. [6] A. Birhane, V. U. Prabhu, and E. Kahembwe, “Multimodal datasets: misogyny, pornography, and malignant stereotypes,” arXiv preprint arXiv:2110.01963, 2021. [7] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in neural information processing systems, vol. 35, p. 36 479–36 494, 2022. [8] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan et al., “Scaling autoregressive models for content-rich text-to-image generation,” arXiv preprint arXiv:2206.10789, 2022. [9] H. Liu, Y. Wu, S. Zhai, B. Yuan, and N. Zhang, “Riatig: Reliable and imperceptible adversarial text-to-image generation with natural prompts,” in Conference on computer vision and pattern recognition, 2023, p. 20 585–20 594. [10] G. Daras and A. G. Dimakis, “Discovering the hidden vocabulary of dalle-2,” arXiv preprint arXiv:2206.00169, 2022. [11] R. Millière, “Adversarial attacks on image generation with made-up words,” arXiv preprint arXiv:2208.04135, 2022. [12] Y. Yang, R. Gao, X. Wang, N. Xu, and Q. Xu, “Mma-diffusion: Multimodal attack on diffusion models,” arXiv:2311.17516, 2023. [13] P. Schramowski, M. Brack, B. Deiseroth, and K. Kersting, “Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models,” in Conference on computer vision and pattern recognition, 2023, p. 22 522–22 531. [14] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in conference on computer vision and pattern recognition, 2022, p. 10 684–10 695. [15] “Midjourney.” https://w.mi djourney.com/home. [16] “Leonardo.ai.” https://leonardo.ai/. [17] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in ACM on asia conference on computer and communications security, 2017, p. 506–519. [18] L. Struppek, D. Hintersdorf, and K. Kersting, “The biased artist: Exploiting cultural biases via homoglyphs in text-guided image generation models,” arXiv preprint arXiv:2209.08891, 2022. [19] D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, “Is bert really robust? a strong baseline for natural language attack on text classification and entailment,” in AAAI conference on artificial intelligence, vol. 34, 2020, p. 8018–8025. [20] Y. Yang, B. Hui, H. Yuan, N. Gong, and Y. Cao, “Sneakyprompt: Jailbreaking text-to-image generative models,” in Symposium on security and privacy, 2024, p. 123–123. [21] M. S. Daoud, M. Shehab, H. M. Al-Mimi, L. Abualigah, R. A. Zitar, and M. K. Y. Shambour, “Gradient-based optimizer (gbo): a review, theory, variants, and applications,” Archives of computational methods in engineering, vol. 30, no. 4, p. 2431–2449, 2023. [22] W. Zhou, P. Wang, A. A. Heidari, X. Zhao, H. Turabieh, and H. Chen, “Random learning gradient based optimization for efficient design of photovoltaic models,” Energy conversion and management, vol. 230, p. 113751, 2021. [23] D. Golovin, B. Solnik, S. Moitra, G. Kochanski, J. Karro, and D. Sculley, “Google vizier: A service for black-box optimization,” in ACM SIGKDD international conference on knowledge discovery and data mining, 2017, p. 1487–1495. [24] C. Oh, H. Hwang, H.-y. Lee, Y. Lim, G. Jung, J. Jung, H. Choi, and K. Song, “Blackvip: Black-box visual prompting for robust transfer learning,” in Conference on computer vision and pattern recognition, 2023, p. 24 224–24 235. [25] H. Huang, S. Wu, X. Liang, B. Wang, Y. Shi, P. Wu, M. Yang, and T. Zhao, “Towards making the most of llm for translation quality estimation,” in International conference on natural language processing and chinese computing, 2023, p. 375–386. [26] D. Peng, Q. Ke, and J. Liu, “Upam: Unified prompt attack in text-to-image generation models against both textual filters and visual checkers,” in International conference on machine learning, 2024. [27] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013. [28] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv:1412.6572, 2014. [29] J. C. Costa, T. Roxo, H. Proença, and P. R. Inácio, “How deep learning sees the world: A survey on adversarial attacks & defenses,” IEEE Access, 2024. [30] S.-i. Amari, “Backpropagation and stochastic gradient descent method,” Neurocomputing, vol. 5, no. 4-5, p. 185–196, 1993. [31] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in Symposium on security and privacy, 2017. [32] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in Conference on computer vision and pattern recognition, 2016. [33] E. Wong, L. Rice, and J. Z. Kolter, “Fast is better than free: Revisiting adversarial training,” arXiv preprint arXiv:2001.03994, 2020. [34] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017. [35] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in Conference on computer vision and pattern recognition, 2018, p. 9185–9193. [36] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, “Square attack: a query-efficient black-box adversarial attack via random search,” in European conference on computer vision, 2020. [37] M. Wicker, X. Huang, and M. Kwiatkowska, “Feature-guided black-box safety testing of deep neural networks,” in Tools and algorithms for the Construction and analysis of systems: international conference, 2018, p. 408–426. [38] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu, “Jailbreaking chatgpt via prompt engineering: An empirical study,” arXiv preprint arXiv:2305.13860, 2023. [39] H. Tu, C. Cui, Z. Wang, Y. Zhou, B. Zhao, J. Han, W. Zhou, H. Yao, and C. Xie, “How many are in this image a safety evaluation benchmark for vision llms,” in European conference on computer vision. Springer, 2025, p. 37–55. [40] H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y. Song, “Multi-step jailbreaking privacy attacks on chatgpt,” arXiv preprint arXiv:2304.05197, 2023. [41] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu, “Masterkey: Automated jailbreak across multiple large language model chatbots,” arXiv preprint arXiv:2307.08715, 2023. [42] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023. [43] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” Advances in neural information processing systems, vol. 36, 2024. [44] E. Shayegani, Y. Dong, and N. Abu-Ghazaleh, “Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,” in International conference on learning representations, 2023. [45] X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal, “Visual adversarial examples jailbreak aligned large language models,” in AAAI conference on artificial intelligence, vol. 38, no. 19, 2024, p. 21 527–21 536. [46] E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng et al., “Sleeper agents: Training deceptive llms that persist through safety training,” arXiv preprint arXiv:2401.05566, 2024. [47] W. Yang, X. Bi, Y. Lin, S. Chen, J. Zhou, and X. Sun, “Watch out for your agents! investigating backdoor threats to llm-based agents,” arXiv preprint arXiv:2402.11208, 2024. [48] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, vol. 1, 2022. [49] H. Li, S. Shan, E. Wenger, J. Zhang, H. Zheng, and B. Y. Zhao, “Blacklight: Scalable defense for neural networks against query-based black-box attacks,” in USENIX security symposium, 2022. [50] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021. [51] K. S. Kalyan, “A survey of gpt-3 family large language models including chatgpt and gpt-4,” Natural language processing journal, vol. 6, p. 100048, 2024. [52] C. Finlay, A.-A. Pooladian, and A. Oberman, “The logbarrier adversarial attack: making effective use of decision boundary information,” in International conference on computer vision, 2019. [53] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, p. 8748–8763. [54] J. C. Spall, “Multivariate stochastic approximation using a simultaneous perturbation gradient approximation,” Transactions on automatic control, vol. 37, no. 3, p. 332–341, 1992. [55] J. C. Spall, “A one-measurement form of simultaneous perturbation stochastic approximation,” Automatica, vol. 33, p. 109–112, 1997. [56] W. Nowak, “Introduction to stochastic search and optimization. estimation, simulation, and control,” Transactions on neural networks, vol. 18, p. 964–965, 2007. [57] J. C. Spall, “Adaptive stochastic approximation by the simultaneous perturbation method,” Transactions on automatic control, vol. 45, p. 1839–1853, 2000. [58] J. C. Spall, “Accelerated second-order stochastic optimization using only function measurements,” in Conference on decision and control, vol. 2, 1997, p. 1417–1424. [59] Y. E. Nesterov, “A method of solving a convex programming problem with convergence rate O(1/k 2),” in Doklady akademii nauk, vol. 269, 1983, p. 543–547. [60] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.00234, 2022. [61] Y. Liu, X. Chen, C. Liu, and D. Song, “Delving into transferable adversarial examples and black-box attacks,” arXiv preprint arXiv:1611.02770, 2016. [62] J. Rando, D. Paleka, D. Lindner, L. Heim, and F. Tramèr, “Red-teaming the stable diffusion safety filter,” arXiv preprint arXiv:2210.04610, 2022. [63] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision, 2014, p. 740–755. [64] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [65] “Deepfloyd if.” https://github.com/deep-floyd/IF. [66] “Stablestudio.” https://github.com/Stability-AI/StableStudio. [67] “Invokeai.” https://github.com/invoke-ai/InvokeAI. [68] “Dreamlike.” https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0. [69] “Waifu.” https://huggingface.co/hakurei/waifu-diffusion. [70] “Openjourney.” https://huggingface.co/prompthero/openjourney. [71] W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in International conference on machine learning, 2021, p. 5583–5594. [72] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar et al., “Universal sentence encoder,” arXiv preprint arXiv:1803.11175, 2018. [73] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, p. 9, 2019. [74] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma et al., “Scaling instruction-finetuned language models,” Journal of machine learning research, vol. 25, p. 1–53, 2024. [75] L. H. Yan, “Personal information privacy: Investigating the need of amending the personal data (privacy) ordinance,” Ph.D. dissertation, Hong Kong Baptist University Hong Kong, 2010. [76] E. Fournier-Tombs, “Towards a united nations internal regulation for artificial intelligence,” Big data & society, vol. 8, 2021. [77] A. Witze, “Nsf rolls out stringent anti-harassment policy,” Nature, vol. 561, p. 444–445, 2018. [78] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” Advances in neural information processing systems, vol. 32, 2019. [79] Y. Liu, M. Fan, C. Chen, X. Liu, Z. Ma, L. Wang, and J. Ma, “Backdoor defense with machine unlearning,” in Conference on computer communications, 2022. [80] K.-J. Lee, C.-A. Liu, and C.-C. Wu, “A dynamic-key based secure scan architecture for manufacturing and in-field ic testing,” Transactions on emerging topics in computing, vol. 10, p. 373–385, 2020. [81] Y. Miao, Y. Zhu, Y. Dong, L. Yu, J. Zhu, and X.-S. Gao, “T2vsafetybench: Evaluating the safety of text-to-video generative models,” arXiv preprint arXiv:2407.05965, 2024. [82] C.-Y. Yeh, H.-W. Chen, H.-H. Shuai, D.-N. Yang, and M.-S. Chen, “Attack as the best defense: Nullifying image-to-image translation gans via limit-aware adversarial attack,” in International conference on computer vision, 2021, p. 16 188–16 197. [83] C. Li, S. Gao, C. Deng, W. Liu, and H. Huang, “Adversarial attack on deep cross-modal hamming retrieval,” in International conference on computer vision, 2021, p. 2218–2227.

Instant research discovery