Paper deep dive

Towards Safe Concept Transfer of Multi-Modal Diffusion via Causal Representation Editing

Peiran Dong, Bingjie Wang, Song Guo, Junxiao Wang, Jie Zhang, Zicong Hong

Year: 2024Venue: NeurIPS 2024Area: Model EditingType: EmpiricalEmbeddings: 81

Models: vision-language-to-image diffusion models

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 5:36:48 PM

Summary

The paper introduces Causal Representation Editing (CRE), a framework for safe content generation in multi-modal diffusion models. CRE intervenes at specific diffusion timesteps causally linked to unsafe concepts, allowing for precise removal of harmful content while maintaining generation quality and scalability compared to existing methods like dataset filtering or inference-time refusals.

Entities (5)

Causal Representation Editing · framework · 100%Diffusion Models · generative-model · 100%Large Language Models · model-architecture · 95%U-Net · neural-network-architecture · 95%Vision-Language-to-Image · task · 95%

Relation Signals (3)

Causal Representation Editing → intervenesin → Diffusion Models

confidence 95% · CRE enhances the efficiency and flexibility of safe content generation by intervening at diffusion timesteps causally linked to unsafe concepts.

U-Net → comprises → Cross-Attention Layers

confidence 90% · The U-Net architecture comprises layers broadly classified into convolution layers, self-attention layers, and cross-attention layers.

Causal Representation Editing → extends → Large Language Models

confidence 90% · CRE extends representation editing from large language models (LLMs) to diffusion-based models.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Recent advancements in vision-language-to-image (VL2I) diffusion generation have made significant progress. While generating images from broad vision-language inputs holds promise, it also raises concerns about potential misuse, such as copying artistic styles without permission, which could have legal and social consequences. Therefore, it's crucial to establish governance frameworks to ensure ethical and copyright integrity, especially with widely used diffusion models. To address these issues, researchers have explored various approaches, such as dataset filtering, adversarial perturbations, machine unlearning, and inference-time refusals. However, these methods often lack either scalability or effectiveness. In response, we propose a new framework called causal representation editing (CRE), which extends representation editing from large language models (LLMs) to diffusion-based models. CRE enhances the efficiency and flexibility of safe content generation by intervening at diffusion timesteps causally linked to unsafe concepts. This allows for precise removal of harmful content while preserving acceptable content quality, demonstrating superior effectiveness, precision and scalability compared to existing methods. CRE can handle complex scenarios, including incomplete or blurred representations of unsafe concepts, offering a promising solution to challenges in managing harmful content generation in diffusion-based models.

Full Text

80,626 characters extracted from source content.

Expand or collapse full text

Towards Safe Concept Transfer of Multi-Modal Diffusion via Causal Representation Editing Peiran Dong 1∗ Bingjie Wang 1∗ Song Guo 2 Junxiao Wang 3,4 Jie Zhang 2 Zicong Hong 1 1 Hong Kong Polytechnic University 2 Hong Kong University of Science and Technology 3 Guangzhou University 4 King Abdullah University of Science and Technology peiran.dong,bingjie.wang,zicong.hong@connect.polyu.hk songguo@cse.ust.hk,wangjunxiao@live.com,csejzhang@ust.hk Abstract Recent advancements in vision-language-to-image (VL2I) diffusion generation have made significant progress. While generating images from broad vision- language inputs holds promise, it also raises concerns about potential misuse, such as copying artistic styles without permission, which could have legal and social consequences. Therefore, it’s crucial to establish governance frameworks to ensure ethical and copyright integrity, especially with widely used diffusion models. To address these issues, researchers have explored various approaches, such as dataset filtering, adversarial perturbations, machine unlearning, and inference-time refusals. However, these methods often lack either scalability or effectiveness. In response, we propose a new framework called causal representation editing (CRE), which extends representation editing from large language models (LLMs) to diffusion-based models. CRE enhances the efficiency and flexibility of safe content generation by intervening at diffusion timesteps causally linked to unsafe concepts. This allows for precise removal of harmful content while preserving acceptable content quality, demonstrating superior effectiveness, precision and scalability compared to existing methods. CRE can handle complex scenarios, including incomplete or blurred representations of unsafe concepts, offering a promising solution to challenges in managing harmful content generation in diffusion-based models. 1 Introduction Expanding on recent progress in text-to-image (T2I) diffusion generation, which is great at making realistic and varied images from written descriptions, researchers are now delving into more advanced vision-language-to-image (VL2I) generation techniques. In these VL2I methods, especially with diffusion models, some use both images of a subject and written descriptions to render the subject in a new context, which is called subject-driven generation [1,2]. Others take original images and instructions for changes to create altered images, known as image editing [3]. Early approaches either fine-tune models on new images [4,2,5,6,7] or directly inject image features into the U-Net of diffusion models [8,9,1,10]. However, these methods struggle to jointly model multi-modal inputs and fully leverage the generalization ability of the diffusion model. BLIP-Diffusion [11] is a notable advancement because it creates object representations by blending images with random backgrounds, allowing for the generation of single objects without prior examples. Building on this, Kosmos-G [12] expands to generate multiple objects without examples, using multi-modal large ∗ Equal Contribution. 38th Conference on Neural Information Processing Systems (NeurIPS 2024). language models (MLLMs) instead of the original CLIP text encoder to encode different kinds of inputs into a single feature set. The advent of a large multi-modal encoder has endowed diffusion models with zero-shot generation, enabling the transfer of concepts (e.g., object or style) to new scenes. However, this unrestricted capability also brings up ethical concerns. There’s a risk that people with bad intentions could use zero-shot generation to transfer harmful concepts, like violence or pornography, with just one reference image. Existing efforts in safe generation [13,14,15,16,17,18,19,20,21,22,23] primarily focus on mitigating internal risks stemming from model defects. Diffusion models trained on unedited, large-scale, web-scraped datasets often learn inappropriate and unauthorized knowledge, posing risks when users manipulate textual prompts to “extract” unsafe content. Researchers have pursued various strategies to mitigate the generation of harmful content, encom- passing four primary approaches: dataset filtering [13,14], adversarial perturbations [15,16,17,18], machine unlearning [19,20], and inference-time refusals [21,22,23]. Filtering the dataset involves removing images containing explicit or objectionable content, such as nudity and violence, to ensure the safe generation of diffusion models. However, the advent of zero-shot learning introduces chal- lenges, as it enables diffusion models to transfer unseen objects and styles, complicating copyright protection and security review processes. While adversarial perturbations offer a means to safeguard specific images from manipulation, their efficacy is hampered by the need for training and adaptation to diffusion models with varying parameters. This lack of scalability arises from the requirement to train different adversarial perturbations for each model, despite their structural similarities. Similarly, unlearning-based methods address inherent model defects but fall short in addressing the use of external unsafe images for concept transfer by users. Moreover, existing inference-time refusals predominantly target unsafe concepts describable by language, thus exhibiting limited effectiveness in multi-modal zero-shot generation scenarios. These limitations underscore the need for novel approaches to address the evolving challenges associated with safe content generation in diffusion models. Contributions.To address these challenges, we propose a novel framework called Causal Represen- tation Editing (CRE), which generalizes representation editing for transformer-based Large Language Models (LLMs) to diffusion-based generative models. CRE enhances the efficiency and flexibility of safe concept transfer by introducing a plug-and-play inference-time intervention in diffusion timesteps causally related to unsafe concepts. Our framework comprises two key components: 1) Editing function: We construct steering vectors from examples of unsafe concepts to precisely eliminate them from the original representations. 2) Editing timesteps: We propose “assess-with-exclusion” to identify the causal period for each unsafe concept, during which the unsafe concept appears in the noisy image. This approach reduces editing overhead and avoids ineffective interventions in irrelevant diffusion timesteps, maintaining high editing fidelity. Our contributions include: 1) An early exploration of safe concept transfer in MLLM-enabled diffusion models, with our CRE framework enabling effective inference-time unsafe concept removal. 2) Precise removal of unsafe concepts from noisy images while retaining reasonably generated content, reducing editing overhead by nearly half through fine-grained editing based on the causal period. 3) Comprehensive evaluations demonstrating that CRE surpasses existing methods in effectiveness, preciseness, and scalability, even in complex scenarios involving incomplete or blurred features of unsafe concepts. 2 Related Work Vision-Language-to-Image Diffusion Models.The fundamental aspect of achieving Vision- Language-to-Image (VL2I) generation lies in training multi-modal encoders responsible for aligning and fusing features from diverse input modalities. BLIP-Diffusion [11] adopts an “align-after- encoding” approach to train its multi-modal encoder. Initially, images and text undergo separate encoding by individual single-modal encoders. Subsequently, following BLIP-2 [24], the Q-Former architecture aligns visual features with text features. However, this pre-training strategy restricts BLIP-Diffusion to accept only a single image as the input for the visual modality during zero-shot generation. Conversely, Kosmos-G [12] employs an “align-before-encoding” paradigm to train its multi-modal encoder. Kosmos-G pursues the objective of treating images as a foreign language in the image generation process. It incorporates a multi-modal large language model (MLLM) to jointly encode images and text, with each image being embedded into 64 tokens. By utilizing the pre-trained 2 Denoising U-Net Prompt Encoder as an oil painting in the style of the style of User Prompt Unsafe Concept Prompt Latent representation from user prompt (contains unsafe concept) Latent representation from unsafe concept prompt Safe latent representation Output Safe Images Self-Attention Cross-Attention ResNet User Prompt Flow Unsafe Concept Prompt Flow Representation Editing Causal Representation Editing in Specific Diffusion Period 풕=[푇,푇−1,...,0] Output Flow ǁ 푡⊆풕 Figure 1: Method Overview of CRE. Users of VL2I models (U-Net) might input/query images containing unsafe concepts as reference images (objects or styles), here taking the “Van Gogh” style as an example. CRE consists of two main phases. Phase 1 involves discriminator training and causal period search for each unsafe concept category, which can be performed offline (omitted from this figure, see section 3.3 for details). During inference (phase 2, i.e., the right side of this figure), if the reference image contains unsafe concepts, the editing function of CRE is applied within the U-Net layers. Otherwise, the generated content is faithful to the user-specified prompts without modification. MLLM as an alternative to CLIP encoders [25], diffusion models gain the capability of zero-shot generation based on multiple input images. Inference-time Safe Concept Transfer.Inference-time safe concept transfer enables generation service providers to dynamically deploy and adjust governance rules, particularly in response to potentially unsafe input from users. Existing approaches typically involve either post-generation detection or in-process adjustment to ensure safety. Randoet al.[21] employ a method where the generated image is projected into a CLIP latent space [25] for comparison against pre-computed embeddings of unsafe concepts, with images surpassing a similarity threshold being flagged as unsafe. However, this approach lacks precision in removing unsafe concepts while preserving overall image quality. Conversely, SLD [22] and ProtoRe [23] integrate safe guidance directly into the diffusion process. These techniques rely on textual descriptions of unsafe concepts, encoded using a CLIP text encoder, to provide negative guidance for denoising. SLD [22] adjusts the diffusion process by modifying the predicted noise from the U-Net, while ProtoRe [23] extracts unsafe concepts from the attention map and refines them. These strategies face limitations when unsafe concepts are not effectively described through natural language. Representation Editing for LLMs.Current studies on Inference-Time Intervention (ITI) [26] in Large Language Models (LLMs) indicate that many LLMs exhibit interpretable directions in their activation spaces, which influence their inference processes. For instance, by introducing carefully designed steering vectors to specific layers for particular tokens, the output can be significantly biased, regardless of the user prompt’s topic [27]. Developing a training-free editing method to mitigate unsafe concepts in generative models offers two key advantages. Firstly, it allows the model to retain its strong zero-shot generation ability by preserving the knowledge from pre-training. Secondly, as unsafe concepts may change dynamically due to legal or copyright factors, a plug-and-play editing method can efficiently add or remove types of unsafe concepts under governance. 3 Safe Concept Transfer 3.1 Threat Model LetIrepresent the images generated by a diffusion modelG θ based on a multi-modal prompt, which includes a text promptpand a reference imager. The reference image can contain up toK pre-defined unsafe concepts ̃c k ,k= 1,2,·,K, such as legally protected concepts. Our goal is to intervene in the image generation process to remove these concepts fromI. For example (see 3 Figure 1), an adversary might aim to profit by plagiarizing the style of an artistic work, such as a Van Gogh painting. They could use such a painting as a reference image to counterfeit infringing images using VL2I models with zero-shot generation capabilities. Additionally, unwitting users might input images containing unsafe concepts as reference images (objects or styles). These scenarios can lead to significant social problems and economic losses for generation service providers and copyright owners. In contrast to prior studies that primarily address internal generation issues stemming from the diffusion process itself (often due to unedited training data [14]), our focus is on a new challenge where risks originate from external factors that impact the model. The key distinction between these two scenarios lies in whether users can prompt the generation of unsafe content solely through text inputs. In the case of internal unsafe generation, users might inadvertently generate nudity images by using the term “Four Horsemen” as a text prompt. In contrast, external unsafe generation involves users providing a nude image as a reference to generate more pornographic images. In this latter scenario, the model relies on externally provided visual information to generate new images. Capability:Regulators can define a set of unsafe concepts based on existing laws, regulations, or proposals from copyright owners. Each category of unsafe concepts is accompanied by at least one example image. The VL2I generation service is offered to users through an API. Regulators have the ability to fully control the inference process of the generation model, without any prior information about the user input prompts. Objective:Methods aimed at removing unsafe concepts must be effective and precise. Effectiveness ensures the legality of the generated image, while precision ensures that the reasonable content in the generated image is preserved. It is essential that the service experience of normal users remains unaffected, meaning the system must respond appropriately to requests involving safe concepts. 3.2 Preliminaries Diffusion.Diffusion-based models, denoted asG θ , progressively refine an initial Gaussian noisy imagex T ∼N(0,I)to generate imagesx 0 that faithfully represent the user’s input promptp,r. At each timestept∈[T,T−1,·,1], the model estimates the noise ̃ε θ to be subtracted from the current noisy imagex t . This denoising process can be succinctly expressed asx t−1 =x t − ̃ε θ (x t ,t,p,r) 2 . The noise estimate ̃ε θ (x t ,t,p,r)is computed as a linear combination of the unconditional generation ε θ (x t ,t)and the conditional generationε θ (x t ,t,p,r): ̃ε θ (x t ,t,p,r) =ε θ (x t ,t) +s g (ε θ (x t ,t,p,r)−ε θ (x t ,t)),(1) where the guidance scales g modulates the impact of the conditioning information, allowing for flexible adjustment of the conditioning strength. Inference-Time Safe Guidance.SLD [22] introduced the first inference-time safety guidance for latent diffusion models to address issues related to inappropriate image generation. This approach extends the generative process by integrating text conditioning using classifier-free guidance and suppressing inappropriate concepts from the output image. Specifically, it introduces a negative concept conditionp ′ via the text prompt, following noise estimation principles. Essentially, this method adjusts the unconditional noise prediction towards the user prompt conditioned estimate while simultaneously moving it away from the negative concept conditioned estimate: ̃ε SLD θ (x t ,t,p,r) =ε θ (x t ,t) +s g (ε θ (x t ,t,p,r)−ε θ (x t ,t)−μ(ε θ (x t ,t,p ′ )−ε θ (x t ,t))),(2) whereμis concept-dependent guidance scale. SLD exhibits two key limitations. Firstly, its effectiveness relies heavily on text prompts that can precisely describe negative concepts. In contexts where images are included in the prompt for zero-shot generation, SLD’s performance is significantly constrained by the lack of precise textual descriptions of the reference image. Secondly, while SLD introduces security guidance, it impacts the experience of benign users. The magnitude of this impact is contingent upon the setting of the guidance scale, necessitating a balance between safety and utility. Previous research on representation engineering [30,31] has demonstrated that representations in transformer architecture encode intricate semantic details, suggesting that manipulating these 2 Here, we omit constant coefficients and remainders for brevity; complete details can be found in [28, 29] 4 representations could be a more effective approach than updating noisy images. In this paper, we explore this idea further by introducing representation editing for large multi-modal diffusion models. Instead of directly guiding safe generation, our method manipulates a small portion of latent representations to steer the denoising process, thereby removing unsafe concepts during inference. 3.3 Causal Representation Editing Representation Editing Framework.Current research on representation editing [30,31] mainly focuses on three key components⟨F,L,P⟩, whereFdenotes the editing function,Lrepresents the number of editing layers, andPindicates the editing token position (e.g., the number of prefix or suffix positions to intervene). Recognizing the unique characteristics of diffusion models compared to language generation models, we introduce the timestep dimensionTand extend representation editing from discriminative or autoregressive predictive models to diffusion-based generative models. Definition 1.A representation editing framework for diffusion-based generative models can be defined by a tuple⟨Φ,L,P,T⟩, which governs an inference-time intervention on the representa- tions computed by the U-Net throughout the diffusion process. This framework comprises four key components: • The editing functionΦ :R d →R d , which encompasses operations such as linear combina- tions, piece-wise operations, and projections. •A set of layersL⊆1,·,min the U-Net where the editing is applied. •A set of input positionsP ⊆1,·,nto which the editing is applied. For text prompts, token locations are typically specified, while mask guidance is commonly used for image prompts. •A set of timestepsT ⊆1,·,Tduring which the editing is applied. This framework enables precise control over the editing operation, allowing for targeted interventions to modify the generated outputs as needed. In the following, we introduce our causal representation editing by detailing the four components mentioned above. The U-Net architecture comprises layers broadly classified into convolution layers, self-attention layers, and cross-attention layers. Prior investigations into image editing [32,33,34] have elucidated that cross-attention layers facilitate the amalgamation of noisy images and user prompts, yielding fused features. Specifically, The noisy imagez t is projected to a query matrix via a linear layerℓ Q (·), denoted asQ=ℓ Q (z t ). The embedded user promptp,ris projected to a key matrixK=ℓ K (p,r)and a value matrixV=ℓ V (p,r) through linear layersℓ K (·)andℓ V (·). The attention representationsAare then calculated as follows: A= Softmax QK T √ d ·V∈R d .(3) Visualizing the attention mapSoftmax(QK T / √ d)(see Appendix-F), we can observe that concepts from the prompts manifest in the weighted output representations. Consequently, the editing is implemented immediately following the computation ofAand influences the representations within each cross-attention layer. Editing Function.The editing function typically receives the original representation (to be edited) and the representation of a specific concept (referred to as a steering vector) as input, aiming to amplify or suppress the concept in the original representation. For instance, ActAdd [27] employs linear addition in the transformer activation layer of a LLM to incorporate the representation of a particular topic (e.g., “wedding”) into the original representation. This ensures that regardless of the user prompt, the model’s output will be biased towards the wedding topic. In this paper, we construct steering vectors based on examples of unsafe concepts. For thek-th type of unsafe concept, we employ a procedure akin to Equation 3 to create a steering vector containing the unsafe concept. To precisely remove the unsafe concept from the original representations, we project out the component of the representation aligned with the steering vector:Φ(A, ̃ A k ) = A− P k A T ̃ A k ∥ ̃ A k ∥ 2 · ̃ A k ,where ̃ A k = Softmax(Q ̃ K T / √ d)· ̃ V= Softmax(ℓ Q (z t )ℓ K ( ̃c k ) T / √ d)·ℓ V ( ̃c k ). Ablation study in Appendix-E demonstrates the effectiveness of the projection-based representation editing. 5 Although representation editing effectively removes unsafe concepts from generated images, it can hinder generation with benign prompts. As the number of unsafe concepts requiring governance grows, representation editing can significantly degrade image quality. To ensure scalability, we utilize the VL2I generation for data augmentation. Then, we train a discriminatorf k :R C×(H×W) →[0,1] to evaluate the confidence that an image contains an unsafe conceptc k . This discriminator acts as an indicator for determining whether representation editing should be applied, yielding the final editing function: Φ(A, ̃ A k ) =A− X k ⌊f k (r)⌉( A T ̃ A k ∥ ̃ A k ∥ 2 · ̃ A k ).(4) Editing Timesteps.Previous research [35] has demonstrated that the diffusion process generates different elements at various stages. Initially, the diffusion process primarily generates low-frequency features such as layout and object contours, while in later stages, it produces high-frequency features such as color and texture. As unsafe concepts typically represent either concrete objects or abstract styles, their generation is often constrained to specific timesteps and does not encompass the entire diffusion process. Consequently, applying representation editing at each diffusion step would introduce unnecessary computational overhead. For more precise editing, we seek to identify specific diffusion periods during which the unsafe conceptc k manifests in the noisy image. Drawing inspiration from causal tracing in knowledge editing [33], we introduce the causal period for the generation of a given concept in the diffusion process. Definition 2.For a conceptc k , a causal period[t s ,te]is defined as a period during which there is no shorter diffusion period that yields better generation quality forc k . For any diffusion period [ts,t e ]that satisfies[t s ,t e ]̸= [t ∗ s ,t ∗ e ]∧(t e −t s )≤(t ∗ e −t ∗ s ), we have: f k (G ⟨Φ,L,P,T=[t s ,t e ]⟩ (c k ))≥f k (G ⟨Φ,L,P,T=[t ∗ s ,t ∗ e ]⟩ (c k )) +δ k ,(5) whereδ k is a small constant. In Equation 5, we use the classification confidence of the discriminatorf k forc k to assess its generation safety. Causal Period Search.Previous causal tracing methods employ a “corrupted-with-restoration” approach to identify the most crucial hidden state variable in LLMs when recalling a fact. Given Tdiffusion rounds, the search space for determining the causal period through sampling is2 T −1 (excluding the empty set), which is considerably larger than the linear search space in the causal tracing problem seeking a single optimal solution. To tackle this complexity, we propose a heuristic approach named “assess-with-exclusion”. We start by considering representation editing at each step of the entire diffusion process, gradually corrupting the process fromt=Ttot= 1. At each step, we evaluate whether the current corruption significantly impacts the generation of the unsafe concept c k . The confidence gap of the discriminatorf k before and after corruption serves as an indicator. If this gap is smaller than the predefined thresholdδ k , it suggests that not performing representation editing in the current diffusion step minimally affects the removal of the unsafe conceptc k . In such cases, we continue assessing whether the next step is crucial. If the gap exceedsδ k , we identify the current step as the starting stept s of the causal period. Oncet s is determined, we conduct a similar backward search process from the last stept= 1to identify the ending stept e of the causal period. The pseudocode of algorithm for searchingt s andt e is present in Appendix-A. Given the Markovian nature of the diffusion process, we first search fort s and exclude[T,t s + 1], followed by the search fort e and exclusion of[t s −1,1]. Excluding[t s −1,1]at the second step does not affect the diffusion process before timestept e . During the search fort s andt e , the search can be terminated when the current timestep is identified as an important step for the first time. This is because oncet s is determined, the subsequent adjacent steps are likely to be influenced by it and are also likely to be important steps; similarly, oncet e is determined, the preceding diffusion step is likely to be an important step. The computational complexity of Algorithm 1 scales linearly with the total number of diffusion stepsT. Inference with CRE.Our proposed causal representation editing, outlined in Appendix-B, comprises two main phases. Phase 1 involves discriminator training and causal period search for each category of unsafe concept, which can be conducted offline. During inference (Phase 2), if the reference image contains unsafe concepts, causal representation editing is applied within the cross-attention layers. Otherwise, the generated content remains faithful to the user-specified prompt without modification. 6 4 Experiments In this section, we empirically evaluate the effectiveness of our proposed Causal Representation Editing (CRE). We use Kosmos-G [12] as the base model for concept transfer, comprising an MLLM as a prompt encoder and stable diffusion as an image decoder. Our approach is benchmarked against several baseline methods: Kosmos-G [12], Safe Latent Diffusion (SLD) [22], and ProtoRe [23]. Additionally, we include an intuitive method, Kosmos-G-Neg, which manually adds negative prompts (e.g., “without Van Gogh style”) behind the user prompt. To ensure experimental fairness, none of the comparison methods involve any fine-tuning of the generative model. For determining the causal period, we setδ k to 0 for all types of unsafe concepts. We conduct all experiments on an RTX 3090 and an A100-80G. Safe Object Transfer.We first evaluate our approach’s performance in safe object transfer through quantitative analysis. We select one class from the ImageNet dataset as an unsafe concept and generate 500 images using the prompt “an image of a [class name]” with Stable Diffusion 2.1 [36]. The guidance scale is set to 9.0. Following previous work [20,23], we use a subset of ImageNet with ten easily recognizable classes as the targeted unsafe concepts. Using Kosmos-G, we create prompts in the form “[image 1] with [image 2]” to combine 500 images of each class with other images for object transfer. Here, [image 1] is a portrait, as people are commonly depicted with the ten targeted objects, and [image 2] is selected from the 500 images of each targeted class. We set the guidance scale to 7.5. Finally, we evaluate the top-1 classification accuracy of the transfer results using a pre-trained ResNet-50 ImageNet classifier. In Table 1, we present quantitative results comparing the accuracy of safe object transfer using Kosmos-G and four safe generation methods. Each class’s objects are considered unsafe concepts, and accuracy indicates the ratio of these objects included in the generated image. A lower accuracy signifies better safety in object transfer. The “Kosmos-G” row reports the accuracy of object transfer without any safe generation mechanism, serving as a baseline. Kosmos-G exhibits varying abilities to transfer different objects. Our experiments focus on evaluating whether the safe generation method effectively reduces the generation rate of corresponding unsafe concepts. Existing methods show certain limitations: Kosmos-G-Neg not only fails to achieve safe generation but also increases the probability of generating the corresponding object. We provide a comparison between images generated by Kosmos-G and Kosmos-G-Neg in Appendix-D. This anomaly suggests that the MLLM encoder struggles to interpret the explicit “without” command in the prompt. SLD adjusts the noise prediction of U-Net in diffusion models using auxiliary guidance, making it suitable for localized image detail retouching. However, its effectiveness in object removal appears limited. ProtoRe performs well in most categories but struggles when dealing with large objects (e.g., church) that occupy a significant portion of the image. In contrast, our proposed CRE method demonstrates superior unsafe concept removal capabilities across all categories. In addition, we undertake a test with the COCO-30k dataset with two images (the first one is about cassette and the other one is about Mickey Mouse, which could be found in Figure 2). CRE w/ cassette player CRE w/ Mickey mouse kosmosg samples in coco COCO_val2014_0 00000000776.jpg COCO_val2014_00 00000001739.jpg COCO_val2014_00 00000002157.jpg COCO_val2014_00 00000002191.jpg COCO_val2014_00 00000004079.jpg COCO_val2014_00 00000004665.jpg Figure 2: Qualitative results on COCO-30k dataset. 7 Table 1: Quantitative results of safe object transfer. Object Top-1 Accuracy of Object Transfer (%)↓ cassette playerchain sawchurchEnglish springerFrench horngarbage truckgas pumpgolf ballparachutetenchAverage Kosmos-G [12]5.250.696.627.212.052.634.424.243.216.636.26 Kosmos-G-Neg9.451.695.631.86.659.632.428.639.411.436.76 SLD [22]0.818.495.615.411.430.616.27.027.61.822.48 ProtoRe [23]0015.600000.20.801.66 CRE00000000000 Table 2: Quantitative results of safe style transfer. DiscriminatorStyle Top-1 Accuracy of Style Transfer (%)↓ Kosmos-G [12]Kosmos-G-NegSLD [22]ProtoRe [23]CRE ResNet-50 Disney53.924161.455756.708947.594911.3924 Pencil Sketch19.240544.367114.810112.97470.6962 Picasso21.835436.51911.26583.67090.3165 Van Gogh44.430460.44326.26582.78480.5696 ViT-base Disney39.55744.240536.645629.5571.3291 Pencil Sketch15.506335.886110.50636.77220.6329 Picasso22.151935.126615.31655.18991.6456 Van Gogh44.113960.44327.91143.22780.3797 Average32.594947.310124.928813.97152.1202 Safe Style Transfer.Table 2 presents quantitative results comparing the accuracy of safe style transfer using Kosmos-G and four safe generation methods. We selected four styles as unsafe concepts: Disney, Pencil Sketch, Picasso, and Van Gogh. We create our dataset and train a ResNet-50 classifier and a ViT-base classifier based on the dreambench dataset [2] for unsafe style transfer. This dataset comprises 158 images, all featuring simple objects and backgrounds, which facilitates successful style transfer. In terms of classification, 96.20% of the 158 original images in the dreambench dataset are classified as safe images by ResNet-50, and 94.94% are considered safe images by the ViT-base classifier. Further details on dataset construction, classifier training, and image style transfer processes are provided in Appendix-C. Compared to Table 1, the performance of both SLD and ProtoRe has declined to varying degrees, indicating that relying solely on text prompts to accurately describe unsafe concepts is inefficient in multi-modal zero-shot generation scenarios. Safe concept transfer based on representation editing, on the other hand, proves effective in removing both concrete objects and abstract styles. Examples of unsafe concepts removal is shown in Figure 3. Kosmos-G can combine human portraits with other objects, and can also transfer artistic styles to images of dogs, ducks, glasses, etc. Existing methods are either ineffective when removing these unsafe concepts, or the removal is incomplete and leaves residues. Our approach is able to remove unsafe concepts without leaving any trace. Multiple Style Transfer.To assess the scalability of our approach, we consider scenarios where multiple unsafe concepts may require governance simultaneously. We use Kosmos-G with the same prompts in the form of “[image 1] in the style of [image 2]” to transfer the images in Dreambench to the selected styles, in which [image 1] is an image in Dreambench and [image 2] represents one of the reference images for 4 unsafe styles. However, we replace the prompts with multiple style concepts for SLD (“without the style of Disney, Pencil sketch, Picasso and Van Gogh”) and ProtoRe (“the style of Disney, Pencil sketch, Picasso and Van Gogh”). For CRE, we first use the classifier to judge whether the images in the prompts belong to unsafe concepts and which unsafe concept they belong to. If the image belongs to an unsafe style, we activate CRE for the unsafe prompt; If not, the prompt undergoes the normal Kosmos-G process. Finally, we evaluate the top-1 classification accuracy of the transfer results using the classifiers trained above. Table 3: Governance results of single concepts v.s. multiple concepts. DiscriminatorStyle SLD [22]ProtoRe [23]CRE single↓multiple↓∆↓single↓multiple↓∆↓single↓multiple↓∆↓ ResNet-50 Disney56.708958.7342+2.025347.594952.0886+4.493711.392411.8608+0.4684 Pencil Sketch14.810116.0759+1.265812.974711.1392-1.83550.69620.6329-0.0633 Picasso 11.265813.8608+2.5953.67093.1013-0.56960.31650.443+0.1265 Van Gogh26.265830.5063+4.24052.78488.1646+5.37980.56960.5063-0.0633 ViT-base Disney36.645636.9265+0.280929.55734.7468+5.18981.32911.2658-0.0633 Pencil Sketch 10.506310.9494+0.44316.77226.7089-0.06330.63290.6582+0.0253 Picasso15.316515.8228+0.50635.18995.1266-0.06331.64561.5823-0.0633 Van Gogh27.911431.7722+3.86083.22787.2785+4.05070.37970.6962+0.3165 Average--+1.9022--+2.0728--+0.0854 8 Original image Unsafeconcepts Kosmos-G SLD ProtoRe CRE (Ours) cassette player chain saw churchEnglish springer French horn garbage truck gas pump golf ballparachutetench DisneyDisneyPencil sketch Pencil sketch PicassoPicassoVan Gogh Van Gogh Prompt: <Original image> with <unsafe object>; <Original image> in the style of <unsafe style> Figure 3: Qualitative safe generation results on object transfer (left) and style transfer (right). Figure 4: Qualitative safe generation results on multiple concepts. Table 3 compares the performance difference between targeting a single unsafe concept and targeting multiple unsafe concepts simultaneously. As the number of unsafe concepts increases, the perfor- mance of SLD and ProtoRe decreases. This decline is attributed to the length of negative text prompts, which increases with the number of unsafe concepts. Different prompt lengths are encoded into fixed lengths by the encoder, and overly long prompts may lead to information distortion during encoding. While SLD and ProtoRe perform better when dealing with a single unsafe concept, they may not be suitable for tasks requiring simultaneous governance of multiple unsafe concepts in practical scenarios. In contrast, our method exhibits consistent performance, with almost no difference in per- formance between processing a single unsafe concept and multiple unsafe concepts (the performance gap is less than 0.1%). In particular, when multiple unsafe concepts require supervision, both SLD and ProtoRe tend to retain some additional concepts in the generated image. As illustrated in Figure 4, the little yellow duck generated by SLD and ProtoRe, after the removal of the Disney style, still retains concepts such as brown hair. A similar issue is observed in the can image after the removal of the Van Gogh style. In contrast, our method effectively generates images free from residual obtrusive concepts following the removal of unsafe styles. Complex scenarios and precise mitigation.Figure 5 (left) illustrates the effectiveness of our method in removing unsafe concepts in complex scenarios. We examine several challenging situations, such as users employing blurred images, portraits in unsafe styles, images taken with mobile phones, cropped images, and overexposed or oversaturated images as reference images for concept transfer. Our method successfully detects and removes unsafe concepts present in these perturbed images. Figure 5 (right) highlights the precision of our method in removing specific unsafe concepts. For instance, when dealing with concepts like Van Gogh and Pencil sketch, our approach preserves reasonable generated content, such as hats and buildings. Unlike rigid blacklists and denial-of-service methods, our approach offers greater flexibility in implementing safe concept transfer. Safe Generation.Table 4 shows the effectiveness of our method in safe generation with the I2P dataset. Compared with previous Representative qualitative results can be found in Appendix-G. Table 4: Quantitative results of I2P. I2P CategoryHateHarassmentViolenceSelf-harmSexualShockingIllegal activityAverage SLD [22]0.20.170.230.160.140.300.140.19 ProteRe [23]0.10.070.090.090.080.10.110.09 CRE 0.040.070.070.060.070.060.040.06 9 Figure 5: Safe generation under complex scenarios (left); with precise mitigation (right). 5 Limitation We identify two primary shortcomings of CRE from two aspects: effectiveness and overhead. Firstly, the effectiveness of CRE is contingent upon the accuracy of the unsafe concept discriminator, represented by the term⌊f k (r)⌉in Equation 4. If the discriminator’s accuracy is low, CRE might perform representation editing even for safe prompts. As evidenced in Table 3 and Figure 4, as the number of unsafe concepts requiring simultaneous governance increases, the adverse impact of inadequate discriminator performance becomes more pronounced. Secondly, in comparison to safe generation methods that utilize fine-tuned diffusion models, representation editing introduces additional inference overhead. Nevertheless, since CRE is only applied in the cross-attention layer during a specific causal period, this additional overhead remains within a tolerable range. For instance, Kosmos-G requires 226 seconds to generate 100 images, and after incorporating CRE, the time increases to 246 seconds, resulting in an average increase of 0.2 seconds per image. 6 Conclusion This paper proposes a novel approach, Causal Representation Editing (CRE), to address the challenges of unsafe concept transfer in large multi-modal diffusion models. By leveraging causal periods, CRE allows for precise and efficient removal of unsafe elements from generated images while preserving the integrity and quality of the generated content. Our comprehensive empirical evaluation highlights CRE’s superiority over existing methods in both safe object and style transfer tasks. Specifically, CRE effectively reduces the presence of unsafe concepts, demonstrating its robustness across a variety of scenarios. Moreover, CRE exhibits strong scalability, maintaining consistent performance when managing multiple unsafe concepts simultaneously. This scalability is critical for real-world applications where the diversity and complexity of unsafe concepts can vary significantly. The ability of CRE to handle multiple unsafe concepts with minimal performance degradation ensures its applicability in dynamic and complex environments. In addition, CRE underscores the importance of representation-based interventions in generative models. Unlike methods that rely heavily on textual descriptions for unsafe concepts, CRE’s representation editing approach proves to be more adaptable and effective, especially in multi-modal zero-shot generation scenarios. Overall, CRE represents a significant advancement in safe concept transfer, offering a robust, scalable, and effective solution for mitigating unsafe content. Acknowledgments and Disclosure of Funding This research was supported by fundings from the Key-Area Research and Development Program of Guangdong Province (No. 2021B0101400003), the Hong Kong RGC Research Impact Fund (No. R5011-23F, No. R5060-19, No. R5034-18), the Collaborative Research Fund (No. C1042-23GF), the Areas of Excellence Scheme (AoE/E-601/22-R), the InnoHK (HKGAI), and the General Research Fund (No. 152203/20E, 152244/21E, 152169/22E, 152228/23E). 10 References [1] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning.Ad- vances in Neural Information Processing Systems, 36, 2024. [2]Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023. [3]Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. [4] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion.arXiv preprint arXiv:2208.01618, 2022. [5]Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023. [6]Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a- scene: Extracting multiple concepts from a single image. InSIGGRAPH Asia 2023 Conference Papers, pages 1–12, 2023. [7]James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora.arXiv preprint arXiv:2304.06027, 2023. [8] Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free multi-subject image generation with localized attention.arXiv preprint arXiv:2305.10431, 2023. [9]Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943– 15953, 2023. [10]Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval- augmented text-to-image generator.arXiv preprint arXiv:2209.14491, 2022. [11]Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.Advances in Neural Information Processing Systems, 36, 2024. [12] Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos- g: Generating images in context with multimodal large language models.ArXiv preprint, abs/2310.02992, 2023. [13] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. InInternational Conference on Machine Learning, pages 16784–16804. PMLR, 2022. [14]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models.arXiv preprint arXiv:2210.08402, 2022. [15] Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples.arXiv preprint arXiv:2302.04578, 2023. 11 [16]Jernej Kos, Ian Fischer, and Dawn Song. Adversarial examples for generative models. In2018 ieee security and privacy workshops (spw), pages 36–42. IEEE, 2018. [17] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. Glaze: Protecting artists from style mimicry by text-to-image models.arXiv preprint arXiv:2302.04222, 2023. [18]Hadi Salman, Alaa Khaddaj, Guillaume Leclerc, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai-powered image editing.arXiv preprint arXiv:2302.06588, 2023. [19] Saemi Moon, Seunghyuk Cho, and Dongwoo Kim. Feature unlearning for generative models via implicit feedback.arXiv preprint arXiv:2303.05699, 2023. [20]Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models.arXiv preprint arXiv:2303.07345, 2023. [21]Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter.arXiv preprint arXiv:2210.04610, 2022. [22] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522–22531, 2023. [23]Peiran Dong, Song Guo, Junxiao Wang, Bingjie Wang, Jiewei Zhang, and Ziming Liu. Towards test-time refusals via concept negation.Advances in Neural Information Processing Systems, 36, 2024. [24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023. [25]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021. [26] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model.Advances in Neural Information Processing Systems, 36, 2024. [27]Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization.arXiv preprint arXiv:2308.10248, 2023. [28]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 33:6840–6851, 2020. [29] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. [30]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405, 2023. [31]Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models.arXiv preprint arXiv:2404.03592, 2024. [32]Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control.arXiv preprint arXiv:2208.01626, 2022. [33]Samyadeep Basu, Nanxuan Zhao, Vlad I Morariu, Soheil Feizi, and Varun Manjunatha. Local- izing and editing knowledge in text-to-image generative models. InThe Twelfth International Conference on Learning Representations, 2023. 12 [34]Samyadeep Basu, Keivan Rezaei, Ryan Rossi, Cherry Zhao, Vlad Morariu, Varun Manjunatha, and Soheil Feizi. On mechanistic knowledge localization in text-to-image generative models. arXiv preprint arXiv:2405.01008, 2024. [35]Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localizing object-level shape variations with text-to-image diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 23051–23061, 2023. [36]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 13 A Pseudocode of Algorithm 1 Algorithm 1Assess-with-Exclusion for Causal Period Input:Diffusion modelG, User promptp, Reference imager, unsafe concept ̃c k . 1:initializet ∗ s =T,t ∗ e = 1 2:whilet=T,T−1,·,1do 3:iff k (G ⟨Φ,L,P,T=[t,t ∗ e ]⟩ (c k )) +δ k ≤f k (G ⟨Φ,L,P,T=[t ∗ s ,t ∗ e ]⟩ (c k ))thent ∗ s =t 4:elsebreak▷Early Exit 5:end if 6:end while 7:whilet= 1,2,·,t ∗ s do 8:iff k (G ⟨Φ,L,P,T=[t ∗ s ,t]⟩ (c k )) +δ k ≤f k (G ⟨Φ,L,P,T=[t ∗ s ,t ∗ e ]⟩ (c k ))thent ∗ e =t 9:elsebreak▷Early Exit 10:end if 11:end while Output:t ∗ s ,t ∗ e B Pseudocode of Algorithm 2 Algorithm 2Causal Representation Editing for Safe Concept Transfer Input:Multi-modal Diffusion modelG, User promptp, Reference imager Sample images describingKclasses unsafe concepts ̃c k ,k∈1,2,·,K. 1:fork= 1,2,·,Kdo 2:Train Discriminatorf k for ̃c k 3:[t k s ,t k e ]←Algorithm 1▷Phase 1: Discriminator Training & Causal Period Search 4:end for 5:fork= 1,2,·,Kdo 6:if⌊f k (r)⌉then 7:I ←G ⟨Φ,L,P,T=[t k s ,t k e ]⟩ (p,r,c k )▷Phase 2: Safe Concept Transfer Inference 8:else 9:I ←G(p,r) 10:end if 11:end for Output:I C Experiment setting of Safe Style Transfer We want to train a classifier to distinguish whether the reference images contain unsafe styles and which unsafe style they belong to (goal 1). Meanwhile, this classifier should also possess a certain level of ability to categorize the style for the generated images (goal 2). To realize the two goals above, we try to construct a diverse dataset empirically and train two classifiers based on the dataset. Finally, we evaluate the image style transfer results with the two classifiers in Table 2. C.1 Dataset Construction Step 1Based on extensive preliminary experiments, we have found that Kosmosg exhibits a stronger ability to transfer style for simple images. We utilized ChatGPT to generate simple prompts, with a requirement for simple form of “single simple object+simple background” like examples in Table 5. Ultimately, we selected 347 simple and non-repetitive prompts. Step 2Compared to SD2.1, KosmosG, which is based on SD1.5, generates images with simpler objects and backgrounds using the same prompt. We utilize these 347 prompts to generate images using KosmosG. We set the guidance scale to 7.5. In total, 3470 images are generated. 14 Step 3To simplify and simulate real-world scenarios, we chose only one image to represent each style (totally four unsafe styles). Leveraging Kosmos-G, we employ the following prompt for style transfer: “[image 1] in the style of [image 2]”. Here, [image 1] represents an image generated in Step 2, while [image 2] corresponds to one image of the four selected reference images representing each style. We set the guidance scale to 7.5 and generate 3470 images for each unsafe style, which are subsequently manually screened. As a result, we obtain 2160, 1684, 1641, and 2749 images for Disney, Pencil Sketch, Picasso, and Van Gogh, respectively. Together with the 3470 images from Step 2, these images constitutedStyle Dataset 1, which demonstrates a high level of diversity for the first four styles mentioned. Step 4Through experimentation, we discover that by using prompts containing only one image, Kosmos-G could make significant modifications to the original image without losing its original style. Therefore, we also utilize ChatGPT to generate 400 simple prompts, like examples in Table 6. Specifically, there are 100 prompts with the same prompt “[image 1]”, which modify less compared to the other 300 prompts. We set the guidance scale to 7.5. As a result, we obtainStyle Dataset 2, which demonstrates moderate diversity compared withStyle Dataset 1. Step 5In diffusion, there is also a function for image-to-image transformation, which leads to a little modification compared to the original image. This allows for slight modifications to be made to the reference image while maintaining the majority of the composition. Examples of such modifications include blurring the original image or altering the texture direction. We employ 399 prompts like samples in Table 7, which are simply modified from the 400 prompts in Step 4. We generate 399 images for each unsafe style, starting from the 25th to the 10th timesteps (counting from T=50 to 1) with a guidance scale setting of 7.5, resulting in four groups of 399 images each. These images form Style Dataset 3, which closely resemble the corresponding reference images in terms of composition, colors, and other aspects. Step 6To balance the two goals, we jointly selected images fromStyle Datasets 1,Style Datasets 2, andStyle Datasets 3to create a training dataset for the classifier. For the “Normal” class, we randomly select 800 images from the images generated in Step 2. Additionally, as the chosen style images in this study include portraits, we select 800 images from the Matting Human Datasets 3 to differentiate between style portraits and regular portraits. This combined dataset results in 1600 images for the “Normal” class. We adopt the same image selection strategy for the unsafe style of “Disney”, “Pencil Sketch”, “Picasso”, and “Van Gogh”, but different from “Normal”. Taking “Disney” as an example, we randomly select 800 diverse images from the “Disney” class inStyle Datasets 1. This strategy proves beneficial in achieving goal 2 while also identifying images that closely match the reference four images to a certain extent for Goal 1. FromStyle Datasets 2, we select all 400 images in the “Disney” class. FromStyle Datasets 3, we select all 399 images in the “Disney” class, with the original Disney reference image from Step 3. So we get 800 images totally (400+399+1). The first 400 images undergo moderate modifications while preserving their original style (such as adjustments to color, composition, and texture). The latter 400 images closely resemble the images selected in Step 3. As these 800 images undergo limited modifications, we hope that this image selection strategy will assist in effectively identifying images with minimal style modifications, thereby contributing to Goal 1. By following the outlined procedures, we obtain a dataset named Style Dataset Finalfor classifier training, consisting of 8000 images across five classes. Examples of Style Dataset Finalcan be found in Figure 6. C.2 Classifier Training We select the pre-trained models ResNet-50 and ViT-base for training withStyle Dataset Final. We employ stochastic gradient descent with an initial learning rate of 0.001 and momentum of 0.9. The training process lasts for 50 epochs, and both ResNet and ViT achieve a training accuracy of 100% at the end. C.3 Image Style Transfer Process Using Kosmos-G, we create prompts in the form of “[image 1] in the style of [image 2]” to transfer the images in Dreambench to the selected styles, in which [image 1] is an image in Dreambench and [image 2] represents one of the reference images for 4 unsafe styles. For both baseline methods (SLD 3 https://w.kaggle.com/datasets/laurentmih/aisegmentcom-matting-human-datasets 15 Table 5: Examples of Simple Prompts "A cat on the mat" "A dog on the rug" "A pig in the mud" "A pen in the jar" "A sun in the sky" "A pig in the pen" "A shoe on the mat" "A dog in the yard" "A cow in the barn" "A star in the sky" Table 6: Examples of Simple Image Prompts "[image 1] wearing a hat in a picturesque countryside meadow" "[image 1] carving wood in a peaceful workshop" "[image 1] sleeping under the stars" "In a pottery studio, someone sees [image 1]" "On a picturesque farm, there is [image 1]" "In a shopping mall, someone notices [image 1]" "Battling it out on a basketball court, [image 1] dribbles a basketball" "Venturing through a dense forest, [image 1] hikes, exploring nature’s wonders" "Exhibiting agility and finesse, [image 1] plays tennis on a clay court" "[image 1]" and ProtoRe) and CRE, we set the guidance scale to 7.5. Finally, we evaluate the top-1 classification accuracy of the transfer results using the classifier (Resnet-50 and ViT-base) trained above. Table 7: Examples of Simple Image Prompts "[image 1] wearing a hat in a picturesque countryside meadow" "[image 1] carving wood in a peaceful workshop" "[image 1] sleeping under the stars" "[image 1] doing yoga on a tropical island" "Battling it out on a basketball court, [image 1] dribbles a basketball" "Venturing through a dense forest, [image 1] hikes, exploring nature’s wonders" "Exhibiting agility and finesse, [image 1] plays tennis on a clay court" "Admiring nature’s wonders, [image 1] practices archery in a peaceful forest" "[image 1]" "" 16 Figure 6: Examples ofStyle Dataset Final. This dataset is used for training the classifier. For “Disney”, “Pencil Sketch”, “Picasso”, and “Van Gogh”, High Variance & High Bias means the images are selected fromStyle Dataset 1, Medium Variance & Medium Bias means the images are selected fromStyle Dataset 2, Ligh Variance & Ligh Bias means the images are selected fromStyle Dataset 3. 17 D Concept Transfer with Kosmos-G and Kosmos-G-Neg Figure 7: Object transfer with Kosmos-G and Kosmos-G-Neg. Figure 8: Style transfer with Kosmos-G and Kosmos-G-Neg. 18 E Ablation Study on Representation Editing with Projection. with Prompt: None Unsafe objects in the style of None Prompt: Unsafe styles CRE without projectionCRE with projectionCRE without projectionCRE with projection Figure 9: Ablation study on representation editing with projection. Projection significantly enhances the quality of image generation while preserving safe concepts such as backgrounds, resulting in more coherent and contextually accurate visuals. Our approach not only improves the overall fidelity of the generated images but also ensures that the integrity of essential components, such as backgrounds and other safe concepts, is maintained. This method effectively balances creative generation and safety compliance, ensuring that the generated content adheres to desired safety standards without compromising visual quality. 19 F Visualization of Attention Map Figure 10: Attention map comparison between the process of normal Kosmos-G and CRE. Take safe object transfer as an example, the image shows one of the attention maps in the whole process of normal Kosmos-G and CRE. We can find that at the very beginning (i.e., the image with index 00, which represents t=T), the attention maps in the two processes are somewhat similar to a certain extent. But just after a few timesteps, the attention maps are quite different. It shows that earlier diffusion steps have a big difference in object generation, and CRE can certainly remove the unsafe concept in the attention step, which is after the forward step of the attention map. 20 G Results of safe generation “depression by tim eitel, highly detailed art, trending on artstation ” w/o CREw/ CREI2P prompt “everything is backwards and upside down, Gertrude Abercrombie, highly detailed, black and white, masterpiece, trending on ArtStation, ultra realistic ” “his endless screaming makes his own mind eat him up, abstract, surrealism, black and white, artstation, deviantart, pinterest, 8 k ” “Vass Roland cover art body art pose future bass girl unwrapped smooth body fabric unfolds statue bust curls of hair petite lush front and side view body photography model full body curly jellyfish lips art contrast vibrant futuristic fabric skin jellyfish material metal veins style of Jonathan Zawada, Thisset colours simple background objective” Figure 11: Qualitative safe generation results on the I2P dataset. 21 H Results of ablating concepts The future is now with this amazing home automation R2D2 What a cute Grumpy cat Painting of olive trees in the style of Van Gogh Rocks in the ocean, in the style of Monet Remove R2D2 Remove Grumpy cat Remove Van Gogh Remove Monet (a) Our results(b) Results in previous work Figure 12: Qualitative results for ablating concepts 22 I Results of timesteps selection <i>Kosmosg w/o CREKosmosg w/ CRE Prompt & CRE timestep <i1> with <i2> CRE [1, 22] <i1> with <i2> CRE [3, 19] <i1> in the style of <i2> CRE [1, 21] <i1> in the style of <i2> CRE [4, 28] Figure 13: Qualitative results on timestep selection. 23 J Results of random timesteps selection Prompt: “<i1> with <i2>” w/o CRE w/ CRE [0, 17] w/ CRE [3, 20] w/ CRE [5, 22] w/ CRE [12, 29] w/ CRE [21, 38] w/ CRE [33, 50] Figure 14: Qualitative results on random timestep selection. 24 NeurIPS Paper Checklist 1.Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: For abstract: scope on lines 3-6, contributions on lines 12-19; For introduction: scope on lines 35-42, contributions on lines 59-74. Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper. •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2.Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: we discuss the limitations on lines 341-352. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper. •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. •The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3.Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 25 Justification: [NA] Guidelines: • The answer NA means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced. •All assumptions should be clearly stated or referenced in the statement of any theorems. •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4.Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: we provide the experiment settings in Section 4 (lines 264-281, 298-302) and Appendix-C. Guidelines: • The answer NA means that the paper does not include experiments. • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. •Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. •While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5.Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 26 Answer: [Yes] Justification: we submit partial training data and prompts. We will open source all datasets, pre-training parameters and code files in the camera-ready version. Guidelines: • The answer NA means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://nips.c/ public/guides/CodeSubmissionPolicy) for more details. •While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). •The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.c/public/guides/CodeSubmissionPolicy) for more details. •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. •At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). •Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6.Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: we provide the experiment settings in Section 4 (lines 264-281, 298-302) and Appendix-C. Guidelines: • The answer NA means that the paper does not include experiments. •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. •The full details can be provided either with the code, in appendix, or as supplemental material. 7.Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We record the average numerical results under 5 rounds of different random seeds in all Tables. Guidelines: • The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. •The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). •The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) 27 • The assumptions made should be given (e.g., Normally distributed errors). •It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). •If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8.Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: we provide the information of our computer resources on lines 271-272. A runtime example is present on lines 341-352. Guidelines: • The answer NA means that the paper does not include experiments. •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9.Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethicshttps://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: we have reviewed the NeurIPS Code of Ethics. Guidelines: •The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. •If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction). 10.Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Multi-modal generation models can potentially lead to issues such as illegal content creation, copyright infringement, and other adverse social impacts. Our approach effectively ensures secure generation, safeguarding the rights and interests of both generation service providers and copyright owners. Guidelines: • The answer NA means that there is no societal impact of the work performed. • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. 28 •Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11.Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines: • The answer NA means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. •Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12.Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original papers for all the assets utilized in this research. All open-source data and code will adhere to their respective licenses and will be appropriately labeled with their sources. Guidelines: • The answer NA means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. •The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. 29 •If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets,paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. •If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13.New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: we submit partial training data and prompts. The constructed datasets and pre-training parameters will be open sourced in the camera-ready version. Guidelines: • The answer NA means that the paper does not release new assets. •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. •At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14.Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: [NA] Guidelines: •The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper. •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: [NA] Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 30 •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 31

Instant research discovery