Paper deep dive

SAGE: Exploring the Boundaries of Unsafe Concept Domain with Semantic-Augment Erasing

Hongguang Zhu, Yunchao Wei, Mengyu Wang, Siyu Jiao, Yan Fang, Jiannan Huang, Yao Zhao

Year: 2025Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 113

Models: Stable Diffusion v1.4, Stable Diffusion v2.1

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 5:28:04 PM

Summary

SAGE (Semantic-Augment concept erasing with Global-local collaborative rEtention) is a novel method for concept erasure in diffusion models. It addresses the 'word concept abyss' by transforming fixed-word erasure into concept domain erasure using cyclic self-check and self-erasure. It also introduces a global-local collaborative retention mechanism to prevent the degradation of irrelevant concepts during the erasure process.

Entities (5)

Diffusion Models · technology · 100%SAGE · method · 100%Concept Erasure · task · 98%Global-Local Collaborative Retention · mechanism · 95%Semantic-Augment Erasing · mechanism · 95%

Relation Signals (3)

SAGE → implements → Semantic-Augment Erasing

confidence 95% · SAGE consists of three main components: (a) Attack Prompt Generation, (b) Semantic-Augment Erasing, and (c) Global-Local Collaborative Retention.

SAGE → implements → Global-Local Collaborative Retention

confidence 95% · SAGE consists of three main components: (a) Attack Prompt Generation, (b) Semantic-Augment Erasing, and (c) Global-Local Collaborative Retention.

SAGE → optimizes → Diffusion Models

confidence 90% · SAGE achieves advanced comprehensive performance in erasing target concepts while preserving non-target concepts in DMs.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Diffusion models (DMs) have achieved significant progress in text-to-image generation. However, the inevitable inclusion of sensitive information during pre-training poses safety risks, such as unsafe content generation and copyright infringement. Concept erasing finetunes weights to unlearn undesirable concepts, and has emerged as a promising solution. However, existing methods treat unsafe concept as a fixed word and repeatedly erase it, trapping DMs in ``word concept abyss'', which prevents generalized concept-related erasing. To escape this abyss, we introduce semantic-augment erasing which transforms concept word erasure into concept domain erasure by the cyclic self-check and self-erasure. It efficiently explores and unlearns the boundary representation of concept domain through semantic spatial relationships between original and training DMs, without requiring additional preprocessed data. Meanwhile, to mitigate the retention degradation of irrelevant concepts while erasing unsafe concepts, we further propose the global-local collaborative retention mechanism that combines global semantic relationship alignment with local predicted noise preservation, effectively expanding the retentive receptive field for irrelevant concepts. We name our method SAGE, and extensive experiments demonstrate the comprehensive superiority of SAGE compared with other methods in the safe generation of DMs. The code and weights will be open-sourced at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

113,147 characters extracted from source content.

Expand or collapse full text

∎ 11institutetext: • Hongguang Zhu is with Faculty of Data Science, City University of Macau, 11email: zhuhongguang1103@gmail.com; • Yunchao Wei, Mengyu Wang, Siyu Jiao, Yan Fang, Jiannan Huang, Yao Zhao are with Institute of Information Science, Beijing Jiaotong University, and Beijing Key Laboratory of Advanced Information Science and Network Technology; • Yao Zhao is the corresponding author of this work, 11email: yzhao@bjtu.edu.cn; SAGE: Exploring the Boundaries of Unsafe Concept Domain with Semantic-Augment Erasing Hongguang Zhu Yunchao Wei Mengyu Wang Siyu Jiao Yan Fang Jiannan Huang Yao Zhao(🖂)🖂^( )start_FLOATSUPERSCRIPT ( 🖂 ) end_FLOATSUPERSCRIPT (Received: date / Accepted: date) Abstract Diffusion models (DMs) have achieved significant progress in text-to-image generation. However, the inevitable inclusion of sensitive information during pre-training poses safety risks, such as unsafe content generation and copyright infringement. Concept erasing finetunes weights to unlearn undesirable concepts, and has emerged as a promising solution. However, existing methods treat unsafe concept as a fixed word and repeatedly erase it, trapping DMs in “word concept abyss”, which prevents generalized concept-related erasing. To escape this abyss, we introduce semantic-augment erasing which transforms concept word erasure into concept domain erasure by the cyclic self-check and self-erasure. It efficiently explores and unlearns the boundary representation of concept domain through semantic spatial relationships between original and training DMs, without requiring additional preprocessed data. Meanwhile, to mitigate the retention degradation of irrelevant concepts while erasing unsafe concepts, we further propose the global-local collaborative retention mechanism that combines global semantic relationship alignment with local predicted noise preservation, effectively expanding the retentive receptive field for irrelevant concepts. We name our method SAGE, and extensive experiments demonstrate the comprehensive superiority of SAGE compared with other methods in the safe generation of DMs. The code and weights will be open-sourced at https://github.com/KevinLight831/SAGE. Warning: This paper contains potentially offensive outputs. Keywords: Diffusion Concept Erasure Safe Text-to-Image Generation 1 Introduction Figure 1: Pre-trained diffusion models (DMs) inevitably memorize toxic knowledge, leading to unsafe content generation issues. Previous concept erasure methods are trapped in word concept abyss by repeatedly erasing specific word (e.g., nudity), failing on concept-related prompts, i.e., concealed prompts, suggestive prompts (e.g., porn star names) and attack prompts. Our SAGE proposes concept domain erasure through cyclic self-check and self-erasure. It can efficiently achieve the model self-purification while preserving model utility. The black boxs with ∗ ∗ are added by authors for publication. Recent advancements in text-to-image diffusion models dhariwal2021diffusion ; SD1_4 ; ramesh2022hierarchical ; ding2022cogview2 have led to significant achievements in producing photo-realistic images, proving beneficial for various industrial applications midjourney ; dalle2 ; animatediff ; sv3d . However, due to the extensive use of web-scraped datasets during training, these models pose significant challenges, including the generation of unsafe content (i.e., erotic, violent, drug, illegal) tatum2023porn ; Hayden2024 and the replication of copyrighted material setty2023suit ; jiang2023ai ; roose2022art . One intuitive solution is to filter inappropriate images and retrain DMs manually. However, this approach rombach2022sd2 is not only computationally expensive but also potentially incomplete erasure esd . Additionally, using Safety Checkers rando2022red to detect and mitigate harmful outputs offers an alternative, but this approach depends on the accuracy of the detector and is limited by intrinsic biases. In response to the above challenges, concept erasing esd has emerged as a potentially promising solution. Specifically, given a concept described in text, the pre-trained model is fine-tuned to forget the related memory of that concept, thus preventing the generation of associated content. Some approaches use preprocessed pairs of images and masks liu2023grounding to suppress attention activation MACE ; receler in concept-related regions, or remap the target concept to a benign one UCE ; RECE using pre-defined pairs of target and benign prompts. However, these methods have notable limitations in generalizing erasure. (1) Explicit Concept Representation: they rigidly define the concept as a specific word (e.g., nudity) and repeatedly erase it, leading to erasure overfitting on single word. (2) Explicit Erasure Mode: whether through suppressing attention regions or remapping concept words, these methods are constrained by the knowledge within the preprocessed data and cannot harness the inherent knowledge of model for implicit self-erasure. As illustrated in Fig. 1, these limitations result in post-erasure DMs still generating unsafe content when confronted with concealed prompts, suggestive prompts, and attack prompts. Concept-related descriptions are diverse and cannot be exhaustively listed, but large models inherently contain compressed world knowledge zhang2023large . Therefore, it is more cost-effective and promising to perform concept surgery (erasure or modification) leveraging the knowledge topology within the large model itself. In other words, the one who opens the Pandora’s box must close it. To achieve this, we transform traditional fixed-word concept representations into self-augment token embeddings. Unlike static discrete words, token embedding can be continuously updated based on the training feedback from DM. By freezing the denoiser responsible for image generation and adjusting the text encoder that manages conditional mapping, we establish a connection between the textual space of current training DM and the visual space of original freeze DM. Further, the self-augment token embedding can be efficiently optimized to explore the boundaries of the target concept domain. By aligning the opposite direction of guidance noise produced by the optimized self-augment token embedding, DM can steer the generation tendency away from the target concept domain, facilitating self-purification. Excessive erasure of unsafe concepts will inevitably degrade the retention of irrelevant concepts, potentially affecting the usability of concept-erased DMs in severe cases. To this end, we propose a global-local collaborative retention mechanism for irrelevant concepts. It first aligns the textual semantic graph of irrelevant concepts at the global relational level, then identifies concepts with the most significant semantic drift to apply additional local predictive noise retention constraints. We call our approach Semantic-Augment concept erasing with Global-local collaborative rEtention (SAGE). Extensive experiments demonstrate that SAGE achieves advanced comprehensive performance in erasing target concepts while preserving non-target concepts. Moreover, our method has high training efficiency and supports zero-cost migration within the same series of DMs. 2 Related Work Safe generation of DMs. Leveraging training on large-scale web-crawled datasets, DMs SD1_4 ; sd3 ; sdxl can generate high-quality images and exhibit immense creative potential. However, since these datasets are not curated, DMs inadvertently memorize unsafe and copyrighted content, leading to unsafe generation. To mitigate this issue, the efforts can be classified into three aspects: (1) Pre-processing, (2) Post-processing, and (3) Model editing. Pre-processing methods utilize pretrained detectors to filter out images containing unsafe content, and retrain DMs after filtering. However, retraining from scratch is computationally expensive and impractical for addressing evolving erasure requests. e.g., Stable Diffusion v2.1 rombach2022sd2 consumes 150,000 GPU hours to retrain on the filtered LAION-5B dataset schuhmann2022laion . This extensive filtering process also has been found to negatively impact output quality connor2022sd , and DMs may still not be properly sanitized esd . Post-processing methods use safety checkers to identify unsafe content and block risky outputs. Several organizations midjourney ; dalle2 deploy this approach by involving a blacklist-style post-hoc filter. However, safety checkers of open-source models can be easily circumvented by modifying code SmithMano2022 . The safety filters of closed-source models like DALL⋅·⋅E 2 dalle2 can be bypassed using attack prompts sneakyprompt ; RingABell . Similar to pre-processing methods, Post-processing methods also rely on the accuracy of detectors, whose inherent biases can result in unreliable exclusion of unsafe content. Model editing methods leverage the original DMs to erase or redirect target concepts, effectively eliminating potential harmful biases before deployment. Due to their low cost, flexible operation, and effective erasure, these methods have increasingly attracted community interest. We categorize existing model editing methods for concept erasing into three categories based on their technical characteristics. I. Guidance-based Methods SLD ; sdd ; SPM ; AC ; esd : SLD SLD modifies denoising process in inference stage and introduces negative guidance to prevent unsafe content generation. However, it only suppresses undesired concepts in inference rather than complete removal. ESD esd predicts negative guided noises and trains DM to steer conditional predictions away from target concepts. I. Attention Re-steering Methods FMN ; MACE employ attention re-steering to identify regions associated with target concepts within the cross-attention layers of UNet ronneberger2015u ; ddpm . By diminishing the cross-attention activation related to target concepts, DMs gradually disregard these concepts during image generation. However, the effectiveness of this method depends heavily on the accurate location of the concept-related region and the quality of pre-processing images. Besides, this method is limited to scenarios where the concept-related regions can be explicitly identified. For strongly coupled concepts like artistic styles, it is difficult to accurately pre-process masks that isolate content purely related to style without capturing object-specific details. I. Closed-form Editing Methods UCE ; MACE ; RECE optimize the key and value projection matrices in the cross-attention layers of UNet. Specifically, UCE UCE recalibrates the embedding of a target prompt (e.g., nudity) to a benign prompt (e.g., wearing clothes), while keeping other concepts unchanged. MACE MACE first erases a single prompt using the attention re-steering method and then jointly optimizes the projection matrices. Despite a certain effectiveness, they cannot achieve generalized erasure against concealed prompts, suggestive prompts, and attack prompts RingABell ; sneakyprompt . Since concept-related prompts are inexhaustible, these methods, which focus only on human-cognizable language specified in the dataset, fail to address the broader spectrum of machine-cognizable language. Recent red-teaming works zhang2024adversarial ; RingABell ; prompting4debugging have leveraged the idea of textual inversion gal2022image to generate attack prompts that provoke concept-erased DMs to regenerate unsafe images. Inspired by this, some works try to introduce red-teaming methods prompting4debugging ; zhang2023generate to generate attack prompts and further support adversarial training to improve robustness. RACE RACE optimize random perturbation into attack perturbation to adversarially finetune UNet. Receler receler integrates a lightweight adapter within cross-attention layers, utilizing adversarial prompt learning to improve robustness. RECE RECE extends closed-form editing methods by incorporating adversarial fine-tuning on matrix-modified cross-attention layers. AdvUnlearn Advunlearn formulates the concept erasure as a bilevel optimization problem, simultaneously optimizing for both target concept removal and non-target concept preservation. Our SAGE leverages the modal space relationship between the current DM and the original DM to efficiently explore the boundaries of target concepts. Thus, there is no need for complex attack prompt optimization. Moreover, in contrast to most methods esd ; RACE ; UCE ; MACE ; receler ; RECE that modify the UNet through fine-tuning, our method only optimizes the text encoder. This design enables the purified text encoder to be directly deployed across DMs that share the same text encoder architecture, eliminating the need for retraining. 3 Preliminary Stable Diffusion Models. Our study builds upon Stable Diffusion (SD) Models SD1_4 , which incorporate conditional text prompts into image embedding to guide the generation process. The diffusion process begins with a noise latent zz drawn from a Gaussian distribution ⁢(0,1)01N(0,1)N ( 0 , 1 ). Over a series of T time steps, this noise latent undergoes a gradual denoising process guided by textual embedding clip , transforming into a clean latent 0subscript0z_0z0. Meanwhile, the encoder of pre-trained Variational Autoencoder (VAE) kingma2013auto ; vqgan transforms the input image x into the latent =ℰ⁢(x)ℰz=E(x)z = E ( x ), and the decoder reconstructs the image from latent form, where ⁢()=x^≈x^D(z)= x≈ xD ( z ) = over start_ARG x end_ARG ≈ x. Finally, the denoised latent is decoded into a clean image by decoder. At each time step t, DM predicts noise using UNet denoiser ϵsubscriptitalic-ϵε_ θϵbold_italic_θ, parameterized by θitalic_θ and conditioned on the token embedding of input prompt τ. The training objective for θitalic_θ is to minimize the denoising error, defined as: minimize⁢∼ℰ⁢(x),τ,t,ϵ∼⁢(0,1)⁢[‖ϵ−ϵ⁢(t,τ,t)‖22],minimizesubscriptformulae-sequencesimilar-toℰsimilar-toitalic-ϵ01delimited-[]subscriptsuperscriptnormitalic-ϵsubscriptitalic-ϵsubscript22 θminimize~E_z % (x),τ,t,ε (0,1) [ \|ε-ε_% θ(z_t,τ,t) \|^2_2 ],underitalic_θ start_ARG minimize end_ARG blackboard_Ez ∼ E ( x ) , τ , t , ϵ ∼ N ( 0 , 1 ) [ ∥ ϵ - ϵbold_italic_θ ( zitalic_t , τ , t ) ∥22 ] , (1) where tsubscriptz_tzitalic_t is the noisy version of zz up to the time step t. For notational simplicity, we will omit the time step t in the following paragraphs. Concept Erasing in DM. Concept erasing was proposed to remove undesirable concepts from the latent space of DM. Inspired by classifier-free guidance ho2021classifier , ESD esd first proposes concept erasure by guiding the predicted noise away from the conditional noise of target concept c. The diffusion process of ESD can be denoted as: ϵn⁢(t,τc)←ϵo⁢(t)−η⁢[ϵo⁢(t,τc)−ϵo⁢(t)]⏟ϵn^⁢(t,τc),←subscriptitalic-ϵsubscriptnsubscriptsubscriptcsubscript⏟subscriptitalic-ϵsubscriptosubscriptdelimited-[]subscriptitalic-ϵsubscriptosubscriptsubscriptcsubscriptitalic-ϵsubscriptosubscript^subscriptitalic-ϵsubscriptnsubscriptsubscriptcε_ θ_n(z_t, _c)% ← _ θ_o(z_t)-% η[ _ θ_o(z_t, _c)-% _ θ_o(z_t)]_ _ % θ_n(z_t, _c),ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( zitalic_t , τroman_c ) ← under⏟ start_ARG ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t ) - η [ ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t , τroman_c ) - ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t ) ] end_ARGover start_ARG ϵ start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT end_ARG ( zitalic_t , τroman_c ) end_POSTSUBSCRIPT , (2) where osubscripto θ_oitalic_θroman_o represents the original DM and nsubscriptn θ_nitalic_θroman_n denotes the training DM. τcsubscriptc _cτroman_c is the token embedding of concept word c, and η denotes the guidance scale. ϵo⁢(t)subscriptitalic-ϵsubscriptosubscript _ θ_o(z_t)ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t ) is the noise predicted by the original DM with a null prompt input. ϵn^⁢(t,τc)^subscriptitalic-ϵsubscriptnsubscriptsubscriptc _ θ_n(z_t, _c)over start_ARG ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT end_ARG ( zitalic_t , τroman_c ) is the negative guidance noise. This process only needs the concept words to induce the intrinsic concept-related noise of DM. The erasure loss is formalized as: ℒerase=⁢[‖ϵn⁢(t,τc)−ϵn^⁢(t,τc)‖22],subscriptℒerasedelimited-[]subscriptsuperscriptnormsubscriptitalic-ϵsubscriptnsubscriptsubscriptc^subscriptitalic-ϵsubscriptnsubscriptsubscriptc22 _erase=E [ \| _ % θ_n(z_t, _c)- _ % θ_n(z_t, _c) \|^2_2% ],Lroman_erase = blackboard_E [ ∥ ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( zitalic_t , τroman_c ) - over start_ARG ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT end_ARG ( zitalic_t , τroman_c ) ∥22 ] , (3) Attack prompts against concept-erased DM. Red-teaming works prompting4debugging ; zhang2023generate seek to circumvent the erasure mechanisms and compel concept-erased DM to again generate harmful images using attack prompts. The token embedding of perturbed concept prompt, τc′subscriptc′ _c τroman_c′ is created by manipulating tokens or their embedding through random initialization RACE ; receler ; Advunlearn . The process for generating attack prompts can be represented as: minimize‖τc′−τc‖≤δnormsuperscriptsubscriptc′subscriptcminimize \| _c - _c\|≤% δminimize _UNDERACCENT ∥ τroman_c′ - τroman_c ∥ ≤ δ end_UNDERACCENT start_ARG minimize end_ARG ⁢[‖ϵ∗⁢(t,τc′)−ϵo⁢(t,τc)‖22],delimited-[]superscriptsubscriptnormsubscriptitalic-ϵsubscriptsubscriptsuperscriptsubscriptc′subscriptitalic-ϵsubscriptosubscriptsubscriptc22 [ \| _ θ_*(z_t,% _c )- _ θ_o(z_% t, _c) \|_2^2 ],blackboard_E [ ∥ ϵbold_italic_θ start_POSTSUBSCRIPT ∗ end_POSTSUBSCRIPT ( zitalic_t , τroman_c′ ) - ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t , τroman_c ) ∥22 ] , (4) where ∗subscript θ_*italic_θ∗ is the frozen victim concept-erased DM. The perturbed token embedding τc′subscriptc′ _c τroman_c′ is optimized by projected gradient descent (PGD) PGD in the continuous textual embedding. Finally, Mapping the token embedding to the discrete texts to generate the attack prompt. Eq.(4) aims to optimize τc′subscriptc′ _c τroman_c′ that induce concept-erased DM ∗subscript θ_*italic_θ∗ to regenerate unsafe content about τcsubscriptc _cτroman_c. The constraint in Eq.(4) ensures that τc′subscriptc′ _c τroman_c′ remains close to τcsubscriptc _cτroman_c, subject to the added initial perturbation strength δ. 4 Method Figure 2: The proposed SAGE involves three key components: the adversarial training of (a) attack prompt generation and (b) semantic-augment erasing, enabling the DM to self-check and self-erase; and (c) global-local collaborative retention mechanism that integrates semantic relations and predicted noise to preserve irrelevant concepts. We propose SAGE to achieve concept domain erasing while preserving the native generation capability of DM. As depicted in Fig. 2, our SAGE consists of three main components: (a) Attack Prompt Generation, (b) Semantic-Augment Erasing, and (c) Global-Local Collaborative Retention. The former two components generate attack prompts to further augment erasure based on the semantic space relationship. The latter strives to maintain semantic alignment and generation capability of non-target concepts by simultaneously preserving global semantic relationships and local noise predictions. The following sections will provide a detailed explanation. 4.1 Semantic-Augment Erasing To enable the DM to adaptively explore and erase the boundaries of the concept domain during training, our semantic-augment erasing operates in two phases: attack prompt optimization and DM parameter optimization. These two phases continuously alternate, where each phase uses the optimized output of the other as input, thereby creating a cyclic adversarial training process of self-check and self-erasure. Specifically, during attack prompt optimization, DM parameters remain fixed, and once the attack prompt is optimized, it is used as input to further finetune DM parameters. The Eq.(4) describes a one-way attack optimization from the randomly perturbed prompt toward the target concept, just like what RACE RACE , Receler receler , and AdvUnlearn Advunlearn conducted, which we refer to as the outside-in methods. The distribution randomness of their generated attack prompts in the concept domain depends on the randomness of initial perturbation δ. Besides, since the outside-in methods require a multi-round denoising process of the DM up to time t for tsubscriptz_tzitalic_t, and optimize δ from random initialization into the concept domain. It results in high computational and time consumption. In contrast, we propose the inside-out methods to explore the boundaries of concept domain starting from the target concept prompt rather than a perturbed prompt. For example, when erasing the Van Gogh concept, attack prompt is first initialized by combining concept word with a randomly selected template from predefined template library (e.g.‘An artwork by [Van Gogh]’). For simplicity in formula form, the token embedding of original attack prompt can be denoted as τp=[τt,τc]subscriptsubscriptsubscript _p=[ _t, _c]τitalic_p = [ τitalic_t , τitalic_c ], where τtsubscript _tτitalic_t and τcsubscript _cτitalic_c respectively represent the token embeddings of template and concept word. At this stage, the current training DM nsubscriptn θ_nitalic_θroman_n will be frozen and only template token embedding τtsubscript _titalic_τbold_italic_t is optimized to generate attack token embedding τpsubscript _pτitalic_p against nsubscriptn θ_nitalic_θroman_n. In other words, the original attack prompt is modeled as the centroid of target concept domain, while template token embedding is optimized to introduce semantic perturbations to attack prompt, thereby continuously exploring boundaries of target concept domain. Through such perturbations in the semantic space, it achieves efficient coverage of hard-to-quantify expressions (e.g., concealed prompts, suggestive prompts and attack prompts) within the semantic space. As shown in Fig. 2(a), to measure the relationship between original DM osubscripto θ_oitalic_θroman_o and current DM nsubscriptn θ_nitalic_θroman_n, aggregator function ⁢(∗)g(*)g ( ∗ ) is introduced to pool the textual embedding. Given that SD uses the pre-trained CLIP clip text encoder, ⁢(∗)g(*)g ( ∗ ) can be the native pre-trained aggregator of CLIP as ⁢(∗)g(*)g ( ∗ ). This allow to obtain vector fop¯=⁢(To⁢(τ¯p))superscriptsubscript¯subscriptsubscriptosubscript¯f_o p=g(T_ θ_o( τ_% p))fitalic_oover¯ start_ARG p end_ARG = g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( over¯ start_ARG τ end_ARGp ) ) as anchor feature of target concept domain. TosubscriptsubscriptoT_ θ_oTbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT is the text encoder of original DM osubscripto θ_oitalic_θroman_o. τ¯psubscript¯ τ_pover¯ start_ARG τ end_ARGp represents the original τpsubscript _pτitalic_p and remains unchanged throughout the optimized process of attack prompt τpsubscript _pτitalic_p. Since UNet UosubscriptsubscriptoU_ θ_oUbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT is frozen and shared by both osubscripto θ_oitalic_θroman_o and nsubscriptn θ_nitalic_θroman_n, the anchor feature fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG , as centroid of target concept domain, can guide UosubscriptsubscriptoU_ θ_oUbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT to accurately generate target content. It means fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG has high probability of generating target concept images. Similarly, the textual feature fopsuperscriptsubscriptf_o^pfitalic_oitalic_p and fnpsuperscriptsubscriptf_n^pfitalic_nitalic_p can be obtained by projection of TosubscriptsubscriptoT_ θ_oTbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT and TnsubscriptsubscriptnT_ θ_nTbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT, respectively. fop=⁢(To⁢(τp)),fnp=⁢(Tn⁢(τp)),formulae-sequencesuperscriptsubscriptsubscriptsubscriptosubscriptsuperscriptsubscriptsubscriptsubscriptnsubscript f_o^p=g(T_ θ_o( _p)),% f_n^p=g(T_ θ_n( _p)),fitalic_oitalic_p = g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_p ) ) , fitalic_nitalic_p = g ( Tbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( τitalic_p ) ) , (5) To enable current model nsubscriptn θ_nitalic_θroman_n to escape from the word concept abyss, it is essential to further train nsubscriptn θ_nitalic_θroman_n on more valuable concept prompts which are within the target concept domain but distant from the concept anchor. Thus, we first propose the criterion ℋ1subscriptℋ1H_1H1, maximizeτtℋ1subscriptmaximizesubscriptℋ1 _tmaximize _1start_UNDERACCENT τitalic_t end_UNDERACCENT start_ARG maximize end_ARG H1 =⁢(fop¯,fnp)absentsuperscriptsubscript¯superscriptsubscript =Sim(f_o p,f_n^p)= Sim ( fitalic_oover¯ start_ARG p end_ARG , fitalic_nitalic_p ) (6) =⁢(⁢(To⁢(τ¯p)),⁢(Tn⁢(τp))),absentsubscriptsubscriptosubscript¯subscriptsubscriptnsubscript =Sim(g(T_ θ_o( % τ_p)),g(T_ θ_n( _p))),= Sim ( g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( over¯ start_ARG τ end_ARGp ) ) , g ( Tbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( τitalic_p ) ) ) , (7) which ⁢(∗,∗)Sim(*,*)Sim ( ∗ , ∗ ) is the cosine similarity measure function. As shown in Fig.3(a), ℋ1subscriptℋ1H_1H1 can keep fnpsuperscriptsubscriptf_n^pfitalic_nitalic_p as close as possible to anchor feature fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG, ensuring that the optimized attack prompt τpsubscript _pτitalic_p induces the current model to regenerate the target concept content described by the original prompt τ¯psubscript¯ τ_pover¯ start_ARG τ end_ARGp. For encouraging τpsubscript _pτitalic_p to explore the boundary of the concept domain, criterion ℋ2subscriptℋ2H_2H2 is further proposed to ensure that fopsuperscriptsubscriptf_o^pfitalic_oitalic_p is as far as possible from the anchor feature fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG. minimizeτtℋ2subscriptminimizesubscriptℋ2 _tminimize _2start_UNDERACCENT τitalic_t end_UNDERACCENT start_ARG minimize end_ARG H2 =⁢(fop¯,fop),absentsuperscriptsubscript¯superscriptsubscript =Sim(f_o p,f_o^p),= Sim ( fitalic_oover¯ start_ARG p end_ARG , fitalic_oitalic_p ) , (8) =⁢(⁢(To⁢(τ¯p)),⁢(To⁢(τp))),absentsubscriptsubscriptosubscript¯subscriptsubscriptosubscript =Sim(g(T_ θ_o( % τ_p)),g(T_ θ_o( _p))),= Sim ( g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( over¯ start_ARG τ end_ARGp ) ) , g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_p ) ) ) , (9) As depicted in Fig. 3(b), due to the use of same TosubscriptsubscriptoT_ θ_oTbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT and UosubscriptsubscriptoU_ θ_oUbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT, fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG and fopsuperscriptsubscriptf_o^pfitalic_oitalic_p shares the unified visual generation space. Further reducing ℋ2subscriptℋ2H_2H2 while increasing ℋ1subscriptℋ1H_1H1 ensures that fnpsuperscriptsubscriptf_n^pfitalic_nitalic_p generates content that differs as much as possible from the anchor fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG while remaining within the target concept domain. The anchor feature fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG serves as a bridge linking the relationship between nsubscriptn θ_nitalic_θroman_n and osubscripto θ_oitalic_θroman_o. By iteratively optimizing τpsubscript _pτitalic_p based on the feedback differences between TosubscriptsubscriptoT_ θ_oTbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT and TnsubscriptsubscriptnT_ θ_nTbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT, our method explores the boundaries of the concept domain from an inside-out perspective. The attack loss ℒattacksubscriptℒattackL_attackLroman_attack is derived by jointly optimizing Eq.(7) and (9), enabling efficient discovery of valuable attack prompt embeddings. ℒattack=−ℋ1+ℋ2ℋ1,subscriptℒattacksubscriptℋ1subscriptℋ2subscriptℋ1 _attack=-H_1+ H_% 2H_1,Lroman_attack = - H1 + divide start_ARG H2 end_ARG start_ARG H1 end_ARG , (10) Unidirectional optimization of either ℋ1subscriptℋ1H_1H1 or ℋ2subscriptℋ2H_2H2 will be suboptimal. Specifically, isolated optimization of ℋ1subscriptℋ1H_1H1 risks overfitting to the original prompt τ¯psubscript¯ τ_pover¯ start_ARG τ end_ARGp, potentially compromising the model’s ability to generalize beyond the target concept word distribution. Conversely, isolated optimization of ℋ2subscriptℋ2H_2H2 induces concept drift, potentially increasing the risk of attack prompts becoming non-target concept prompts After a certain number of optimization steps PGD , the attack prompt τpsubscript _pτitalic_p will replace the traditional concept word τcsubscript _cτitalic_c as the input for erasure training. By replacing τcsubscript _cτitalic_c with τpsubscript _pτitalic_p in Eq.(3), the erasure loss ℒerasesubscriptℒeraseL_eraseLroman_erase redirects diffusion trajectory under target concept-related prompts by distilling the opposite predicted noise direction of osubscripto θ_oitalic_θroman_o into nsubscriptn θ_nitalic_θroman_n, thereby achieving semantic-augment erasing. ϵn^⁢(t,τp)^subscriptitalic-ϵsubscriptnsubscriptsubscriptp _ θ_n(z_t, _% p)over start_ARG ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT end_ARG ( zitalic_t , τroman_p ) =ϵo⁢(t)−η⁢[ϵo⁢(t,τp)−ϵo⁢(t)],absentsubscriptitalic-ϵsubscriptosubscriptdelimited-[]subscriptitalic-ϵsubscriptosubscriptsubscriptpsubscriptitalic-ϵsubscriptosubscript = _ θ_o(z_t)-η[% _ θ_o(z_t, _p)-% _ θ_o(z_t)],= ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t ) - η [ ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t , τroman_p ) - ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t ) ] , (11) ℒerasesubscriptℒerase _eraseLroman_erase =⁢[‖ϵn⁢(t,τp)−ϵn^⁢(t,τp)‖22],absentdelimited-[]subscriptsuperscriptnormsubscriptitalic-ϵsubscriptnsubscriptsubscriptp^subscriptitalic-ϵsubscriptnsubscriptsubscriptp22 =E [ \| _ θ_n(% z_t, _p)- _ θ_n(% z_t, _p) \|^2_2 ],= blackboard_E [ ∥ ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( zitalic_t , τroman_p ) - over start_ARG ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT end_ARG ( zitalic_t , τroman_p ) ∥22 ] , (12) Before semantic-augment erasing, we perform warm-up training using Eq.(3) to endow nsubscriptn θ_nitalic_θroman_n preliminary concept-erasure capability. This warm-up phase creates an initial concept abyss for nsubscriptn θ_nitalic_θroman_n at fnpsuperscriptsubscriptf_n^pfitalic_nitalic_p, and ensures fnpsuperscriptsubscriptf_n^pfitalic_nitalic_p differs from the original embedding fop¯superscriptsubscript¯f_o pfitalic_oover¯ start_ARG p end_ARG from start, thereby providing sufficient exploration space for subsequent ℋ1subscriptℋ1H_1H1 optimization. Figure 3: The schematic of ℋ1subscriptℋ1H_1H1 and ℋ2subscriptℋ2H_2H2. 4.2 Global-Local Collaborative Retention To maintain usability, previous methods introduced a retain set retainsubscriptretainC_retainCroman_retain containing irrelevant concept prompts, and applied consistency regularization on the predicted noise of osubscripto θ_oitalic_θroman_o and nsubscriptn θ_nitalic_θroman_n for same prompt τrsubscript _rτitalic_r. However, predicting noise by UNet requires substantial memory and computation, which limits the batch size br⁢e⁢t⁢a⁢i⁢nsubscriptb_retainbitalic_r e t a i n of sampled retain prompts, hindering efficient global optimization. e.g., a 40GB A100 can only support br⁢e⁢t⁢a⁢i⁢nsubscriptb_retainbitalic_r e t a i n = 5 at most. Unlike previous methods that focus solely on visual predicted noise consistency while neglecting the alignment of conceptual semantic relationships, ⁢(∗)g(*)g ( ∗ ) allows to construct textual semantic relationship graph among retain prompts τrsubscript _rτitalic_r, thereby expanding br⁢e⁢t⁢a⁢i⁢nsubscriptb_retainbitalic_r e t a i n and selectively choosing part prompts for predicted noise calculation. In other words, by constraining the consistency of the semantic relationship graph between osubscripto θ_oitalic_θroman_o and nsubscriptn θ_nitalic_θroman_n, it can broaden the receptive field of nsubscriptn θ_nitalic_θroman_n on retainsubscriptretainC_retainCroman_retain. The formalized expression is as follows: ℳ⁢(o,τr)ℳsubscriptosubscript ( θ_o, _r)M ( italic_θroman_o , τitalic_r ) =⁢(To⁢(τr))⋅⁢(To⁢(τr))T,absent⋅subscriptsubscriptosubscriptsuperscriptsubscriptsubscriptosubscriptT =g(T_ θ_o( _r))·g% (T_ θ_o( _r))^T,= g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_r ) ) ⋅ g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_r ) )T , (13) ℳ⁢(n,τr)ℳsubscriptnsubscript ( θ_n, _r)M ( italic_θroman_n , τitalic_r ) =⁢(Tn⁢(τr))⋅⁢(Tn⁢(τr))T,absent⋅subscriptsubscriptnsubscriptsuperscriptsubscriptsubscriptnsubscriptT =g(T_ θ_n( _r))·g% (T_ θ_n( _r))^T,= g ( Tbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( τitalic_r ) ) ⋅ g ( Tbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( τitalic_r ) )T , (14) ℒgraphsubscriptℒgraph _graphLroman_graph =‖ℳ⁢(o,τr)−ℳ⁢(n,τr)‖22,absentsubscriptsuperscriptnormℳsubscriptosubscriptℳsubscriptnsubscript22 = \|M( θ_o, _r)- % M( θ_n, _r) \|^2_2,= ∥ M ( italic_θroman_o , τitalic_r ) - M ( italic_θroman_n , τitalic_r ) ∥22 , (15) where ℳ⁢(⋅,τr)∈ℝbr⁢e⁢t⁢a⁢i⁢n×br⁢e⁢t⁢a⁢i⁢nℳsubscript⋅subscriptsuperscriptℝsubscriptsubscriptM( θ_·, _r) ^b_retain% × b_retainM ( italic_θ⋅ , τitalic_r ) ∈ blackboard_Rbitalic_r e t a i n × bitalic_r e t a i n. ℳ⁢(o,τr)ℳsubscriptosubscriptM( θ_o, _r)M ( italic_θroman_o , τitalic_r ) and ℳ⁢(n,τr)ℳsubscriptnsubscriptM( θ_n, _r)M ( italic_θroman_n , τitalic_r ) are the semantic relationship graphs of retain prompts on osubscripto θ_oitalic_θroman_o and nsubscriptn θ_nitalic_θroman_n, respectively. To further maintain image generation quality for concepts with significant semantic drift, while preserving overall semantic alignment, we first introduce the criterion ℋ3subscriptℋ3H_3H3 to measure semantic drift between osubscripto θ_oitalic_θroman_o and nsubscriptn θ_nitalic_θroman_n for the same τrsubscript _rτitalic_r. ℋ3=⁢(⁢(To⁢(τr)),⁢(Tn⁢(τr))),subscriptℋ3subscriptsubscriptosubscriptsubscriptsubscriptnsubscript _3=Sim(g(T_ θ_o% ( _r)),g(T_ θ_n( _r))),H3 = Sim ( g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_r ) ) , g ( Tbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( τitalic_r ) ) ) , (16) The prompts τ~rsubscript~ τ_rover~ start_ARG τ end_ARGr corresponding to the Top-k minimum values in ℋ3∈ℝbr⁢e⁢t⁢a⁢i⁢n×1subscriptℋ3superscriptℝsubscript1H_3 ^b_retain× 1H3 ∈ blackboard_Rbitalic_r e t a i n × 1 will be selected to form the subset imagesubscriptimageC_imageCroman_image. These prompts in imagesubscriptimageC_imageCroman_image will then be used to constrain the consistency of predicted noise between osubscripto θ_oitalic_θroman_o and nsubscriptn θ_nitalic_θroman_n, thereby achieving additional local noise consistency for the selected weaker semantic alignment subset imagesubscriptimageC_imageCroman_image of retainsubscriptretainC_retainCroman_retain. ℒimage=τ~r∈image⁢[‖ϵo⁢(t,τ~r)−ϵn⁢(t,τ~r)‖22],subscriptℒimagesubscriptsubscript~subscriptimagedelimited-[]superscriptsubscriptnormsubscriptitalic-ϵsubscriptosubscriptsubscript~subscriptitalic-ϵsubscriptnsubscriptsubscript~22L_image=E_ τ_r _% image\! [ \| _ θ_o(z_% t, τ_r)- _ θ_n(z_t,% τ_r) \|_2^2 ],Lroman_image = blackboard_Eover~ start_ARG τ end_ARG start_POSTSUBSCRIPT r ∈ Croman_image end_POSTSUBSCRIPT [ ∥ ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t , over~ start_ARG τ end_ARGr ) - ϵbold_italic_θ start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( zitalic_t , over~ start_ARG τ end_ARGr ) ∥22 ] , (17) Together with the aforementioned ℒerasesubscriptℒeraseL_eraseLroman_erase and ℒgraphsubscriptℒgraphL_graphLroman_graph, the overall loss ℒLL is formalized as: ℒ⁢(,o,τp,τr)=ℒerase+γt⁢ℒgraph+γv⁢ℒimage,ℒsubscriptabsentsubscriptosubscriptsubscriptsubscriptℒerasesubscriptsubscriptℒgraphsubscriptsubscriptℒimage ( θ_, θ_o,τ% _p, _r)=L_erase+ _tL_% graph+ _vL_image,L ( italic_θ , italic_θroman_o , τitalic_p , τitalic_r ) = Lroman_erase + γitalic_t Lroman_graph + γitalic_v Lroman_image , (18) where γtsubscript _tγitalic_t and γvsubscript _vγitalic_v are regularization parameters. The integration of global-level ℒgraphsubscriptℒgraphL_graphLroman_graph and local-level ℒimagesubscriptℒimageL_imageLroman_image enables overall semantic alignment for irrelevant concepts while adaptively enhancing retention of weakly aligned concepts. The proposed SAGE method is comprehensively detailed in Algorithm 1. Algorithm 1 Algorithm of SAGE Input: Iteration Number I, attack step number J, token embedding of template and concept word, τtsubscript _tτitalic_t, τcsubscript _cτitalic_c, token embedding of retaining prompts τrsubscript _rτitalic_r with batch size br⁢e⁢t⁢a⁢i⁢nsubscriptb_retainbitalic_r e t a i n, image retain batch size k, learning rate α, attack learning rate β, regularization weights γvsubscript _vγitalic_v, γtsubscript _tγitalic_t. Model: the i-step training DM isubscripti θ_iitalic_θroman_i, the frozen original DM osubscripto θ_oitalic_θroman_o, the text encoder of DM T⁢(∗)T(*)T ( ∗ ). 1:for i=1,⋯,I1⋯i=1,·s,Ii = 1 , ⋯ , I do 2: τp=τcsubscriptsubscript _p= _cτitalic_p = τitalic_c. 3: if i >=>=> = warmup iterations then 4: τp=[τt,τc],fop¯=⁢(To⁢(τp))formulae-sequencesubscriptsubscriptsubscriptsuperscriptsubscript¯subscriptsubscriptosubscript _p=[ _t, _c],f_o p=g(T_ θ_% o( _p))τitalic_p = [ τitalic_t , τitalic_c ] , fitalic_oover¯ start_ARG p end_ARG = g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_p ) ). 5: for j=1,2,…,J12…j=1,2,…,Jj = 1 , 2 , … , J do 6: fop=⁢(To⁢(τp)),fip=⁢(Ti⁢(τp))formulae-sequencesuperscriptsubscriptsubscriptsubscriptosubscriptsuperscriptsubscriptsubscriptsubscriptisubscriptf_o^p=g(T_ θ_o( _p)),f_i^p= % g(T_ θ_i( _p))fitalic_oitalic_p = g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_p ) ) , fitalic_iitalic_p = g ( Tbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( τitalic_p ) ) 7: ℋ1=⁢(fop¯,fip),ℋ2=⁢(fop¯,fop)formulae-sequencesubscriptℋ1superscriptsubscript¯superscriptsubscriptsubscriptℋ2superscriptsubscript¯superscriptsubscriptH_1=Sim(f_o p,f_i^p),H_2=% Sim(f_o p,f_o^p)H1 = Sim ( fitalic_oover¯ start_ARG p end_ARG , fitalic_iitalic_p ) , H2 = Sim ( fitalic_oover¯ start_ARG p end_ARG , fitalic_oitalic_p ) 8: ℒattack⁢(i,τt)=−ℋ1+ℋ2/ℋ1subscriptℒattacksubscriptisubscriptsubscriptℋ1subscriptℋ2subscriptℋ1L_attack( θ_i, _t)=-H_% 1+H_2/H_1Lroman_attack ( italic_θroman_i , τitalic_t ) = - H1 + H2 / H1 9: τt←τt−β⁢∇τtℒattack⁢(i,τt)←subscriptsubscriptsubscript∇subscriptsubscriptℒattacksubscriptisubscript _t← _t-β _ _tL_attack% ( θ_i, _t)τitalic_t ← τitalic_t - β ∇τ start_POSTSUBSCRIPT t end_POSTSUBSCRIPT Lroman_attack ( italic_θroman_i , τitalic_t ) 10: end for 11: end if 12: ℳ⁢(o,τr)=⁢(To⁢(τr))⋅⁢(To⁢(τr))Tℳsubscriptosubscript⋅subscriptsubscriptosubscriptsuperscriptsubscriptsubscriptosubscriptTM( θ_o, _r)=g(T_ θ_% o( _r))·g(T_ θ_o( _r))% ^TM ( italic_θroman_o , τitalic_r ) = g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_r ) ) ⋅ g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_r ) )T 13: ℳ⁢(i,τr)=⁢(Ti⁢(τr))⋅⁢(Ti⁢(τr))Tℳsubscriptisubscript⋅subscriptsubscriptisubscriptsuperscriptsubscriptsubscriptisubscriptTM( θ_i, _r)=g(T_ θ_% i( _r))·g(T_ θ_i( _r))% ^TM ( italic_θroman_i , τitalic_r ) = g ( Tbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( τitalic_r ) ) ⋅ g ( Tbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( τitalic_r ) )T 14: ℒgraph=‖ℳ⁢(o,τr)−ℳ⁢(i,τr)‖22subscriptℒgraphsubscriptsuperscriptnormℳsubscriptosubscriptℳsubscriptisubscript22L_graph= \|M( θ_o,% _r)-M( θ_i, _r) \|^2_2Lroman_graph = ∥ M ( italic_θroman_o , τitalic_r ) - M ( italic_θroman_i , τitalic_r ) ∥22 15: ℋ3=⁢(⁢(To⁢(τr)),⁢(Ti⁢(τr)))subscriptℋ3subscriptsubscriptosubscriptsubscriptsubscriptisubscriptH_3=Sim(g(T_ θ_o( _r)% ),g(T_ θ_i( _r)))H3 = Sim ( g ( Tbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( τitalic_r ) ) , g ( Tbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( τitalic_r ) ) ) 16: The prompts accord with top-k minimum of ℋ3subscriptℋ3H_3H3 are selected as the image retain set imagesubscriptimageC_imageCroman_image. 17: ℒimage=τ~r∼retain⁢[‖ϵi⁢(t,τ~r)−ϵo⁢(t,τ~r)‖22]subscriptℒimagesubscriptsimilar-tosubscript~subscriptretaindelimited-[]superscriptsubscriptnormsubscriptitalic-ϵsubscriptisubscriptsubscript~subscriptitalic-ϵsubscriptosubscriptsubscript~22L_image=E_ τ_r _% retain [ \| _ θ_i(z_% t, τ_r)- _ θ_o(z_t, % τ_r) \|_2^2 ]Lroman_image = blackboard_Eover~ start_ARG τ end_ARG start_POSTSUBSCRIPT r ∼ Croman_retain end_POSTSUBSCRIPT [ ∥ ϵbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( zitalic_t , over~ start_ARG τ end_ARGr ) - ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( zitalic_t , over~ start_ARG τ end_ARGr ) ∥22 ] 18: ϵi^⁢(t,τp)^subscriptitalic-ϵsubscriptisubscriptsubscript _ θ_i(z_t, _p)over start_ARG ϵbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_ARG ( zitalic_t , τitalic_p ) is obtained by Eq.(11). 19: ℒerase=⁢[‖ϵ⁢(t,τp)−ϵi^⁢(t,τp)‖22]subscriptℒerasedelimited-[]subscriptsuperscriptnormsubscriptitalic-ϵsubscriptsubscriptsubscript^subscriptitalic-ϵsubscriptisubscriptsubscript22L_erase=E [ \| _ _% i(z_t, _p)- _ θ_i% (z_t, _p) \|^2_2 ]Lroman_erase = blackboard_E [ ∥ ϵbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ( zitalic_t , τitalic_p ) - over start_ARG ϵbold_italic_θ start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_ARG ( zitalic_t , τitalic_p ) ∥22 ] 20: ℒ⁢(i,o,τp,τr)=ℒerase+γt⁢ℒgraph+γv⁢ℒimageℒsubscriptisubscriptosubscriptsubscriptsubscriptℒerasesubscriptsubscriptℒgraphsubscriptsubscriptℒimageL( θ_i, θ_o, _p, _r% )=L_erase+ _tL_graph+ _% vL_imageL ( italic_θroman_i , italic_θroman_o , τitalic_p , τitalic_r ) = Lroman_erase + γitalic_t Lroman_graph + γitalic_v Lroman_image 21: i+1←i−α⁢∇ℒ⁢(i,o,τp,τr)←subscripti1subscriptisubscript∇ℒsubscriptisubscriptosubscriptsubscript θ_i+1← θ_i-α _ % θL( θ_i, θ_o, _p% , _r)italic_θroman_i + 1 ← italic_θroman_i - α ∇italic_θ L ( italic_θroman_i , italic_θroman_o , τitalic_p , τitalic_r ) 22:end for 5 Experiments 5.1 Experiment Setups Tasks and Datasets. In addressing the two real-world challenges, such as unsafe generation and copyright infringement, we focus on erasing nudity and artistic style. Nudity erasing aims to prevent DM from generating nude content subject to nudity-related prompts. The test set is the Inappropriate Image Prompt (I2P) dataset SLD , which comprises 4,703 inappropriate prompts about violence and sexual content. The 10k prompts sampled from the COCO dataset coco are used to verify the retention of unrelated concepts. The COCO dataset covers various common concepts while avoiding unsafe concepts, making it suitable to evaluate the generation capability of common concepts. Style erasing focuses on eliminating the influence of specific artistic styles in DM. We choose Van Gogh and Claude Monet, who have distinct artistic styles, as erasure targets. Following prior worksesd , the erasure test set contains 50 prompts about the erased artistic style. The 129-class style classifier from zhang2023generate is used to determine the artistic style of images. We further select 34 other artists who have the highest style classification accuracy rate in the generated image of original DM, and create 170 prompts as the other-style set to test the retention of other artistic styles. Baseline Methods. We conduct a comprehensive evaluation for SAGE compared with other 9 open-sourced baselines, including SD v2.1 rombach2022sd2 , SLD-Max SLD , ESD esd , RACE RACE , UCE UCE , MACE MACE , Receler receler , RECE RECE and AdvUnlearn Advunlearn . Since not all methods have been tested on both nudity and style erasure, we use publicly available model weights for the corresponding tasks. For tasks without prior testing, we reimplement the methods to conduct evaluations. To ensure fair comparison and consistency with previous works, we fine-tune SD v1.4 and generate images using the 50-step DDIM sampler song2020denoising . Training Setups. All experiments are conducted on a single A100 GPU. The text encoder is finetuned for 1,000 steps using the Adam optimizer, with 10−5superscript10510^-510- 5 learning rate and erasing guidance parameter η=1.01.0η=1.0η = 1.0. The first 200 steps serve as a warm-up training stage. The template library is generated by GPT-4 gpt2023 and the template token embedding τtsubscript _tτitalic_t is updated for 30 steps with a step size of 10−3superscript10310^-310- 3. The retain set retainsubscriptretainC_retainCroman_retain includes 243 different objects from COCO dataset, and each retain prompt is constructed using template ‘a photo of [object]’. Each iteration uses br⁢e⁢t⁢a⁢i⁢n=32subscript32b_retain=32bitalic_r e t a i n = 32 retain prompts, with γt=0.4subscript0.4 _t=0.4γitalic_t = 0.4 for nudity erasing and 3.03.03.03.0 for style erasing. The Top-4 prompts with lowest semantic similarity are selected as subset imagesubscriptimageC_imageCroman_image to calculate ℒimagesubscriptℒimageL_imageLroman_image, with γv=1.0subscript1.0 _v=1.0γitalic_v = 1.0. Table 1: Comprehensive evaluation (HosubscriptHoH_oHo) of nudity concept erasure methods on three aspects: erasure of nudity concept (RER), retention of unrelated concepts (visual similarity FID and semantic consistency CLIP-S), and model robustness (ASR). F: Female. M: Male. Method Detected Quantity on I2P dataset RER (↑ ↑) FID (↓ ↓) CLIP-S (↑ ↑) ASR (↓ ↓) HosubscriptHo~~H_o~~Ho (↑ ↑) Breasts (F) Genitalia (F) Genitalia (M) Buttocks Total (↓ ↓) SD v1.4 SD1_4 267 8 7 21 303 0.00 16.70 31.09 - - SD v2.1 rombach2022sd2 188 4 6 9 207 31.68 18.19 31.21 - - SLD-Max SLD [CVPR23] 37 3 0 5 45 85.14 29.85 28.85 75.35 64.63 ESD esd [ICCV23] 25 2 7 0 34 88.77 18.18 30.17 40.84 84.21 RACE RACE [ECCV24] 11 1 2 1 15 95.05 21.41 29.29 21.83 86.36 UCE UCE [WACV24] 43 4 4 2 53 82.51 17.10 30.89 23.24 89.07 MACE MACE [CVPR24] 27 5 3 1 36 88.11 17.83 29.11 4.92 92.62 Receler receler [ECCV24] 12 0 5 6 23 92.41 18.28 30.15 9.86 92.72 RECE RECE [ECCV24] 16 0 3 2 21 93.06 17.96 30.20 11.97 92.80 AdvUnlearn Advunlearn [NIPS24] 4 0 0 0 4 98.67 19.34 29.03 6.33 93.01 Ours: SAGE 1 0 4 1 6 98.01 19.21 29.53 2.81 94.28 Evaluation Setups. For nudity erasure, we employ the NudeNet detector nudenet with a detection threshold of 0.6 to identify sensitive body regions. Following RingABell RingABell , NudeNet detects and counts 4 erotically sensitive regions (female breasts, female/male genitalia, and buttocks), while excluding less sensitive areas like belly or feet. It maintains safety standards of evaluation without excessive conservatism. The Relative Erasure Ratio (RER) evaluates the percentage decrease in the number of exposed body parts detected by the erasure model compared to the original SD v1.4. To evaluate the ability of erased DMs to retain common concepts, FID FID is used to evaluate the visual similarity between generated images and original images, while CLIPScore (CLIP-S) clipscore measures the semantic consistency of generated images and prompt descriptions. To evaluate the generalization and robustness of the erased DM, Ring-A-Bell RingABell , a widely applicable and low-cost black-box red-teaming method, is employed to assess the safeguard capability via Attack Success Rate (ASR) on 142 attack nudity-related prompts SLD . Given inconsistent setups and evaluation systems of the current nudity concept erasure field, we build HosubscriptHoH_oHo metric to unify evaluation, covering three criteria: erasure effectiveness, retention, and robustness. Specifically, all metrics are normalized to ensure higher values indicate better performance. Then we introduce the indicator HosubscriptHoH_oHo to average all metrics, which is defined as: Ho=RER+FID⁢(o)FID⁢(n)+CLIP-S⁢(n)CLIP-S⁢(o)+(1−ASR)4.subscriptHoRERFIDsubscriptoFIDsubscriptnCLIP-SsubscriptnCLIP-Ssubscripto1ASR4 _o= RER+ FID( % θ_o)FID( θ_n)+ CLIP-% S( θ_n)CLIP-S( θ_o)+(1-% ASR)4.Ho = divide start_ARG RER + divide start_ARG FID ( italic_θroman_o ) end_ARG start_ARG FID ( italic_θroman_n ) end_ARG + divide start_ARG CLIP-S ( italic_θroman_n ) end_ARG start_ARG CLIP-S ( italic_θroman_o ) end_ARG + ( 1 - ASR ) end_ARG start_ARG 4 end_ARG . (19) where osubscripto θ_oitalic_θroman_o represents the original DM, which refers to SD v1.4 in experiment, and nsubscriptn θ_nitalic_θroman_n represents the erased DM. For style erasure, both style classification accuracy Acc and perceptual distance LPIPS are used to evaluate concept-erased DMs. Correct classification is defined as the target artist appearing in the top-3 classification results. For the erased style, lower accuracy AccesubscriptAcceAcc_eAcce indicates better, while higher accuracy AccrsubscriptAccrAcc_rAccr means better for other unerased styles. Thus, the overall classification metric can be calculated by HA=Accr−AccesubscriptHAsubscriptAccrsubscriptAcceH_A=Acc_r-Acc_eHA = Accr - Acce. LPIPS evaluates the perceptual distance between images of concept-erased DM and original DM, where a higher value indicates greater difference and a lower value indicates more similarity. The overall perceptual metric can be calaulate by HL=LPIPSe−LPIPSrsubscriptHLsubscriptLPIPSesubscriptLPIPSrH_L=LPIPS_e-LPIPS_rHL = LPIPSe - LPIPSr. 5.2 Nudity Erasure Quatitative Results. Table 1 provides a comprehensive evaluation of state-of-the-art methods as well as ours, assessing erasure performance of the nudity concept, preservation performance of common concepts, and erasure robustness against red-teaming attack prompts. SD v2.1, despite its extensive retraining on filtered data, shows only a modest RER improvement compared to SD v1.4 (↑ ↑31.68%). The reason is that real-world data often contains a mix of concepts, making it difficult to completely remove specific concepts through detection filtering. Compared to the original SD v1.4, our SAGE has significantly reduced the probability of generating nudity content by 98.01%. The outstanding RER and ASR performance indicates that SAGE is no longer trapped in the word concept abyss, achieving a more generalized concept-related erasure. The results of Table 1 also highlight a key trade-off in existing methods: while some approaches (e.g., SLD-MAX SLD and RACE RACE ) excel in safe generation, they struggle with content retention and safety robustness. Conversely, methods like UCE UCE and MACE MACE maintain strong consistency for common concepts but underperform in erasing the nudity concept. To enable a fair and comprehensive comparison, HosubscriptHoH_oHo is introduced as a unified evaluation metric, assessing erasure effectiveness, retention ability, and safety robustness. Our SAGE achieves the highest HosubscriptHoH_oHo score, demonstrating a superior balance across all three dimensions compared to existing approaches. Table 2: Training efficiency comparison between AdvUnlearn and SAGE. All experiments are tested on a single A100. Time AdvUnlearn SAGE Relative Ratio Attack Prompt 57.75s 0.93s ↑ ↑ 62.1× Total Time 16.32h 2.4h ↑ ↑ 6.8× Training Efficiency. Benefiting from the inside-out semantic-augment erasure, our method also demonstrates superior training efficiency compared to the most competitive method AdvUnlearn Advunlearn . As shown in Table 2, SAGE generates one attack prompt 62.1 × faster than AdvUnlearn, and training time is also improved by 6.8 ×. This is because SAGE directly leverages text feature relationships to mine valuable attack prompts. In contrast, the outside-in method adopted by AdvUnlearn requires the time-consuming multi-step UNet denoising to generate attack prompts. Table 3: Comparison of generated inappropriate images proportions for different concept erasure methods on I2P dataset (where lower values are better). The “overall” represents the proportion of all generated images that contain inappropriate concepts. The best performances are bolded. Method Inappropriate Proportions (%) ↓ ↓ 9 Hate Harassment Violence Self-harm Sexual Shocking Illegal Activity Overall SD v1.4 SD1_4 31.17 28.40 30.03 32.83 28.14 32.13 29.99 30.30 SD v2.1 rombach2022sd2 30.30 27.79 32.14 30.09 31.15 31.54 27.92 30.19 UCE UCE 22.94 19.17 22.09 22.35 22.56 21.14 22.42 21.65 Receler receler 23.81 20.63 19.58 19.98 20.41 20.91 19.94 20.37 ESD esd 18.18 17.84 20.37 16.73 18.15 18.69 16.51 18.22 MACE MACE 11.69 10.56 14.29 9.99 11.28 10.86 11.97 11.52 SLD-Max SLD 9.96 9.83 10.85 9.36 11.06 7.59 11.97 10.23 RACE RACE 7.36 8.50 8.33 8.36 8.27 8.41 7.02 7.97 RECE RECE 10.82 6.19 7.01 7.12 6.87 7.36 7.70 7.29 AdvUnlearn Advunlearn 3.03 4.85 5.03 5.49 5.26 5.84 4.26 5.19 Ours: SAGE 1.73 1.70 1.98 3.75 2.79 2.80 2.20 2.61 Table 4: Quantitative evaluation of artistic style erasure. Method Erasing “Van Gogh” Erasing “Claude Monet” 13 AccesubscriptAcceAcc_eAcce ↓ ↓ AccrsubscriptAccrAcc_rAccr ↑ ↑ HAsubscriptHAH_AHA ↑ ↑ LPIPSesubscriptLPIPSeLPIPS_eLPIPSe ↑ ↑ LPIPSrsubscriptLPIPSrLPIPS_rLPIPSr ↓ ↓ HLsubscriptHLH_LHL ↑ ↑ AccesubscriptAcceAcc_eAcce ↓ ↓ AccrsubscriptAccrAcc_rAccr ↑ ↑ HAsubscriptHAH_AHA ↑ ↑ LPIPSesubscriptLPIPSeLPIPS_eLPIPSe ↑ ↑ LPIPSrsubscriptLPIPSrLPIPS_rLPIPSr ↓ ↓ HLsubscriptHLH_LHL ↑ ↑ SLD-Max SLD 0.00 27.06 27.06 54.43 49.99 4.44 0.00 16.47 16.47 56.88 47.31 9.57 ESD esd 14.00 70.00 56.00 44.14 27.07 17.07 4.00 54.71 50.71 43.45 30.50 12.95 RACE RACE 0.00 55.88 55.88 49.52 30.18 19.34 0.00 40.00 40.00 51.45 34.24 17.21 UCE UCE 78.00 94.12 16.12 21.87 5.52 16.35 14.00 93.53 79.53 26.29 5.02 21.27 MACE MACE 36.00 90.00 54.00 32.53 11.38 21.15 8.00 89.12 81.12 28.89 12.20 16.69 Receler receler 6.00 55.88 49.88 57.56 33.95 23.61 2.00 29.41 27.41 48.93 36.77 12.16 RECE RECE 44.00 90.59 46.59 29.48 6.71 22.77 10.00 92.35 82.35 32.41 7.53 24.87 AdvUnlearn Advunlearn 6.00 75.29 69.29 50.36 26.24 24.12 2.00 62.94 60.94 43.95 30.80 13.15 Ours: SAGE 8.00 92.35 84.35 45.21 19.12 26.09 2.00 85.88 83.88 45.06 19.88 25.18 Extended Concepts Erasure. We further evaluated multiple methods for erasing a broader range of unsafe concepts. The I2P dataset SLD includes prompts corresponding to multiple inappropriate classes such as hate, harassment, violence, self-harm, sexual, shocking, and illegal activity. Following the setting of ESD esd and SLD SLD , we utilized the Q16 classifier q16 as the inappropriate concept detector. Q16 is a conservative dual classifier that marks an image as inappropriate if it belongs to any of the specified categories. For ensuring accurate identification of generalized unsafe concepts, we use the fine-tuned weight of Q16 classifier from qu2023unsafe and set the detection threshold to 0.6. Table 3 presents the proportions of detected inappropriate content for each category of the I2P dataset. The results reveal that both SD v1.4 and SD v2.1 exhibit high probabilities of generating inappropriate concepts, with overall detection rates of 30.30% and 30.19%, respectively. In comparison, our SAGE not only achieves the lowest detection rates across all inappropriate categories but also demonstrates the lowest overall detection rate of 2.61%. It shows the effectiveness and generalization of our approach in erasing multiple inappropriate concepts. 5.3 Artistic Style Erasure Quatitative Results. The quantitative results for artistic style erasure are shown in Table 4. For the target styles (Van Gogh or Monet), the low classification accuracy AccesubscriptAcceAcc_eAcce and the larger perceptual distance LPIPSesubscriptLPIPSeLPIPS_eLPIPSe demonstrate that SAGE effectively erases the target styles. For non-target styles, the high classification accuracy AccrsubscriptAccrAcc_rAccr and small perceptual distance LPIPSrsubscriptLPIPSrLPIPS_rLPIPSr indicate SAGE also maintains strong retention capabilities for non-target styles. For the comprehensive evaluation considering both erasure and retention, our method SAGE consistently outperforms other approaches across both evaluation metrics: the coarse-grained classification score (HAsubscriptHAH_AHA) and the fine-grained perceptual distance measure (HLsubscriptHLH_LHL). This dual-metric superiority demonstrates that our approach effectively unlearns the patterns associated with the target style while maintaining the integrity of other stylistic content. Figure 4: Qualitative results of different nudity concept erasing methods. Below each row of images is the corresponding text prompt. Nudity-related prompts are from the I2P dataset, while the common concept prompt is from the COCO dataset. Generated images with issues are highlighted using red border. Figure 5: Qualitative results on erasing the style of Van Gogh. Generated images with issues are highlighted using red border. 5.4 Qualitative Results. Nudity Erasure. Fig. 4 presents the visualizations of generated images of various methods in response to both the nudity-related prompts and the nudity-irrelevant prompt (common concept). For the concealed prompt of 1st row, SD v2.1, SLD-Max, ESD, and UCE still generate unsafe nude content (e.g., female breasts). Due to excessive erasure, the generated image of AdvUnlearn deviates significantly from the semantics of the given prompt, no longer containing female-related content. For suggestive prompts containing nudity-related scenes and names in the 2nd and 3rd rows, MACE, Receler, and AdvUnlearn generate corresponding explicit content. However, SAGE effectively blocks the risk of generating unsafe content while maintaining the overall semantic consistency with nudity-related prompts. The 4th row shows the preservation ability of different methods to common concept. It can be observed that SLD-Max, ESD, RACE, MACE, and AdvUnlearn all partially forget the concepts of kitchen or refrigerator, leading to the absence of the related objects in the generated images. In contrast, SAGE still maintains the semantic consistency on common concept prompt. These visual differences demonstrate that SAGE effectively erases nudity-related concepts while maintaining unrelated concepts, ensuring the safe generation. Artistic Style Erasure. The first two rows of Fig. 5 show the generated images for erasing the target Van Gogh style, while the last two rows show the images for other non-target artistic styles. It can be observed that SLD-Max and Receler, while erasing the Van Gogh style, also cause strong forgetting of other non-target styles. When confronted with the term Starry Night, which is strongly associated with Van Gogh’s art concept, UCE, MACE, and RECE fail to erase the Van Gogh style. Meanwhile, ESD, RACE, and AdvUnlearn forget some patterns of other artistic styles, leading to significant perceptual shifts in the generated images compared to the original SD v1.4 on non-target artistic styles. e.g., Marilyn Monroe is transformed into a male in the 3rd row, and the Monet style is transformed from Impressionism to realistic photographic style in the 4th row. In comparison, our SAGE not only effectively unlearns target Van Gogh style but also preserves the generation quality of other non-target styles as possible. 5.5 Ablation Study Ablation about Components. To study the impact of each component, we conduct ablation studies on erasing nudity concept task and present results in Table 6. The high RER and low ASR of Config. 1 and 2 indicate that only training with attack prompts allows the DM to effectively erase nudity concept. However, the extremely high FID and extremely low CLIP-S also mean the erased DM suffers a significant generation degradation for common concepts. Comparing Config. 3 with 4, we observe that adding either ℒimagesubscriptℒimageL_imageLroman_image or ℒgraphsubscriptℒgraphL_graphLroman_graph improves the retain ability to common concepts. ℒimagesubscriptℒimageL_imageLroman_image is more effective in maintaining local-level image generation quality (FID) but has a limited effect on maintaining global-level semantic alignment (CLIP-S). ℒgraphsubscriptℒgraphL_graphLroman_graph can simultaneously maintain both generation quality and semantic alignment, but may slightly reduce the erasure effect. Config. 5 shows that warm-up training slightly improves the erasure effect. It may be because the warm-up phase initially directs DM to focus on erasing concept words, which helps the subsequent exploration of the concept domain boundaries. Our SAGE effectively unlearns unsafe concepts through semantic-augment erasure and mitigates the generation degradation caused by over-erasure through global-local collaborative retention, thereby achieving better comprehensive performance. Table 5: Ablation study on erasing nudity concept. τpsubscript _pτitalic_p: attack prompts. wu: warm-up training. ℒgsubscriptℒgL_gLroman_g: semantic graph consistency loss. ℒisubscriptℒiL_iLroman_i: noise prediction consistency loss. Config Components RER (↑ ↑) FID (↓ ↓) CLIP-S (↑ ↑) ASR (↓ ↓) HosubscriptHo~~H_o~~Ho (↑ ↑) τpsubscript _pτitalic_p wu ℒgsubscriptℒgL_gLroman_g ℒisubscriptℒiL_iLroman_i SD v1.4 - - - - 0.00 16.64 31.09 - - ESD × × × × 88.77 18.18 30.17 40.84 84.21 1 ✓ × × × 100.00 73.38 15.48 0.00 68.11 2 ✓ ✓ × × 99.34 66.39 15.34 0.00 68.43 3 ✓ ✓ × ✓ 97.02 19.63 25.99 2.81 90.64 4 ✓ ✓ ✓ × 93.06 18.24 29.04 5.63 93.01 5 ✓ × ✓ ✓ 96.70 19.46 29.31 2.81 93.69 SAGE ✓ ✓ ✓ ✓ 98.01 19.21 29.53 2.81 94.28 Table 6: Ablation study on the design of ℒattacksubscriptℒattackL_attackLroman_attack. ℒattacksubscriptℒattackL_attackLroman_attack RER (↑ ↑) FID (↓ ↓) CLIP-S (↑ ↑) ASR (↓ ↓) HosubscriptHo~~H_o~~Ho (↑ ↑) SD v1.4 0.00 16.64 31.09 - - ESD 88.77 18.18 30.17 40.84 84.21 ℋ2subscriptℋ2H_2H2 85.81 18.91 30.09 30.99 84.90 −ℋ1subscriptℋ1-H_1- H1 93.73 19.77 28.57 7.04 90.69 −ℋ1+ℋ2subscriptℋ1subscriptℋ2-H_1+H_2- H1 + H2 96.03 19.51 29.29 2.11 93.35 −ℋ1+ℋ2ℋ1subscriptℋ1subscriptℋ2subscriptℋ1-H_1+ H_2H_1- H1 + divide start_ARG H2 end_ARG start_ARG H1 end_ARG 98.01 19.21 29.53 2.81 94.28 Ablations about ℒattacksubscriptℒattackL_attackLroman_attack. To explore valuable attack prompts which are within the target concept domain but distant from the target concept anchor, ℒattacksubscriptℒattackL_attackLroman_attack (Eq.(10)) is defined for attack prompt generation. The ablation experiments are conducted on the attack loss design, with results shown in Table 6. Optimizing only ℋ2subscriptℋ2H_2H2 pushes attack prompts randomly away from the concept anchor. It underperforms compared to ESD in both erasure and retention. This decline may arise from semantic drift in randomly directed prompts, leading to imprecise concept erasure. While this method improves robustness by approximately 10%, which is likely due to diverse attack prompts enhancing anti-attack capability. In contrast, optimizing only ℋ1subscriptℋ1H_1H1 strictly aligns attack prompts with the concept anchor, improving both erasure efficacy and model robustness. The gains in erasure stem from the concept anchor providing a consistent optimization target during adversarial training. However, this approach degrades retention performance, as excessive unidirectional erasure weakens the ability to preserve unrelated concepts. Linear combination of ℋ1subscriptℋ1H_1H1 and ℋ2subscriptℋ2H_2H2 achieves superior performance, balancing erasure efficacy, retention, and robustness. This suggests that introducing controlled variance is useful during the unidirectional erasure process toward the concept anchor. The attack loss used in this work increases ℋ1subscriptℋ1H_1H1 while further decreasing ℋ2subscriptℋ2H_2H2, generating attack prompts that remain within the concept domain yet explore its boundaries. These prompts refine the erasure process, ultimately yielding the best overall performance. Table 7: ASR evaluation of different methods. Method RAB MMA P4D UnlearnDiff Receler receler 9.86 17.9 42.96 42.96 RECE RECE 11.97 19.6 31.69 38.73 RACE RACE 21.83 0.70 26.06 24.65 AdvUnlearn Advunlearn 6.33 0.60 5.63 5.63 SAGE 2.81 0.50 9.15 9.86 Table 8: Comparison of nudity detection before and after transfering the text encoder of SAGE. Model Origin Transfer Relative Ratio LCM Dreamshaper v7 777 105 ↓ ↓ 86.49% Dreamlike Photoreal v2.0 420 14 ↓ ↓ 96.67% Openjourney v4 162 7 ↓ ↓ 95.68% SDXL v1.0 (base) 140 71 ↓ ↓ 49.29% 5.6 Extended Experiments More Red-teaming Methods. In the domain of multimodal generative safety, an ongoing arms race persists between jailbreak attacks and defense methods. To evaluate nudity-erasure robustness from more dimensions, we employ diverse red-teaming methods to assess robustness-specific approaches, e.g., Recelerc receler , RECE RECE , RACE RACE , and AdvUnlearn Advunlearn . Red-teaming methods include black-box attack methods such as Ring-A-Bell (RAB) RingABell and MMA-Diffusion (MMA) mma , which cannot access the parameters of victim models, and white-box attack methods such as Prompting4Debugging (P4D) prompting4debugging and UnlearnDiff zhang2023generate , which leverage gradient information and intermediate representations of victim models to optimize attacks precisely. As shown in Table 7, our method SAGE demonstrates strong robustness against both white-box and black-box attacks. Besides, it achieves the best anti-attack performance against black-box attacks, which have a wider application scope and lower deployment costs than white-box attacks. More Base Models. Unlike prior methods esd ; RACE ; UCE ; MACE ; receler ; RECE that modify the UNet, our approach updates the text encoder. This key difference allows our trained text encoder to be shared across multiple text-to-image models that use the same text encoder architecture, eliminating redeployment costs. To quantitatively assess this zero-shot transfer capability, we evaluated our method on three Stable Diffusion v1.4 variants (LCM Dreamshaper v7 LCM , Dreamlike Photoreal v2.0 dreamlike , Openjourney v4 Openjourney ) and Stable Diffusion XL (SDXL) v1.0 sdxl , which uses dual text encoders (OpenCLIP-ViT/G openclip and CLIP-ViT/L clip ). We replace the CLIP-ViT/L text encoder in these models with our SAGE nudity-erasure model’s text encoder and measure the reduction in unsafe content generation on the I2P dataset. As shown in Table 8, our method reduces the probability of generating nudity by 86.49%∼similar-to ∼96.67% for single-text-encoder models. For SDXL, replacing only one text encoder still achieves a 49.29% reduction. This training-free cross-model transfer capability indicates that our method has flexible adaptability in practical applications. 6 Conclusion We propose a novel concept erasing method SAGE. It breaks the convention of modeling concepts as fixed words and achieves the generalized concept domain erasing by the iterative self-check and self-erasure. Meanwhile, global-local collaborative retention provides dual protection mechanism for non-target concepts to ensure the model usability. Extensive experiments demonstrate that SAGE effectively and efficiently unlearns target concepts while maintaining high-quality image generation and semantic alignment. Data Availability Statements. The authors declare that the data supporting the experiments in this study are available within the paper. The code will be available at https://github.com/KevinLight831/SAGE. References (1) Bedapudi, P.: Nudenet: Neural nets for nudity classification, detection and selective censoring (2019) (2) Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2818–2829 (2023) (3) Chin, Z.Y., Jiang, C.M., Huang, C.C., Chen, P.Y., Chiu, W.C.: Prompting4debugging: Red-teaming text-to-image diffusion models by finding problematic prompts. In: International Conference on Machine Learning (ICML) (2024) (4) CNBC: Microsoft AI engineer says Copilot designer creates ’disturbing’ images (2024). Accessed: 2024-08-15 (5) Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems 34, 8780–8794 (2021) (6) Ding, M., Zheng, W., Hong, W., Tang, J.: Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems 35, 16890–16902 (2022) (7) dreamlike.art: Dreamlike photoreal v2.0. https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0 (2023) (8) Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Forty-first International Conference on Machine Learning (2024) (9) Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12873–12883 (2021) (10) Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 (2022) (11) Gandikota, R., Materzynska, J., Fiotto-Kaufman, J., Bau, D.: Erasing concepts from diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2426–2436 (2023) (12) Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., Bau, D.: Unified concept editing in diffusion models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 5111–5120 (2024) (13) Gong, C., Chen, K., Wei, Z., Chen, J., Jiang, Y.G.: Reliable and efficient concept erasure of text-to-image diffusion models. In: European Conference on Computer Vision, p. 73–88. Springer (2024) (14) Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 (2023) (15) Hero, P.: Openjourney v4. https://huggingface.co/prompthero/openjourney-v4 (2023) (16) Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 (2021) (17) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017) (18) Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020) (19) Ho, J., Salimans, T.: Classifier-free diffusion guidance. In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (2021) (20) Huang, C.P., Chang, K.P., Tsai, C.T., Lai, Y.H., Yang, F.E., Wang, Y.C.F.: Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. In: European Conference on Computer Vision, p. 360–376. Springer (2024) (21) Jiang, H.H., Brown, L., Cheng, J., Khan, M., Gupta, A., Workman, D., Hanna, A., Flowers, J., Gebru, T.: Ai art and its impact on artists. In: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, p. 363–374 (2023) (22) Kim, C., Min, K., Yang, Y.: Race: Robust adversarial concept erasure for secure text-to-image diffusion model. In: European Conference on Computer Vision, p. 461–478. Springer (2024) (23) Kim, S., Jung, S., Kim, B., Choi, M., Shin, J., Lee, J.: Towards safe self-distillation of internet-scale text-to-image diffusion models. arXiv preprint arXiv:2307.05977 (2023) (24) Kingma, D.P.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013) (25) Kumari, N., Zhang, B., Wang, S.Y., Shechtman, E., Zhang, R., Zhu, J.Y.: Ablating concepts in text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22691–22702 (2023) (26) Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, p. 740–755. Springer (2014) (27) Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023) (28) Lu, S., Wang, Z., Li, L., Liu, Y., Kong, A.W.K.: Mace: Mass concept erasure in diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6430–6440 (2024) (29) Luo, S., Tan, Y., Huang, L., Li, J., Zhao, H.: Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378 (2023) (30) Lyu, M., Yang, Y., Hong, H., Chen, H., Jin, X., He, Y., Xue, H., Han, J., Ding, G.: One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7559–7568 (2024) (31) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: International Conference on Learning Representations (2018) (32) MidJourney, I.: Midjourney. https://w.midjourney.com (2023). V5 (33) OpenAI: Dall·e 2. https://openai.com/dall-e-2 (2022). V1 (34) OpenAI: Gpt-4 technical report (2023) (35) O’Connor, R.: Stable diffusion 1 vs 2 - what you need to know (2022) (36) Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023) (37) Post, T.W.: Ai porn and deepfakes are a growing threat to women’s consent (2023). Accessed: 2024-08-15 (38) Qu, Y., Shen, X., He, X., Backes, M., Zannettou, S., Zhang, Y.: Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, p. 3403–3417 (2023) (39) Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning, p. 8748–8763. PMLR (2021) (40) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022) (41) Rando, J., Paleka, D., Lindner, D., Heim, L., Tramèr, F.: Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610 (2022) (42) Rombach, R.: Stable diffusion 2.0 release (2022) (43) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695 (2022) (44) Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part I 18, p. 234–241. Springer (2015) (45) Roose, K.: An ai-generated picture won an art prize. artists aren’t happy (2022) (46) Schramowski, P., Brack, M., Deiseroth, B., Kersting, K.: Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22522–22531 (2023) (47) Schramowski, P., Tauchmann, C., Kersting, K.: Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content? In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, p. 1350–1361 (2022) (48) Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35, 25278–25294 (2022) (49) Setty, R.: Ai art generators hit with copyright suit over artists’ images (2023) (50) SmithMano: Tutorial: How to remove the safety filter in 5 seconds (2022) (51) Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020) (52) Tsai, Y.L., Hsu, C.Y., Xie, C., Lin, C.H., Chen, J.Y., Li, B., Chen, P.Y., Yu, C.M., Huang, C.Y.: Ring-a-bell! how reliable are concept removal methods for diffusion models? In: The Twelfth International Conference on Learning Representations (2024) (53) Voleti, V., Yao, C.H., Boss, M., Letts, A., Pankratz, D., Tochilkin, D., Laforte, C., Rombach, R., Jampani, V.: Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In: European Conference on Computer Vision, p. 439–457. Springer (2025) (54) Yang, Y., Gao, R., Wang, X., Ho, T.Y., Xu, N., Xu, Q.: Mma-diffusion: Multimodal attack on diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7737–7746 (2024) (55) Yang, Y., Hui, B., Yuan, H., Gong, N., Cao, Y.: Sneakyprompt: Jailbreaking text-to-image generative models. arXiv preprint arXiv:2305.12082 (2023) (56) Zhang, C., Hu, M., Li, W., Wang, L.: Adversarial attacks and defenses on text-to-image diffusion models: A survey. Information Fusion p. 102701 (2024) (57) Zhang, G., Wang, K., Xu, X., Wang, Z., Shi, H.: Forget-me-not: Learning to forget in text-to-image diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1755–1764 (2024) (58) Zhang, Y., Chen, X., Jia, J., Zhang, Y., Fan, C., Liu, J., Hong, M., Ding, K., Liu, S.: Defensive unlearning with adversarial training for robust concept erasure in diffusion models. arXiv preprint arXiv:2405.15234 (2024) (59) Zhang, Y., Jia, J., Chen, X., Chen, A., Zhang, Y., Liu, J., Ding, K., Liu, S.: To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images… for now. ECCV (2024) (60) Zhang, Z., Fang, M., Chen, L., Namazi-Rad, M.R., Wang, J.: How do large language models capture the ever-changing world knowledge? a review of recent advances. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 8289–8311 (2023)

Instant research discovery