Paper deep dive

Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models

Zhihua Tian, Sirun Nan, Ming Xu, Shengfang Zhai, Wenjie Qu, Jian Liu, Kui Ren, Ruoxi Jia, Jiaheng Zhang

Year: 2025Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 100

Models: Stable Diffusion

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/12/2026, 6:13:18 PM

Summary

The paper introduces 'Interpret then Deactivate' (ItD), a framework for concept erasure in text-to-image diffusion models. ItD uses a sparse autoencoder (SAE) to interpret text embeddings as combinations of sparse features. By identifying and deactivating features specific to unwanted concepts, the model achieves precise, expandable concept removal without retraining or degrading performance on normal generation tasks.

Entities (4)

ItD · framework · 100%Sparse Autoencoder · model-architecture · 100%Text-to-Image Diffusion Models · generative-model · 100%Concept Erasure · task · 95%

Relation Signals (3)

ItD → performs → Concept Erasure

confidence 100% · enable precise concept removal in T2I diffusion models

ItD → utilizes → Sparse Autoencoder

confidence 100% · ItD first employs a sparse autoencoder (SAE) to interpret each concept

Sparse Autoencoder → operateson → Text-to-Image Diffusion Models

confidence 95% · We adopt SAE to identify and deactivate features specific to the target concepts to disable the diffusion model

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images but also raise people's concerns about generating harmful or misleading content. While extensive approaches have been proposed to erase unwanted concepts without requiring retraining from scratch, they inadvertently degrade performance on normal generation tasks. In this work, we propose Interpret then Deactivate (ItD), a novel framework to enable precise concept removal in T2I diffusion models while preserving overall performance. ItD first employs a sparse autoencoder (SAE) to interpret each concept as a combination of multiple features. By permanently deactivating the specific features associated with target concepts, we repurpose SAE as a zero-shot classifier that identifies whether the input prompt includes target concepts, allowing selective concept erasure in diffusion models. Moreover, we demonstrate that ItD can be easily extended to erase multiple concepts without requiring further training. Comprehensive experiments across celebrity identities, artistic styles, and explicit content demonstrate ItD's effectiveness in eliminating targeted concepts without interfering with normal concept generation. Additionally, ItD is also robust against adversarial prompts designed to circumvent content filters. Code is available at: this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

99,644 characters extracted from source content.

Expand or collapse full text

Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models Zhihua Tian Sirun Nan Ming Xu Shengfang Zhai Wenjie Qu Jian Liu Ruoxi Jia Jiaheng Zhang Abstract Text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images but also raise people’s concerns about generating harmful or misleading content. While extensive approaches have been proposed to erase unwanted concepts without requiring retraining from scratch, they inadvertently degrade performance on normal generation tasks. In this work, we propose Interpret then Deactivate ( ItDtypewriter_ItD), a novel framework to enable precise concept removal in T2I diffusion models while preserving overall performance. ItD\ typewriter_ItDfirst employs a sparse autoencoder (SAE) to interpret each concept as a combination of multiple features. By permanently deactivating the specific features associated with target concepts, we repurpose SAE as a zero-shot classifier that identifies whether the input prompt includes target concepts, allowing selective concept erasure in diffusion models. Moreover, we demonstrate that ItD\ typewriter_ItDcan be easily extended to erase multiple concepts without requiring further training. Comprehensive experiments across celebrity identities, artistic styles, and explicit content demonstrate ItD\ typewriter_ItD’s effectiveness in eliminating targeted concepts without interfering with normal concept generation. Additionally, ItD\ typewriter_ItDis also robust against adversarial prompts designed to circumvent content filters. Code is available at: https://github.com/NANSirun/Interpret-then-deactivate. Machine Learning, ICML 1 Introduction Text-to-image (T2I) diffusion models have achieved remarkable success in generating images that faithful reflect the input text descriptions (Dhariwal & Nichol, 2021; Saharia et al., 2022; Ruiz et al., 2023; Rombach et al., 2022) while simultaneously raising concerns about being used to generate images containing inappropriate content such as offensive, pornographic, copyrighted, or Not-Safe-For-Work (NSFW) content (Schramowski et al., 2023; Rando et al., 2022; Gandikota et al., 2023). To mitigate the issue, concept erasing has been proposed to prevent T2I diffusion models from generating images relevant to the unwanted concepts without requiring retraining from scratch. A recent line of research proposes fine-tuning model parameters to remove the unwanted knowledge learned by diffusion models (Gandikota et al., 2023; Kumari et al., 2023; Fan et al., 2023; Huang et al., 2023; Gandikota et al., 2024; Orgad et al., 2023). However, even only modifying the cross-attention (CA) layers within diffusion models would inadvertently degrade the generation quality of normal concepts (Gandikota et al., 2023; Huang et al., 2023; Zhang et al., 2024b; Bui et al., 2024b). For instance, erasing the concept of “nudity” could impair the model’s ability to generate an image of a person. To mitigate the issue, some approaches incorporate regularization techniques during fine-tuning to preserve the generation quality of normal concepts (Kumari et al., 2023; Fan et al., 2023; Ko et al., 2024). However, fine-tuning with a subset of concepts may introduce new biases into the model, leading to unpredictable performance degradation when generating other concepts (Bui et al., 2024b). As an alternative solution, some approaches integrate customized modules into the model to enable concept erasing without modifying the original model parameters(Lyu et al., 2024; Lu et al., 2024; Lee et al., 2025). However, each module may also affect the generation of normal concepts due to its limited generalization capability. Moreover, erasing new concepts requires training additional modules, resulting in substantial computational overhead. In this work, we aim to overcome the above limitation by introducing a novel framework, Interpret-then-Deactivate ( ItD\ typewriter_ItD), to enable precise and expandable concept erasure in T2I diffusion models. Precise refers to the erasure only influences the generation of the target concept while Expandable means that the approach can be easily extended to erase multiple concepts without further training. ItD\ typewriter_ItDemploys sparse autoencoder (SAE) (Olshausen & Field, 1997), an unsupervised model, to learn the sparse features that constitute the semantic space of the text encoder. Within this space, we interpret each concept as a linear combination of sparse features, which may overlap with the feature sets between the target and normal concepts. We hypothesize that this overlap is a key factor causing unintended effects on the normal concepts during erasure. To this end, we propose to selectively erase the features unique to the target concept, enabling the precise erasure. This can be achieved by first encoding the text embedding of the concept into the sparse feature space, deactivating the specific features, and then decoding it back into the embedding space. To enable expandability in erasing multiple concepts, we can deactivate concept-specific features without requiring additional retraining. In summary, we make the following contributions: (1) We propose ItD\ typewriter_ItD, a novel framework to enable precise and expandable concept erasure in T2I diffusion models. (2) To the best of our knowledge, we are the first to adopt SAE to concept erasing tasks in T2I diffusion models. (3) Extensive experiments across various datasets demonstrate that ItD\ typewriter_ItDeffectively erases target concepts while preserving the diversity of remaining concepts, outperforming baselines by large margins. 2 Related Works 2.1 Sparse Autoencoder (SAE) The internal of the neural network (N) is hard to explain due to its polysemanticity nature, where neurons appear to activate in multiple, semantically distinct contexts. Recently, SAE has emerged as an effective tool for interpreting mechanisms of N by breaking done the intermediate results into features interpretable to specific concepts (Huben et al., 2024; Kissane et al., 2024). Let ∈ℝdi⁢nsuperscriptℝsubscriptx ^d_inx ∈ ℝitalic_ditalic_i n denote the input vector, the decoder and encoder of SAE can be formalized as: =ReLU⁡(Wenc⁢+)absentReLUsubscriptenc =ReLU(W_encx+b)= ReLU ( Wenc x + b ) ^ xover start_ARG x end_ARG =Wdec⁢absentsubscriptdec =W_decz= Wdec z =∑i=0dnid −1zi⁢iabsentsuperscriptsubscript0subscriptnid 1subscriptsubscript = _i=0^d_nid -1z_if_i= ∑i = 0dnid - 1 zitalic_i fitalic_i where Wenc∈ℝdhid×dinsubscriptencsuperscriptℝsubscripthidsubscriptinW_enc ^d_hid× d_inWenc ∈ ℝitalic_dhid × din and Wdec∈ℝdin×dhidsubscriptdecsuperscriptℝsubscriptinsubscripthidW_dec ^d_in× d_hidWdec ∈ ℝitalic_din × dhid are learned matrices of encoder and decoder. ∈ℝdhidsuperscriptℝsubscripthidb ^d_hidb ∈ ℝitalic_dhid is the learned bias. The loss to train the autoencoder is ℒ⁢()=‖−^‖22+α⁢ℒa⁢u⁢xℒsuperscriptsubscriptnorm^22subscriptℒL(x)=\|x- x\|_2^2+α % L_auxL ( x ) = ∥ x - over start_ARG x end_ARG ∥22 + α Litalic_a u x, where ℒa⁢u⁢xsubscriptℒL_auxLitalic_a u x is the loss to control the sparsity of the reconstruction (Huben et al., 2024) or to prevent dead features that do not been fired on a large number of training samples (Gao et al., 2024), scaled with coefficient α. In our work, ℒa⁢u⁢xsubscriptℒL_auxLitalic_a u x is the reconstruction error using only the largest KauxsubscriptauxK_auxKaux feature activations following (Gao et al., 2024). We adopt SAE to identify and deactivate features specific to the target concepts to disable the diffusion model to generate related images. 2.2 Concept Erasing in T2I Diffusion Model Diffusion models can be used to generate inappropriate content (Zhang et al., 2025; Schramowski et al., 2023). To address this issue, several approaches have been proposed, such as dataset censoring (Face & CompVis, 2023b), post-generation filtering (Face & CompVis, 2023a), and safety-guided generation (Schramowski et al., 2023). However, these methods either demand significant computational resources (Face & CompVis, 2023b), introduce new biases, or remain vulnerable to adversarial prompts (Yang et al., 2024; Rando et al., 2022). To address these limitations, fine-tuning-based approaches have been extensively explored to erase target concepts. FMN (Zhang et al., 2024a) efficiently erases target concepts by re-steering cross-attention (CA) layers. UCE (Gandikota et al., 2024) and TIME (Orgad et al., 2023) modify the projection layer within CA layers with a closed-form solution. ESD (Gandikota et al., 2023) reduces the probability of generating images that are labeled as target concepts, and AC (Kumari et al., 2023) aligns the distribution of target concepts with surrogate concepts for concept erasure. To improve the effectiveness of erasing, SalUn (Fan et al., 2023) and Scissorhands (Wu & Harandi, 2025) identify the most sensitive neurons related to target concepts and update only those neurons. 2.3 Forgetting on Remaining Concepts While the above approaches demonstrate good performance in erasing target concepts, they inadvertently degrade the generation of remaining concepts (Zhang et al., ). To alleviate the issue, many approaches (Huang et al., 2023; Heng & Soh, 2024; Ko et al., 2024; Zhang et al., 2024b) utilize a regularization loss on remaining concepts to preserve their generation capability. EAP (Bui et al., 2024b) further investigate the impact of selecting different concepts for regularization, and OTE (Bui et al., 2024a), which employs a similar training objective to AC (Kumari et al., 2023), adaptively selecting the optimal surrogate concept during erasing. However, the effectiveness of regularization on unseen concepts remains unclear due to the enormous scale of normal concepts. As an alternative solution, some approaches incorporate customized modules into the intermediate layers of the diffusion model without modifying original model parameters. we summarize them inference-based approaches. SPM (Lyu et al., 2024) applies one-dimensional LoRA (Hu et al., 2021) to the intermediate layers of diffusion models and proposes an anchoring loss for distant concepts. MACE (Lu et al., 2024) propose using LoRA for erasing each target concept and introduced a loss integrating LoRAs from multiple target concepts, enabling the massive concept erasing while mitigating forgetting of remaining concepts. CPE (Lee et al., 2025) introduces a residual attention gate, a module inserted into each CA layer of the diffusion model to control whether erasure should be applied to a given concept. However, erasing multiple concepts requires training separate modules for each target concept, resulting in significant computational overhead. 3 Sparse Autoencoder for T2I Diffusion Models In this section, we introduce the training of an SAE for T2I diffusion model. We first discuss where to apply SAE for effective and efficient concept erasing in diffusion models in Section 3.1. We then introduce how an SAE is trained in Section 3.2. 3.1 Where to apply Spare Autoencoder. A T2I diffusion model comprises multiple modules that work jointly to generate an image. As a short preliminary, we take the Latent DM (LDM) (Rombach et al., 2022) as an example, wherein the text encoder and U-net are two main modules. The text encoder transforms input text prompts into embeddings E, which is used to guide the image generation process. The U-net module works as a noise predictor, taking the text embedding E, timestep t, and the noised latent representation xtsubscriptx_txitalic_t as inputs to predict the noise added at time t. Specifically, with an initial noise xT∼⁢(,)similar-tosubscript0x_T (0,I)xitalic_T ∼ N ( 0 , I ), the generation process iteratively performs denoising operations on xTsubscriptx_Txitalic_T, ultimately producing the final image x0subscript0x_0x0. Take DDPM (Ho et al., 2020) as an example, the denoise step can be formulated as: xt−1=1αt⁢(xt−1−αt1−α¯t⁢ϵθ⁢(xt,E,t))+σt⁢ϵ,subscript11subscriptsubscript1subscript1subscript¯subscriptitalic-ϵsubscriptsubscriptitalic-ϵx_t-1= 1 _t (x_t- 1- _t 1-% α_t _θ (x_t,E,t ) )+ _tε,xitalic_t - 1 = divide start_ARG 1 end_ARG start_ARG square-root start_ARG αitalic_t end_ARG end_ARG ( xitalic_t - divide start_ARG 1 - αitalic_t end_ARG start_ARG square-root start_ARG 1 - over¯ start_ARG α end_ARGt end_ARG end_ARG ϵitalic_θ ( xitalic_t , E , t ) ) + σitalic_t ϵ , (1) where ϵθsubscriptitalic-ϵ _θϵitalic_θ is realized by an U-net model, αt,α¯t,σtsubscriptsubscript¯subscript _t, α_t, _tαitalic_t , over¯ start_ARG α end_ARGt , σitalic_t are pre-defined values and ϵ∼⁢(,)similar-toitalic-ϵ0ε (0,I)ϵ ∼ N ( 0 , I ). To disable the model’s ability to generate unwanted images, numerous approaches propose modifying the U-net module to remove unwanted knowledge (Zhang et al., 2024a; Kumari et al., 2023; Gandikota et al., 2023; Fan et al., 2023; Huang et al., 2023; Bui et al., 2024b; Lu et al., 2024; Li et al., 2024; Lee et al., 2025). However, due to (1) the complexity of the U-Net architecture and (2) the iterative nature of the denoising process, which typically requires multiple steps (e.g., 50 in DDIM), even minor modifications to the U-Net can lead to unexpected outcomes, potentially degrading overall performance. To mitigate the issue, we propose applying SAE to the text encoder to remove unwanted knowledge from the text embedding before feeding it into the U-Net. The key motivations behind this are as follows: ∙ ∙ During the image generation process, the text embedding E plays a dominant role in encoding the semantic information to the generated images (cf. Equation 1). Consequently, erasing concepts at the text embedding level is sufficient to prevent their appearance in generated images. ∙ ∙ While previous works perform concept erasure within U-Net, their modifications mainly target text information processed within the CA layers (i.e., Key and Value projections, as detailed in the Appendix) (Lu et al., 2024; Lee et al., 2025), which demonstrates the effectiveness of performing erasure on text embedding. ∙ ∙ Prior studies demonstrate that performing unlearning on the text encoder achieves the best robustness against adversarial attacks (Zhang et al., 2024b). In the following section, we detail the training process of SAE. 3.2 Training of SAE ((a)) Figure 1: Unsupervised training of SAE, which takes a token embedding obtained from the residual streamer in text encoder as an input and aims to reconstruct it with sparse features The text encoder comprises a series of transformer blocks (cf. Figure 1). Denote lsubscriptT_lTitalic_l as the l-th transformer block, the text encoder with L blocks can be roughly formulated as TextEncoder=L∘…∘1TextEncodersubscript…subscript1TextEncoder=T_L ... _1TextEncoder = Titalic_L ∘ … ∘ T1. To train an SAE for concept erasure, we focus on residual stream (Elhage et al., 2021), which is the output of a transformer block. The residual stream of l-th layer can be represented as: el=l∘…∘1⁢(e0),el∈ℝH×dinformulae-sequencesubscriptesubscript…subscript1subscripte0subscriptesuperscriptℝsubscriptine_l=T_l ... _1(e_0),\ % e_l ^H× d_inel = Titalic_l ∘ … ∘ T1 ( e0 ) , el ∈ ℝitalic_H × din (2) where 0subscript0e_0e0 is the embedding of the tokenized prompt, H is the number of tokens composing the prompt, and dinsubscriptind_indin is the output dimension. We assume that dinsubscriptind_indin are the same across different layers for notation simplicity. We train an SAE that aims to learn the sparse features for each token embedding elh,h∈[1,..,H]e_l^h,h∈[1,..,H]elitalic_h , h ∈ [ 1 , . . , H ]. Therefore, for a prompt with H tokens, we get H samples to train SAE. In our work, we adopt K-sparse autoencoder (K-SAE) (Makhzani & Frey, 2013), which could explicitly control the number of active latents by only keeping the K largest activations and zeros the rest for reconstruction. Let e∈ℝdinesuperscriptℝsubscriptine ^d_ine ∈ ℝitalic_din refers to a single SAE training example. The encoder and decoder within K-SAE are then defined as follows: z =TopK⁡(Wenc⁢(e−bp⁢r⁢e))absentTopKsubscriptencesubscriptb =TopK (W_enc (e-% b_pre ) )= TopK ( Wenc ( e - bp r e ) ) (3) e^^e eover start_ARG e end_ARG =Wdec⁢z+bpreabsentsubscriptdeczsubscriptbpre =W_decz+b_pre= Wdec z + bpre where Wenc∈ℝdin×dhidsubscriptencsuperscriptℝsubscriptinsubscripthidW_enc ^d_in× d_hidWenc ∈ ℝitalic_din × dhid and Wdec∈ℝdhid×dinsubscriptdecsuperscriptℝsubscripthidsubscriptinW_dec ^d_hid× d_inWdec ∈ ℝitalic_dhid × din are learned matrix of encoder and decoder, bp⁢r⁢esubscriptb_prebitalic_p r e is the bias term. dhidsubscripthidd_hiddhid is significantly larger than dinsubscriptind_indin to enforce the sparsity of learned features. The training objective is: ℒ⁢()=‖−^‖22+α⁢ℒa⁢u⁢x,ℒsuperscriptsubscriptnorm^22subscriptℒL(x)=\|x- x\|_2^2+α % L_aux,L ( x ) = ∥ x - over start_ARG x end_ARG ∥22 + α Litalic_a u x , (4) where ℒa⁢u⁢xsubscriptℒL_auxLitalic_a u x is the reconstruction error using top Ka⁢u⁢xsubscriptK_auxKitalic_a u x (Ka⁢u⁢x>KsubscriptK_aux>KKitalic_a u x > K) feature activations, to prevent dead features that have not been fired on a large number of training samples (Gao et al., 2024), scaled with coefficient α. We refer to the activation of the ρ-th learned feature as zρ∈ℝsuperscriptℝz^ρ _ρ ∈ ℝ. Its associated feature vector fρ∈ℝdi⁢nsubscriptsuperscriptℝsubscriptf_ρ ^d_infitalic_ρ ∈ ℝitalic_ditalic_i n is a column in the decoder matrix Wdec=(1⁢|⋯|⁢nf)∈ℝdhid×dinsubscriptdecsubscript1⋯subscriptsubscriptsuperscriptℝsubscripthidsubscriptinW_dec= (f_1|·s|f_n_f )∈ % R^d_hid× d_inWdec = ( f1 | ⋯ | fitalic_n start_POSTSUBSCRIPT f end_POSTSUBSCRIPT ) ∈ ℝitalic_dhid × din. As a result, we can represent each token embedding as a sparse sum e≈∑ρ=1dh⁢i⁢dzρ⁢ρ, with ⁢‖z‖0≤K.formulae-sequenceesuperscriptsubscript1subscriptℎsuperscriptsubscript with subscriptnormz0e≈ _ρ=1^d_hidz^ρf_ρ, with % ||z||_0≤ K.e ≈ ∑ρ = 1ditalic_h i d zitalic_ρ fitalic_ρ , with | | z | |0 ≤ K . (5) The details of training SAE are presented in Appendix. 4 Methodology ((a)) Figure 2: (a) With a well-trained SAE, we identify unique features of target concepts by contrast with normal concepts; (b) we wrap SAE as a deactivation block and insert it into the text encoder for concept eraser. In this section, we introduce ItD\ typewriter_ItD, a framework to erase multiple concepts in a pretrained T2I diffusion model. 4.1 Overview With a well-trained SAE, a straightforward approach to erasing unwanted knowledge within the text embedding is to deactivate features associated with the target concepts during the inference of the text encoder. However, target concepts often share certain features learned by SAE with normal concepts (e.g., ”nudity” and ”person” both contain information related to the ”human body”). As a result, indiscriminately deactivating all related features could affect the generation of normal concepts. To solve the problem, we propose a simple yet effective contrast-based method to identify features specific to the target concepts (Section 4.2). By only deactivating carefully selected features, ItD\ typewriter_ItDcould effectively remove unwanted knowledge while having limited effects on normal concepts. Furthermore, we find that SAE can be exploited as a zero-shot classifier to distinguish between target and remaining concepts. This enables selective concept erasure, further reducing the impact on normal concepts (Section 4.3). Figure 2 depicts the workflow of ItD\ typewriter_ItD. Overall, it is designed to meet the following criteria: ∙ ∙ Effectiveness: The unlearned model could not generate images of target concepts even when prompted with texts related to target concepts. ∙ ∙ Robustness: The model should also prevent the generation of images that are semantically related to synonyms of the targeted concepts, ensuring that erasure is not restricted to exact prompt wording. ∙ ∙ Specificity: The erasure should target only the specified concepts, with minimal or no impact on the remaining concepts. ∙ ∙ Expandability: When erasing new concepts, the algorithm can be easily extended without the need for additional training. 4.2 Feature Selection. Select Features for a Concept. A concept C is typically composed of multiple tokens, denoted as C1,…,CHsubscript1…subscriptC_1,…,C_HC1 , … , Citalic_H where H≥11H≥ 1H ≥ 1. For example, ”Bill Clinton” consists of the tokens ”Bill” and ”Clinton”. SAE decomposes the embedding of each token into a set of features. To select features that can represent the concept, we collect features from all token embedding and select the top KselsubscriptselK_selKsel features based on their activation values sρsuperscripts^ρsitalic_ρ. Formally, we denote F as the set of indices of features specific to the concept C, F=ρ|sCρ∈TopK⁡(sC1,…,sCdh⁢i⁢d),conditional-setsuperscriptsubscriptTopKsuperscriptsubscript1…superscriptsubscriptsubscriptℎ F=\ρ|s_C^ρ (s_C^1,...,s_C^% d_hid)\,F = ρ | sitalic_Citalic_ρ ∈ TopK ( sitalic_C1 , … , sitalic_Citalic_ditalic_h i d ) , (6) where ⁢sCρ=max⁡(s1ρ,…,sHρ).where superscriptsubscriptmaxsuperscriptsubscript1…superscriptsubscript s_C^ρ=max(s_1^ρ,...,s_% H^ρ).where sitalic_Citalic_ρ = max ( s1italic_ρ , … , sitalic_Hitalic_ρ ) . Select Features for Concept Erasing. After selecting features associated with target concepts, we propose using the features that are specifically activated by the target concepts for erasure. To achieve that, we suggest a simple yet effective contrast-based approach. Specifically, given a retain set retainsubscriptretainC_retainCretain comprising normal concepts, the features specific to the target concept (i.e. feature to deactivate) F^tarsubscript^tar F_tarover start_ARG F end_ARGtar can be found by eliminating features that can be activated by normal concepts in retainsubscriptretainC_retainCretain: F^tar =Ftar \⋃Cr∈Cretain FCr.subscript^tar subscriptsubscriptsubscriptretain subscriptsubscript F_tar =F_tar _C_r∈ C_% retain F_C_r.over start_ARG F end_ARGtar = Ftar \ ⋃C start_POSTSUBSCRIPT r ∈ Cretain end_POSTSUBSCRIPT Fitalic_C start_POSTSUBSCRIPT r end_POSTSUBSCRIPT . (7) In our experiments, we use the concepts employed for the utility-preserving regularization term in previous works (Kumari et al., 2023; Zhang et al., 2024b; Fan et al., 2023; Lu et al., 2024) as the retain set for comparison. Notably, our approach does not require fine-tuning on these concepts, thereby avoiding the introduction of new biases and making more effective use of them. When erasing multiple concepts, we select features for erasing as the union of the specific features associated with each target concept. Specifically, let tarsubscripttarC_tarCtar denote the set of target concepts. Ferase=⋃C∈tarF^C.subscripterasesubscriptsubscripttarsubscript^F_erase= _C _tar F_C.Ferase = ⋃C ∈ C start_POSTSUBSCRIPT tar end_POSTSUBSCRIPT over start_ARG F end_ARGC . (8) 4.3 Concept Erasing To erase knowledge of target concepts within the text embedding, we first encode the text embeddings into their activations s using SAE. We then modify each activation component as follows: s^ρ=sρ⋅τ,if ⁢ρ∈Ferasesρ,otherwisesuperscript^cases⋅superscriptif subscripterasesuperscriptotherwise s^ρ= casess^ρ·τ,&if ρ∈ F_% erase\\ s^ρ,&otherwise casesover start_ARG s end_ARGρ = start_ROW start_CELL sitalic_ρ ⋅ τ , end_CELL start_CELL if ρ ∈ Ferase end_CELL end_ROW start_ROW start_CELL sitalic_ρ , end_CELL start_CELL otherwise end_CELL end_ROW (9) where FerasesubscripteraseF_eraseFerase represents the set of features selected before, and τ is a scaling factor controlling the degree of deactivation. Finally, the decoder reconstructs the modified embedding as ^=⁢(s^)^ e= Dec( s)over start_ARG e end_ARG = sansserif_Dec ( over start_ARG s end_ARG ). To build an unlearned model, we wrap SAE as a deactivation block inserted into the intermediate layers of the text encoder (Figure 2 (b)). Any text embedding inputted to the block would erase the information of target concepts. As a result, the generated image guided by the text embedding would not contain target concepts. The block is plug-and-play, which means we do not need to fine-tune the diffusion model, making it highly efficient. Moreover, since the selected features are not activated by normal concepts, Equation 10 does not influence normal concepts. Selective feature deactivation The inherent reconstruction loss of SAE may still influence the generation of normal concepts. To alleviate the issue, we propose a mechanism to selectively apply SAE for concept erasure. Given a text embedding ee and its reconstructed version ^ eover start_ARG e end_ARG, we construct a classifier G to identify whether ee contains information about target concepts. The classification is based on the reconstruction loss between ee and ^ eover start_ARG e end_ARG: G⁢()=1,if ⁢‖−^‖2<τ0,if ⁢‖−^‖2≥τcases1if superscriptnorm^20if superscriptnorm^2G(e)= cases1,&if ||e- e||^2<% τ\\ 0,&if ||e- e||^2≥τ casesG ( e ) = start_ROW start_CELL 1 , end_CELL start_CELL if | | e - over start_ARG e end_ARG | |2 < τ end_CELL end_ROW start_ROW start_CELL 0 , end_CELL start_CELL if | | e - over start_ARG e end_ARG | |2 ≥ τ end_CELL end_ROW (10) where τ is the threshold. Finally, the deactivation block outputs ee for subsequent computation if it does not contain target concept information; otherwise, it outputs ^ eover start_ARG e end_ARG. Figure 3 demonstrates the effectiveness of the classifier. We select 50 celebrities as target concepts for erasure. For the remaining concepts, we include 100 different celebrities, 100 artistic styles, and the COCO-30K dataset. It shows that there is a clear boundary in the reconstruction loss between the target concepts and the remaining concepts. ((a)) Figure 3: Histogram of reconstruction MSE loss on target concepts and remaining concepts. ((a)) Figure 4: Qualitative results of our ItD\ typewriter_ItDand baselines on multiple concepts erasing. We erased 50 celebrities at once. The remaining celebrity concepts serve as surrogate concepts in the baselines and as training data for SAE in ItD\ typewriter_ItD, whereas the concepts in DiffusionDB-10K are used solely during the generation process. Table 1: Quantitative results on celebrities erasure. We use CLIP Score (CS) and GCD accuracy (ACC) for target celebrities. We measured CS and FID for COCO-30K and DiffusionDB-10K, or KID for the other remaining concepts. Methods Target Concepts Remaining Concepts Unpredictable Concepts 50 Celebrities 100 Celebrities 100 Artistic Styles COCO-30K DiffusionDB-10K CS ↓ ↓ ACC%percent\%% ↓ ↓ CS ↑ ↑ ACC%percent\%% ↑ ↑ KID(×100absent100× 100× 100) ↓ ↓ CS ↑ ↑ KID(×100absent100× 100× 100) ↓ ↓ CS ↑ ↑ FID(×100absent100× 100× 100) ↓ ↓ CS ↑ ↑ FID ↓ ↓ ESD-x (Gandikota et al., 2023) 24.41 7.30 26.23 10.39 2.66 28.23 0.01 29.55 14.40 28.94 9.46 AC (Kumari et al., 2023) 33.63 90.16 33.98 87.04 1.86 28.47 0.38 30.91 16.91 31.28 8.55 AdvUn (Zhang et al., 2024b) 16.71 0.00 17.61 2.84 14.80 19.29 10.29 18.25 47.38 14.53 63.61 Receler (Huang et al., 2023) 12.84 0.00 13.37 86.45 18.09 24.80 5.23 29.98 15.85 27.85 18.96 MACE (Lu et al., 2024) 24.60 3.29 34.39 84.64 0.23 27.75 0.47 30.38 14.03 29.59 11.10 CPE (Lee et al., 2025) 20.79 0.37 34.82 88.26 0.08 29.01 0.01 30.86 14.62 31.84 4.18 ItD (Ours) 19.65 0.00 34.87 89.12 0 29.02 0 31.02 14.72 32.06 0.91 SD1.4 (Rombach et al., 2022) 34.49 90.56 34.87 89.12 - 29.02 - 31.02 14.73 32.17 - 5 Experiments In this section, we conduct comprehensive experiments to evaluate the effectiveness of our approach. Setting. We consider four domains for concept erasing tasks: celebrities, artistic styles, COCO-30K captions, and DiffusionDB-10K captions, where the last dataset contains 10K captions collected from DiffusionDB (Wang et al., 2022). These captions are constructed by collecting chat messages from Stable Diffusion Discord channels, representing the user prompts in a real-world setting. Next, we conduct experiments on the removal of explicit contents and evaluate the efficacy on I2P dataset (Schramowski et al., 2023). We also evaluate the robustness against adversarial prompts using the red-teaming tools Unlearning-Diff (Zhang et al., 2025). We generate images using SD1.4 (Rombach et al., 2022) with DDIM in 50 steps. SAE Training Setting For celebrity and artistic style erasure, we train an SAE using 200 celebrity names and 200 artist names from MACE (Lu et al., 2024). Additionally, we incorporate the full captions from COCO-30K. The celebrity and artist names are embedded into 100 and 35 prompt templates, respectively, following the setup in CPE (Lee et al., 2025). This results in a total of 57,000 text samples for training the SAE. For nudity erasure, we collect 10,000 captions from DiffusionDB (Wang et al., 2022) with an NSFW score above 0.8, along with all captions from COCO-30K, to train the SAE. Further details on the training process can be found in the Appendix B. Baselines. We compare with six baselines including four fine-tuning-based approaches: ESD (Gandikota et al., 2023), AC (Kumari et al., 2023), AdvUn (Zhang et al., 2024b), Receler (Huang et al., 2023). and two inference-based approaches: MACE (Lu et al., 2024), and CPE (Lee et al., 2025). The implementation details are provided in Appendix A. Metrics. We adopt CLIP score (Hessel et al., 2021) to evaluate the quality of generated images, where a lower score indicates an effective erasure for target concepts, and a higher score refers to better preservation for the remaining concepts. Additionally, for celebrity erasure experiments, we utilize the GIPHY Celebrity Detector (Nick Hasty & Korduban, 2025) to measure the top-1 accuracy (ACC) of generated celebrity images. A lower accuracy is better for target concepts, and a high accuracy is better for remaining concepts. For COCO-30K and DiffusionDB-10K captions, we evaluate their Frechet Inception Distance (FID) (Heusel et al., 2017). A lower value is better. Following (Lee et al., 2025), we evaluate the Kernel Inception Distance (KID) for other remaining concepts, which is more stable and reliable for a smaller number of images. 5.1 Celebrity Erasure We select 50 celebrities as the target concepts from the list of celebrities provided by (Lu et al., 2024), which consists of 200 celebrities. For the remaining concepts, we consider two domains: 100 celebrities and 100 artistic styles. We generated 25 images using 5 prompt templates with 5 random seeds, resulting in 2,500 images for each remaining domain. We also use COCO-30K and DiffusionDB-10K as remaining concepts. Notably, the captions in DiffusionDB-10K are not used for training SAE, allowing us to evaluate the effectiveness of our approach on unpredictable concepts. Figure 4 presents the qualitative results of erasing multiple celebrities. The results indicate that ItD\ typewriter_ItDand all baselines, except CA, effectively remove the target concepts. CA struggles in this scenario due to the large number of target concepts, exceeding its capacity for erasure. For the remaining concepts, AdvUn and Receler cause significant degradation in image quality, likely because adversarial unlearning introduces broader disruptions to the generation process. In contrast, MACE and CPE preserve the quality of remaining celebrity images, as they are trained within the same domain. However, images from DiffusionDB-10K exhibit noticeable deviations from those generated by the original model. Among all methods, ItD\ typewriter_ItDis the only approach that maintains the same generation quality as the original model. Table 1 presents quantitative results on erasing target concepts while evaluating the impact on remaining concepts. The results demonstrate that ItD\ typewriter_ItDhas the least effect on remaining concepts and outperforms all baselines by a large margin in preserving them, while still achieving strong erasure performance. ((a)) Figure 5: Qualitative results of our ItD\ typewriter_ItDand baselines on multiple concepts erasing. We erased 100 artist styles at once. The remaining artist style concepts serve as surrogate concepts in the baselines and as training data for SAE in ItD\ typewriter_ItD, whereas the concepts in DiffusionDB-10K are used solely during the generation process. Table 2: Quantitative results on artistic styles erasure. We used CLIP Score (CS)for target artistic styles. We measured CS and FID for COCO-30K and DiffusionDB-10K, or KID for the other remaining concepts. Methods Target Concepts Remaining Concepts Unpredictable Concepts 100 Artistic Styles 100 Artistic Styles 100 Celebrities COCO-30K DiffusionDB-10K CS ↓ ↓ CS ↑ ↑ KID(×100absent100× 100× 100) ↓ ↓ CS ↑ ↑ ACC%percent\%% ↑ ↑ KID(×100absent100× 100× 100) ↓ ↓ CS ↑ ↑ FID(×100absent100× 100× 100) ↓ ↓ CS ↑ ↑ FID ↓ ↓ ESD-x (Gandikota et al., 2023) 20.89 28.90 0.65 30.42 81.86 0.81 29.52 15.19 28.11 11.54 AC (Kumari et al., 2023) 28.91 28.33 1.25 34.77 93.71 0.25 30.97 16.19 31.21 10.20 AdvUn (Zhang et al., 2024b) 18.94 19.28 9.65 17.78 0.0 13.90 18.11 43.24 12.63 62.84 Receler (Huang et al., 2023) 24.90 24.96 2.11 32.76 86.68 0.44 29.29 16.25 26.83 22.00 MACE (Lu et al., 2024) 22.59 28.95 0.25 26.87 10.79 0.25 29.51 12.71 25.77 16.70 CPE (Lee et al., 2025) 20.67 28.95 0.01 34.81 89.80 0.04 30.95 14.77 31.60 5.72 ItD (Ours) 19.88 29.02 0.00 34.87 89.12 0.00 31.02 14.71 31.91 1.07 SD1.4 (Rombach et al., 2022) 29.72 29.02 - 34.87 89.12 - 31.02 14.73 32.17 - Table 3: Robust concept erasure against adversarial attack: UnlearnDiff (Zhang et al., 2025) Method ASR (↓ ↓) FMN(Zhang et al., 2024a) 97.89 ESD (Gandikota et al., 2023) 76.05 UCE (Gandikota et al., 2024) 79.58 MACE (Lu et al., 2024) 66.90 AdvUnlearn (Zhang et al., 2024b) 21.13 ItD\ typewriter_ItD(Ours) 12.61 Table 4: Results of the detected number of explicit contents using NudeNet detector on I2P and preservation performance on COCO 30K with CS, FID. Methods Number of nudity detected on I2P (Detected Quantity) COCO-30K Armpits Belly Buttocks Feet Breasts (F) Genitalia (F) Breasts (M) Genitalia (M) Total CS ↑ ↑ FID ↓ ↓ FMN (Zhang et al., 2024a) 43 117 12 59 155 17 19 2 424 30.39 13.52 ESD-x (Gandikota et al., 2023) 59 73 12 39 100 4 30 8 315 30.69 14.41 AC (Kumari et al., 2023) 153 180 45 66 298 22 67 7 838 31.37 16.25 AdvUn (Zhang et al., 2024b) 8 0 0 13 1 1 0 0 28 28.14 17.18 Receler (Huang et al., 2023) 48 32 3 35 20 0 17 5 160 30.49 15.32 MACE (Lu et al., 2024) 17 19 2 39 16 0 9 7 111 29.41 13.40 CPE (Lee et al., 2025) 10 8 2 8 6 1 3 2 40 31.19 13.89 ItD (Ours) 0 2 3 3 0 0 0 10 18 30.42 14.64 SD v1.4 (Rombach et al., 2022) 148 170 29 63 266 18 42 7 73 31.02 14.73 SD v2.1 (Face & CompVis, 2023b) 105 159 17 60 177 9 57 2 586 31.53 14.87 5.2 Artistic Styles Erasure We select 100 artistic styles as the target concepts following (Lu et al., 2024) and utilize the same remaining concepts in Section 5.1, including 100 celebrity, 100 artistic styles, COCO-30K, and DiffusionDB-10K. We utilize the same SAE used for celebrity erasure. Figure 5 shows the qualitative results of erasing multiple artist styles. Among all baselines, Receler fails to erase the target artistic style, possibly because it adopts a mask to identify pixels for erasure within the image, which fails when erasing styles, leading to unsuccessful erasure. Table 2 shows results for artistic style erasure that are consistent with those from celebrity erasure. ItD\ typewriter_ItDeffectively distinguishes between target artistic styles, remaining artistic styles, and celebrities, ensuring strong erasure performance while preventing degradation of remaining concepts. Additionally, it has the least impact on DiffusionDB-10K and outperforms all baselines by a large margin. 5.3 Explicit Content Erasure We evaluate the effectiveness of erasing explicit contents on the I2P dataset (Schramowski et al., 2023). It consists of 4,703 prompts without inappropriate words but would generate explicit images with stable diffusion. To erase explicit concepts, we adopt four keywords as target concepts to select features following (Lu et al., 2024): ’nudity’, ‘naked’, ‘erotic’, and ‘sexual’. We employ the NudeNet detector (Bedapudi, 2025) to measure the frequency of explicit content. The threshold is set to 0.60.60.60.6 (Lu et al., 2024). For preservation performance evaluation, we utilize COCO-30K. We train an SAE using the captions of COCO-30K. Table 4 shows the number of explicit contents detected by the NudeNet detector. The results show that ItD\ typewriter_ItDresulted in the fewest detected explicit contents compared with baselines. 5.4 Robustness To evaluate the robustness against the adversarial attack prompts, we utilize UnlearnDiff (Zhang et al., 2025) as the red-teaming tool to verify the robustness of our ItD\ typewriter_ItD. We evaluate on I2P dataset and report the attack success rate (ASR) as the evaluation metric to measure the ratio of generated images containing explicit content. The results are shown in Table 3. From Table 3, we can find that ItD\ typewriter_ItDdemonstrates robust erasure of target concepts competitive to recent robust methods AdvUnlearn (Zhang et al., 2024b) and MACE (Lu et al., 2024). In particular, ItD\ typewriter_ItDsuccessfully defends attacks by UnlearnDiff, significantly outperforming the existing approaches, verifying its robustness. 6 Conclusion In this work, we show that only fine-tuning CA layers for concept erasing in diffusion models could sometimes fail in preserving remaining concepts. As one solution, we proposed our framework, ItD\ typewriter_ItD(Interpret-then-deactivate), simple and effective approach to remove the regulation constraint constraint aiming to erase target concepts while maintaining diverse remaining concepts. We integrate the Sparse Autoencoder (SAE) into the the diffusion models, making it capable of adaptively adjusting the text embeddings. To robustly erase target concepts without forgetting on remaining concepts, we also comprehensively compare the unique features unique to the target concepts, and deactive them, removing the effect on the remaining concepts. Through extensive experiments on erasure of celebrities, artistic styles, and explicit concepts, the empirical results ensure the robust deletion of target concepts and protection of diverse remaining concepts. 7 Impact Statements: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, for example, improving the safety and ethical use of generative models by preventing the creation of harmful or explicit content. This could help mitigate the spread of inappropriate or harmful imagery, ensuring that AI systems align more closely with societal values and legal standards, particularly in areas such as content moderation, education, and creative industries. References Bedapudi (2025) Bedapudi, P. Nudenet: Neural nets for nudity detection and censoring., 2025. URL https://nudenet.notai.tech/. Bui et al. (2024a) Bui, A., Vu, T., Vuong, L., Le, T., Montague, P., Abraham, T., and Phung, D. Fantastic targets for concept erasure in diffusion models and where to find them. Preprint, 2024a. Bui et al. (2024b) Bui, A., Vuong, L., Doan, K., Le, T., Montague, P., Abraham, T., and Phung, D. Erasing undesirable concepts in diffusion models with adversarial preservation. arXiv preprint arXiv:2410.15618, 2024b. Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. Du et al. (2020) Du, Y., Li, S., and Mordatch, I. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637–6647, 2020. Du et al. (2021) Du, Y., Li, S., Sharma, Y., Tenenbaum, J., and Mordatch, I. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34:15608–15620, 2021. Efron (2011) Efron, B. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011. Elhage et al. (2021) Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021. Face & CompVis (2023a) Face, H. and CompVis. Stable diffusion safety checker, 2023a. URL https://huggingface.co/CompVis/stable-diffusion-safety-checker. Face & CompVis (2023b) Face, H. and CompVis. Stable diffusion 2, 2023b. URL https://huggingface.co/stabilityai/stable-diffusion-2. Fan et al. (2023) Fan, C., Liu, J., Zhang, Y., Wong, E., Wei, D., and Liu, S. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint arXiv:2310.12508, 2023. Gandikota et al. (2023) Gandikota, R., Materzynska, J., Fiotto-Kaufman, J., and Bau, D. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2426–2436, 2023. Gandikota et al. (2024) Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., and Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 5111–5120, 2024. Gao et al. (2024) Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/abs/2406.04093. Heng & Soh (2024) Heng, A. and Soh, H. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36, 2024. Hessel et al. (2021) Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2021. Heusel et al. (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. Hu et al. (2021) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Huang et al. (2023) Huang, C.-P., Chang, K.-P., Tsai, C.-T., Lai, Y.-H., and Wang, Y.-C. F. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. arXiv preprint arXiv:2311.17717, 2023. Huben et al. (2024) Huben, R., Cunningham, H., Riggs, L., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=F76bwRSLeK. Jin et al. (2024) Jin, M., Yu, Q., Huang, J., Zeng, Q., Wang, Z., Hua, W., Zhao, H., Mei, K., Meng, Y., Ding, K., et al. Exploring concept depth: How large language models acquire knowledge at different layers? arXiv preprint arXiv:2404.07066, 2024. Kinga et al. (2015) Kinga, D., Adam, J. B., et al. A method for stochastic optimization. In International conference on learning representations (ICLR), volume 5, p. 6. San Diego, California;, 2015. Kissane et al. (2024) Kissane, C., Krzyzanowski, R., Bloom, J. I., Conmy, A., and Nanda, N. Interpreting attention layer outputs with sparse autoencoders, 2024. URL https://arxiv.org/abs/2406.17759. Ko et al. (2024) Ko, M., Li, H., Wang, Z., Patsenker, J., Wang, J. T., Li, Q., Jin, M., Song, D., and Jia, R. Boosting alignment for post-unlearning text-to-image generative models. arXiv preprint arXiv:2412.07808, 2024. Kumari et al. (2023) Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., and Zhu, J.-Y. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22691–22702, 2023. Lee et al. (2025) Lee, B. H., Lim, S., Lee, S., Kang, D. U., and Chun, S. Y. Concept pinpoint eraser for text-to-image diffusion models via residual attention gate. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=ZRDhBwKs7l. Li et al. (2024) Li, X., Yang, Y., Deng, J., Yan, C., Chen, Y., Ji, X., and Xu, W. Safegen: Mitigating sexually explicit content generation in text-to-image models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 4807–4821, 2024. Liu et al. (2024) Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, p. 38–55. Springer, 2024. Lu et al. (2024) Lu, S., Wang, Z., Li, L., Liu, Y., and Kong, A. W.-K. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6430–6440, 2024. Lyu et al. (2024) Lyu, M., Yang, Y., Hong, H., Chen, H., Jin, X., He, Y., Xue, H., Han, J., and Ding, G. One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7559–7568, 2024. Makhzani & Frey (2013) Makhzani, A. and Frey, B. K-sparse autoencoders. arXiv preprint arXiv:1312.5663, 2013. Nick Hasty & Korduban (2025) Nick Hasty, Ihor Kroosh, D. V. and Korduban, D. Giphy celebrity detector, 2025. URL https://github.com/Giphy/celeb-detection-oss. Olshausen & Field (1997) Olshausen, B. A. and Field, D. J. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997. Orgad et al. (2023) Orgad, H., Kawar, B., and Belinkov, Y. Editing implicit assumptions in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7053–7061, 2023. Rando et al. (2022) Rando, J., Paleka, D., Lindner, D., Heim, L., and Tramèr, F. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610, 2022. Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695, 2022. Ruiz et al. (2023) Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 22500–22510, 2023. Saharia et al. (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022. Schramowski et al. (2023) Schramowski, P., Brack, M., Deiseroth, B., and Kersting, K. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22522–22531, 2023. Wang et al. (2022) Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., and Chau, D. H. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arXiv:2210.14896 [cs], 2022. URL https://arxiv.org/abs/2210.14896. Wu & Harandi (2025) Wu, J. and Harandi, M. Scissorhands: Scrub data influence via connection sensitivity in networks. In European Conference on Computer Vision, p. 367–384. Springer, 2025. Yang et al. (2024) Yang, Y., Hui, B., Yuan, H., Gong, N., and Cao, Y. Sneakyprompt: Jailbreaking text-to-image generative models. In 2024 IEEE symposium on security and privacy (SP), p. 897–912. IEEE, 2024. Zhang et al. (2024a) Zhang, G., Wang, K., Xu, X., Wang, Z., and Shi, H. Forget-me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1755–1764, 2024a. (45) Zhang, Y., Fan, C., Zhang, Y., Yao, Y., Jia, J., Liu, J., Zhang, G., Liu, G., Kompella, R. R., Liu, X., et al. Unlearncanvas: Stylized image dataset for enhanced machine unlearning evaluation in diffusion models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Zhang et al. (2024b) Zhang, Y., Chen, X., Jia, J., Zhang, Y., Fan, C., Liu, J., Hong, M., Ding, K., and Liu, S. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. arXiv preprint arXiv:2405.15234, 2024b. Zhang et al. (2025) Zhang, Y., Jia, J., Chen, X., Chen, A., Zhang, Y., Liu, J., Ding, K., and Liu, S. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images… for now. In European Conference on Computer Vision, p. 385–403. Springer, 2025. Appendix A Preliminaries on baseline approaches We consider six baseline approaches, including 4 fine-tuning-based approaches:ESD (Gandikota et al., 2023), AC (Kumari et al., 2023), AdvUn (Zhang et al., 2024b), Receler (Huang et al., 2023), and two inference-based approaches MACE (Lu et al., 2024), CPE (Lee et al., 2025). A.1 Cross-Attention in T2I Diffusion Model We first introduce the fundamentals of the cross-attention (CA) layer in T2I diffusion models, which is widely utilized in previous works. The CA layer serves as a key mechanism for integrating textual information, represented by text embeddings EE, with image features, represented as image latents xtsubscriptx_txitalic_t, within T2I diffusion models. Specifically, the image feature ϕ⁢(xt)italic-ϕsubscriptφ(x_t)ϕ ( xitalic_t ) is linearly transformed into query matrix Q=lQ(ϕ(xt)Q=l_Q(φ(x_t)Q = litalic_Q ( ϕ ( xitalic_t ), while the text embedding is mapped into key matrix K=lK⁢()subscriptK=l_K(E)K = litalic_K ( E ) and value matrix V=lV⁢()subscriptV=l_V(E)V = litalic_V ( E ) through separate linear transformations. The attention map can be calculated as: M=Softmax⁡(Q⁢KTd)∈ℝu×(h×w)×n,SoftmaxsuperscriptsuperscriptℝℎM=Softmax ( QK^T d ) ^u% ×(h× w)× n,M = Softmax ( divide start_ARG Q Kitalic_T end_ARG start_ARG square-root start_ARG d end_ARG end_ARG ) ∈ ℝitalic_u × ( h × w ) × n , where d is the projection dimension, u is the number of attention heads, h×wℎh× wh × w represents the spatial dimension of the image, and n is the number of text tokens. The final output of the cross-attention is M⁢VMVM V, representing the weighted average of the values in V. A.2 Fine-tuning-based approaches ESD is inspired by energy-based composition (Du et al., 2020, 2021). It aims to reduce the probability of generating images belonging to the target concept. Specifically, it aims to minimize Pθ⁢(x)∝Pθo⁢(x)Pθo⁢(c∣x)η,proportional-tosubscriptsubscriptsubscriptosubscriptsubscriptosuperscriptconditionalP_θ(x) P_ _o(x)P_ _o(% c x)^η,Pitalic_θ ( x ) ∝ divide start_ARG Pitalic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( x ) end_ARG start_ARG Pitalic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( c ∣ x )η end_ARG , (11) where Pθo⁢(x)subscriptsubscriptoP_ _o(x)Pitalic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( x ) represents the distribution generated by the original model, c is the target concept to erase, and η is scale parameter. Inspired by Equ. 11, and Tweedie’s formula (Efron, 2011) ϵ~θ⁢(xt∣ce)←ϵθo⁢(xt∣∅)−η⁢(ϵθo⁢(xt∣c)−ϵθo⁢(xt∣∅)),←subscript~italic-ϵconditionalsubscriptsubscriptsubscriptitalic-ϵsubscriptconditionalsubscriptsubscriptitalic-ϵsubscriptconditionalsubscriptsubscriptitalic-ϵsubscriptconditionalsubscript ε_θ (x_t c_e )← _% _o (x_t )-η ( _ _o% (x_t c )- _ _o (x_t )% ),over~ start_ARG ϵ end_ARGθ ( xitalic_t ∣ citalic_e ) ← ϵitalic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( xitalic_t ∣ ∅ ) - η ( ϵitalic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( xitalic_t ∣ c ) - ϵitalic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( xitalic_t ∣ ∅ ) ) , (12) The training loss to optimize θ is: minθℓESD(θ,c):=[∥ϵθ(xt∣c)−(ϵθ0(xt∣∅)−η(ϵθ0(xt∣c)−ϵθ0(xt∣∅)))∥22]. _θ _ESD(θ,c):=E [ \|% _θ (x_t c )- ( _ _0 (x_% t )-η ( _ _0 (x_t c )% - _ _0 (x_t ) ) ) \|_2% ^2 ].minitalic_θ ℓroman_ESD ( θ , c ) := roman_ [ ∥ ϵitalic_θ ( xitalic_t ∣ c ) - ( ϵitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( xitalic_t ∣ ∅ ) - η ( ϵitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( xitalic_t ∣ c ) - ϵitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( xitalic_t ∣ ∅ ) ) ) ∥22 ] . (13) Empirical results show that updating only the CA layers (ESD-x) effectively erases target concepts while preserving the overall performance of remaining concepts. In contrast, fine-tuning all parameters (ESD-u) can lead to significant degradation of remaining concepts. In our implementation of ESD, we mainly implement ESD-x, as it demonstrates good concept erasure while minimizing its impact on remaining concepts. AdvUn is designed based on ESD. It proposes a bi-level optimization approach that is robust to adversarial attacks. The upper-level optimization aims to minimize the unlearning objective, while lower-level optimization aims to find the optimized adversarial prompt c∗superscriptc^*c∗: minimizeℓu⁢(,c∗) [Upper-level] subject to c∗=argmin‖c∗−c‖0≤ρ[∥ϵθ(t∣c∗)−ϵθ∗(t∣c)∥22]. [Lower-level] array[]l θminimize&% _u ( θ,c^* )& [Upper-level] % \\ subject to &c^*= _ \|c^*-c \|_0≤ρ % E [ \| _θ (x_t c^* )-% _θ^* (x_t c ) \|_2^2 ].% & [Lower-level] arraystart_ARRAY start_ROW start_CELL underitalic_θ start_ARG minimize end_ARG end_CELL start_CELL ℓroman_u ( italic_θ , c∗ ) end_CELL start_CELL [Upper-level] end_CELL end_ROW start_ROW start_CELL subject to end_CELL start_CELL c∗ = arg min∥ c∗ - c ∥ start_POSTSUBSCRIPT 0 ≤ ρ end_POSTSUBSCRIPT roman_ [ ∥ ϵitalic_θ ( xitalic_t ∣ c∗ ) - ϵitalic_θ∗ ( xitalic_t ∣ c ) ∥22 ] . end_CELL start_CELL [Lower-level] end_CELL end_ROW end_ARRAY (14) To maintain the utility on remaining concepts, it adopts a regularization term to penalize the discrepancy between the original model and optimized model on a retain set CretainsubscriptretainC_retainCretain. As a result, the upper-level optimization objective is: ℓu(,c∗)=ℓESD(,c∗)+γc~∼retain [∥ϵ(t∣c~)−ϵo(t∣c~)∥22] _u ( θ,c^* )= _ESD% ( θ,c^* )+ _ c % C_retain [ \| _ θ (% x_t c )- _ θ_o (% x_t c ) \|_2^2 ]ℓroman_u ( italic_θ , c∗ ) = ℓroman_ESD ( italic_θ , c∗ ) + γ over~ start_ARG c end_ARG ∼ C start_POSTSUBSCRIPT retain end_POSTSUBSCRIPT [ ∥ ϵbold_italic_θ ( xitalic_t ∣ over~ start_ARG c end_ARG ) - ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( xitalic_t ∣ over~ start_ARG c end_ARG ) ∥22 ] AdvUn tried to perform erasing on different layers within the text encoder and U-net in SD v1-4 (Rombach et al., 2022). Empirical results show that erasing within the text encoder has the best robustness against adversarial attacks. In our implementation of AdvUn, we perform erasing on the text encoder. Receler is also designed based on ESD and adopts adversarial prompt learning to ensure erasure robustness. To reduce the impact on remaining concepts, it adopts a concept-localized regularization for erasing locality: ℓR⁢e⁢g=1L⁢∑l=1L‖ol⊙(1−M)‖2subscriptℓ1superscriptsubscript1superscriptnormdirect-productsuperscript12 _Reg= 1L _l=1^L \|o^l (1-M) \|^2ℓitalic_R e g = divide start_ARG 1 end_ARG start_ARG L end_ARG ∑l = 1L ∥ oitalic_l ⊙ ( 1 - M ) ∥2 (15) where L is the number of U-Net’s layers, ⊙direct-product ⊙ is the element-wise product, olsuperscripto^loitalic_l is the output of the eraser in the l-th layer, and M is the mask of target concept in image and generated using GroundingDINO (Liu et al., 2024) AC is another approach for unlearning. It prevents the model from generating unwanted images by mapping target concepts to an anchor concept, which can be either a generic concept, such as ’dog’ to replace ’English springer’ or a null concept, such as an empty text prompt. Moreover, it also utilizes a regularization term to penalize the discrepancy between the original model and the optimized model on a set of retained concepts. The training objective can be formulated as follows: minℓAC(θ,c):=[∥ϵθ(xt∣c)−ϵθo(xt∣ca)∥22]+γc~∼retain [∥ϵ(t∣c~)−ϵo(t∣c~)∥22], θmin _AC (θ,c ):=% E [ \| _θ (x_t c )- _% _o (x_t c_a ) \|_2^2 ]+ % _ c _retain [ \| _% θ (x_t c )- _% θ_o (x_t c )% \|_2^2 ],underθ start_ARG min end_ARG ℓroman_AC ( θ , c ) := roman_ [ ∥ ϵitalic_θ ( xitalic_t ∣ c ) - ϵitalic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( xitalic_t ∣ citalic_a ) ∥22 ] + γ over~ start_ARG c end_ARG ∼ C start_POSTSUBSCRIPT retain end_POSTSUBSCRIPT [ ∥ ϵbold_italic_θ ( xitalic_t ∣ over~ start_ARG c end_ARG ) - ϵbold_italic_θ start_POSTSUBSCRIPT o end_POSTSUBSCRIPT ( xitalic_t ∣ over~ start_ARG c end_ARG ) ∥22 ] , (16) where casubscriptc_acitalic_a is the surrogate concept. A.3 Inference-based approaches MACE adopts a closed-form solution to refine CA layers within U-net to erase unwanted knowledge. Briefly, it finds the linear projections W′ of keys and values in the cross-attention layers such that: new =arg⁡min′⁢∑n=1N‖′⁢tar n−o⁢sur n‖F2+λ⁢∑m=1M‖′⁢retain m−o⁢retain m‖F2,subscriptnew subscriptsuperscript′subscript1superscriptsubscriptnormsuperscript′subscripttar subscriptosuperscriptsubscriptsur 2superscriptsubscript1superscriptsubscriptnormsuperscript′subscriptretain subscriptosuperscriptsubscriptretain 2W_new = _W _n=1^N \|% W E_tar ^n-W_oE% _sur ^n \|_F^2+λ _m=1^M \|W^% E_retain ^m-W_oE_% retain ^m \|_F^2,Wnew = arg minbold_W′ ∑n = 1N ∥ W′ Etaritalic_n - Wo Esuritalic_n ∥F2 + λ ∑m = 1M ∥ W′ Eretainitalic_m - Wo Eretainitalic_m ∥F2 , (17) where osubscriptoW_oWo is the original key/value projection. tar,sursubscripttarsubscriptsurE_tar,E_surEtar , Esur and retainsubscriptretainE_retainEretain represent the text embedding of target, surrogate, and retain concepts, respectively. To mitigate the impact on the overall parameters, MACE inserts LoRA modules into the CA layers of the model for each target concept. Then, the multiple LoRAs are integrated by a loss function to enable erasing multiple concepts. CPE also works on the linear projections of keys and values in the cross-attention layers. It inserts a customized modular, named residual attention gate (ResAG), into each CA layer within U-net. ResAG is trained to make the projection output of tarsubscripttarE_tarEroman_tar similar to the output of sursubscriptsurE_surEroman_sur. Formally, the erasing objective is: minRtar⁢ℓera=(tar,sur)⁢‖(tar+Rtar⁢(tar))−(sur−η⁢(tar−sur))‖2,subscripttarminsubscriptℓerasubscriptsubscripttarsubscriptsursuperscriptnormsubscripttarsubscripttarsubscripttarsubscriptsursubscripttarsubscriptsur2 R_tarmin _era=E_% (E_tar,E_sur ) \| (% WE_tar+R_tar (E_% tar ) )- (WE_sur- % (E_tar-E_sur ) ) % \|^2,start_UNDERACCENT Rroman_tar end_UNDERACCENT start_ARG min end_ARG ℓera = roman_( E start_POSTSUBSCRIPT tar , Eroman_sur ) end_POSTSUBSCRIPT ∥ ( WEroman_tar + Rroman_tar ( Eroman_tar ) ) - ( WEroman_sur - η W ( Eroman_tar - Eroman_sur ) ) ∥2 , (18) where RtarsubscripttarR_tarRroman_tar is the ResAG for target concept. To prevent undesirable degradation on remaining concepts, CPE adopts a regularization term to minimize the deviation induced by ResAG on retain concepts: ℒatt=retain⁢‖Rtar⁢(retain)‖F.subscriptℒattsubscriptsubscriptretainsubscriptnormsubscripttarsubscriptretainL_att=E_E_retain \|R_% tar (E_retain ) \|_F.Latt = bold_E start_POSTSUBSCRIPT retain end_POSTSUBSCRIPT ∥ Rroman_tar ( Eretain ) ∥F . (19) Each ResAG is trained specifically to one target concept. Therefore, to erase multiple concepts, it is required to train multiple ResAGs. Appendix B Training details of SAE We train an SAE using the output of the 8th transformer block of the text encoder layers.8, which we experimentally show has the best performance. We set K=6464K=64K = 64 and dhid=219subscripthidsuperscript219d_hid=2^19dhid = 219 following (Gao et al., 2024). We adopt Adam (Kinga et al., 2015) as the optimizer with the learning rate of 5⁢e−5555e-55 e - 5 and a constant scheduler without warmup. Following Gao et al. (2024), we set α=132132α= 132α = divide start_ARG 1 end_ARG start_ARG 32 end_ARG and Kaux=256subscriptaux256K_aux=256Kaux = 256. We train the SAE while simultaneously generating training samples with the text encoder, which does not require additional storage space to save samples. The batch size of prompts input into the text encoder is 50, which results in about 1000 samples to train SAE each time. We train SAE on a single H100. For celebrity and artistic style erasure, where we train an SAE using celebrity and artist styles and the captions of COCO-30K, the training time is 56 minutes. For nudity erasure, we train an SAE using the captions of COCO-30 and 10K prompts from DiffusionDB (Wang et al., 2022) with an NSFW score larger than 0.8, the training time is 40 minutes. Appendix C Efficiency study The structure of SAE used in ItD\ typewriter_ItDis very simple, with only two linear layers, two bias layers, and the TopKTopKTopKTopK activation function. While it contains a large number of parameters due to the large hidden size, it is efficient during inference as it mainly requires two matrix multiplication operations. We report the inference time of SAE along with its time ratio relative to the entire image generation process in Table 5. The inference time is measured per prompt, which consists of 77 tokens (the maximum length allowed in SD v1.4 and SD v2.1). The time required to generate a single image is computed by repeating the process 10 times and taking the average. The number of inference steps is set to 50. The results indicate that SAE is highly efficient, accounting for less than 1% of the total image generation time. We note that unlike other module-based approaches that require increased inference time as more concepts are erased (Lyu et al., 2024; Lee et al., 2025), the inference time of ItD\ typewriter_ItDis independent of the number of concepts being erased. SD v1.4 SD v2.1 Inference Time (1000 prompts) 5.05s 6.37s Time Ratio (per prompt) 0.22% 0.13% Table 5: Efficiency study. The inference time of SAE along with its time ratio relative to the entire image generation process. Appendix D Implementation details D.1 Celebrity Erasure We select 50 celebrities from 200 celebrities provide in MACE (Lu et al., 2024) as target concepts to erase. The celebrities can be accurately generated by Stable Diffusion v1.4 (Rombach et al., 2022), which have over 99% accuracy of the GIPHY Celebrity Detector (GCD) (Nick Hasty & Korduban, 2025). The 50 target celebrities are listed in Table 6. We also select 100 celebrities as remaining concepts to preserve, as listed in Table 7. To generate their images, we used 5 prompt templates with 5 random seeds (1-5). The prompt templates are distinct for celebrities and artistic styles. We used 0 as a seed generating 5 images from a prompt for characters. The prompt templates are listed in Table 9. To select features specific to the target celebrities, we adopt the remaining 100 celebrities as well as 1000 captions from COCO-30K as the retain set. In the case of celebrities erasure, we set the following negative prompts to improve image quality: ”bad anatomy, watermark, extra digit, signature, worst quality, jpeg artifacts, normal quality, low quality, long neck, lowres, error, blurry, missing fingers, fewer digits, missing arms, text, cropped, humpbacked, bad hands, username” Table 6: List of target celebrities. We adopt the same 50 celebrities following (Lee et al., 2025). The selected celebrities have over 99% accuracy by the GIPHY Celebrity Detector (GCD) (Nick Hasty & Korduban, 2025). # of Celebrities to be erased Celebrity 50 ’Adam Driver’, ’Adriana Lima’, ’Amber Heard’, ’Amy Adams’, ’Andrew Garfield’, ’Angelina Jolie’, ’Anjelica Huston’, ’Anna Faris’, ’Anna Kendrick’, ’Anne Hathaway’, ’Arnold Schwarzenegger’, ’Barack Obama’, ’Beth Behrs’, ’Bill Clinton’, ’Bob Dylan’, ’Bob Marley’, ’Bradley Cooper’, ’Bruce Willis’, ’Bryan Cranston’, ’Cameron Diaz’, ’Channing Tatum’, ’Charlie Sheen’, ’Charlize Theron’, ’Chris Evans’, ’Chris Hemsworth’,’Chris Pine’, ’Chuck Norris’, ’Courteney Cox’, ’Demi Lovato’, ’Drake’, ’Drew Barrymore’, ’Dwayne Johnson’, ’Ed Sheeran’, ’Elon Musk’, ’Elvis Presley’, ’Emma Stone’, ’Frida Kahlo’, ’George Clooney’, ’Glenn Close’, ’Gwyneth Paltrow’, ’Harrison Ford’, ’Hillary Clinton’, ’Hugh Jackman’, ’Idris Elba’, ’Jake Gyllenhaal’, ’James Franco’, ’Jared Leto’, ’Jason Momoa’, ’Jennifer Aniston’, ’Jennifer Lawrence’ Table 7: List of celebrities to preserve. We adopt the same 100 celebrities following (Lee et al., 2025). The selected celebrities have over 99% accuracy by the GIPHY Celebrity Detector (GCD) (Nick Hasty & Korduban, 2025). # of Celebrities to be preserve Celebrity 100 ’Aaron Paul’, ’Alec Baldwin’, ’Amanda Seyfried’, ’Amy Poehler’, ’Amy Schumer’, ’Amy Winehouse’, ’Andy Samberg’, ’Aretha Franklin’, ’Avril Lavigne’, ’Aziz Ansari’, ’Barry Manilow’, ’Ben Affleck’, ’Ben Stiller’, ’Benicio Del Toro’, ’Bette Midler’, ’Betty White’, ’Bill Murray’, ’Bill Nye’, ’Britney Spears’, ’Brittany Snow’, ’Bruce Lee’, ’Burt Reynolds’, ’Charles Manson’, ’Christie Brinkley’, ’Christina Hendricks’, ’Clint Eastwood’, ’Countess Vaughn’, ’Dane Dehaan’, ’Dakota Johnson’, ’David Bowie’, ’David Tennant’, ’Denise Richards’, ’Doris Day’, ’Dr Dre’, ’Elizabeth Taylor’, ’Emma Roberts’, ’Fred Rogers’, ’George Bush’, ’Gal Gadot’, ’George Takei’, ’Gillian Anderson’, ’Gordon Ramsey’, ’Halle Berry’, ’Harry Dean Stanton’, ’Harry Styles’, ’Hayley Atwell’, ’Heath Ledger’, ’Henry Cavill’, ’Jackie Chan’, ’Jada Pinkett Smith’, ’James Garner’, ’Jason Statham’, ’Jeff Bridges’, ’Jennifer Connelly’, ’Jensen Ackles’, ’Jim Morrison’, ’Jimmy Carter’, ’Joan Rivers’, ’John Lennon’, ’Jon Hamm’, ’Judy Garland’, ’Julianne Moore’, ’Justin Bieber’, ’Kaley Cuoco’, ’Kate Upton’, ’Keanu Reeves’, ’Kim Jong Un’, ’Kirsten Dunst’, ’Kristen Stewart’, ’Krysten Ritter’, ’Lana Del Rey’, ’Leslie Jones’, ’Lily Collins’, ’Lindsay Lohan’, ’Liv Tyler’, ’Lizzy Caplan’, ’Maggie Gyllenhaal’, ’Matt Damon’, ’Matt Smith’, ’Matthew Mcconaughey’, ’Maya Angelou’, ’Megan Fox’, ’Mel Gibson’, ’Melanie Griffith’, ’Michael Cera’, ’Michael Ealy’, ’Natalie Portman’, ’Neil Degrasse Tyson’, ’Niall Horan’, ’Patrick Stewart’, ’Paul Rudd’, ’Paul Wesley’, ’Pierce Brosnan’, ’Prince’, ’Queen Elizabeth’, ’Rachel Dratch’, ’Rachel Mcadams’, ’Reba Mcentire’, ’Robert De Niro’ D.2 Artist Style Erasure We select 100 artist styles as target concepts to erase, and 100 artist styles as remaining concepts to preserve. The 100 target artistic styles are listed in Table 8 and the remaining concepts are listed in Table 10. To generate their images, we used 5 prompt templates with 5 random seeds (1-5). The prompt templates are listed in Table 9, which are different from celebrity erasure. To select features specific to the target artist styles, we adopt the remaining 100 artist styles as well as 1000 captions from COCO-30K as the retain set. Table 8: List of target artist styles. We adopt the same 100 artist styles following (Lee et al., 2025). All artistic styles in these images were successfully generated using SD v1.4. # of Artist Styles to be erased Artist Style 100 ’Brent Heighton’, ’Brett Weston’, ’Brett Whiteley’, ’Brian Bolland’, ’Brian Despain’, ’Brian Froud’, ’Brian K. Vaughan’, ’Brian Kesinger’, ’Brian Mashburn’, ’Brian Oldham’, ’Brian Stelfreeze’, ’Brian Sum’, ’Briana Mora’, ’Brice Marden’, ’Bridget Bate Tichenor’, ’Briton Riviere’, ’Brooke Didonato’, ’Brooke Shaden’, ’Brothers Grimm’, ’Brothers Hildebrandt’, ’Bruce Munro’, ’Bruce Nauman’, ’Bruce Pennington’, ’Bruce Timm’, ’Bruno Catalano’, ’Bruno Munari’, ’Bruno Walpoth’, ’Bryan Hitch’, ’Butcher Billy’, ’C. R. W. Nevinson’, ’Cagnaccio Di San Pietro’, ’Camille Corot’, ’Camille Pissarro’, ’Camille Walala’, ’Canaletto’, ’Candido Portinari’, ’Carel Willink’, ’Carl Barks’, ’Carl Gustav Carus’, ’Carl Holsoe’, ’Carl Larsson’, ’Carl Spitzweg’, ’Carlo Crivelli’, ’Carlos Schwabe’, ’Carmen Saldana’, ’Carne Griffiths’, ’Casey Weldon’, ’Caspar David Friedrich’, ’Cassius Marcellus Coolidge’, ’Catrin WelzStein’, ’Cedric Peyravernay’, ’Chad Knight’, ’Chantal Joffe’, ’Charles Addams’, ’Charles Angrand’, ’Charles Blackman’, ’Charles Camoin’, ’Charles Dana Gibson’, ’Charles E. Burchfield’, ’Charles Gwathmey’, ’Charles Le Brun’, ’Charles Liu’, ’Charles Schridde’, ’Charles Schulz’, ’Charles Spencelayh’, ’Charles Vess’, ’Charles-Francois Daubigny’, ’Charlie Bowater’, ’Charline Von Heyl’, ’Cha ”im Soutine’, ’Chen Zhen’, ’Chesley Bonestell’, ’Chiharu Shiota’, ’Ching Yeh’, ’Chip Zdarsky’, ’Chris Claremont’, ’Chris Cunningham’, ’Chris Foss’, ’Chris Leib’, ’Chris Moore’, ’Chris Ofili’, ’Chris Saunders’, ’Chris Turnham’, ’Chris Uminga’, ’Chris Van Allsburg’, ’Chris Ware’, ’Christian Dimitrov’, ’Christian Grajewski’, ’Christophe Vacher’, ’Christopher Balaskas’, ’Christopher Jin Baron’, ’Chuck Close’, ’Cicely Mary Barker’, ’Cindy Sherman’, ’Clara Miller Burd’, ’Clara Peeters’, ‘Clarence Holbrook Carter’, ‘Claude Cahun’, ’Claude Monet’, ‘Clemens Ascher’ Table 9: Prompt templates used for celebrity image generation. For each prompt, we generate 5 images with seed ranges from 1-5. Additionally, for celebrities, we set the following negative prompts to improve image quality: ”bad anatomy, watermark, extra digit, signature, worst quality, jpeg artifacts, normal quality, low quality, long neck, lowres, error, blurry, missing fingers, fewer digits, missing arms, text, cropped, humpbacked, bad hands, username” Domain Seed Prompt Celebrity 1-5 A portrait of celebrity name A sketch of celebrity name An oil painting of celebrity name celebrity name in an official photo An image capturing celebrity name at a public event Artist Style 1-5 Image in the style of artist name Art inspired by artist name Painting in the style of \ artist name \ A reproduction of art by artist name \ A famous artwork by artist name Table 10: List of artist styles to preserve. We adopt the same 100 artist styles following (Lee et al., 2025). All artistic styles in these images were successfully generated using SD v1.4. # of Artist styles to preserve Artist Style 100 ‘A.J.Casson’, ‘Aaron Douglas’, ‘Aaron Horkey’, ‘Aaron Jasinski’, ‘Aaron Siskind’, ‘Abbott Fuller Graves’, ’Abbott Handerson Thayer’, ’Abdel Hadi Al Gazzar’, ’Abed Abdi’, ’Abigail Larson’, ’Abraham Mintchine’, ’Abraham Pether’, ’Abram Efimovich Arkhipov’, ’Adam Elsheimer’, ’Adam Hughes’, ’Adam Martinakis’, ’Adam Paquette’, ’Adi Granov’, ’Adolf Hiremy-Hirschl’, ’Adolph Got- ’tlieb’, ’Adolph Menzel’, ’Adonna Khare’, ’Adriaen van Ostade’, ’Adriaen van Outrecht’, ’Adrian Donoghue’, ’Adrian Ghenie’, ’Adrian Paul Allinson’, ’Adrian Smith’, ’Adrian Tomine’, ’Adrianus Eversen’, ’Afarin Sajedi’, ’Affandi’, ’Aggi Erguna’, ’Agnes Cecile’, ’Agnes Lawrence Pelton’, ’Agnes Martin’, ’Agostino Arrivabene’, ’Agostino Tassi’, ’Ai Weiwei’, ’Ai Yazawa’, ’Akihiko Yoshida’, ’Akira Toriyama’, ’Akos Major’, ’Akseli Gallen-Kallela’, ’Al Capp’, ’Al Feldstein’, ’Al Williamson’, ’Alain Laboile’, ’Alan Bean’, ’Alan Davis’, ’Alan Kenny’, ’Alan Lee’, ’Alan Moore’, ’Alan Parry’, ’Alan Schaller’, ’Alasdair McLellan’, ’Alastair Magnaldo’, ’Alayna Lemmer’, ’Albert Benois’, ’Albert Bierstadt’, ’Albert Bloch’, ’Albert Dubois-Pillet’, ’Albert Eckhout’, ’Albert Edelfelt’, ’Albert Gleizes’, ’Albert Goodwin’, ’Albert Joseph Moore’, ’Albert Koetsier’, ’Albert Kotin’, ’Albert Lynch’, ’Albert Marquet’, ’Albert Pinkham Ryder’, ’Albert Robida’, ’Albert Servaes’, ’Albert Tucker’, ’Albert Watson’, ’Alberto Biasi’, ’Alberto Burri’, ’Alberto Giacometti’, ’Alberto Magnelli’, ’Alberto Seveso’, ’Alberto Sughi’, ’Alberto Vargas’, ’Albrecht Anker’, ’Albrecht Durer’, ’Alec Soth’, ’Alejandro Burdisio’, ’Alejandro Jodorowsky’, ’Aleksey Savrasov’, ’Aleksi Briclot’, ’Alena Aenami’, ’Alessandro Allori’, ’Alessandro Barbucci’, ’Alessandro Gottardo’, ’Alessio Albi’, ’Alex Alemany’, ’Alex Andreev’ ’Alex Colville’, ’Alex Figini’, ’Alex Garant’ Appendix E Ablation studies for ItD\ typewriter_ItD E.1 The Effect of strength λ ((a)) ((b)) Figure 6: Ablation study on the effect of strength λ in SAE classification for distinguishing target and remaining concepts. The threshold τ is set to ensure zero false negatives; therefore, we primarily report accuracy on the remaining concepts. A higher accuracy indicates that SAE would not misclassify normal concepts as target concepts. In this section, we investigate the impact of the strength parameter λ on SAE’s effectiveness as a classifier for distinguishing between target and remaining concepts, as well as its ability to erase unwanted knowledge during text encoder inference. We vary λ from -8 to 0 and conduct experiments on celebrity and artistic style erasure tasks. Specifically, we measure SAE’s accuracy in correctly identifying remaining concepts, conditioned on successfully identifying all target concepts. This is analogous to the true negative rate (TN) under the condition that the false negative rate (FN) is zero. A higher value indicates greater effectiveness, ensuring that SAE would not misclassify normal concepts as target concepts. The results, presented in Figure 6, demonstrate that our approach is robust to the choice of τ. When τ<−11τ<-1τ < - 1, SAE performs well across the remaining 100 celebrities, 100 artistic styles, and COCO-30K. Moreover, when τ<−22τ<-2τ < - 2, SAE correctly classifies over 95% of prompts in DiffusionDB-10K as normal concepts. To demonstrate the effectiveness of λ in erasing target knowledge, we present generated images of target concepts with different λ. Figure 7 show the results experiment on sd v1.4 (Rombach et al., 2022) and and Figure 8 show the results experiment on sd v2.1 (Face & CompVis, 2023b). The results show that λ=−22λ=-2λ = - 2 is sufficient to erase knowledge about celebrity and artist style concepts. The knowledge about ”nudity” can be erased when λ=−44λ=-4λ = - 4. ((a)) Figure 7: Ablation study on the effect of strength λ in erasing unwanted knowledge. Experiments on SD v1.4 (Rombach et al., 2022). ((a)) Figure 8: Ablation study on the effect of strength λ in erasing unwanted knowledge. Experiments on SD v2.1 (Face & CompVis, 2023b). E.2 Selection of Residual Stream to perform SAE. The text encoder consists of 12 transformer blocks (layers.0-11) connected sequentially. Each residual stream of these layers can be used to train SAE. However, since different layers may capture different types of knowledge (Jin et al., 2024), applying SAE to different residual streams results in varying performance. To identify the most approximate layer for concept erasing, we conduct experiments on different residual streams. The training setup follows the details provided in Appendix B, with the only variation being the choice of residual stream. We evaluate the task of erasing 50 celebrities and 100 artist styles separately. The remaining concepts consist of 100 other celebrities and 100 other artistic styles, COCO-30K, and DiffusionDB-10K. For efficiency, similar to Figure 6, we use accuracy as a metric to assess the impact of different layers. This metric quantifies the proportion of correctly identified remaining concepts when SAE is used as a classifier. The results are summarized in Table 9. For remaining celebrities, artistic styles, and COCO-30K, the accuracy is approximately 100%, demonstrating that SAE, as a classifier, effectively identifies remaining concepts regardless of the training layer. Even for DiffusionDB-10K, which exhibits the lowest performance, the accuracy remains at least 96%. We also provide quantitative results to better understand the effects of erasing at different layers. Our experiments cover three domains: celebrities, artistic styles, and nudity content. The erasure strength τ is set to -6 for celebrities and nudity and -2 for artistic styles. As shown in Figure 11, all layers can be used to erase target concepts. However, certain layers (e.g., layers.2-4) still generate images with exposed chests when attempting to erase the concept of ”nudity”. For the celebrity domain, we use the prompts ”An oil painting of Adam Driver” and ”Anna Kendrick in an official photo.”. The results indicate that applying SAE to layers.6-10 still allows the model to generate a person within the corresponding context (”oil painting” and ”official photo”), suggesting that erasing at these layers preserves the structural integrity of the scene while removing the target identity. For artistic style erasure, applying SAE to layers.6-10 also results in the generation of a photo of a person, even though the original images did not contain any people. ((a)) ((b)) Figure 9: The log feature density is calculated as log⁡(n/N+10−9)superscript109 (n/N+10^-9)log ( n / N + 10- 9 ), where n represents the number of tokens that activate the feature, and N denotes the total number of tested tokens. ((a)) ((b)) ((c)) Figure 10: The log feature density and the reconstruction loss for SAEs trained with different hidden sizes. ((a)) Figure 11: Qualitative results of performing ItD\ typewriter_ItDon different transformer blocks. E.3 The effect of hidden size dhidsubscripthidd_hiddhid We investigate the effect of SAE with different hidden sizes for concept erasing. We vary dhidsubscripthidd_hiddhid from 215superscript2152^15215 to 219superscript2192^19219, which is about 43434343 to 683683683683 times larger than the hidden dimension of SD v1-4 (Rombach et al., 2022). We present the feature density of the SAE as well as the reconstruction loss for target and remaining concepts in Figure 10. The results indicate that the log feature density is relatively higher for smaller hidden sizes compared to larger hidden sizes, suggesting that SAE with small hidden layers learns denser features. However, this increased feature density leads to poorer performance in distinguishing between target and remaining concepts, as shown in the histogram of reconstruction loss. This may be because larger hidden layers are capable of learning more fine-grained features, which in turn enhance the ability to differentiate between different concepts. Appendix F Additional Qualitative results F.1 Example 1 of celebrities erasure. Figure 12: Qualitative comparison on celebrities erasure. The images on the same row are generated using the same seed. F.2 Example 2 of celebrities erasure. ((a)) Figure 13: Qualitative comparison on celebrities erasure. The images on the same row are generated using the same seed. F.3 Example 1 of artist styles erasure. ((a)) Figure 14: Qualitative comparison on artist styles erasure. The images on the same row are generated using the same seed. F.4 Example 2 of artist styles erasure. ((a)) Figure 15: Qualitative comparison on artist styles erasure. The images on the same row are generated using the same seed.

Instant research discovery