Paper deep dive

SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders

Enrico Cassano, Riccardo Renzulli, Marco Nurisso, Mirko Zaffaroni, Alan Perotti, Marco Grangetto

Year: 2025Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 67

Models: Stable Diffusion 1.5

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/12/2026, 5:19:48 PM

Summary

SAEmnesia is a supervised sparse autoencoder framework designed for concept unlearning in diffusion models. By enforcing one-to-one mappings between concepts and neurons through supervised labeling, it achieves feature centralization, which allows for efficient, targeted concept erasure without modifying base model weights. The method outperforms existing state-of-the-art approaches on the UnlearnCanvas benchmark, improves scalability in sequential unlearning, and enhances robustness against adversarial attacks.

Entities (5)

Diffusion Models · model-architecture · 100%SAEmnesia · framework · 100%Sparse Autoencoders · technique · 100%UNLEARNCANVAS · benchmark · 100%Stable Diffusion v1.5 · model · 95%

Relation Signals (3)

SAEmnesia → appliesto → Diffusion Models

confidence 100% · We introduce SAEmnesia, a supervised sparse autoencoder framework that overcomes this [feature splitting] in diffusion models.

Sparse Autoencoders → usedby → SAEmnesia

confidence 100% · We introduce SAEmnesia, a supervised sparse autoencoder framework

SAEmnesia → improvesperformanceon → UNLEARNCANVAS

confidence 95% · SAEmnesia achieves a 9.2% improvement over the state-of-the-art on the UnlearnCanvas benchmark.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Concept unlearning in diffusion models is hampered by feature splitting, where concepts are distributed across many latent features, making their removal challenging and computationally expensive. We introduce SAEmnesia, a supervised sparse autoencoder framework that overcomes this by enforcing one-to-one concept-neuron mappings. By systematically labeling concepts during training, our method achieves feature centralization, binding each concept to a single, interpretable neuron. This enables highly targeted and efficient concept erasure. SAEmnesia reduces hyperparameter search by 96.7% and achieves a 9.2% improvement over the state-of-the-art on the UnlearnCanvas benchmark. Our method also demonstrates superior scalability in sequential unlearning, improving accuracy by 28.4% when removing nine objects, establishing a new standard for precise and controllable concept erasure. Moreover, SAEmnesia mitigates the possibility of generating unwanted content under adversarial attack and effectively removes nudity when evaluated with I2P.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

66,330 characters extracted from source content.

Expand or collapse full text

SAEMNESIA: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders Enrico Cassano 1 Riccardo Renzulli 1 Marco Nurisso 2,3 Mirko Zaffaroni 3 Alan Perotti 3 Marco Grangetto 1 1 University of Turin, Italy 2 Politecnico di Torino, Italy 3 CENTAI Institute, Italy name.surname@unito.it,polito.it,centai.eu Abstract Concept unlearning in diffusion models is hampered by fea- ture splitting, where concepts are distributed across many latent features, making their removal challenging and com- putationally expensive. We introduce SAEMNESIA, a su- pervised sparse autoencoder framework that overcomes this by enforcing one-to-one concept-neuron mappings. By sys- tematically labeling concepts during training, our method achieves feature centralization, binding each concept to a single, interpretable neuron. This enables highly targeted and efficient concept erasure. SAEMNESIA reduces hyper- parameter search by 96.7% and achieves a 9.2% improve- ment over the state-of-the-art on the UnlearnCanvas bench- mark. Our method also demonstrates superior scalability in sequential unlearning, improving accuracy by 28.4% when removing nine objects, establishing a new standard for pre- cise and controllable concept erasure. Moreover, SAEM- NESIA mitigates the possibility of generating unwanted con- tent under adversarial attack and effectively removes nudity when evaluated with I2P. 1. Introduction Text-to-image diffusion models have achieved remarkable success in generating high-quality images from textual de- scriptions, with applications across diverse domains [37]. However, they can also produce harmful, inappropriate, or copyrighted content when given specific prompts, raising safety concerns. This has spurred growing interest in ma- chine unlearning, which aims to selectively remove unde- sired concepts from trained models while preserving their generative abilities [52]. A core challenge in concept unlearning is identifying where and how concepts are represented inside these mod- els. Each neuron can encode multiple unrelated concepts si- multaneously. This phenomenon is known as polysemantic- Concepts to erase with SAEmnesia ArchitecturesBearsSandwiches An image of Architectures in Picasso style An image of Bears in Blossom Season style An image of Sandwiches in Van Gogh style SD w/o SAE SAE activations with SAEmnesia An image of Architectures in Abstractionism style Diffusion Model each latent is active for a single concept each concept is mapped to a single latent monosemanticity feature centralization Figure 1. SAEMNESIA enables precise concept-level manipula- tion: each latent activates for a single concept (monosemanticity), and each concept is embedded in a single latent (feature central- ization). So, to erase a target concept, we only need to steer a single latent. The removed concepts correctly disappear in the di- agonal images (“Architectures”, “Bears”, “Sandwiches”) while the corresponding style is preserved. Note that they remain present in the non-diagonal ones, thereby preserving the fidelity and diver- sity when unlearning unrelated content. ity, making interpretability even more challenging. Mecha- nistic interpretability (MI) seeks to understand the internal workings of neural networks by analyzing their representa- tions. Sparse Autoencoders (SAEs) provide a particularly effective MI tool by decomposing model activations into sparse and interpretable concept-level features [5]. In this work, we adopt the notion of features as the fundamental units of neural network representations that cannot be fur- arXiv:2509.21379v2 [cs.CV] 28 Nov 2025 ther decomposed into simpler independent factors, as de- fined by Bereska and Gavves [4]. Neural networks can cap- ture natural abstractions [7] through their learned features, which serve as building blocks of their internal representa- tions, aiming to capture the concepts underlying the data. For simplicity, we use the terms “concepts” and “features” interchangeably, as well as “neurons” and “latents”. SAEs aim to learn monosemantic latents, meaning that they acti- vate almost exclusively for a specific concept. On the other hand, to improve models’ interpretability even further, indi- vidual concepts should not be spread across many latents, aiming for a one-to-one mapping. Yet, in practice, multiple latents often respond to the same concept, this phenomenon is also known as feature splitting [5]. This means modify- ing one concept requires changing multiple neurons. This distributed nature poses two key challenges: (i) exhaustive searches across many latent combinations to find the right subset to modify, making unlearning computationally ex- pensive; and (i) overlapping latents blur concept bound- aries, so interventions risk unintended side effects on related concepts. To overcome feature splitting, we introduce SAEMNE- SIA, that enriches the SAE training framework enforcing one-to-one mappings between concepts and latents through supervised labeling. Therefore, our method achieves fea- ture centralization, localizing each concept into a sin- gle latent and thereby preventing splitting across multiple neurons. This binding simplifies mechanistic unlearning, thanks to a precise single-latent intervention. Fig. 1 il- lustrates the effect of SAEMNESIA for concept removal in generative models, demonstrating the possibility of pre- cise control over what the model generates, while retaining overall quality and diversity. On the UnlearnCanvas bench- mark [52], SAEMNESIA achieves an 9.22% improvement over the state-of-the-art mechanistic approaches [9]. In se- quential unlearning tasks, we demonstrate superior scalabil- ity with a 28.4% improvement in unlearning accuracy for 9- object removal. Furthermore, at inference, this interpretable representation reduces hyperparameter search by 96.67%. In summary, our key contributions are as follows: (i) we introduce a supervised sparse autoencoder that explic- itly enforces one-to-one concept–latent mappings, eliminat- ing feature splitting and enabling transparent, interpretable control over concept representations; (i) we show that this structure makes concept erasure significantly more effi- cient: each concept can be removed by steering a single latent, substantially reducing inference-time hyperparame- ter search; (i) we achieve state-of-the-art performance on the UnlearnCanvas benchmark and demonstrate improved sequential unlearning, stronger robustness to adversarial at- tacks, and more effective NSFW-content suppression. 2. Related Work Machine unlearning in diffusion models. Machine un- learning was first introduced by Cao and Yang [6], who transformed neural network models through additional sim- ple layers into formats where output is a summation of inde- pendent features, allowing unlearning by blocking selected summation weights or nodes. However, recent works fo- cusing on unlearning for diffusion models typically employ fine-tuning approaches to unlearn specific concepts. ED- iff [48] formulates this problem as bi-level optimization, while ESD [14] leverages negative classifier-free guidance for concept removal. FMN [49] introduces a re-steering loss applied only to attention layers, and SalUn [11] and SHS [47] select parameters to adapt through saliency maps or connection sensitivity. SA [19] replaces unwanted data distribution with surrogate distributions, with an extension to selected anchor concepts in CA [25]. SPM [32] takes a different approach, using small linear adapters added af- ter each linear and convolutional layer to directly block un- wanted content propagation. Methods that do not rely on fine-tuning include SEOT [29], which removes unwanted content from text embeddings, and UCE [15], which adapts cross-attention weights using closed-form solutions.In contrast to these approaches, our work leverages SAEs to achieve unlearning through interpretable feature manipula- tion during inference, without modifying the base model weights and providing full transparency into which specific features are being targeted for removal. Beyond individual methods, recent surveys synthesize objectives, taxonomies, and evaluation protocols for gener- ative model unlearning, offering broader context for method design and assessment [8, 12]. Furthermore, recent analyses highlight instability and concept resurgence after unlearn- ing (such as revival under subsequent fine-tuning or adver- sarial prompting) reinforcing the need for interpretable and stable interventions [16, 30, 40]. SAEs background. Our aim is to enable effective con- cept unlearning in diffusion models by selectively remov- ing unwanted concepts while preserving generative quality. To achieve this, we decompose the high-dimensional, en- tangled activations from Stable Diffusion (SD) into sparse, interpretable feature directions that correspond to meaning- ful visual concepts. SAEs serve as the key tool for this decomposition, enabling us to map individual neurons to specific semantic concepts and subsequently intervene on them for targeted unlearning. A standard single-layer ReLU SAE [34] operates on d-dimensional activation vectors. Let x∈ R d denote the input activation vector andn be the latent dimension, typically set to d multiplied by a positive expan- sion factor. The encoder and decoder are defined as [5]: v = ReLU(W enc (x− b pre ) + b enc ) ˆ x = W dec v + b pre , (1) where v is the sparse hidden representation, ˆ x is the re- constructed input, W enc ∈ R n×d and W dec ∈ R d×n are the encoder and decoder weight matrices respectively, and b pre ∈ R d and b enc ∈ R n are learnable bias terms. TopK SAEs. In our work, we employ TopK SAEs [33] that provide enhanced sparsity control. The TopK activation function identifies the k largest pre-activations and sets all others to zero, ensuring sparsity while preserving the most significant features: z = TopK(v), ˆ x = W dec z + b pre . (2) Here, v represents the pre-TopK activations, while z repre- sents the post-TopK sparse activations. Training objective. Given a mini-batch of sizeB, the TopK SAE loss combines reconstruction error with an auxiliary loss to prevent dead latents: L unsupSAE = 1 B B X b=1 x (b) − ˆ x (b) 2 2 + αL aux ,(3) where ||x− ˆ x|| 2 2 is the reconstruction error and L aux is an auxiliary loss using only the largest k aux feature activations that have not fired on a large number of training samples (so-called dead latents). The auxiliary loss prevents dead latents from occurring and is scaled by coefficient α. Interpretability of vision models. SAEs have recently gained traction as a tool for uncovering human-interpretable structure in high-dimensional representations. Early appli- cations focused on discriminative models, such as inter- preting features in CLIP [10, 13, 36] or traditional clas- sifiers [17, 41]. More recent work has extended SAEs to vision–language settings, enabling tasks such as hallu- cination mitigation [22] and interpretable report genera- tion [1]. Across these domains, SAEs provide explicit con- cept–neuron mappings that move beyond post-hoc expla- nations and allow more precise control over learned repre- sentations. Unlike previous work on discriminative models, our approach uses SAEs with diffusion models to directly identify and control specific concepts. Within diffusion models, interpretability research has primarily examined how semantic information propagates through the architecture. Studies have identified meaning- ful directions in UNet bottlenecks [18, 21, 24, 26, 35], an- alyzed cross-attention to link prompts with spatial activa- tions [2, 3, 42], and even used intermediate text encoder states for generation [44]. Although these advances shed light on the internals of the model, they generally do not enable targeted interventions. Our approach yields sparse representations that support both interpretation and control- lable unlearning of specific concepts. SAE-Based unlearning. Recent work has explored apply- ing SAEs to diffusion models for concept manipulation. Kim and Ghadiyaram [23] introduced Concept Steerers, training SAEs on text embedding representations to identify concept-specific directions before cross-attention process- ing. While their approach achieves effective concept manip- ulation, working only on text encoders can lead to subopti- mal results, especially when facing adversarial attacks that exploit deeper model representations. Cywi ́ nski and Deja [9] introduced SAeUron, a post-cross-attention approach using SAEs trained on diffusion model activations in an un- supervised manner. While SAeUron achieves state-of-the- art performance on UnlearnCanvas, its unsupervised train- ing creates weak concept-latent associations. This means that concepts are still represented across multiple neurons. This distributed representation requires computationally ex- pensive feature threshold searches to identify which combi- nation of latent features must be modified to unlearn each concept. In contrast, we only need to steer a single feature per concept. 3. Methodology Unlike unsupervised SAE training methods [5] that require post-hoc discovery of concept-relevant features, our su- pervised approach directly enforces concept-latent assign- ments during training to achieve stronger one-to-one map- pings (see Fig. 2). Although our approach requires super- vision, the labels come at no additional cost, as they are directly derived from the text prompts used to generate or condition the images: i.e., the same concepts that the SAE aims to forget. SAEMNESIA for diffusion models. We apply SAEM- NESIA to diffusion models by training on activations ex- tracted from every timestep t of the denoising diffusion process. These activations are obtained from the cross- attention blocks of the diffusion model and form feature maps. Each feature map extracted at timestep t is a spa- tially structured tensor of shape F t ∈ R h×w×d , where h and w denote the height and width of the feature map, and d is the dimensionality of each feature vector. Each spatial position within the feature map corresponds to a patch in the input image. As a single SAE training sample, we consider an individual d-dimensional feature vector x, disregarding the information about its spatial position. Therefore, from each feature map, we obtain h× w training samples. Note that our method is architecture agnostic, meaning that it can be transferred to various text-to-image (T2I) models. Concept-latent assignment. Here we introduce supervised training that directly assigns concepts to specific latents. To determine which latent should be assigned to each concept during the supervised phase and to validate the quality of these assignments after training, we utilize the score func- tion [9], defined in Eq. (4), measuring feature-concept cor- respondence. Given a dataset of activations D = D c ∪D ¬c , whereD c contains data of the target conceptc andD ¬c does Prompts corpus Cats Abstractionism Trees SAE activations SAE activations  ❄ Diffusion Model SAE activations  ❄ Diffusion Model Score-based concept dictionary Architecture Dogs Blossom season Flame Unsupervised SAE Training Inference An image of Architectures in Abstractionism style Non-Architectures neurons stays unchanged Erase Architectures Negative multiplier ❄ Diffusion Model Supervised SAE ❄ With SAEmnesia Without erasure Figure 2. SAEMNESIA pipeline. Training comprises two phases: (i) establishing sparse representations via standard unsupervised SAE training, (i) applying supervised losses to strengthen specific concept-neuron associations. During inference, we need to steer a single latent per concept. not, the score function is defined as: score(i,t,c,D) = μ(i,t,D c ) P n j=1 μ(j,t,D c ) + δ − μ(i,t,D ¬c ) P n j=1 μ(j,t,D ¬c ) + δ (4) where δ prevents division by zero and μ(i,t,D) = 1 |D| P x∈D z i denotes the average activation of the i-th fea- ture on activations from timestep t (we omit t from z i for simplicity). Features with high scores exhibit strong ac- tivation for concept c while remaining weakly activated for other concepts. Formally, in order to achieve feature centralization, for a given concept c, the score function score(i,t,c,D) achieves a high value for a single latent in- dex i, while remaining low for all other indices j ̸= i: score(i,t,c,D)≫ score(j,t,c,D), ∀j ̸= i.(5) Given a setC =c 1 ,...,c K of concepts to unlearn, we define a mapping Φ : C → 1,...,n assigning each con- cept c to the latent index Φ(c) = i c with the highest score. For training samples containing multiple concepts, we as- sign multiple target indices corresponding to each present concept. By enforcing one-to-one concept-to-latent map- pings, the cardinality of the SAE’s hidden layer directly corresponds to the number of concepts that can be explicitly represented and manipulated. SAEMNESIA loss function. We employ a composite loss function designed to maintain SAE reconstruction capabili- ties while strengthening concept-latent associations: L SAEMNESIA =L unsupSAE + βL supSAE + λL L1 .(6) WhereL unsupSAE is the loss as defined in Eq. (3),L supSAE is the supervised loss that enforces the desired concept-latent relationships and L L1 is the sparsity regularization term. L supSAE consist of our Concept Assignment (CA) loss with a weighted additional Decorrelation (DC) loss: L supSAE =L CA + γL DC .(7) Concept assignment loss. Unlike traditional approaches that only apply reconstruction loss globally across all la- tents, our method strengthen concept-latent bonds that en- able one-to-one mappings. Our CA loss is computed ex- clusively for latents that are assigned to concepts present in each training sample, encouraging these specific latents to activate strongly. Given a training sample, we define the binary ground-truth vector y = [y 1 ,...,y K ] ⊤ ∈ 0, 1 K , where y k = 1 if concept c k is present in the sample, and 0 otherwise. We denote withT the set of concepts present in that sample. The CA loss measures how much the as- signed latents activate when their corresponding concepts are present: L CA = 1 B B X b=1 1 |T (b) | X c∈T (b) − logσ(v (b) i c ) ,(8) where v i c represents the pre-activation (logit), and σ(·) is the sigmoid function. Decorrelation constraint. To promote disentanglement across multiple macro-categories of concepts, we generalize the decorrelation constraint to M disjoint concept groups. Specifically, we partition the full concept set into non- overlapping subsets C = S M m=1 C m , with C m ∩C m ′ = ∅ for m ̸= m ′ , where eachC m represents a high-level group of related concepts (e.g., objects, styles, materials, or other semantic categories depending on the dataset). We denote by I C m = Φ(c) | c ∈ C m the set of latent indices as- signed to the concepts in groupC m . Given a mini-batch of size B, we define the activation vector of each concept c as a c = [v (1) i c ,v (2) i c ,...,v (B) i c ] ⊤ . The multi-group decorrela- tion constraint is then formulated as: L DC = X m<m ′ X i∈I C m X j∈I C m ′ ρ(a i , a j ) X m<m ′ |I C m ||I C m ′ | .(9) Here, ρ(a i , a j ) denotes the Pearson correlation coefficient between activation vectors a i and a j . This constraint pe- nalizes correlations between activation patterns of latents assigned to different concept groups. Sparsity regularization. To encourage sparse activations in the latent representation, we incorporate an L1 regular- ization term that penalizes the magnitude of latent activa- tions v. This sparsity constraint promotes the emergence of interpretable features by encouraging most latents to re- main inactive for any given input, thereby improving the disentanglement of learned representations. Feature centralization. As defined in Eq. (5), feature cen- tralization occurs when only one feature achieves a high score for a specific concept. However, computing the score function requires the entire dataset, making it impractical during training. Instead, we use L supSAE as a proxy to achieve feature centralization. L CA directly enforces that assigned latents activate strongly for their target concepts, while L L1 maintains sparsity across all latents. This com- bination concentrates a concept’s information in the desig- nated latent. Inference and concept unlearning. To unlearn a concept c, SAEMNESIA only needs the single latent i c to erase the concept. The activation of the selected feature is multi- plied by a negative value γ c < 0 normalized by the average activation μ(i c ,t,D c ) on concept samples of a validation dataset D. This removes the influence of the targeted con- cept on the activation vector z. Each i c -th latent feature activation is modified as follows: z i c = ( γ c μ(i c ,t,D c )z i c , if z i c > μ(i c ,t,D) z i c ,otherwise (10) The condition z i c > μ(i c ,t,D) prevents random feature ablation when scores are low. During inference, we can use the original pretrained model for the first t steps, setting the multipliers to 1 and retaining the pretrained model priors. We can then turn on SAEMNESIA for the remaining steps. 4. Experiments and Results We conduct comprehensive experiments to evaluate SAEMNESIA across multiple dimensions: unlearning ef- fectiveness, generation quality, concept separation, perfor- mance robustness, incremental unlearning capabilities, ad- versarial resilience and nudity removal. Note that our focus in this work is on object erasure, as it presents a consider- ably greater challenge than style erasure [9]. 4.1. Experimental setup We report here the evaluation setup for the experiments of our proposed method, including datasets, architectures and evaluation metrics. Dataset. We extract activations from SD v1.5 [37] within the UnlearnCanvas Benchmark [52]. SD v1.x family re- mains the standard evaluation setting for concept-erasure research: the benchmark itself is built on SD v1.5, and most contemporary methods report results on SD v1.x [9, 27, 39, 43, 45, 46]. This makes SD v1.x the appropriate and widely-accepted testbed for fair comparison and repro- ducibility. We construct labeled training data by generat- ing activations from structured prompts with known concept compositions. For each object classc (e.g., “Bears”, “Cats”) and style s (e.g., “Impressionism”, “Cubism”), we gener- ate 80 prompts of the form An image of object in style and collect feature maps from selected U- Net cross-attention blocks across all 50 denoising timesteps during text-conditioned generation. Each activation is di- rectly labeled with its corresponding object and style based on the prompt structure, creating explicit concept-activation pairs. The employed objects and styles are taken from the UnlearnCanvas benchmark, which consists of 20 different objects and 50 different styles. We focus our analysis on block up.1.1 for object-related features, as this block has been empirically demonstrated to specialize in generating specific visual aspects [2]. This controlled labeling strat- egy provides clean supervision signals that enable direct concept-neuron mapping during SAE training, contrasting with unsupervised approaches that must discover concept representations through post-hoc analysis. For evaluation, we employed the same setting of SAeUron [9]. SAE model. Our best-performing method starts with a pre-trained unsupervised SAE and fine-tunes it using SAEMNESIA loss in Eq. (6).Unless otherwise stated, SAEMNESIA is applied across all denoising steps for a fair comparison with [9]. For the decorrelation constraint in Eq. (9), we chooseC obj andC sty , corresponding to object and style concepts, to enforce concept separation. Loss function hyperparameters and additional model setups can be found in the Appendix 6.5, 6.6, 6.7 and 6.10. Evaluation metrics.Our primary evaluation uses the UnlearnCanvas benchmark [52] with Vision Transformer- based classifiers to measure three key metrics: Unlearn- ing Accuracy (UA), which quantifies the proportion of samples from target concept prompts that are misclassi- fied (i.e., successful unlearning); In-domain Retain Accu- racy (IRA), measuring classification accuracy on retained concepts within the same domain; and Cross-domain Re- tain Accuracy (CRA), assessing accuracy on concepts from different domains such as object accuracy during style unlearning. To evaluate generation quality, we compute Fr ́ echet Inception Distance (FID) scores across different un- learning configurations and multiplier values, quantifying both the quality and diversity of generated images. 4.2. Quantitative results UnlearnCanvas benchmark performance.Tab. 1 presents our performance compared to the SAeUron base- line. An inconsistency in the computation of the metrics was discovered in the public implementation of SAeUron. We therefore recomputed the metrics ourselves using the corrected procedure. SAEMNESIA achieves 91.51% aver- age score with hyperparameter search, improving over the baseline’s 82.29%. Standard deviations are reported in Ap- pendix 6.1. The performance gains stem from the stronger concept-latent associations that enable more targeted inter- ventions. Therefore, SAEMNESIA is particularly effective for maintaining concept separation. The performance gains stem from the stronger concept-latent associations that en- able more targeted interventions. Table 1. Evaluation metrics (%) of SAEMNESIA against state- of-the-art methods on object concept unlearning using the Un- learnCanvas benchmark. The best result for each metric is highlighted in bold. SAEMNESIA achieves superior performance across all evaluation metrics with 91.51% average score, repre- senting an 9.22% improvement over the previous state-of-the-art. MethodUAIRACRAAvg. ESD [14]92.1555.7844.2364.05 FMN [49]45.6490.6373.4669.91 UCE [15]94.3139.3534.6756.11 CA [25]46.6790.1181.9772.92 SEOT [29]23.2595.5782.7167.18 SPM [32]71.2590.7981.6581.23 EDiff [48]86.6794.0348.4876.39 SHS [47]80.7381.1567.9976.62 SAeUron [9]87.1685.5774.1482.29 SAEMNESIA94.6591.3988.4891.51 We also conduct specialized concept separation analy- sis by examining latent overlap between different concepts, with particular focus on the challenging “Dogs vs. Cats” classification case, a known limitation of current SAE- based unlearning approaches [9] that leads to concept in- terference. Results are presented in Appendix 6.8. Computational efficiency through interpretable repre- sentations. The one-to-one concept-neuron mappings di- rectly eliminate the feature combinations search complex- ity that affects existing methods. The methodology with unsupervised training requires exploring different num- bers of top latent features for unlearning, creating a two- dimensional search space with m× l computations, where m = 7 possible multiplier values and l = 30 number of possible latent combinations required to unlearn a concept. This results in precisely 210 evaluations. In contrast, our ap- proach requires only m computations since we only search multiplier values, as each concept maps to one neuron. This means exactly 7 evaluations are needed. This represents a 96.67% reduction in computational cost, achieved through interpretable concept localization. Effect of uniform multipliers. Fig. 3 compares the effect of applying uniform multipliers (so the same multiplier for all objects) to the latent steering for SAeUron and SAEM- NESIA. Across all evaluation metrics, SAEMNESIA con- sistently achieves higher and more stable scores over the full range of multipliers. This demonstrates that SAEM- NESIA is less sensitive to the exact steering strength across different objects and provides more robust control of con- cept removal. Additional quantitative plots are included in the Appendix 6.9. -30-25-20-15-10-5-10.0 Multiplier 0 20 40 60 80 100 UA -30-25-20-15-10-5-10.0 Multiplier IRA -30-25-20-15-10-5-10.0 Multiplier CRA -30-25-20-15-10-5-10.0 Multiplier Mean (UA, IRA, CRA) SAeUronSAEmnesia Figure 3. Effect of uniform multipliers on unlearning perfor- mance for SAeUron and SAEMNESIA. SAEMNESIA maintains higher and more stable performance across all multipliers com- pared to SAeUron, indicating greater robustness to the steering strength. Concept-latent association distribution. Our core contri- bution is creating interpretable one-to-one concept-neuron mappings. Fig. 4 is an example of the effectiveness of our supervised training in promoting feature centralization by comparing feature importance score distributions before and after training for the “Flowers” concept. The original SAeUron model exhibits relatively uniform, low-magnitude scores across the entire latent space (max score: 0.0166), in- dicating distributed concept representation. Our supervised SAEMNESIA model produces a clear dominant peak at neu- ron 11979 with a maximum score of 0.0404, 2.43 times higher than the baseline. This concentrated activation pat- tern confirms that supervised training successfully enforces strong one-to-one concept-latent relationships, with the as- signed neuron becoming highly specialized for the target concept while other neurons remain largely inactive. This transformation from distributed to concentrated representa- tions is the foundation that enables all subsequent improve- ments in computational efficiency and unlearning perfor- mance. The higher score values are also tightly linked to the lower absolute values of the optimal unlearning mul- tipliers (a deeper analysis can be found in Appendix 6.2). Additional examples of score distributions are reported in the Appendix 6.3. Feature centralization validation via K-N classifica- tion. To quantitatively validate that SAEMNESIA achieves feature centralization, where each concept’s information is concentrated in a single latent, we conducted k-nearest neighbors (k-N) classification experiments on the latent representations across all 20 object concepts in the Unlearn- Canvas benchmark. For each object, we compared classi- fication accuracy across the denoising process using four strategies: (1) the top-scoring latent identified by SAEM- 05000100001500020000 Neuron Index 0.00 0.01 0.02 0.03 0.04 Scores SAEmnesia - Max: 0.0404 SAeUron - Max: 0.0166 Figure 4. Feature importance score distributions for “Flowers” concept. SAeUron shows dispersed, low-magnitude scores across all neurons with a maximum of 0.0166. SAEMNESIA shows a clear dominant peak at neuron 11979 with maximum score of 0.0404 (2.43× improvement). 020406080100 Timestep 0 5 20 30 40 50 60 70 80 90 100 Accuracy [%] Top-1 single feature All features Random features Random guess Figure 5. K-N classification across denoising timesteps, aver- aged over 20 object concepts. Using only the top-scoring latent identified by SAEMNESIA, performances are similiar to using all features, demonstrating that supervised training successfully con- centrates concept-relevant information into single interpretable la- tents across diverse object categories. NESIA (Eq. (4)), (2) all latent features, (3) randomly se- lected features, and (4) random guess baseline. As shown in Fig. 5, the score-based selection achieves nearly identical classification accuracy to using all available latents through- out most of the denoising timesteps. 4.3. Qualitative results Concepts removal. Figs. 1 and 6 show qualitative ex- amples of SAEMNESIA applied to a representative subset of object–style combinations. In this setting, we use con- cept multipliers of γ c = −1 and restrict the application of SAEMNESIA to the final 25 denoising steps to reduce arti- facts. Thanks to the one–to–one mapping between concepts and latent units, steering the sparse autoencoder selectively removes the targeted concept: the removed concepts vanish in the diagonal images, while the corresponding style is pre- served. When unlearning unrelated content, the objects in the prompts remain present in the non-diagonal ones. Ad- ditional qualitative examples covering a wider range of ob- jects and styles are provided in Appendix 6.4. Concepts to erase with SAEmnesia BirdsFrogsHumansFlowersWaterfalls An image of Birds in Blossom Season style An image of Frogs in Byzantine style An image of Flowers in Comic Etch style An image of Human in Dadaism style An image of Waterfalls in Gorgeous Love style Figure 6.Qualitative examples of concept removal with SAEMNESIA. Each row shows a different style, and each col- umn a different object concept. SAEMNESIA effect on discovered concepts. We examine the activation of the concept selected feature on correspond- ing image patches across two objects, “Architectures” and “Rabbits”. Fig. 7 shows the difference between unsuper- vised SAE training and our SAEMNESIA approach. In the unsupervised setting (first rows), we can see that unsuper- vised SAEs decompose concepts into multiple distributed features of comparable importance. In contrast, SAEMNE- SIA (second rows) produces a markedly different activation structure. This concentrated representation, where object- level semantics are encoded in a single, highly interpretable latent, facilitates more precise concept manipulation and en- ables effective machine unlearning through targeted inter- ventions. 4.4. Additional experiments Sequential unlearning scalability. To demonstrate how interpretable representations enable scalability, we test se- quential unlearning (where unlearning requests arrive se- quentially) across 9 objects: “Bears”, “Cats”, “Flowers”, “Frogs”, “Jellyfish”, “Sea”, “Statues”, “Sandwiches” and “Waterfalls”. We selected these objects because the Un- learnCanvas benchmark focuses only on the sequential un- learning of styles. In this setting, we apply the unlearn- ing multipliers cumulatively: first γ Bears alone, then γ Bears and γ Cats , and so on. Fig. 8 shows that SAEMNESIA sig- nificantly outperforms the baseline. SAEMNESIA achieves t=47t=30t=10t=1 Unsupervised SAE SAEmnesia Unsupervised SAE SAEmnesia 1.0 0.8 0.6 0.4 0.2 0.0 Figure 7. SAEMNESIA shifts attention toward patches most responsible for the target concept. Visualization of the most im- portant patches for the objects “Architectures” (top) and “Rabbits” (bottom) across timesteps. 123456789 Number of Concepts 60 80 100 UA 123456789 Number of Concepts 40 50 60 RA SAeUronSAEmnesia Figure 8. Evaluation of SAEMNESIA against SAeUron baseline on sequential object unlearning tasks. SAEMNESIA achieves higher UA and RA. 92.4% accuracy for 9-object removal compared to the base- line’s 64%.This scalability results from having inter- pretable, specialized neurons rather than distributed repre- sentations that interfere with each other. SAEMNESIA also achieves higher retention accuracy (RA, average of IRA and CRA), reflecting the model’s ability to preserve all non- removed concepts. When removing all objects, SAEMNE- SIA attains RA of 60.9%, while the baseline 48.4%. Adversarial robustness.We follow Zhang et al. [51] and evaluate against UnlearnDiffAtk attacks, optimizing 5- token adversarial prefixes for 40 iterations with learning rate 0.01 to provide fair comparison with baseline SAE methods [9].The concentrated concept representations also improve adversarial robustness, as shown in Fig. 9. SAEMNESIA demonstrates improved resilience, dropping from 73.30% to 28.30% unlearning accuracy. This con- trasts sharply with the state-of-the-art significant drop from 26.19% to 2.22%. The supervised concept-neuron mapping approach in SAEMNESIA shows substantially improved ro- bustness against adversarial perturbations, with the stronger ESD FMN UCE CA SalUn SEOT SPM EDiff SHS SAeUron SAEmnesia 0 50 100 Unlearning Accuracy (%) Before UnlearnDiffAtkAfter UnlearnDiffAtk Figure 9. Unlearning accuracy before and after UnlearnDif- fAtk adversarial attacks. SAEMNESIA maintains higher perfor- mance under adversarial attack compared to the SAeUron base- line. concept-neuron bonds making it significantly harder for at- tacks to disrupt the specialized representations. Nudity unlearning. To highlight the potential of SAEM- NESIA in real-world applications such as NSFW content re- moval, we evaluate our method on the I2P benchmark [38] on SD v1.4.We follow the same experimental setting as Cywi ́ nski and Deja [9]. We train SAEs on SD-v1.4 acti- vations gathered from a random 30K captions from COCO train 2014. We employ the NudeNet detector for nudity detection, filtering out outputs with confidence less than 0.6. Our best model, which steers only the top-2 concept neurons, achieves state-of-the-art results (9 detections vs. SAeUron’s 18), while preserving the model’s overall qual- ity. We adopt two neurons because the SAE is trained on only two nudity-related prompts (“naked man” and “naked woman”). Complete per-category results and comparisons against all baselines are provided in Appendix 6.11. 5. Conclusions We introduced SAEMNESIA, a supervised sparse au- toencoder framework for concept unlearning in diffusion models.By preventing feature splitting and enforcing concept-aligned latent structure, SAEMNESIA produces more reliable concept–latent mappings and stronger erasure behavior than unsupervised SAE baselines. On UnlearnCanvas, it attains 91.51% average performance while preserving competitive generation quality.The method also reduces hyperparameter search by 96.67%, substantially lowering tuning cost. In sequential unlearn- ing, SAEMNESIA scales more effectively, delivering a 28.4% improvement when removing nine objects. We also demonstrate the practical relevance of our approach on nu- dity removal, where a two-neuron variant of SAEMNESIA achieves state-of-the-art NSFW content suppression on I2P while retaining overall image fidelity. Looking forward, as richer benchmarks and regulatory requirements emerge, we see SAEMNESIA as a step toward principled, interpretable, and controllable concept unlearning in generative models, with capabilities increasingly critical for trustworthy and safe deployment. AcknowledgmentsWeacknowledgetheCINECA award under the ISCRA initiative, for the availability of high performance computing resources and support. References [1] Ahmed Abdulaal, Hugo Fry, Nina Monta ̃ na-Brown, Ayodeji Ijishakin, Jack Gao, Stephanie Hyland, Daniel C Alexander, and Daniel C Castro. An x-ray is worth 15 features: Sparse autoencoders for interpretable radiology report generation. arXiv preprint arXiv:2410.03334, 2024. 3 [2] Samyadeep Basu, Nanxuan Zhao, Vlad I Morariu, Soheil Feizi, and Varun Manjunatha. Localizing and editing knowl- edge in text-to-image generative models. In The Twelfth In- ternational Conference on Learning Representations, 2023. 3, 5 [3] Samyadeep Basu, Keivan Rezaei, Priyatham Kattakinda, Vlad Morariu, Nanxuan Zhao, R A Rossi, Varun Manju- natha, and Soheil Feizi. On mechanistic knowledge local- ization in text-to-image generative models. In ICML, 2024. 3 [4] Leonard Bereska and Stratis Gavves.Mechanistic inter- pretability for AI safety - a review. Transactions on Machine Learning Research, 2024. Survey Certification, Expert Cer- tification. 2 [5] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yi- fan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Ka- rina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language mod- els with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic- features/index.html. 1, 2, 3 [6] Yinzhi Cao and Junfeng Yang. Towards making systems for- get with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015. 2 [7] Lawrence Chan, Leon Lang, and Erik Jenner. Natural ab- stractions: Key claims, theorems, and critiques, 2023. AI Alignment Forum. 2 [8] Aobo Chen, Yangyi Li, Chenxu Zhao, and Mengdi Huai. A survey of security and privacy issues of machine unlearning. AI Magazine, 46, 2025. 2 [9] Bartosz Cywi ́ nski and Kamil Deja. Saeuron: Interpretable concept unlearning in diffusion models with sparse autoen- coders. arXiv preprint arXiv:2501.18052, 2025. 2, 3, 5, 6, 8, 12, 13, 15, 19 [10] Gytis Daujotas.Case study:Interpreting,ma- nipulating,andcontrollingclipwithsparseau- toencoders,2024.URLhttps://w.lesswrong. com/posts/iYFuZo9BMvr6GgMs5/case-study-interpreting- manipulating-and-controlling-clip. Accessed, pages 09–24, 2024. 3 [11] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Den- nis Wei, and Sijia Liu. Salun: Empowering machine unlearn- ing via gradient-based weight saliency in both image classi- fication and generation. arXiv preprint arXiv:2310.12508, 2023. 2 [12] Xiaohua Feng, Jiaming Zhang, Fengyuan Yu, Chengye Wang, Li Zhang, Kaixiang Li, Yuyuan Li, Chaochao Chen, and Jianwei Yin. A survey on generative model unlearning: Fundamentals, taxonomy, evaluation, and future direction. arXiv preprint arXiv:2507.19894, 2025. 2 [13] Hugo Fry. Towards multimodal interpretability: Learning sparse interpretable features in vision transformers, 2024. 3 [14] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto- Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 2426–2436, 2023. 2, 6, 19 [15] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzy ́ nska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pages 5111–5120, 2024. 2, 6, 19 [16] Naveen George, Karthik Nandan Dasaraju, Rutheesh Reddy Chittepu, and Konda Reddy Mopuri. The illusion of unlearn- ing: The unstable nature of machine unlearning in text-to- image diffusion models. In CVPR, 2025. 2 [17] Liv Gorton. The missing curve detectors of inceptionv1: Ap- plying sparse autoencoders to inceptionv1 early vision. arXiv preprint arXiv:2406.03662, 2024. 3 [18] Jaehoon Hahm, Junho Lee, Sunghyun Kim, and Joonseok Lee. Isometric representation learning for disentangled latent space of diffusion models. arXiv preprint arXiv:2407.11451, 2024. 3 [19] Alvin Heng and Harold Soh. Selective amnesia: A contin- ual learning approach to forgetting in deep generative mod- els. Advances in Neural Information Processing Systems, 36: 17170–17194, 2023. 2 [20] Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung- Hsuan Lai, Fu-En Yang, and Yu-Chiang Frank Wang. Re- celer: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. In European Conference on Computer Vision, pages 360–376. Springer, 2024. 19 [21] Ayodeji Ijishakin, Ming Liang Ang, Levente Baljer, Daniel Chee Hian Tan, Hugo Laurence Fry, Ahmed Abdulaal, Aen- gus Lynch, and James H Cole. H-space sparse autoencoders. In Neurips Safe Generative AI Workshop 2024, 2024. 3 [22] Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language rep- resentations to mitigate hallucinations.arXiv preprint arXiv:2410.02762, 2024. 3 [23] Dahye Kim and Deepti Ghadiyaram.Concept steerers: Leveraging k-sparse autoencoders for controllable genera- tions. arXiv preprint arXiv:2501.19066, 2025. 3 [24] Dahye Kim, Xavier Thomas, and Deepti Ghadiyaram. Reve- lio: Interpreting and leveraging semantic information in dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 4659– 4669, 2025. 3 [25] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating con- cepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 22691–22702, 2023. 2, 6, 19 [26] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.10960, 2022. 3 [27] Byung Hyun Lee, Sungjin Lim, and Se Young Chun. Local- ized concept erasure for text-to-image diffusion models us- ing training-free gated low-rank adaptation. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 18596–18606, 2025. 5 [28] Byung Hyun Lee, Sungjin Lim, Seunggyu Lee, Dong Un Kang, and Se Young Chun. Concept pinpoint eraser for text- to-image diffusion models via residual attention gate. arXiv preprint arXiv:2506.22806, 2025. 19 [29] Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shah- baz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Get what you want, not what you don’t: Image content sup- pression for text-to-image diffusion models. arXiv preprint arXiv:2402.05375, 2024. 2, 6 [30] Kevin Lu, Nicky Kriplani, Rohit Gandikota, Minh Pham, David Bau, Chinmay Hegde, and Niv Cohen. When are concepts erased from diffusion models?arXiv preprint arXiv:2505.17013, 2025. 2 [31] Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong.Mace: Mass concept erasure in diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430– 6440, 2024. 19 [32] Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7559–7568, 2024. 2, 6 [33] Alireza Makhzani and Brendan J. Frey. k-sparse autoen- coders. CoRR, abs/1312.5663, 2013. 3 [34] Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vi- sion Research, 37:3311–3325, 1997. 2 [35] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 36: 24129–24142, 2023. 3 [36] Sukrut Rao, Sweta Mahajan, Moritz B ̈ ohle, and Bernt Schiele. Discover-then-name: Task-agnostic concept bottle- necks via automated concept discovery. In European Con- ference on Computer Vision, pages 444–461. Springer, 2024. 3 [37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ̈ orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 5 [38] Patrick Schramowski, Manuel Brack, Bj ̈ orn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappro- priate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522–22531, 2023. 8, 19 [39] Koushik Srivatsan, Fahad Shamshad, Muzammal Naseer, Vishal M. Patel, and Karthik Nandakumar. Stereo: A two- stage framework for adversarially robust concept erasing from text-to-image diffusion models.In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 23765–23774, 2025. 5 [40] Vinith M. Suriyakumar, Rohan Alur, Ayush Sekhari, Manish Raghavan, and Ashia C. Wilson. Unstable unlearning: The hidden risk of concept resurgence in diffusion models. arXiv preprint arXiv:2410.08074, 2024. 2 [41] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015. 3 [42] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffu- sion using cross attention. arXiv preprint arXiv:2210.04885, 2022. 3 [43] Kartik Thakral, Tamar Glaser, Tal Hassner, Mayank Vatsa, and Richa Singh.Fine-grained erasure in text-to-image diffusion-based foundation models.In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 9121–9130, 2025. 5 [44] Michael Toker, Hadas Orgad, Mor Ventura, Dana Arad, and Yonatan Belinkov. Diffusion lens: Interpreting text encoders in text-to-image pipelines. arXiv preprint arXiv:2403.05846, 2024. 3 [45] Yuan Wang, Ouxiang Li, Tingting Mu, Yanbin Hao, Kuien Liu, Xiang Wang, and Xiangnan He. Precise, fast, and low- cost concept erasure in value space: Orthogonal complement matters. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 28759–28768, 2025. 5 [46] Zihao Wang, Yuxiang Wei, Fan Li, Renjing Pei, Hang Xu, and Wangmeng Zuo.Ace: Anti-editing concept erasure in text-to-image models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 23505–23515, 2025. 5 [47] Jing Wu and Mehrtash Harandi. Scissorhands: Scrub data in- fluence via connection sensitivity in networks. In European Conference on Computer Vision, pages 367–384. Springer, 2024. 2, 6 [48] Jing Wu, Trung Le, Munawar Hayat, and Mehrtash Harandi. Erasediff: Erasing data influence in diffusion models. arXiv preprint arXiv:2401.05779, 2024. 2, 6 [49] Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi.Forget-me-not:Learning to for- get in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1755–1764, 2024. 2, 6, 19 [50] Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. Advances in neu- ral information processing systems, 37:36748–36776, 2024. 19 [51] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yi- hua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To gener- ate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now. In European Con- ference on Computer Vision, pages 385–403. Springer, 2024. 8 [52] Yihua Zhang, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Xiaoming Liu, and Sijia Liu. Unlearncan- vas: A stylized image dataset to benchmark machine un- learning for diffusion models. arXiv e-prints, pages arXiv– 2402, 2024. 1, 2, 5 6. Appendix 6.1. Standard deviations We report here the standard deviations of UA, IRA and CRA metrics across 4 different seeds. Table 2. Mean and standard deviations (%) of SAEMNESIA on the UnlearnCanvas benchmark. MethodUAIRACRAAvg. SAEMNESIA94.65± 2.691.39± 1.388.48± 0.591.51± 0.4 6.2. Multipliers Comparison The comparison reveals significant differences in unlearn- ing strategies between other SAE-based methods [9] and our SAEMNESIA variants.SAeUron employs substan- tially larger multiplier magnitudes (as per Tab. 3), averag- ing -21.25, while our SAEMNESIA-OS-CA-FS and SAEM- NESIA-OS-DC-CA-FT models achieve effective unlearning with more conservative multipliers, averaging -6.20 and - 6.60 respectively. These results align well with the findings on latents score distributions shown in Figs. 4 and 10 to 12. 6.3. Additional Scores Histograms In this section are provided additional examples similar to Fig. 4, to further demonstrate the robustness of our meth- ods throughout different concepts, such as Architectures Fig. 10, Rabbits Fig. 11 and Sea Fig. 12. For all the objects, we show a clear improvement, going from distributed scores (in orange) produced by SAeUron [9], to well defined peaks (in green) produced by our method SAEMNESIA. 05000100001500020000 Neuron Index 0.00 0.01 0.02 Scores SAEmnesia - Max: 0.0229 SAeUron - Max: 0.0054 Figure 10. Feature importance score distributions for Architec- tures concept. SAeUron shows dispersed concept scores, while SAEMNESIA shows a clear dominant peak. 6.4. Additional unlearning visualization Figs. 13 and 14 provides additional visualization with ran- domly sampled styles when unlearning with SAEMNESIA. We can see that SAEMNESIA performs well in most cases; however, it struggles in some scenarios with more challeng- ing objects that are more blended with the styles, for exam- ple, flames and jellyfishes. 05000100001500020000 Neuron Index 0.00 0.01 0.02 0.03 Scores SAEmnesia - Max: 0.0283 SAeUron - Max: 0.0206 Figure 11. Feature importance score distributions for Rabbits con- cept. SAeUron shows dispersed concept scores, while SAEMNE- SIA shows a clear dominant peak. 05000100001500020000 Neuron Index 0.00 0.01 0.02 0.03 Scores SAEmnesia - Max: 0.0339 SAeUron - Max: 0.0117 Figure 12. Feature importance score distributions for Sea con- cept. SAeUron shows dispersed concept scores, while SAEMNE- SIA shows a clear dominant peak. Figs. 15 and 16 show the unlearning accuracies for each object in the columns when unlearning the objects in the rows. This visualization highlights entanglements between different objects that can lead to poor unlearning perfor- mances. 6.5. Global Cross-Entropy We employ Cross-entropy (CE) loss with softmax applied to all pre-TopK latent activations as an alternative to the Concept-Assignment Loss (Eq. (8)). This way, we treat concept assignment as a classification problem across the entire latent space: L global-ce =− 1 B B X i=1 log exp(v i c i ) P n j=1 exp(v i j ) ! ,(11) where B is the batch size, v i j represents the pre-TopK acti- vation value for sample i at latent position j, c i denotes the assigned concept latent index for sample i, and n is the total number of latents. This approach, however, led to suboptimal performance as reported in Tabs. 5 and 6 w.r.t. O-GCE-FS and O- GCE-FT. Table 3. Comparison of Object Unlearning Multipliers Across Three Models ObjectSAeUronS-OS-CA-FSS-OS-DC-CA-FT Architectures-20.0-5.0-5.0 Bears-30.0-10.0-5.0 Birds-10.0-5.0-5.0 Butterfly-15.0-5.0-5.0 Cats-15.0-1.0-10.0 Dogs-20.0-5.0-5.0 Fishes-30.0-5.0-5.0 Flame-25.0-1.0-1.0 Flowers-20.0-5.0-5.0 Frogs-5.0-5.0-10.0 Horses-25.0-10.0-15.0 Human-20.0-5.0-5.0 Jellyfish-15.0-1.0-1.0 Rabbits-30.0-10.0-5.0 Sandwiches-15.0-25.0-5.0 Sea-30.0-5.0-5.0 Statues-30.0-1.0-10.0 Towers-20.0-5.0-5.0 Trees-25.0-5.0-5.0 Waterfalls-30.0-10.0-20.0 Average-21.25± 7.45-6.20± 5.31-6.60± 4.48 6.6. Post-TopK Loss Analysis OS-TK-CA-FT is the only model variant that applies CA loss after the Top-K, yet demonstrates competitive perfor- mance without major degradation. In fine-tuned configu- rations with hyperparameter search (Tab. 5), OS-TK-CA- FT achieves 85.52% average performance compared to the baseline’s 82.29%, representing a meaningful 3.2% im- provement. Similarly, in from-scratch training (Tab. 6), OS- TK-CA-FT maintains 86.93% performance, indicating that post Top-K supervision remains viable. 6.7. Exploratory Variants In Tab. 4, all the variants tested for this work are pre- sented. SAEmnesia-OS-DC-CA-FT is the version reported as SAEMNESIA in the main paper. 6.8. Concept Interference Mitigation The Dogs vs. Cats overlap analysis (Tabs. 7 and 8) provides crucial insights into concept interference patterns, a known limitation of the current approaches [9]. From-Scratch Models generally show better concept separation, with O-GCE-FS achieving zero overlapping timesteps. Fine-Tuned Models show more variable performance, with some variants (O-GCE-FT, OS-TK-CA-FT) achiev- ing zero overlap while others (OS-CA-FT) perform worse than the baseline. 6.9. Uniform Multipliers Analysis The uniform multiplier sweep analysis (Figs. 17 and 18) reveals varying performance characteristics across model versions and evaluation metrics. In the fine-tuned mod- els (Fig. 17), SAEMNESIA variants show competitive per- formance with some variations across different multiplier ranges, with certain variants like S-OS-DC-CA-FT main- taining strong performance while others such as S-OS-TK- CA-FT exhibit sensitivity to specific multiplier settings. The from-scratch models (Fig. 18) demonstrate different behavior patterns, with variants like S-O-CA-FS show- ing particular sensitivity to moderate multiplier values be- fore recovering at gentler settings. Both figures indicate that SAEMNESIA variants achieve reasonable performance across various hyperparameter ranges. The results suggest that different SAEMNESIA configurations may be better suited for different multiplier ranges, highlighting the im- portance of hyperparameter selection in optimizing unlearn- ing performance. 6.10. Lambda Coefficient Analysis for SAEMNESIA Loss The choice of value for β in Eq. (6) as the coefficient for theL supSAE loss critically affects model performance. Our best experimental results were achieved with β = 3. When β = 10, the model produces completely white images and loses generative capabilities. In our experimental setup, λ Architectures Color Fantasy ArchitecturesBearsBirdsButterflyCatsDogsFishesFlameFlowersFrogsHorsesHumanJellyfishRabbitsSandwichesSeaStatuesTowersTreesWaterfalls Bears Artist Sketch Birds Red Blue Ink Butterfly Joy Cats Van Gogh Dogs Comic Etch Fishes On Fire Flame Structuralism Flowers Warm Smear Frogs Picasso Horses Abstractionism Human Bricks Jellyfish Warm Love Rabbits Mosaic Sandwiches Pastel Sea Pencil Drawing Statues Vibrant Flow Towers Watercolor Trees Meteor Shower Waterfalls Rust Figure 13. Qualitative examples of concept removal with SAEMNESIA. Each row shows a different randomly sampled style, and each column a different object concept. Table 4. SAE Experiment Versions. Feature/MetricOO-GCE-FSOO-GCE-FTOS-DC-CA-FSOS-DC-CA-FTOO-CA-FSOO-CA-FTOS-CA-FSOS-CA-FTOS-TK-CA-FSOS-TK-CA-FT Object labels✓ Styles labels×✓×✓ Decorrelation×✓× Global CE✓× CA before Top-K×✓× CA after Top-K×✓ Finetuned×✓×✓×✓×✓×✓ Architectures Cold Warm ArchitecturesBearsBirdsButterflyCatsDogsFishesFlameFlowersFrogsHorsesHumanJellyfishRabbitsSandwichesSeaStatuesTowersTreesWaterfalls Bears Meta Physics Birds Blossom Season Butterfly On Fire Cats Sponge Dabbed Dogs Pencil Drawing Fishes Red Blue Ink Flame Bricks Flowers Fauvism Frogs Cubism Horses Cartoon Human Neon Lines Jellyfish Vibrant Flow Rabbits Byzantine Sandwiches Superstring Sea Impressionism Statues Gorgeous Love Towers Abstractionism Trees Rust Waterfalls Winter Figure 14. Qualitative examples of concept removal with SAEMNESIA. Each row shows a different randomly sampled style, and each column a different object concept. from Eq. (6) was set to λ = 0.01, and γ from Eq. (9) was set to γ = 0.1. 6.11. Nudity Unlearning For nudity unlearning, we follow the same experimental set- ting as Cywi ́ nski and Deja [9] as described in the main pa- per. Our base unlearning setup uses only the top scoring la- tent to erase an object. As per Tab. 9, for nudity unlearning, SAEMNESIA achieves weak performance (47 detections vs. SAeUron’s 18 detections). However, when we instead steer two latents in our variant SAEMNESIA-TOP2, performance substantially improves (9 detections). The highly imbal- anced distribution of nudity-related content in the training dataset may lead to a weaker concept centralization. Table 5. SAEMNESIA Experimental Results - Fine Tuned Models - Searched HP MetricSAeUronOO-GCE-FTOS-DC-CA-FTOO-CA-FTOS-CA-FTOS-TK-CA-FT UA (%)↑87.1694.5591.7595.7590.9066.85 IRA (%)↑85.5787.4493.1685.9292.1196.76 CRA (%)↑74.1457.7188.6075.0986.4692.95 Avg. (%)↑82.2979.991.5185.5989.8285.52 FID (%)↓124.11155.17111.16111.70110.84110.24 Table 6. SAEMNESIA Experimental Results - Trained From Scratch with Hyperparameters Search MetricSAeUronOO-GCE-FSOS-DC-CA-FSOO-CA-FSOS-CA-FSOS-TK-CA-FS UA (%)↑87.1695.9892.9578.4586.0069.45 IRA (%)↑85.5768.2091.4864.2592.7697.38 CRA (%)↑74.1442.4888.1257.7687.0593.97 Avg. (%)↑82.2969.2290.8566.8288.9386.93 FID (%)↓124.11150.15110.25119.46110.49109.84 Nevertheless, SAEMNESIA maintains a significant prac- tical advantage over unsupervised alternatives. SAeUron requires steering the 205 top-scoring latents to achieve such performances, while SAEMNESIA-TOP2 only needs 2. Table 7. Timesteps with Cats-Dogs Overlapping Latent as Most Active - Models Trained from scratch MetricSAeUronOO-GCE-FSOS-DC-CA-FSOO-CA-FSOS-CA-FSOS-TK-CA-FS Overlap. timesteps10036102 Table 8. Timesteps with Cats-Dogs Overlapping Latent as Most Active - Finetuned Models MetricSAeUronOO-GCE-FTOS-DC-CA-FTOO-CA-FTOS-CA-FTOS-TK-CA-FT Overlap. timesteps100103140 Architectures Bears Birds Butterfly Cats Dogs Fishes Flame Flowers Frogs Horses Human Jellyfish Rabbits Sandwiches Sea Statues Towers Trees Waterfalls Evaluated Class Architectures Bears Birds Butterfly Cats Dogs Fishes Flame Flowers Frogs Horses Human Jellyfish Rabbits Sandwiches Sea Statues Towers Trees Waterfalls Unlearned Class 0.0100.096.1100.0100.0100.0100.084.3100.098.0100.086.3100.0100.0100.0100.031.423.598.0100.0 100.00.084.3100.0100.0100.098.0100.098.094.1100.094.1100.0100.0100.0100.084.372.5100.0100.0 98.0100.00.096.196.1100.098.098.0100.056.9100.096.196.198.098.0100.076.576.598.098.0 98.098.031.468.670.696.198.0100.086.358.898.070.692.270.6100.0100.078.478.498.098.0 94.1100.080.4100.00.090.296.198.0100.092.2100.096.1100.094.1100.0100.090.276.5100.0100.0 96.198.041.2100.045.10.094.198.051.088.284.362.782.496.192.296.176.582.4100.0100.0 100.098.088.2100.098.0100.00.098.096.190.2100.098.0100.0100.0100.098.084.370.698.0100.0 100.098.086.3100.098.0100.0100.09.896.196.1100.094.198.0100.098.0100.072.584.3100.0100.0 98.0100.086.3100.096.1100.098.094.10.082.4100.076.590.298.098.098.080.480.4100.0100.0 98.0100.088.2100.0100.0100.0100.0100.098.02.0100.094.198.0100.0100.0100.082.492.2100.0100.0 100.096.184.3100.0100.0100.098.098.0100.098.00.084.3100.0100.0100.0100.086.376.5100.0100.0 60.854.95.996.111.872.556.980.49.833.366.721.60.052.956.986.30.017.623.582.4 94.1100.086.3100.0100.0100.0100.098.0100.094.1100.090.20.0100.0100.0100.080.478.4100.0100.0 100.0100.088.2100.098.0100.0100.098.098.094.1100.096.1100.00.0100.0100.078.478.4100.0100.0 100.0100.090.2100.098.0100.098.096.1100.082.4100.090.296.1100.00.0100.082.482.498.0100.0 96.1100.086.3100.0100.0100.080.4100.0100.098.0100.092.2100.0100.096.115.786.382.4100.098.0 98.098.086.3100.0100.0100.098.0100.0100.090.2100.088.2100.0100.0100.0100.00.082.4100.0100.0 84.3100.094.1100.0100.0100.098.0100.0100.0100.0100.090.2100.0100.0100.0100.033.32.094.1100.0 100.0100.086.398.098.0100.0100.096.1100.092.296.184.398.098.098.098.076.578.411.8100.0 96.1100.094.1100.0100.0100.098.0100.0100.094.1100.092.2100.0100.0100.0100.074.568.6100.00.0 SAEmnesia-OS-DC-CA-FT - HP Searched Multiplier 0 20 40 60 80 100 Classification Accuracy (%) Figure 15. OS-DC-CA-FT UA and IRA classes disentanglement. Architectures Bears Birds Butterfly Cats Dogs Fishes Flame Flowers Frogs Horses Human Jellyfish Rabbits Sandwiches Sea Statues Towers Trees Waterfalls Evaluated Class Architectures Bears Birds Butterfly Cats Dogs Fishes Flame Flowers Frogs Horses Human Jellyfish Rabbits Sandwiches Sea Statues Towers Trees Waterfalls Unlearned Class 17.6100.080.4100.0100.0100.0100.096.1100.094.1100.088.2100.098.0100.098.078.482.498.0100.0 100.054.988.2100.0100.0100.0100.0100.0100.098.0100.088.2100.0100.0100.0100.098.082.4100.0100.0 100.0100.025.5100.098.0100.092.298.096.194.1100.086.398.098.0100.0100.086.386.3100.0100.0 98.0100.082.4100.098.0100.0100.098.0100.0100.0100.078.4100.098.0100.098.088.288.2100.098.0 98.0100.086.3100.00.0100.098.0100.0100.0100.0100.090.298.096.1100.0100.090.284.3100.098.0 98.0100.084.3100.084.319.6100.0100.096.190.2100.082.4100.0100.098.098.090.286.3100.0100.0 100.0100.090.2100.0100.0100.02.098.0100.0100.0100.088.298.0100.0100.098.092.284.398.0100.0 98.0100.088.2100.0100.0100.0100.064.798.096.1100.086.394.1100.0100.098.090.282.498.098.0 96.1100.094.1100.0100.0100.0100.0100.098.098.0100.096.1100.0100.0100.098.088.278.4100.0100.0 100.0100.080.4100.098.0100.0100.0100.098.027.5100.090.298.0100.0100.0100.088.286.3100.098.0 100.098.094.1100.098.0100.098.0100.0100.098.098.082.4100.0100.0100.0100.090.280.4100.0100.0 82.4100.090.2100.098.0100.0100.0100.0100.094.1100.088.2100.0100.0100.096.186.382.498.0100.0 100.0100.088.2100.0100.0100.098.0100.098.0100.0100.088.20.0100.0100.098.090.284.3100.0100.0 98.0100.086.3100.098.0100.0100.0100.098.098.0100.092.2100.05.9100.096.188.282.4100.0100.0 98.0100.084.3100.0100.0100.098.0100.0100.0100.0100.094.1100.098.02.0100.092.280.498.0100.0 100.0100.086.3100.098.0100.0100.098.098.096.1100.090.298.0100.0100.09.898.082.498.098.0 96.198.092.2100.098.0100.098.0100.098.096.1100.084.396.1100.0100.0100.00.086.398.098.0 98.0100.094.1100.096.1100.098.0100.0100.096.1100.086.398.0100.0100.0100.090.227.596.1100.0 100.0100.088.2100.098.0100.0100.0100.0100.098.098.090.296.1100.0100.098.080.478.439.2100.0 96.1100.092.2100.0100.0100.098.0100.098.098.0100.092.296.1100.0100.096.184.384.398.00.0 SAEmnesia-OS-CA-FS - HP Searched Multiplier 0 20 40 60 80 100 Classification Accuracy (%) Figure 16. OS-CA-FS UA and IRA classes disentanglement. -30-25-20-15-10-5-10.0 Multiplier 0 20 40 60 80 100 UA -30-25-20-15-10-5-10.0 Multiplier IRA -30-25-20-15-10-5-10.0 Multiplier CRA -30-25-20-15-10-5-10.0 Multiplier Mean (UA, IRA, CRA) SAeUronS-OS-DC-CA-FTS-O-CA-FTS-OS-CA-FTS-OS-TK-CA-FT Figure 17. All Model’s performances with uniform multipliers - fine tuned. -30-25-20-15-10-5-10.0 Multiplier 0 20 40 60 80 100 UA -30-25-20-15-10-5-10.0 Multiplier IRA -30-25-20-15-10-5-10.0 Multiplier CRA -30-25-20-15-10-5-10.0 Multiplier Mean (UA, IRA, CRA) SAeUronS-OS-DC-CA-FSS-O-CA-FSS-OS-CA-FSS-OS-TK-CA-FS Figure 18. All Model’s performances with uniform multipliers - from scratch. MethodArmpits Belly Buttocks Feet Breasts (F) Genitalia (F) Breasts (M) Genitalia (M) Total CLIPScore (↑) FID (↓) FMN [49]43117125915517192424 30.3913.52 CA [25]153180456629822677838 31.3716.25 AdvUn [50]8001311002828.1417.18 Receler [20]4832335200175160 30.4915.32 MACE [31]171923916097111 29.4113.42 CPE [28]1082861324031.1913.89 UCE [15]2962729355114182 30.8514.07 SLD-M [38]4772321391263212 30.9016.34 ESD-x [14]597312391006188315 30.6914.41 ESD-u [14]323021927382123 30.2115.10 SAeUron [9]713240011830.8914.37 SAEMNESIA71725112214730.9814.72 SAEMNESIA-TOP2 13104000930.9814.72 SD v1.4148170296326618427743 31.3414.04 SD v2.110515917601779572586 31.5314.87 Table 9. SAEMNESIA-TOP2 is the presented SAEMNESIA setup, but with two latents steered instead of one. Multiplier for SAEMNESIA (one latent affected): -49. Multiplier for SAEMNESIA-TOP2 (two latents affected): -60.

Instant research discovery