Paper deep dive

GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention

Amro Abdalla, Ismail Shaheen, Dan DeGenaro, Rupayan Mallick, Bogdan Raita, Sarah Adel Bargal

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 64

Models: Stable Diffusion v1.5

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 5:28:07 PM

Summary

GIFT is a gradient-aware immunization framework for text-to-image diffusion models that defends against malicious fine-tuning. It uses a bi-level optimization approach to simultaneously maximize loss on harmful concepts (via loss maximization and representation noising) and minimize loss on safe data, ensuring robust resistance to adversarial adaptation while preserving generative quality.

Entities (5)

GIFT · method · 100%Stable Diffusion · model · 98%DreamBooth · fine-tuning-technique · 95%IMMA · method · 95%LoRA · fine-tuning-technique · 95%

Relation Signals (3)

GIFT → defendsagainst → Malicious Fine-Tuning

confidence 95% · GIFT: a Gradient-aware Immunization technique to defend diffusion models against malicious Fine-Tuning

GIFT → appliesto → Stable Diffusion

confidence 90% · Let θ represent the U-Net parameters of a pre-trained T2I model (e.g., Stable Diffusion)

GIFT → improvesupon → IMMA

confidence 90% · GIFT outperforms existing baselines (ESD, IMMA) in resisting malicious fine-tuning while preserving safe model utility.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We present GIFT: a {G}radient-aware {I}mmunization technique to defend diffusion models against malicious {F}ine-{T}uning while preserving their ability to generate safe content. Existing safety mechanisms like safety checkers are easily bypassed, and concept erasure methods fail under adversarial fine-tuning. GIFT addresses this by framing immunization as a bi-level optimization problem: the upper-level objective degrades the model's ability to represent harmful concepts using representation noising and maximization, while the lower-level objective preserves performance on safe data. GIFT achieves robust resistance to malicious fine-tuning while maintaining safe generative quality. Experimental results show that our method significantly impairs the model's ability to re-learn harmful concepts while maintaining performance on safe content, offering a promising direction for creating inherently safer generative models resistant to adversarial fine-tuning attacks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

63,641 characters extracted from source content.

Expand or collapse full text

GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention Amro Abdalla1 Ismail Shaheen1 Dan DeGenaro1 Rupayan Mallick1 Bogdan Rai ă2 Sarah Adel Bargal1 1Department of Computer Science, Georgetown University 2Department of Mathematics and Statistics, Georgetown University a654, ias68, drd92, rupayan.mallick, br607, sarah.bargal@georgetown.edu Abstract We present GIFT: a Gradient-aware Immunization technique to defend diffusion models against malicious Fine-Tuning while preserving their ability to generate safe content. Existing safety mechanisms like safety checkers are easily bypassed, and concept erasure methods fail under adversarial fine-tuning. GIFT addresses this by framing immunization as a bi-level optimization problem: the upper-level objective degrades the model’s ability to represent harmful concepts using representation noising and maximization, while the lower-level objective preserves performance on safe data. GIFT achieves robust resistance to malicious fine-tuning while maintaining safe generative quality. Experimental results show that our method significantly impairs the model’s ability to re-learn harmful concepts while maintaining performance on safe content, offering a promising direction for creating inherently safer generative models resistant to adversarial fine-tuning attacks. Warning: This paper contains NSFW content. Reader discretion is advised. 1 Introduction Text-to-image (T2I) models have emerged as powerful generative tools capable of producing high-quality images faithful to input prompts [22, 19, 12, 20]. However, their accessibility and adaptability make them vulnerable to malicious fine-tuning, where adversaries adapt pre-trained models to generate harmful or copyrighted content. Methods like DreamBooth [24], LoRA [11] , and Textual Inversion [5] enable this adaptation with minimal resources and without needing to train from scratch. This vulnerability persists even when existing safety mechanisms, such as safety checkers [21] or concept erasure methods [6, 7, 9], are in place, as they can be bypassed [32, 36, 8], disabled, or undone through lightweight adaptation techniques. This creates a significant risk: once a model is open-sourced, it becomes difficult to guarantee its continued alignment with safety goals. Current defenses either degrade the model’s generative capabilities or fail to withstand adversarial fine-tuning. While safety checkers and licensing agreements offer a first line of defense [3], they are not an inherent property of T2I models and are easily circumvented [21]. To enhance the inherent safety of T2I models, concept erasure techniques have been proposed to remove undesirable concepts by modifying the model’s internal representations. Although such techniques can suppress the generation of undesired concepts, they are vulnerable to circumvention [18, 36]. Moreover, as we show in our experiments, simple fine-tuning can reintroduce the erased concepts, undermining the long-term effectiveness of concept erasure methods as a safety mechanism. To address the limitations of concept erasure and defend against its circumvention, model immunization has been proposed as a proactive defense against malicious fine-tuning of T2I models. IMMA [37], for example, introduces a bi-level optimization approach inspired by MAML [4], aiming to learn poor model initializations that hinder adaptation to undesirable concepts. By simulating the fine-tuning process during immunization, IMMA makes it more difficult for adversaries to reintroduce harmful content through fine-tuning. However, IMMA’s framework significantly compromises the model’s performance on safe concepts, degrading both its generative quality and its ability to be fine-tuned for benign applications as we will show in our experiments. To this end, we propose GIFT—a Gradient-aware Immunization framework to defend T2I diffusion models against malicious Fine-Tuning while preserving their ability to generate safe content. Inspired by IMMA [37] and MAML [4], we formulate GIFT as a bi-level optimization problem: the lower-level task minimizes a prior preservation objective to retain performance on safe concepts, while the upper-level task minimizes an immunization objective that prevents adaptation to harmful concepts. The immunization objective is composed of two parts: (1) a loss maximization term and (2) a representation noising term inspired by [23] from the language domain. We demonstrate that immunizing a model with GIFT significantly impairs its ability to re-learn harmful content while maintaining generative ability across a wide range of safe concepts. Our evaluation covers several concept categories—including objects, art styles, and NSFW content—and considers multiple fine-tuning strategies, e.g., LoRA and DreamBooth. Our main contributions are: • We propose GIFT, a novel framework that immunizes text-to-image diffusion models against malicious fine-tuning while preserving their generative quality and utility on safe concepts. • We formulate immunization as a bi-level optimization problem where the lower-level task uses a prior preservation loss to maintain generation quality on safe concepts, and the upper-level task employs an immunization loss to resist adaptation to harmful ones. • We conduct extensive experiments across diverse concept types (objects, art styles, and NSFW content), demonstrating that GIFT outperforms existing baselines (ESD, IMMA) in resisting malicious fine-tuning while preserving safe model utility. 2 Related Work The advancements of text-to-image (T2I) generative models, such as Stable Diffusion [22], have democratized content creation but also introduced significant risks associated with their misuse. A key concern is the vulnerability of these models to malicious fine-tuning, where bad actors can adapt pre-trained models to generate harmful, copyrighted, or otherwise undesirable content, prompting growing interest in developing safeguards for T2I models. Existing approaches to mitigate these risks can be broadly categorized. One line of work focuses on concept erasure or unlearning, aiming to remove specific concepts from a pre-trained model. Erased Stable Diffusion (ESD) [6] fine-tunes model weights using textual descriptions of the undesired concept to prevent the model from generating it. Other methods explore unlearning by modifying specific model components like the text encoder or attention layers [13, 34], sometimes using few-shot unlearning techniques [30], by adding lightweight eraser modules [9], or through data unlearning [1]. While effective at removal, some erasure techniques can be circumvented by further fine-tuning [18], as the underlying knowledge might not be entirely eliminated or can be easily relearned [38, 36]. Additionally, a significant challenge is preserving the model’s utility on unrelated concepts, as aggressive erasure can lead to “catastrophic forgetting” of desired knowledge [27, 31]. Some recent works attempt to address this by focusing on concept-localized regularization or mitigating conflicting gradients during unlearning [17, 30]. Another paradigm is model immunization, which seeks to make the model inherently resistant to adaptation towards malicious concepts before it is released. IMMA (Immunizing text-to-image Models against Malicious Adaptation) [37] proposes learning model parameters that are difficult for adaptation methods to fine-tune on malicious content, framed as a bi-level optimization problem. While IMMA demonstrates effectiveness against various adaptation methods like LoRA [11], Textual Inversion [5], and DreamBooth [24], it can be overly aggressive, potentially degrading the model’s performance on safe, unrelated concepts. Other defense strategies include methods akin to data poisoning (e.g., Glaze [26], which protects artistic styles from mimicry). Other defense approaches focus on safe decoding or generation, often by modifying the diffusion process [25] or employing external classifiers and adaptive guards to filter outputs [33, 28]. However, these can sometimes be bypassed by users with white-box access to the model or through carefully crafted adversarial prompts and jailbreaking methods [21, 32, 8, 16]. Techniques from the Large Language Model (LLM) domain are also being explored and adapted. Representation Noising (RepNoise) [23], for instance, has been proposed as a defense mechanism against harmful fine-tuning in LLMs by removing information about harmful representations across model layers, making them difficult to recover. GIFT draws inspiration from this by adapting representation noising to T2I models. Unlike some erasure methods that can be easily reversed or circumvented [18, 36], and in contrast to immunization methods like IMMA [37] that may overly degrade general utility, GIFT aims for a better trade-off. GIFT’s bi-level formulation helps coordinate the optimization to prevent the immunization objective from detrimentally affecting the prior preservation objective. 3 Methodology 3.1 Problem Formulation Our goal is to prevent adaptation methods from reintroducing malicious concepts into a pre-trained T2I diffusion model. We formulate this as a bi-level optimization problem with two objectives: (1) immunization against harmful concepts and (2) preservation of model performance on safe data. Let θ represent the U-Net parameters of a pre-trained T2I model (e.g., Stable Diffusion [22]), and ψ⊂θψ⊂θψ ⊂ θ denote the subset of parameters corresponding to cross-attention layers. Let (xm,cm)∈DMsubscriptsubscriptsubscript(x_m,c_m)∈ D_M( xitalic_m , citalic_m ) ∈ Ditalic_M and (xs,cs)∈DSsubscriptsubscriptsubscript(x_s,c_s)∈ D_S( xitalic_s , citalic_s ) ∈ Ditalic_S denote image–concept pairs from the malicious and safe datasets, respectively. We aim to derive an immunized model θIsuperscriptθ^Iθitalic_I that resists adaptation to malicious concepts under any subsequent fine-tuning while maintaining its ability to learn and generate safe concepts. 3.2 Bi-level Optimization Framework The authors of IMMA [37] employ a meta-learning algorithm inspired by MAML [4] to immunize T2I models. They simulate malicious adaptation steps by minimizing the adaptation loss in the lower-level task, while maximizing the same loss in the upper-level task to achieve immunization. We employ a similar framework to immunize a T2I model against malicious concepts while retaining performance on safe data. We define the upper-level task as the immunization objective over DMsubscriptD_MDitalic_M and the lower-level task as the prior preservation objective over DSsubscriptD_SDitalic_S. We show our algorithm at 1. To perform optimization, we compute parameters θ∗superscriptθ^*θ∗ via a gradient step using the lower-level task on DSsubscriptD_SDitalic_S, followed by an optimization step of said parameters using the upper-level task on DMsubscriptD_MDitalic_M. We formulate this as the following bi-level optimization problem: ψI=arg⁢minψ⊂θ∗⁡ℒimmunize⁢(xm,cm;θ∗)⏟upper-level taskwhereθ∗=arg⁢minθ⁡ℒprior⁢(xs,cs;θ)⏟lower-level task.subscript⏟superscriptsubscriptargminsuperscriptsubscriptℒimmunizesubscriptsubscriptsuperscriptupper-level taskwheresubscript⏟superscriptsubscriptargminsubscriptℒpriorsubscriptsubscriptlower-level task ψ^I= *arg\,min_ψ⊂θ^% *L_immunize(x_m,c_m;θ^*)_upper-level % task θ^*= *arg\,min_% θL_prior(x_s,c_s;θ)_lower-level task% .under⏟ start_ARG ψitalic_I = start_OPERATOR arg min end_OPERATORψ ⊂ θ∗ Limmunize ( xitalic_m , citalic_m ; θ∗ ) end_ARGupper-level task where under⏟ start_ARG θ∗ = start_OPERATOR arg min end_OPERATORθ Lprior ( xitalic_s , citalic_s ; θ ) end_ARGlower-level task . (1) In the upper-level task, we minimize the immunization loss ℒimmunizesubscriptℒimmunizeL_immunizeLimmunize with respect to ψ (i.e., cross-attention layers). This encourages the model to resist adapting to harmful concepts from the malicious dataset DMsubscriptD_MDitalic_M. In the lower-level task, we minimize the prior preservation loss ℒpriorsubscriptℒpriorL_priorLprior with respect to the U-Net parameters θ, which includes ψ. This selection reflects the intuition that cross-attention layers play a central role in encoding and manipulating concepts [15], while optimizing the U-Net in the inner loop ensures that the model adapts safely while incorporating the immunization updates. To further explore how this bi-level setup benefits the immunization approach more than naive addition of all losses, we examine intermediate gradient updates. We implement the bi-level scheme in Eq. (1) by iterating the gradient updates: θ′=θ−αP⁢∇ℒP⁢(θ)andψ′=ψ′−αI⁢∇ℒI⁢(ψ′),formulae-sequencesuperscript′subscript∇subscriptℒandsuperscript′subscript∇subscriptℒsuperscript′θ =θ- _P _P(θ) % ψ =ψ - _I _I(ψ^% ),θ′ = θ - αitalic_P ∇ Litalic_P ( θ ) and ψ′ ′ = ψ′ - αitalic_I ∇ Litalic_I ( ψ′ ) , (2) where ψ′⊂θ′superscript′ψ ⊂θ ψ′ ⊂ θ′, and we abbreviate ℒprior⁢(xs,cs;θ)=ℒP⁢(θ)subscriptℒpriorsubscriptsubscriptsubscriptℒL_prior(x_s,c_s;θ)=L_P(θ)Lroman_prior ( xitalic_s , citalic_s ; θ ) = Litalic_P ( θ ) and ℒimmunize⁢(xm,cm;θ)=ℒI⁢(ψ)subscriptℒimmunizesubscriptsubscriptsubscriptℒL_immunize(x_m,c_m;θ)=L_I(ψ)Lroman_immunize ( xitalic_m , citalic_m ; θ ) = Litalic_I ( ψ ). We then use the Taylor Series expansion for ∇ℒI⁢(ψ′)∇subscriptℒsuperscript′ _I(ψ )∇ Litalic_I ( ψ′ ) as seen in Eq (3), which gives us the total update for ψ′ψ ψ′ ′ in Eq (4). Full derivation can be found in the appendix. ∇ℒI⁢(ψ′)≈∇ℒI⁢(ψ)−αP⁢∇2ℒI⁢(ψ)⁢∇ψℒP⁢(θ)∇subscriptℒsuperscript′∇subscriptℒsubscriptsuperscript∇2subscriptℒsubscript∇subscriptℒ _I(ψ )≈ _I(ψ)- _% P∇^2L_I(ψ) _ψL_P(θ)∇ Litalic_I ( ψ′ ) ≈ ∇ Litalic_I ( ψ ) - αitalic_P ∇2 Litalic_I ( ψ ) ∇ψ Litalic_P ( θ ) (3) ψ′≈ψ−αP⁢∇ψℒP⁢(θ)−αI⁢∇ℒI⁢(ψ)+αP⁢αI⁢∇2ℒI⁢(ψ)⁢∇ψℒP⁢(θ)superscript′subscriptsubscript∇subscriptℒsubscript∇subscriptℒsubscriptsubscriptsuperscript∇2subscriptℒsubscript∇subscriptℒψ ≈ψ- [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1 _P _ψL_P(θ)-% [rgb]1,.5,0 [named]pgfstrokecolorrgb1,.5,0 _I% _I(ψ)+ [rgb].75,0,.25 [named]% pgfstrokecolorrgb.75,0,.25 _P _I∇^2L_I(% ψ) _ψL_P(θ)ψ′ ′ ≈ ψ - αitalic_P ∇ψ Litalic_P ( θ ) - αitalic_I ∇ Litalic_I ( ψ ) + αitalic_P αitalic_I ∇2 Litalic_I ( ψ ) ∇ψ Litalic_P ( θ ) (4) The final equation shows that our current immunization gradient update is equivalent to doing an update in the prior preservation direction plus an update in the immunization direction plus an additional term. That term is the directional curvature of ℒIsubscriptℒL_ILitalic_I along ∇ψℒP⁢(θ)subscript∇subscriptℒ _ψL_P(θ)∇ψ Litalic_P ( θ ), which is crucial for our approach. This term adds a second order correction which helps coordinate the gradient descent so that minimizing ℒIsubscriptℒL_ILitalic_I does not make minimizing ℒPsubscriptℒL_PLitalic_P harder. This improves the retention of our model significantly by making the immunization update “aware” of previous prior preservation updates. 3.3 Immunization Loss The immunization loss that we employ in the upper-level task consists of two components: (1) loss maximization and (2) representation noising. Loss Maximization. We maximize the loss with respect to the malicious concept as follows: ℒmax=−t,ϵ∼⁢(0,I)⁢[‖ϵθ⁢(xm,cm,t)−ϵ‖22].subscriptℒmaxsubscriptsimilar-toitalic-ϵ0delimited-[]superscriptsubscriptnormsubscriptitalic-ϵsubscriptsubscriptitalic-ϵ22L_max=-E_t,ε (0,I) [\|% _θ(x_m,c_m,t)-ε\|_2^2 ].Lmax = - blackboard_Et , ϵ ∼ N ( 0 , I ) [ ∥ ϵitalic_θ ( xitalic_m , citalic_m , t ) - ϵ ∥22 ] . (5) This maximization aims to push the model parameters θ to perform poorly on the target malicious data (xm,cm)subscriptsubscript(x_m,c_m)( xitalic_m , citalic_m ). However, loss maximization on malicious content is not sufficient on its own. Representation Noising. While maximizing the loss on malicious concepts reduces the model’s ability to generate them, it does not necessarily prevent the model from re-adapting to these concepts with further fine-tuning. This is because when maximizing ℒmaxsubscriptℒmaxL_maxLmax, the mutual information MI⁢(xm|cm;ym)MIconditionalsubscriptsubscriptsubscriptMI(x_m|c_m;y_m)MI ( xitalic_m | citalic_m ; yitalic_m ) between conditioned malicious inputs xm|cmconditionalsubscriptsubscriptx_m|c_mxitalic_m | citalic_m and malicious model outputs ymsubscripty_myitalic_m is targeted, but the mutual information MI⁢(xm|cm;zm)MIconditionalsubscriptsubscriptsubscriptMI(x_m|c_m;z_m)MI ( xitalic_m | citalic_m ; zitalic_m ) between inputs xm|cmconditionalsubscriptsubscriptx_m|c_mxitalic_m | citalic_m and intermediate representations zmsubscriptz_mzitalic_m can remain, which may allow the malicious concept to return [23]. The data processing inequality states: MI⁢(xm|cm;zm)≥MI⁢(xm|cm;ym).MIconditionalsubscriptsubscriptsubscriptMIconditionalsubscriptsubscriptsubscriptMI(x_m|c_m;z_m) (x_m|c_m;y_m).MI ( xitalic_m | citalic_m ; zitalic_m ) ≥ MI ( xitalic_m | citalic_m ; yitalic_m ) . (6) That is, information shared between inputs xm|cmconditionalsubscriptsubscriptx_m|c_mxitalic_m | citalic_m and intermediate representations zmsubscriptz_mzitalic_m is an upper bound on information shared between those inputs and the outputs ymsubscripty_myitalic_m. As such, it is useful to directly reduce MI⁢(xm|cm;zm)MIconditionalsubscriptsubscriptsubscriptMI(x_m|c_m;z_m)MI ( xitalic_m | citalic_m ; zitalic_m ) which implies a reduction in MI⁢(xm|cm;ym)MIconditionalsubscriptsubscriptsubscriptMI(x_m|c_m;y_m)MI ( xitalic_m | citalic_m ; yitalic_m ). To this end, we adapt the LLM immunization technique of [23], representation noising, to T2I models. Let L(j)superscriptL^(j)L( j ) denote the j-th layer of the U-Net, where j∈1,…,n1…j∈\1,…,n\j ∈ 1 , … , n . For an input to the U-Net conditioned on the malicious concept xm|cmconditionalsubscriptsubscriptx_m|c_mxitalic_m | citalic_m, we define the first intermediate representation as zm(1)=L(1)⁢(xm|cm)superscriptsubscript1superscript1conditionalsubscriptsubscriptz_m^(1)=L^(1)(x_m|c_m)zitalic_m( 1 ) = L( 1 ) ( xitalic_m | citalic_m ) and further intermediate representations as zm(j)=L(j)⁢(zm(j−1))superscriptsubscriptsuperscriptsuperscriptsubscript1z_m^(j)=L^(j)(z_m^(j-1))zitalic_m( j ) = L( j ) ( zitalic_m( j - 1 ) ) for j∈2,…,n2…j∈\2,…,n\j ∈ 2 , … , n . We then minimize the loss between these activations and random noise: ℒnoise=∑j=1nMSE⁢(zm(j),ϵm(j)),where ⁢ϵm(j)∼⁢(μzm(j),σzm(j)2),formulae-sequencesubscriptℒnoisesuperscriptsubscript1MSEsuperscriptsubscriptsuperscriptsubscriptitalic-ϵsimilar-towhere superscriptsubscriptitalic-ϵsubscriptsuperscriptsubscriptsubscriptsuperscript2superscriptsubscriptL_noise= _j=1^nMSE (z_m^(j% ), _m^(j) ), _m^(j) % N ( _z_m^(j),σ^2_z_m^(j) ),Lnoise = ∑j = 1n MSE ( zitalic_m( j ) , ϵitalic_m( j ) ) , where ϵitalic_m( j ) ∼ N ( μitalic_z start_POSTSUBSCRIPT m( j ) end_POSTSUBSCRIPT , σ2italic_z start_POSTSUBSCRIPT m( j ) end_POSTSUBSCRIPT ) , (7) where (μzm(j),σzm(j)2)subscriptsuperscriptsubscriptsubscriptsuperscript2superscriptsubscript ( _z_m^(j),σ^2_z_m^(j) )( μitalic_z start_POSTSUBSCRIPT m( j ) end_POSTSUBSCRIPT , σ2italic_z start_POSTSUBSCRIPT m( j ) end_POSTSUBSCRIPT ) are the computed mean and variance of the sample zm(j)superscriptsubscriptz_m^(j)zitalic_m( j ). This loss aims to destroy undesirable information contained in the model while preventing model parameters from diverging significantly from their pre-training. Total Immunization Loss. The final immunization objective combines the loss maximization term with the representation noising loss, weighted by a hyperparameter β: ℒimmunize=ℒmax+β⋅ℒnoise.subscriptℒimmunizesubscriptℒmax⋅subscriptℒnoiseL_immunize=L_max+β·L_% noise.Limmunize = Lmax + β ⋅ Lnoise . (8) This loss is applied specifically to the cross-attention layers in the upper-level optimization to target the parts of the model most responsible for concept encoding. 3.4 Prior Preservation Loss The immunization loss can degrade the model’s performance on safe tasks. To mitigate this effect, we employ the original T2I model training objective for safe data preservation: ℒprior=t,ϵ∼⁢(0,I)⁢[‖ϵθ⁢(xs,cs,t)−ϵ‖22],subscriptℒpriorsubscriptsimilar-toitalic-ϵ0delimited-[]superscriptsubscriptnormsubscriptitalic-ϵsubscriptsubscriptitalic-ϵ22L_prior=E_t,ε (0,I) [\|% _θ(x_s,c_s,t)-ε\|_2^2 ],Lprior = blackboard_Et , ϵ ∼ N ( 0 , I ) [ ∥ ϵitalic_θ ( xitalic_s , citalic_s , t ) - ϵ ∥22 ] , (9) which helps to maintain performance on safe concepts while immunizing against malicious ones. Algorithm 1 Our method (GIFT) 1:Malicious dataset DMsubscriptD_MDitalic_M, safe dataset DSsubscriptD_SDitalic_S 2:Model parameters θ with cross-attention subset ψ⊂θψ⊂θψ ⊂ θ 3:Learning rates αinnersubscriptinner _innerαinner, αoutersubscriptouter _outerαouter, Noising weight β 4:for each training iteration do 5: if inner loop step then 6: Sample batch (xs,cs)subscriptsubscript(x_s,c_s)( xitalic_s , citalic_s ) from DSsubscriptD_SDitalic_S ▷ ▷ Lower-level task: Prior Preservation 7: θ←θ−αinner⁢∇θℒprior⁢(xs,cs;θ)←subscriptinnersubscript∇subscriptℒpriorsubscriptsubscriptθ←θ- _inner _θL_% prior(x_s,c_s;θ)θ ← θ - αinner ∇θ Lprior ( xitalic_s , citalic_s ; θ ) 8: else 9: Sample batch (xm,cm)subscriptsubscript(x_m,c_m)( xitalic_m , citalic_m ) from DMsubscriptD_MDitalic_M ▷ ▷ Upper-level task: Immunization 10: ℒmax←−t,ϵ∼⁢(0,I)⁢[‖ϵθ⁢(xm,cm,t)−ϵ‖22]←subscriptℒmaxsubscriptsimilar-toitalic-ϵ0delimited-[]superscriptsubscriptnormsubscriptitalic-ϵsubscriptsubscriptitalic-ϵ22L_max←-E_t,ε (0,I)% [\| _θ(x_m,c_m,t)-ε\|_2^2 ]Lmax ← - blackboard_Et , ϵ ∼ N ( 0 , I ) [ ∥ ϵitalic_θ ( xitalic_m , citalic_m , t ) - ϵ ∥22 ] 11: Extract intermediate activations zm(j)superscriptsubscriptz_m^(j)zitalic_m( j ) for layers j=1,…,n1…j=1,…,nj = 1 , … , n 12: Sample noise ϵm(j)∼⁢(μzm(j),σzm(j)2)similar-tosuperscriptsubscriptitalic-ϵsubscriptsuperscriptsubscriptsubscriptsuperscript2superscriptsubscript _m^(j) ( _z_m^(j),σ^2_z_m^(j% ) )ϵitalic_m( j ) ∼ N ( μitalic_z start_POSTSUBSCRIPT m( j ) end_POSTSUBSCRIPT , σ2italic_z start_POSTSUBSCRIPT m( j ) end_POSTSUBSCRIPT ) ▷ ▷ Mean and Var. from zm(j)superscriptsubscriptz_m^(j)zitalic_m( j ) 13: ℒnoise←∑j=1nMSE⁢(zm(j),ϵm(j))←subscriptℒnoisesuperscriptsubscript1MSEsuperscriptsubscriptsuperscriptsubscriptitalic-ϵL_noise← _j=1^nMSE (z_m^(j),% _m^(j) )Lnoise ← ∑j = 1n MSE ( zitalic_m( j ) , ϵitalic_m( j ) ) 14: ℒimmunize←ℒmax+β⋅ℒnoise←subscriptℒimmunizesubscriptℒmax⋅subscriptℒnoiseL_immunize _max+β·% L_noiseLimmunize ← Lmax + β ⋅ Lnoise 15: ψ←ψ−αouter⁢∇ψℒimmunize←subscriptoutersubscript∇subscriptℒimmunizeψ←ψ- _outer _ψL_% immunizeψ ← ψ - αouter ∇ψ Limmunize 16: end if 17:end for 18:return Immunized model parameters θ as θIsuperscriptθ^Iθitalic_I 4 Experiments In this section, we show GIFT’s ability to immunize the T2I model Stable Diffusion v1.5 (SD) [22] on objects, art styles, and NSFW content. We evaluate our method against IMMA and ESD. Experimental Setup. For object immunization, we select 26 objects from the Custom Concept 101 dataset [14], each with more than 8 images split into 2 disjoint sets: DMsubscriptD_MDitalic_M and DAsubscriptD_ADitalic_A. The defense (malicious) split DMsubscriptD_MDitalic_M is used during immunization, and the attack split DAsubscriptD_ADitalic_A is used to simulate malicious fine-tuning with DreamBooth. For prior preservation, we generate 500 safe images per object using category-level prompts to form a safe set, DSsubscriptD_SDitalic_S. For each object, we compare GIFT to IMMA and an undefended baseline. Similarly, for artistic styles, we test on 10 styles (e.g., Van Gogh, Picasso) by generating 40 images per artist with prompts like <a painting in [artist] style>, splitting them equally into disjoint DMsubscriptD_MDitalic_M and DAsubscriptD_ADitalic_A sets. We compare GIFT to ESD and IMMA and we use <a painting of a cat in [artist] style> as a validation prompt. Finally, for NSFW content, we use the porn subset of the NSFW-T2I dataset [39], sampling 40 images and dividing them into DMsubscriptD_MDitalic_M and DAsubscriptD_ADitalic_A sets. We used a single NVIDIA L40 GPU with 40GB of memory in an internal cluster for each experiment. Evaluation Metrics. We evaluate GIFT using four metrics: CLIP similarity for prompt-image alignment [10], LPIPS for perceptual fidelity [35], DINO similarity for feature-level consistency [24], and NudeNet [2] to quantify explicit content after immunization. Together, these capture semantic alignment, visual quality, safe concept retention, and NSFW suppression. 4.1 Objects Attack Results. We find that GIFT performs similarly to IMMA in terms of immunizing SD against particular concepts, and generally achieves CLIP and LPIPS scores ranging between those of the undefended model and those of a model defended with IMMA. In cases such as Fig. 1, GIFT outperforms IMMA by producing images with lower CLIP scores when prompted for the concept against which the model is immunized. Averaged per-epoch metrics across all 26 objects can be seen in Fig. 3. We do not view this overall quantitative difference in our results as compared with IMMA’s as a weakness; rather, it indicates a less aggressive, but still functional immunization technique that preserves the model’s generative capabilities to a great extent. Figure 1: GIFT Immunizes Similarly to IMMA. Here, we treat the tortoise plushie as a malicious concept using the prompt <a *s tortoise plushie on the beach> where *s is DreamBooth’s special token. Top row: Reference images used to fine-tune via DreamBooth. Second row: Results of fine-tuning the undefended SD. Third row: Results of fine-tuning after 1K steps of immunization with GIFT. Bottom row: Results of fine-tuning after 1K steps of immunization with IMMA. Preservation Results. Models immunized with GIFT generally outperform those immunized with IMMA when tasked with generating images of a safe concept as can be seen qualitatively in Fig. 2. Models immunized with GIFT achieve CLIP and LPIPS scores similar to the undefended SD checkpoint. Averaged per-epoch metrics across all 26 objects can be seen in Fig. 3. Models immunized with IMMA generally achieve much lower similarity scores. Figure 2: GIFT Retains Safe Concepts Better than IMMA. Here, we treat the bass guitar as a safe concept unrelated to the malicious concept from Figure 1 using the prompt <a *s bass guitar on the beach>. Top row: Reference images used to fine-tune via DreamBooth. Second row: Results of fine-tuning the undefended SD. Third row: Results of fine-tuning after 1K steps of immunization against the plushie from Figure 1 with GIFT. Bottom row: Results of fine-tuning after 1K steps of immunization against the plushie from Figure 1 with IMMA. Figure 3: GIFT Finds a Middle Ground. Averaged per-epoch CLIP Score and LPIPS Similarity across 26 immunized models during fine-tuning. GIFT achieves significantly higher CLIP and LPIPS similarities between images of the safe concept and its corresponding prompt than models immunized with IMMA, indicating preservation of generative capabilities. It additionally achieves similar CLIP scores to IMMA on malicious concepts, and significantly lower LPIPS scores than the undefended model, indicating successful immunization. Qualitatively, as seen in Figure 1, GIFT’s scores still indicate sufficient immunization. 4.2 Art Styles Figure 4: GIFT Prevents Artistic Style Adaption. We fine-tune each model on a dataset of 20 Van Gogh generations (reference images included) and validate using the prompt <a painting of a cat in [artist] style>. On the left is the CLIP score for each method over the duration of training. On the right are qualitative results for each method at the 100, 500, 900, and 1300 step mark. ESD isn’t able to prevent adaption to the protected art style. IMMA consistently produces noise for the protected model at the expense of degraded model performance. GIFT prevents the adaption to the protected art style by producing noise for the first part of the attack then overfitting at the end. Attack Results. As shown in Fig. 4, ESD rapidly reacquires Van Gogh’s style (by step 100), including its application to unseen concepts, e.g., cats. Then, the model enters a corruption phase, where overfitting becomes apparent. This is evidenced by a decline in CLIP score alongside increasing similarity to the training data. Near step 1300, we observe a transient improvement phase, followed by further degradation behavior consistent with previously observed fine-tuning dynamics in diffusion models [29]. Thus, from an attacker’s perspective, fine-tuning an ESD-erased model to reintroduce the erased concept is essentially equivalent to fine-tuning a standard SD model. In contrast, immunization methods (e.g., IMMA) cause fine-tuning to continually produce pure noise, preventing the re-emergence of the concept. However, as discussed earlier in Section 4.1 and further in Section 4.3, IMMA significantly degrades model performance on unrelated, safe concepts. GIFT prevents Van Gogh-style generation entirely up to approximately step 600. Beyond this point, GIFT produces results that lie in a sweet spot between those of erasure-based (ESD) and immunization-based (IMMA) methods. Notably, GIFT allows limited re-learning from the data, which is beneficial when fine-tuning on benign inputs. The model appears to map prompts to training images, but it does not recover the generalizable ability to generate in the artist’s style. This is evident from the outputs in Fig. 4, where generated images closely resemble training examples but fail to match the prompt, resulting in lower CLIP scores. The slight increase in CLIP reflects that a Van Gogh-like image is produced, but it does not align with the intended subject (e.g., a cat). This overall trend holds across all evaluated artists, as illustrated in Fig. 5. GIFT consistently yields generations with lower LPIPS and DINO similarity compared to ESD, indicating reduced memorization and less precise replication of the training data. GIFT exhibits slightly higher similarity than IMMA, due to its capacity to overfit on individual samples without fully re-acquiring the erased concept. Despite this, GIFT fails to produce prompt-aligned generations throughout the training process, as evidenced by the much lower CLIP scores. This confirms that while GIFT permits limited data memorization, it successfully impedes the model from regaining the protected artistic style. Figure 5: Quantitative Results for All Artists. Comparison of CLIP Score, LPIPS Similarity, and DINO Similarity over fine-tuning steps for all evaluated artists. GIFT maintains lower LPIPS and DINO similarity than ESD, indicating reduced memorization of training data. The CLIP score remains substantially lower for GIFT, demonstrating its effectiveness in preventing prompt-consistent generation of the protected artistic styles. 4.3 NSFW Content Figure 6: GIFT Blocks Malicious Fine-Tuning While Preserving Safe Adaptation. We fine-tune each NSFW-immune model on a safe concept (A) and a malicious one (B), evaluating at steps 500, 1000, and 2500. ESD permits safe learning but fails to block unsafe content; IMMA blocks unsafe content but harms safe fidelity; IMMA+ESD fails at both. GIFT retains safe learning while resisting malicious adaptation. GIFT+PI further improves both. Attack Results. During the malicious fine-tuning attack, we observe that ESD quickly allows the model to recover explicit content. IMMA prevents re-learning but does so by significantly degrading the model’s learning ability across all concepts, not just NSFW. In contrast, GIFT consistently suppresses such malicious adaptation, yielding noisy or failed generations when prompted with NSFW content. It does so while preserving the ability to learn safe concepts as shown in Fig. 6. To further enhance performance on safe concepts, we apply a post-immunization (PI) fine-tuning step to the GIFT-immunized model. This involves training on a generic, benign prompt (e.g., <A photo of a barn and mountains>) for 1000 steps. Interestingly, this additional step not only improves the model’s ability to retain safe generation quality but also strengthens its resistance to malicious NSFW re-adaptation. We leave a deeper investigation of this effect to future work. These results demonstrate GIFT’s ability to impose robust and persistent resistance to harmful concept injection without compromising general generation quality. 5 Additional Analysis Attacking with Different Adaptation Method. Prior methods like IMMA perform a separate immunization process for each attack technique: immunize with DreamBooth to protect against DreamBooth (resp. LoRA) fine-tuning. GIFT, on the other hand, does not depend on the attack algorithm during immunization. We show in Fig. 7 how our model performs with the same immunization technique used in prior sections against a different attack method, namely LoRA. We fine-tune a LoRA adapter once using an erased Stable Diffusion v1.5 (ESD) and once using our own immunized model. Fig. 7 shows that an adapter trained using an un-immunized model can easily re-acquire erased knowledge, while using GIFT it cannot. Figure 7: GIFT Immunization with LoRA. GIFT can prevent model adaption using LoRA. 6 Limitations and Negative Impacts While GIFT effectively immunizes text-to-image diffusion models against malicious fine-tuning, several limitations remain. First, our approach assumes access to clearly defined and representative unsafe concept datasets. In real world, such representative datasets may be hard to curate. Second, our immunization loss may still impact generation quality for safe concepts, especially when visual features overlap between safe and unsafe categories. Additionally, we focus on single concept immunization in this work. Multi-concept immunization will be explored in future work. From an ethical standpoint, our method is designed to reduce the risk of generating harmful, unsafe, or copyrighted content. However, it does not guarantee full immunity and could potentially be circumvented by future, more sophisticated adaptation techniques. As with any content moderation tool, misuse or overreach (e.g., censoring legitimate creative expression) remains a concern. We encourage the community to treat GIFT as a step toward safer generative models, not a definitive solution, and to accompany its use with broader societal oversight. 7 Conclusion This paper introduces GIFT, a gradient-aware immunization framework for diffusion models, which addresses the critical vulnerability of diffusion models to malicious fine-tuning. While previously developed safety mechanisms either degrade overall model performance (e.g., IMMA) or can be easily circumvented (e.g., ESD), GIFT strikes a balance between immunization effectiveness and preservation of generative capabilities on safe concepts. We formulate immunization as a bi-level optimization problem: the lower-level task focuses on preserving performance on safe concepts, while the upper-level task prevents adaptation to harmful content through a combination of loss maximization and representation noising. Extensive experiments across diverse concepts show that GIFT resists harmful re-learning, maintains generation quality, and remains fine-tunable on safe data. This makes GIFT a practical tool for safer model deployment. Future work will explore multi-concept immunization, efficient scaling, and broader application to other generative architectures. While GIFT is a key step toward model safety, it should be complemented by policy and ethical oversight for responsible AI deployment. References Alberti et al. [2025] Silas Alberti, Kenan Hasanaliyev, Manav Shah, and Stefano Ermon. Data unlearning in diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=SuHScQv5gP. Bedapudi [2019] Praneet Bedapudi. Nudenet: Neural nets for nudity classification, detection and selective censoring. https://github.com/platelminto/NudeNetClassifier, 2019. CompVis [2022] CompVis. Stable diffusion license, 2022. URL https://github.com/CompVis/stable-diffusion/blob/main/LICENSE. CreativeML OpenRAIL-M License. Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017. Gal et al. [2023] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=NAQvF08TcyG. Gandikota et al. [2023] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2426–2436, October 2023. Gandikota et al. [2024] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5111–5120, January 2024. Gao et al. [2024] Sensen Gao, Xiaojun Jia, Yihao Huang, Ranjie Duan, Jindong Gu, Yang Bai, Yang Liu, and Qing Guo. Hts-attack: Heuristic token search for jailbreaking text-to-image models, 2024. URL https://arxiv.org/abs/2408.13896. Gong et al. [2024] Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision – ECCV 2024, pages 73–88, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-73668-1. Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. ArXiv, abs/2104.08718, 2021. URL https://api.semanticscholar.org/CorpusID:233296711. Hu et al. [2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. Imagen-Team-Google et al. [2024] Imagen-Team-Google, :, Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castrejon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Eaton-Rosen, Hongliang Fei, Nando de Freitas, Yilin Gao, Evgeny Gladchenko, Sergio Gómez Colmenarejo, Mandy Guo, Alex Haig, Will Hawkins, Hexiang Hu, Huilian Huang, Tobenna Peter Igwe, Christos Kaplanis, Siavash Khodadadeh, Yelin Kim, Ksenia Konyushkova, Karol Langner, Eric Lau, Rory Lawton, Shixin Luo, Soňa Mokrá, Henna Nandwani, Yasumasa Onoe, Aäron van den Oord, Zarana Parekh, Jordi Pont-Tuset, Hang Qi, Rui Qian, Deepak Ramachandran, Poorva Rane, Abdullah Rashwan, Ali Razavi, Robert Riachi, Hansa Srinivasan, Srivatsan Srinivasan, Robin Strudel, Benigno Uria, Oliver Wang, Su Wang, Austin Waters, Chris Wolff, Auriel Wright, Zhisheng Xiao, Hao Xiong, Keyang Xu, Marc van Zee, Junlin Zhang, Katie Zhang, Wenlei Zhou, Konrad Zolna, Ola Aboubakar, Canfer Akbulut, Oscar Akerlund, Isabela Albuquerque, Nina Anderson, Marco Andreetto, Lora Aroyo, Ben Bariach, David Barker, Sherry Ben, Dana Berman, Courtney Biles, Irina Blok, Pankil Botadra, Jenny Brennan, Karla Brown, John Buckley, Rudy Bunel, Elie Bursztein, Christina Butterfield, Ben Caine, Viral Carpenter, Norman Casagrande, Ming-Wei Chang, Solomon Chang, Shamik Chaudhuri, Tony Chen, John Choi, Dmitry Churbanau, Nathan Clement, Matan Cohen, Forrester Cole, Mikhail Dektiarev, Vincent Du, Praneet Dutta, Tom Eccles, Ndidi Elue, Ashley Feden, Shlomi Fruchter, Frankie Garcia, Roopal Garg, Weina Ge, Ahmed Ghazy, Bryant Gipson, Andrew Goodman, Dawid Górny, Sven Gowal, Khyatti Gupta, Yoni Halpern, Yena Han, Susan Hao, Jamie Hayes, Jonathan Heek, Amir Hertz, Ed Hirst, Emiel Hoogeboom, Tingbo Hou, Heidi Howard, Mohamed Ibrahim, Dirichi Ike-Njoku, Joana Iljazi, Vlad Ionescu, William Isaac, Reena Jana, Gemma Jennings, Donovon Jenson, Xuhui Jia, Kerry Jones, Xiaoen Ju, Ivana Kajic, Christos Kaplanis, Burcu Karagol Ayan, Jacob Kelly, Suraj Kothawade, Christina Kouridi, Ira Ktena, Jolanda Kumakaw, Dana Kurniawan, Dmitry Lagun, Lily Lavitas, Jason Lee, Tao Li, Marco Liang, Maggie Li-Calis, Yuchi Liu, Javier Lopez Alberca, Matthieu Kim Lorrain, Peggy Lu, Kristian Lum, Yukun Ma, Chase Malik, John Mellor, Thomas Mensink, Inbar Mosseri, Tom Murray, Aida Nematzadeh, Paul Nicholas, Signe Nørly, João Gabriel Oliveira, Guillermo Ortiz-Jimenez, Michela Paganini, Tom Le Paine, Roni Paiss, Alicia Parrish, Anne Peckham, Vikas Peswani, Igor Petrovski, Tobias Pfaff, Alex Pirozhenko, Ryan Poplin, Utsav Prabhu, Yuan Qi, Matthew Rahtz, Cyrus Rashtchian, Charvi Rastogi, Amit Raul, Ali Razavi, Sylvestre-Alvise Rebuffi, Susanna Ricco, Felix Riedel, Dirk Robinson, Pankaj Rohatgi, Bill Rosgen, Sarah Rumbley, Moonkyung Ryu, Anthony Salgado, Tim Salimans, Sahil Singla, Florian Schroff, Candice Schumann, Tanmay Shah, Eleni Shaw, Gregory Shaw, Brendan Shillingford, Kaushik Shivakumar, Dennis Shtatnov, Zach Singer, Evgeny Sluzhaev, Valerii Sokolov, Thibault Sottiaux, Florian Stimberg, Brad Stone, David Stutz, Yu-Chuan Su, Eric Tabellion, Shuai Tang, David Tao, Kurt Thomas, Gregory Thornton, Andeep Toor, Cristian Udrescu, Aayush Upadhyay, Cristina Vasconcelos, Alex Vasiloff, Andrey Voynov, Amanda Walker, Luyu Wang, Miaosen Wang, Simon Wang, Stanley Wang, Qifei Wang, Yuxiao Wang, Ágoston Weisz, Olivia Wiles, Chenxia Wu, Xingyu Federico Xu, Andrew Xue, Jianbo Yang, Luo Yu, Mete Yurtoglu, Ali Zand, Han Zhang, Jiageng Zhang, Catherine Zhao, Adilet Zhaxybay, Miao Zhou, Shengqi Zhu, Zhenkai Zhu, Dawn Bloxwich, Mahyar Bordbar, Luis C. Cobo, Eli Collins, Shengyang Dai, Tulsee Doshi, Anca Dragan, Douglas Eck, Demis Hassabis, Sissie Hsiao, Tom Hume, Koray Kavukcuoglu, Helen King, Jack Krawczyk, Yeqing Li, Kathy Meier-Hellstern, Andras Orban, Yury Pinsky, Amar Subramanya, Oriol Vinyals, Ting Yu, and Yori Zwols. Imagen 3, 2024. URL https://arxiv.org/abs/2408.07009. Kumari et al. [2023a] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22691–22702, October 2023a. Kumari et al. [2023b] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1931–1941, June 2023b. Liu et al. [2024a] Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7817–7826, June 2024a. Liu et al. [2024b] Xuannan Liu, Xing Cui, Peipei Li, Zekun Li, Huaibo Huang, Shuhan Xia, Miaoxuan Zhang, Yueying Zou, and Ran He. Jailbreak attacks and defenses against multimodal generative models: A survey. arXiv preprint arXiv:2411.09259, 2024b. Patel and Qiu [2025] Gaurav Patel and Qiang Qiu. Learning to unlearn while retaining: Combating gradient conflicts in machine unlearning, 2025. URL https://arxiv.org/abs/2503.06339. Pham et al. [2024] Minh Pham, Kelly O. Marshall, Niv Cohen, Govind Mittal, and Chinmay Hegde. Circumventing concept erasure methods for text-to-image generative models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=ag3o2T51Ht. Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL https://arxiv.org/abs/2307.01952. Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821–8831. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/ramesh21a.html. Rando et al. [2022] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610, 2022. Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022. Rosati et al. [2024] Domenic Rosati, Jan Wehner, Kai Williams, Ł ukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 12636–12676. Curran Associates, Inc., 2024. Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22500–22510, June 2023. Schramowski et al. [2023] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522–22531, 2023. Shan et al. [2023] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y. Zhao. Glaze: protecting artists from style mimicry by text-to-image models. In Proceedings of the 32nd USENIX Conference on Security Symposium, SEC ’23, USA, 2023. USENIX Association. ISBN 978-1-939133-37-3. Tian et al. [2024] Bozhong Tian, Xiaozhuan Liang, Siyuan Cheng, Qingbin Liu, Mengru Wang, Dianbo Sui, Xi Chen, Huajun Chen, and Ningyu Zhang. To forget or not? towards practical knowledge unlearning for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1524–1537, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.82. URL https://aclanthology.org/2024.findings-emnlp.82/. Wang et al. [2024] Yiming Wang, Jiahao Chen, Qingming Li, Xing Yang, and Shouling Ji. Aeiou: A unified defense framework against nsfw prompts in text-to-image models, 2024. URL https://arxiv.org/abs/2412.18123. Wu et al. [2024] Xiaoyu Wu, Jiaru Zhang, Yang Hua, Bohan Lyu, Hao Wang, Tao Song, and Haibing Guan. Exploring diffusion models’ corruption stage in few-shot fine-tuning and mitigating with bayesian neural networks, 2024. URL https://arxiv.org/abs/2405.19931. Wu et al. [2025] Yongliang Wu, Shiji Zhou, Mingzhuo Yang, Lianzhe Wang, Heng Chang, Wenbo Zhu, Xinting Hu, Xiao Zhou, and Xu Yang. Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient. Proceedings of the AAAI Conference on Artificial Intelligence, 39(8):8496–8504, Apr. 2025. doi: 10.1609/aaai.v39i8.32917. URL https://ojs.aaai.org/index.php/AAAI/article/view/32917. Xu et al. [2023] Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. Machine unlearning: A survey. ACM Comput. Surv., 56(1), August 2023. ISSN 0360-0300. doi: 10.1145/3603620. URL https://doi.org/10.1145/3603620. Yang et al. [2024] Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Jailbreaking text-to-image generative models. In 2024 IEEE Symposium on Security and Privacy (SP), pages 897–912, 2024. doi: 10.1109/SP54263.2024.00123. Yoon et al. [2025] Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, and Mohit Bansal. SAFREE: Training-free and adaptive guard for safe text-to-image and video generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=hgTFotBRKl. Zhang et al. [2024a] Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1755–1764, June 2024a. Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018. Zhang et al. [2024b] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images… for now. In European Conference on Computer Vision, pages 385–403. Springer, 2024b. Zheng and Yeh [2024] Amber Yijia Zheng and Raymond A Yeh. Imma: Immunizing text-to-image models against malicious adaptation. In European Conference on Computer Vision, pages 458–475. Springer, 2024. Zhou et al. [2024] Shiji Zhou, Lianzhe Wang, Jiangnan Ye, Yongliang Wu, and Heng Chang. On the limitations and prospects of machine unlearning for generative ai. ArXiv, abs/2408.00376, 2024. zxbsmk [2024] zxbsmk. Nsfw-t2i. https://huggingface.co/datasets/zxbsmk/NSFW-T2I, 2024.

Instant research discovery