Paper deep dive
Spectral Saliency for Machine Unlearning
Cedar Site Bai, Amber Yijia Zheng, Raymond A. Yeh, Brian Bullins
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/22/2026, 2:27:23 AM
Summary
The paper introduces Spectral Saliency Unlearning (SSU), a method for machine unlearning that applies singular value thresholding to the unlearning gradient. Inspired by the Muon optimizer, SSU suppresses weak spectral components (small singular values) which are theoretically shown to correspond to directions with high 'forget-retain' conflict, thereby improving the trade-off between forgetting specific data and retaining model utility. Experiments on image classifiers, diffusion models, and LLMs demonstrate SSU's effectiveness.
Entities (9)
Relation Signals (7)
Spectral Saliency Unlearning → usestechnique → singular value thresholding
confidence 100% · SSU thresholds weak singular components and updates only those directions supported by a confident unlearning signal.
Spectral Saliency Unlearning → evaluatedon → CIFAR-10
confidence 95% · Experiments across image classifiers... demonstrate SSU's effectiveness... SSU reduced the average gap by 30.6% on CIFAR-10 classification
Spectral Saliency Unlearning → inspiredby → Muon
confidence 95% · Inspired by Muon, we adopt the spectral view for unlearning and propose Spectral Saliency Unlearning (SSU).
Spectral Saliency Unlearning → mitigates → Forget-Retain Conflict
confidence 95% · suppressing them can mitigate utility degradation... characterizing how weak directions relate to forget-retain interference.
SalUn → analogueof → signSGD
confidence 90% · we view the recent unlearning method SalUn [12] as an analogue of SignSGD.
Spectral Saliency Unlearning → evaluatedon → DDPM
confidence 90% · achieving perfect unlearning efficacy with DDPM
Spectral Saliency Unlearning → evaluatedon → TOFU
confidence 90% · consistently improves the utility-forgetting trade-off on LLM unlearning, yielding an average gain of 0.0125 on the TOFU benchmark.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, using gradient-based updates to reduce the influence of a forget-set by counteracting the previously learned behavior. Recently, Muon, a gradient descent variant, has been introduced. Muon applies spectral magnitude normalization to encourage exploration of rare directions and demonstrates promising performance. Inspired by Muon, we adopt the spectral view for unlearning and propose Spectral Saliency Unlearning (SSU). SSU thresholds weak singular components and updates only those directions supported by a confident unlearning signal. We further provide theoretical justification for this thresholding approach from the perspective of the forgetting-retention trade-off. Experiments across image classifiers, diffusion models, and LLMs demonstrate SSU's effectiveness.
Tags
Links
- Source: https://arxiv.org/abs/2608.15548v1
- Canonical: https://arxiv.org/abs/2608.15548v1
Trouble viewing inline? Open PDF directly →
Full Text
89,544 characters extracted from source content.
Expand or collapse full text
Spectral Saliency for Machine Unlearning Cedar Site Bai Thanks: Equal contribution Amber Yijia Zheng Raymond A. Yeh Brian BullinsDepartment of Computer SciencePurdue Universitybai123, zheng709, rayyeh, bbullins@purdue.edu Abstract Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, using gradient-based updates to reduce the influence of a forget-set by counteracting the previously learned behavior. Recently, Muon, a gradient descent variant, has been introduced. Muon applies spectral magnitude normalization to encourage exploration of rare directions and demonstrates promising performance. Inspired by Muon, we adopt the spectral view for unlearning and propose Spectral Saliency Unlearning (SSU). SSU thresholds weak singular components and updates only those directions supported by a confident unlearning signal. We further provide theoretical justification for this thresholding approach from the perspective of the forgetting-retention trade-off. Experiments across image classifiers, diffusion models, and LLMs demonstrate SSU’s effectiveness. 1 Introduction Machine unlearning (MU) studies how to remove the influence of specific training data from a trained model without resorting to full retraining [7]. There is an increasing need for MU to satisfy data-governance requirements, such as the General Data Protection Regulation (GDPR) [11]. Recent methods focus on approximate unlearning, which aims to efficiently reduce the influence of the forget set through a few gradient-based updates [12, 45, 24]. These methods typically optimize a forgetting objective that increases the model’s loss on the forget set or pushes a model’s predictions toward misclassifications or randomization. In this work, we conceptualize MU as the partial “reversal” of the optimization process. In the deep learning era, learning can be viewed as using stochastic gradient descent (SGD) to incrementally accumulate the influence of training examples into the model parameters. Correspondingly, approximate unlearning can be viewed as undoing the effect of gradient descent on the designated forget set. UnrollingSGD [45] provides one concrete instantiation, analyzing unlearning through the lens of the SGD training dynamics. Motivated by the central role of SGD in deep learning, it is natural to ask how variants of SGD that alter the geometry and scaling of updates translate into the design of unlearning updates. A canonical variant is SignSGD [4], which replaces the stochastic gradient g with its coordinate-wise sign, i.e., sign() ( g ), thereby equalizing update magnitudes across coordinates and emphasizing directional information over scale. For matrix-structured parameters [19, 47], recent optimizers such as Muon [26] extend this magnitude normalization beyond coordinates by applying the matrix sign in the spectral domain of layer gradients. This design is motivated in part by the idea that exploring “rare”, weakly expressed directions can be beneficial for learning [26]. The approach is based on analyses of deep learning dynamics showing that training progresses along singular directions in order of singular value magnitude. Smaller singular values associated with weaker and more specific structure are learned later [27, 38, 31]. From this perspective, we view the recent unlearning method SalUn [12] as an analogue of SignSGD. Instead of uniformizing coordinate magnitudes, SalUn thresholds low-magnitude coordinates of the unlearning gradient. As SignSGD and Muon are both SGD-variants, where Muon promotes rare directions by equalizing magnitudes in the gradient spectrum, one might ask whether unlearning could similarly benefit from suppressing weak spectral components. We hypothesize that these weak components may encode interactions where forgetting and retention are entangled. This intuition is formalized through the concept of forgetting-retention conflict in Sec. 4. We propose Spectral Saliency Unlearning (SSU), a singular value thresholding method that suppresses weak spectral components of the unlearning gradient and can be plugged into general gradient-based unlearning methods. We justify SSU from the perspective of the forgetting-retention trade-off, characterizing how weak directions relate to forget-retain interference. We then explain why suppressing them can mitigate utility degradation, i.e., the loss of model performance on data not in the forget set. In addition, we extend the same perspective to coordinate-wise masking and offer a theoretical explanation for the efficacy of SalUn-style unlearning, which has largely been used as a heuristic. Empirically, we evaluate SSU across diverse applications, including the unlearning of image classifiers, diffusion models for image generation [12], and large language models [52]. Results demonstrate consistent improvements when SSU is used with existing MU objectives. Specifically, SSU reduced the average gap by 30.6% on CIFAR-10 classification, achieving perfect unlearning efficacy with DDPM while enhancing the generation quality by 23.6% on retained classes, and consistently improves the utility-forgetting trade-off on LLM unlearning, yielding an average gain of 0.0125 on the TOFU benchmark. Our main contributions are as follows: • Inspired by recent advances in optimization, we introduce spectral saliency for matrix gradients and propose SSU, a novel singular value thresholding approach for gradient-based unlearning. • We provide a theoretical justification for saliency-based thresholding from the forgetting-retention trade-off perspective, and extend the analysis to explain the efficacy of SalUn-style coordinate masking. • Extensive experiments demonstrate the effectiveness of SSU across a range of models, including image classifiers, diffusion models, and LLM unlearning. 2 Preliminaries Unlearning formulation. We consider a general gradient-based approximate unlearning framework [45, 36, 17, 29, 16]. A pre-trained model is updated to reduce the influence of a given forget set f D_f from the training data D while preserving utility on the complementary retain set r≔∖f D_r D D_f. Let W denote model parameters. We define an unlearning objective ℒu()=ℒf(,f)+ℒr(,r) L_u( W)= L_f( W; D_f)+ L_r( W; D_r) (1) composed of a forgetting loss ℒf(,f) L_f( W; D_f) and a retaining loss ℒr(,r) L_r( W; D_r), which are optimized jointly via gradient-based updates to minimize Eq. (1). We further denote the gradient update ∇ℒu()∇ L_u ( W ) at each step as u=f+r G_u= G_f+ G_r, where f=∇ℒf(,f) G_f=∇ L_f( W; D_f) and r=∇ℒr(,r) G_r=∇ L_r( W; D_r). Muon and matrix sign. Muon [26] is a recently proposed optimizer for matrix-parameterized functions. It orthogonalizes the layer gradient by the projection o=argmin‖−‖F:⊤or⊤=. G_o= _ O \ G- O _F: O O\ or\ O O = I \. (2) This operation is equivalent to applying the matrix sign in the gradient’s spectral domain [9], i.e., for =diag() G= U ( σ ) V, then we have msign()≔diag(sign()) ( G ) U ( ( σ ) ) V. We therefore view Muon as a matrix counterpart of SignSGD. 3 Method Algorithm 1 Spectral Saliency Unlearning (SSU) 0: Pre-trained parameters 0 W_0; unlearning objective ℒu()=ℒf()+ℒr() L_u( W)= L_f( W)+ L_r( W), learning rate η, keep ratio γ∈(0,1]γ∈(0,1] 1: for t=0,1,…,T−1t=0,1,…,T-1 do 2: u,t←∇ℒu(t) G_u,t← _ W L_u( W_t). 3: for each matrix-structured gradient u,t(ℓ) G_u,t ( ) do 4: Compute SVD: u,t(ℓ)=diag()⊤ G_u,t ( )= U ( σ) V 5: Set k←⌊γm⌋k← γ m for ∈ℝm σ ^m. 6: ~u,t(ℓ)←[:,1:k]diag([1:k])[:,1:k]⊤ G_u,t^( )← U_[:,1:k]\, ( σ_[1:k])\, V_[:,1:k] 7: end for 8: t+1←t−η~u,t W_t+1← W_t-η G_u,t 9: end for 9: T W_T For a gradient-based unlearning algorithm that minimizes the objective given in Eq. (1), our proposed SSU replaces the base unlearning gradient u G_u with a singular value thresholded counterpart that suppresses weak spectral components and retains only the dominant singular directions for the update. The full procedure is summarized in Alg. 1. In the following, we elaborate on this procedure by first defining spectral saliency and then describing the singular value thresholding step. 3.1 Spectral Saliency We define spectral saliency by measuring the magnitude of the unlearning update u G_u along the singular directions of the layer gradient. For a matrix-structured parameter W, let the unlearning gradient’s SVD be denoted as u=diag()⊤with=[σ1,⋯,σm]⊤, G_u= U\, ( σ)\, V \;\;with\;\; σ= [ _1,·s, _m ] , (3) where σ1≥⋯≥σm≥0 _1≥·s≥ _m≥ 0, so that each rank-one component σiii⊤ _i\, u_i v_i represents the update along the singular direction pair (i,i)( u_i, v_i). Note, we omit layer and block indices for readability. The spectral saliency of direction i is defined as its singular magnitude σi _i, which quantifies how strongly the unlearning objective drives updates along this direction. At a high-level, we view that larger saliency corresponds to a stronger and more reliable unlearning signal, whereas small singular magnitudes indicate weakly supported directions that are less confident for unlearning and may be susceptible to interference between forgetting and retention, an intuition we later formally justify in Sec. 4. 3.2 Singular Value Thresholding To suppress these weak directions, we apply singular value thresholding [6] to u G_u. Given a threshold τ≥0τ≥ 0, we define the thresholded gradient ~u≔diag(⊙[≥τ])⊤,with[≥τ](i)=1if σi≥τ0otherwise. G_u U ( σ _ [ σ≥τ ] ) V ,\;\;with\;\;I_ [ σ≥τ ](i)= cases1&if _i≥τ\\ 0&otherwise\\ cases. (4) Here, [≥τ]I_ [ σ≥τ ] is a vector-valued indicator function such that its coordinate where singular values below τ are set to zero while those above are preserved. Equivalently, since the singular values are already sorted, we may retain only the leading components and discard the tail. Different from coordinate-wise saliency methods, e.g., SalUn [12], which constructs a coordinate selection mask using dataset specific computations (e.g., based on the forget set) either offline or online, SVD provides an intrinsic ordering of singular values. That is, the most salient directions correspond naturally to the leading singular components. In practice, we use a fixed keep ratio γ∈(0,1]γ∈(0,1] and set k=⌊γm⌋k= γ m , retaining the top γ fraction of singular components. We then reconstruct ~u≔∑i=1kσi⊤=[:,1:k]diag([1:k])[:,1:k]⊤ G_u _i=1^k _i\, u_i v_i = U_[:,1:k]\, ( σ_[1:k])\, V_[:,1:k] (5) and proceed with standard gradient-based updates using this thresholded gradient. For vector- and scalar-structured parameters (e.g., biases), we resort to coordinate-wise thresholding as in SalUn, mirroring Muon’s practice of handling these parameters with vector-based optimizers rather than a spectral update. 4 Theoretical Justification Our analysis adopts a directional view of the unlearning update, characterizing when and why suppressing weakly supported directions can improve unlearning from the perspective of the forgetting-retention trade-off. We first develop this justification for spectral thresholding on matrix-structured parameters, then derive an analogous interpretation for coordinate-wise masking. 4.1 Why Spectral Thresholding Helps Motivated by the Muon perspective of operating in the gradient’s spectral domain, and by analyses linking smaller singular values to weaker, more specific structure [27, 38, 31], we take the reverse stance for unlearning. Weak spectral components tend to be the least reliable directions to update for unlearning, as they may reflect forget-retain entanglement and thereby worsen the forgetting-retention trade-off. Accordingly, we use the singular basis of the unlearning gradient to explain why suppressing these components via spectral thresholding can improve unlearning. We start by examining how the forgetting gradient and the retaining gradient contribute to and interact within the unlearning gradient. We consider the compact SVD of the unlearning gradient, retaining only non-zero singular values: u=ediag(e)e⊤=∑i=1reσiii⊤,σ1≥⋯≥σre>0 G_u= U_e ( σ_e)\, V_e = _i=1^r_e _i\, u_i v_i , _1≥·s≥ _r_e>0 (6) where rer_e denotes the effective rank; exact rank in theory and significantly non-zero singular values in practice. We refer to ℰ=span(e)E=span( V_e) as the effective subspace of update directions induced by u G_u. In this subspace, each singular direction i v_i defines a canonical one-dimensional update mode with a magnitude σi _i. To understand how the forgetting and retention gradients interact within the effective subspace, we introduce the notion of forget-retain alignment, which measures the directional alignment between the projected forgetting and retention gradients. Definition 4.1. (Forget-Retain Alignment) We defined the alignment between the forgetting and retention directions of i v_i as a(i)=⟨fi,ri⟩‖fi‖2‖ri‖2. a ( v_i )= G_f v_i,\, G_r v_i G_f v_i _2 G_r v_i _2. (7) Using this notion, Prop. 4.2 shows that when the forgetting and retaining gradients exhibit strong negative alignment in the effective subspace, this indicates that there is significant conflict between the forgetting and retention objectives and thus cannot be simultaneously improved along the same update direction, yielding an unavoidable trade-off. Proposition 4.2. (Informal) Assume ℒf L_f and ℒr L_r are smooth. For unlearning gradient u=η∑i=1reσiii⊤ G_u=η _i=1^r_e _i u_i v_i , consider a single-direction update along the ithi^th singular component i v_i: ′=−ησiii⊤ W = W-η _i u_i v_i . There exist constants a0<0a_0<0 and η0>0 _0>0 such that if the alignment a(i)<a0a ( v_i )<a_0, and the step size η<η0η< _0, then the forgetting progress Δℒf=ℒf(′)−ℒf() L_f= L_f ( W )- L_f ( W ) and the retaining progress Δℒr=ℒr(′)−ℒr() L_r= L_r ( W )- L_r ( W ) satisfies ΔℒfΔℒr<0 L_f L_r<0. That is, along any singular direction i v_i where the retain and forget gradients are in significant conflict, any update that improves one objective must necessarily worsen the other. We refer the readers to Appx. A.1 for the formal statement, assumptions, and proof of this proposition. Next, we show how this conflict is reflected in the spectrum of the unlearning gradient. Proposition 4.3. (a) If the direction carries a nontrivial forget/retain signal, i.e., ‖fi‖+‖ri‖≥ξ G_f v_i + G_r v_i ≥ξ for ξ>0ξ>0 and its singular value is small relative to this signal, i.e., σi≤ρξ _i≤ρξ for ρ∈(0,12)ρ∈ (0, 1 2 ), then its forget-retain alignment satisfies a(i)≤2ρ2−1<0a ( v_i )≤ 2ρ^2-1<0. (b) (Informal) Assume the projections of f G_f and r G_r onto the effective subspace of u G_u have bounded spectral disparity. For two directions i,j v_i, v_j with alignment scores a(i)≤−δia ( v_i )≤- _i and a(j)≥δja ( v_j )≥ _j where δi,δj∈(0,1] _i, _j∈(0,1], if the separation (δi+δj) ( _i+ _j ) exceeds a gap threshold, then σi<σj _i< _j. We refer the readers to Appx. A.2 for the assumption of spectral disparity, the formal statement, and proof. Prop. 4.3 (a)(a) shows that as long as there is a nontrivial forgetting or retention signal, small singular values correspond to directions in which the forgetting and retaining gradients are negatively aligned. If a direction carries neither forgetting nor retention signal, i.e., ‖fi‖=‖ri‖=0 G_f v_i = G_r v_i =0, then i v_i lies in the null space of u G_u, which is not affected by singular value thresholding. Prop. 4.3 (b)(b) complementarily shows that, between sufficiently positively aligned directions and negatively aligned ones, the former attain larger singular values. Together with Prop. 4.2, this analysis shows that by suppressing small singular components of u G_u, SSU preferentially removes directions that are more likely to exhibit strong forget-retain conflict, thereby mitigating unnecessary retention degradation while preserving the dominant unlearning signal. 4.2 Justification for SalUn-Style Masking We further extend the same directional analysis to SalUn-style coordinate thresholding, which operates on the unlearning gradient in the standard coordinate basis. Each coordinate i∈[d]i∈[d] defines a one-dimensional update direction, and the unlearning gradient decomposes as gu,i=gf,i+gr,ig_u,i=g_f,i+g_r,i. A key regime of interest is when a coordinate carries a nontrivial forgetting or retention signal (otherwise updates along that coordinate are negligible), yet the magnitude |gu,i||g_u,i| is small, suggesting cancellation between forgetting and retention effects along that coordinate. This explains why SalUn’s heuristic of suppressing small-magnitude coordinates can be beneficial: updating along such weakly supported coordinates is likely to couple forgetting progress with retention degradation, as characterized by the following proposition whose proof can be found in Appx. A.3. Proposition 4.4. Assume ℒf L_f and ℒr L_r are βf _f- and βr _r-smooth. Consider any coordinate i∈[d]i∈[d] such that the forgetting/retention signal is nontrivial, i.e., ∀i∈[d]∀ i∈[d], |gf,i|+|gr,i|≥ξ g_f,i + g_r,i ≥ξ where ξ>0ξ>0, and the combined unlearning gradient is small, i.e., |gu,i|≤ρξ g_u,i ≤ρξ for some ρ∈(0,1]ρ∈(0,1]. For the coordinate-wise update ′=−ηgu,i w = w-η g_u,i with step size η<min(2|gf,i|βf|gu,i|,2|gr,i|βr|gu,i|)η< ( 2 g_f,i _f g_u,i , 2 g_r,i _r g_u,i ), the induced progress Δℒf=ℒf(′)−ℒf() L_f= L_f ( w )- L_f ( w ) and Δℒr=ℒr(′)−ℒr() L_r= L_r ( w )- L_r ( w ) satisfies ΔℒfΔℒr<0 L_f L_r<0. That is to say, the unlearning update necessarily improves one objective while worsening the other, among forgetting and retention. To our knowledge, existing saliency-masking methods are primarily motivated empirically. The proposition above offers a nontrivial theoretical justification from the forgetting-retention trade-off perspective. 4.3 Limitations and Discussion Our theoretical results justify SSU as a principled mechanism for mitigating forget-retain conflict by suppressing weak spectral components. At the same time, the theory is intended as a characterization of the forgetting-retention trade-off rather than a complete quantitative prediction of end-to-end unlearning performance. Extending the analysis to weaker assumptions and deriving tighter performance-level guarantees remain important directions for future work. A natural variant of SSU would be to threshold directions by forget-retain alignment. However, this requires separately computing f G_f and r G_r, projecting both onto the singular directions of u G_u, and then evaluating their directional inner products, introducing nontrivial computation overhead. Singular values, in contrast, are obtained directly from u G_u and, supported by Prop. 4.3, serve as a principled proxy for forget-retain conflict. While a more fine-grained, alignment-aware criterion may further improve thresholding, we focus on the simple, broadly applicable choice of singular value thresholding. 5 Experiments We conduct experiments across three applications spanning over image classification (Sec. 5.1), image generation (Sec. 5.2), and language modeling (Sec. 5.3) following the benchmarks proposed by existing MU works. 5.1 Random Subset Unlearning in Image Classification Setup and evaluation. Following the setup by 12, we focus on random subset unlearning in image classification tasks using the CIFAR-10 dataset. We employ ResNet-18 [19] as our architecture and compare our method against three simple baselines: fine-tuning (FT) [50], gradient ascent (GA) [45], influence unlearning (IU) [24], and five competitive baselines: ℓ1 _1-sparse [25], SCRUB [30], SSD [15], SFRON [23], and SalUn [12]. Following SalUn’s setup, we regard the retrained oracle as the gold standard of unlearning. We report four evaluation metrics: forgetting set accuracy (FA, lower is better) to measure unlearning efficacy, remaining set accuracy (RA) and test set accuracy (TA) to assess preserved generalization, and the membership inference attack (MIA) [12] success rate on the forgetting set as a privacy metric. Finally, we report the average gap between each method and the retrained oracle model across the four metrics as the overall performance. Implementation details are provided in Appx. B.1. Results. In Tab. 1, we summarize the results for random subset unlearning on CIFAR-10 with ResNet-18, where 10% of the training data is forgotten. Table 1: Random unlearning of ResNet-18 on CIFAR-10, for 10% random data forgetting. Methods FA ↓ RA ↑ TA ↑ MIA ↑ Avg. Gap ↓ Time ↓ Retrain 94.86 100.00 94.14 12.86 0.00 43.29 FT 99.30 99.91 94.44 2.82 3.72 2.37 GA 98.90 99.23 93.83 1.90 4.02 0.13 IU 99.44 99.53 94.72 0.10 4.60 3.22 ℓ1 _1-sparse 95.81 97.74 91.59 9.84 2.20 2.36 SCRUB 99.39 99.76 93.91 3.69 3.54 1.88 SSD 94.46 94.86 88.28 7.80 4.12 2.78 SFRON 99.32 99.96 94.74 1.98 4.00 1.90 SalUn 96.62 99.46 93.44 14.28 1.11 2.61 + SSU 96.24 99.26 93.39 12.66 0.77 2.62 Most baseline methods face a fundamental trade-off: techniques such as FT, GA, IU, SCRUB, and SFRON yield high FA, indicating insufficient forgetting, while methods like SSD that achieve lower FA suffer large drops in RA and TA, compromising model utility. Among the baselines, SalUn achieves the best overall performance with an average gap of 1.11. Our method improves upon SalUn, achieving the lowest average gap of 0.77 while maintaining comparable performance across all metrics. Notably, although SSU employs SVD, it applies only to convolution layers, resulting in negligible computational overhead during unlearning. 5.2 Class-wise Unlearning in Image Generation Setup and evaluation. Following the setup in [12], we evaluate our method on class-wise unlearning for image generation using DDPM [21] on CIFAR-10 and Stable Diffusion V1.4 (SD) [39] on Imagenette [22]. We compare against four baselines: SA [20], ESD [17], SFRON [23], and SalUn [12]. For DDPM, SVD is applied only to the convolution layers of the denoising U-Net, while for Stable Diffusion, SVD is applied only to the cross-attention layers of the U-Net. We evaluate unlearning quality using the FID [42] score to measure generation quality on retained classes, and forgetting set accuracy (FA) of unlearned classes via a pre-trained classifier. Implementation details are provided in Appx. B.2. Results on DDPM. Tab. 2 presents class-wise unlearning results for DDPM on CIFAR-10 across five classes. Table 2: Class-wise unlearning of image generation on CIFAR-10 with DDPM. Method CIFAR-10 Class-wise Unlearning Automobile Cat Dog Horse Truck FA↓ FID↓ FA↓ FID↓ FA↓ FID↓ FA↓ FID↓ FA↓ FID↓ SA 0.00 23.56 14.20 21.34 8.60 21.19 0.00 21.13 0.00 29.04 SFRon 0.00 20.70 7.40 18.44 0.20 18.89 0.00 19.93 0.00 20.61 SalUn 0.20 21.23 1.40 20.29 0.00 20.18 0.60 20.70 0.80 20.45 + SSU 0.00 15.46 1.00 16.01 0.00 15.19 0.00 15.60 0.00 16.32 While all methods achieve near-perfect forgetting, they differ significantly in generation quality. Baseline methods yield FID scores ranging from 18.44 to 29.04, indicating moderate image quality on retained classes. Our method achieves substantial improvements, with FID scores between 15.19 and 16.32 across all classes, which is a 25-30% reduction compared to baselines. Notably, SSU maintains perfect forgetting on all classes while delivering the best generation quality, with the most significant improvements on Automobile, Cat, and Truck. These results show that our approach can effectively unlearn target data while preserving and even improving the model’s generative capabilities. While the method requires SVD computation during training, we note that the method has minimal overhead. For U-Net architectures, gradients are reshaped to moderate-sized 2D matrices (typically ≤512×512≤ 512× 512) for SVD, which can be efficiently decomposed using optimized GPU implementations. In practice, our method takes approximately 40 seconds per 100 training steps on one NVIDIA L40S, nearly identical to SalUn, while providing better unlearning through spectral filtering. Results on SD. Tab. 3 shows class-wise unlearning performance on Stable Diffusion [39] with the Imagenette dataset [22]. We generate 300 images per class for the computation of both FA and FID. All methods achieve strong forgetting performance with average FA below 0.30%, confirming effective removal of target concepts. In terms of generation quality, ESD achieves an average FID of 1.71, while SalUn improves this to 1.55. Table 3: Class-wise unlearning of image generation on ImageNette with SD. Forget Class ESD SalUn + SSU FA (↓ ) FID (↓ ) FA (↓ ) FID (↓ ) FA (↓ ) FID (↓ ) Tench 0.00 2.37 0.00 0.91 0.00 0.76 English Springer 0.00 1.70 0.00 0.92 0.00 1.17 Cassette Player 0.00 1.34 0.67 1.30 0.67 1.16 Chain Saw 0.00 1.48 0.00 1.38 0.00 1.55 Church 2.33 2.56 0.00 1.73 0.00 1.65 French Horn 0.00 1.61 0.00 1.36 0.00 1.14 Garbage Truck 0.00 1.57 0.00 1.01 0.00 1.00 Gas Pump 0.00 1.34 0.00 2.78 0.00 1.48 Golf Ball 0.67 1.35 1.33 2.44 1.33 1.12 Parachute 0.00 1.75 0.00 1.68 0.00 1.35 Average 0.30 1.71 0.20 1.55 0.20 1.24 Our method further advances the state-of-the-art with an average FID of 1.24, achieving the best overall performance. SSU demonstrates consistent improvements across most classes, with larger gains on Tench, French Horn, Gas Pump, and Golf Ball. Moreover, SSU achieves slightly better forgetting with an average FA of 0.20% compared to 0.30% for ESD, matching SalUn’s 0.20%. These results validate the generalizability of our approach across different diffusion architectures and demonstrate its effectiveness in balancing unlearning efficacy with generation quality. tench springer cassette chain saw church French horn garbage truck gas pump golf ball parachute Unlearn Retain Figure 1: Qualitative results of unlearning and retaining images across Imagenette classes using the StableDiffusion Model. The top block shows generated samples for each class when it is targeted for unlearning, while the bottom block displays samples for the same classes when they are retained. Fig. 1 shows qualitative results for class-wise unlearning on Stable Diffusion with the Imagenette dataset. We observe that SSU effectively removes the model’s ability to generate images corresponding to the unlearned classes, as evidenced by the lack of recognizable features on those classes, e.g., tench turns into a dog. At the same time, the generations of retained classes show high-fidelity and remain semantically accurate. This demonstrates that our approach preserves generative quality for non-targeted concepts. These results visually verify the effectiveness of our method in selectively unlearning specific classes while maintaining overall model utility. 5.3 Random Subset Unlearning in LLM Going beyond unlearning on computer vision tasks, we now evaluate SSU for unlearning LLM, demonstrating the general applicability of the method across models and tasks. Experimental setup and evaluation. Following the setup by 52, we evaluate SSU on LLM unlearning using the TOFU benchmark [35] with Llama2-7B [46]. The TOFU benchmark consists of three scenarios: forget01, forget05, and forget10, corresponding to forgetting 1%, 5%, and 10% of the training data, respectively. We consider two types of unlearning tasks: untargeted unlearning, including gradient-based methods (GA+GD, GA+KL), preference-based methods (NPO+GD, NPO+KL), and knowledge manipulation methods (ME+GD); and targeted unlearning, where we evaluate on preference-based methods (DPO+GD, DPO+KL) and knowledge manipulation methods (IDK+GD, IDK+AP). The baselines are adopted from 52. SSU is built on top of the best-performing baseline for each task. As in prior work [52], we evaluate performance using two aggregated metrics: Model Utility (MU) and Forget Efficacy (FE). These metrics combine multiple complementary signals, including ROUGE, prediction probability, truth ratio, token entropy, semantic similarity, and entailment score, to jointly assess lexical accuracy, semantic consistency, generation quality, and factual correctness. MU is computed on the retain set using the harmonic mean to measure overall utility preservation, while FE is computed on the forget set as one minus the arithmetic mean (excluding token entropy) to quantify the strength of forgetting. We further report their average as a single summary score. Implementation details are provided in Appx. B.3. Results. Tab. 4 presents random subset unlearning on Llama2-7B across three forgetting scenarios. We apply SSU on top of the best baseline methods for each unlearning type. For untargeted unlearning, ME+GD emerges as the strongest baseline, significantly outperforming gradient-based (GA) and preference-based (NPO) alternatives. Applying SSU on top of ME+GD further enhances performance. SSU improves model utility for forget01 from 0.7245 to 0.7477 and forget10 from 0.7312 to 0.7480 while simultaneously improving forget efficacy across all scenarios, demonstrating superior balance between preserving model capabilities and effective unlearning. For targeted unlearning, IDK+AP serves as the best baseline with consistent performance. Building upon IDK+AP, SSU achieves the best results with average scores of 0.7856, 0.7562, and 0.7475 for the three scenarios. The improvements are particularly noticeable for forget01, where SSU enhances both model utility and forget efficacy. These results demonstrate that SSU generalizes effectively to large language models, consistently improving state-of-the-art unlearning methods for both untargeted and targeted scenarios across different forgetting ratios. Table 4: Results of different unlearning methods on the TOFU benchmark with Llama2-7B. MU and FE represent Model Utility and Forget Efficacy, respectively, and we indicate the best results in bold. Dataset Metric Method GA+GD GA+KL NPO+GD NPO+KL ME+GD + SSU DPO+GD DPO+KL IDK+GD IDK+AP + SSU forget01 MU 0.6671 0.6385 0.6402 0.7404 0.7245 0.7477 0.7554 0.7601 0.6704 0.7579 0.7769 FE 0.5935 0.6028 0.6137 0.4943 0.9156 0.9599 0.5260 0.3013 0.7700 0.7625 0.7943 Avg. 0.6303 0.6206 0.6269 0.6174 0.8201 0.8538 0.6407 0.5307 0.7202 0.7602 0.7856 forget05 MU 0.2913 0.0000 0.5718 0.5470 0.7521 0.7277 0.0000 0.4536 0.0000 0.7522 0.7515 FE 0.9135 0.8927 0.6967 0.6192 0.9262 0.9407 0.8242 0.7831 0.7948 0.7480 0.7608 Avg. 0.6024 0.4463 0.6342 0.5831 0.8391 0.8342 0.4121 0.6183 0.3974 0.7501 0.7562 forget10 MU 0.5001 0.0000 0.5660 0.4904 0.7312 0.7480 0.0000 0.0000 0.0527 0.7444 0.7463 FE 0.9449 0.9484 0.7530 0.7449 0.9505 0.9557 0.8043 0.8346 0.7602 0.7432 0.7487 Avg. 0.7225 0.4742 0.6595 0.6177 0.8409 0.8519 0.4022 0.4173 0.4065 0.7438 0.7475 5.4 Ablation Studies Effect of the keep ratio. To investigate the impact of the keep ratio on unlearning performance, Table 5: Ablating top singular value ratio on random unlearning 10% data on CIFAR-10. Keep Ratio FA ↓ RA ↑ TA ↑ MIA ↑ 0.10 98.06 99.67 94.00 10.36 0.30 96.24 99.26 93.39 12.66 0.50 94.68 97.94 91.81 14.90 0.60 95.46 98.56 92.74 14.64 0.70 95.50 98.45 92.74 14.42 0.90 94.54 97.69 91.96 13.98 we conduct an ablation study by varying the ratio parameter in our method for random subset unlearning on CIFAR-10. As shown in Tab. 5, increasing the ratio generally improves forgetting efficacy, but excessively high values can degrade remaining and test accuracy, indicating a trade-off between unlearning strength and model utility. Our default setting of ratio=0.3ratio=0.3 achieves the best balance, confirming the importance of careful ratio selection. Impact of SVD-applied weight selection. We additionally ablate the effect of applying SVD to different subsets of weights in the U-Net architecture for text-to-image class-wise unlearning. Tab. 6 reports results on Stable Diffusion with Imagenette, comparing SVD applied to all layers, only cross-attention layers, only convolutional layers, and only MLP layers. We observe that restricting SVD to cross-attention layers achieves the best trade-off between forgetting and generation quality. Table 6: Ablating different modules adapting SVD on class-wise unlearning on Stable Diffusion. Conv MLP Full XAttn FA ↓ 0.17 0.23 0.23 0.20 FID ↓ 1.29 1.29 1.28 1.24 In contrast, applying SVD to all layers slightly degrades generation quality, likely because it perturbs low-level visual features and global denoising dynamics that are less directly related to semantic concepts. This suggests that effective unlearning requires targeted modification of concept-carrying components rather than uniformly across the entire network. These results motivate our design choice of selectively applying SVD to cross-attention layers, which maximizes unlearning efficacy while preserving overall generative fidelity. 6 Related Work Forgetting-retention trade-off and spectral structure. The forgetting-retention trade-off is a central consideration in machine unlearning [43]. As updates that remove forget-set influence can degrade retained performance, unlearning inherits the same interference phenomenon studied as catastrophic forgetting in continual learning [32]. Many methods, therefore, use SVD-based projection to project the unlearning update onto a subspace orthogonal to an estimated retain subspace [49, 33, 5, 13, 8], following parameter-isolation strategies from continual learning [14, 3, 40]. SSU differs from these approaches in both object and mechanism. It operates directly on the singular basis of the unlearning gradient and suppresses weak spectral components. This yields an optimizer-agnostic mechanism that targets forget-retain interaction directly, rather than constraining updates solely through an estimated retain subspace. Related work also studies forget-retain conflict via Euclidean gradient alignment between the forgetting and retaining gradients [37, 48, 1], whereas SSU analyzes their interaction in the spectral basis of the unlearning gradient and uses the resulting spectrum as a structured proxy for interference. In addition, [44] applies SVD-based low-rank parameterization for efficient updates, while SSU uses singular values as a saliency signal for thresholding. Saliency-based MU. Existing saliency-based unlearning selectively updates a subset of parameters deemed most responsible for forget-set behavior. Motivated by evidence that model sparsity improves unlearning [41], SalUn [12] computes a coordinate-wise saliency map from the forget-set gradient evaluated with the pretrained model, and thresholds low-magnitude coordinates to sparsify unlearning updates. Building on SalUn, [10] further argues via a fine-tuning analysis that saliency should instead be derived from retained data to better preserve overlapping features and mitigate the forgetting-retention trade-off. Unlike coordinate masking, SSU defines spectral saliency using the singular values of the unlearning gradient, selecting dominant matrix-valued update directions that reflect signals from both forgetting and retention. [23] develops a unified view of saliency by decomposing the unlearning objective into a forgetting term, a retention term, and an explicit weight-saliency matrix that modulates the unlearning direction. Overall, while saliency-driven sparsification is empirically effective, it remains largely a heuristic. Our analysis offers a complementary theoretical explanation for why saliency-based masking can improve the forgetting-retention trade-off. General MU has been explored through a range of optimization-based approaches, including gradient ascent methods that increase the forget-set loss [45, 36], influence-function-based approximations to leave-one-out retraining [24, 28], and Fisher-information-based selective updates [18, 2]. More recently, practical approximate unlearning has increasingly relied on fine-tuning [50, 18, 30], particularly for large models where full retraining is prohibitive. These techniques have been developed across modalities, including image classification models [25, 12], generative image models such as diffusion models [17, 53, 20, 51], and LLMs [54, 52, 34]. Despite their differences, many existing methods can be expressed under a common template that jointly optimizes a forgetting loss and a retention or utility-preserving loss [12, 52, 55]. SSU can be used as a plug-in enhancement to methods of this structure, as we have demonstrated in Sec. 5. 7 Conclusion Motivated by a spectral view of gradient-based optimization, we proposed Spectral Saliency Unlearning (SSU), which applies singular-value thresholding to suppress weak spectral components of the unlearning gradient and can be used as a drop-in enhancement for gradient-based unlearning pipelines. We provided a theoretical justification for SSU from the perspective of the forgetting-retention trade-off, characterizing how weak directions relate to forget-retain interference and suppressing them can mitigate utility degradation. We further extended this perspective to coordinate-wise thresholding, offering a nontrivial explanation for the empirical effectiveness of SalUn-style masking. Empirically, we evaluated SSU across diverse unlearning settings, including image classification, diffusion-model unlearning, and large language model unlearning. Across these settings, SSU consistently demonstrates its effectiveness in unlearning and improves the forgetting-utility trade-off. References [1] S. Asif and M. M. Amiri (2026) OFMU: optimization-driven framework for machine unlearning. In Proc. ICLR, Cited by: §6. [2] A. Becker and T. Liebig (2022) Evaluating machine unlearning via epistemic uncertainty. arXiv preprint arXiv:2208.10836. Cited by: §6. [3] M. A. Bennani, T. Doan, and M. Sugiyama (2020) Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006.11942. Cited by: §6. [4] J. Bernstein, Y. Wang, K. Azizzadenesheli, and A. Anandkumar (2018) SignSGD: compressed optimisation for non-convex problems. In Proc. ICML, Cited by: §1. [5] S. D. Biswas, A. Roy, and K. Roy (2025) CURE: concept unlearning via orthogonal representation editing in diffusion models. In Proc. NeurIPS, Cited by: §6. [6] J. Cai, E. J. Candès, and Z. Shen (2010) A singular value thresholding algorithm for matrix completion. SIOPT. Cited by: §3.2. [7] Y. Cao and J. Yang (2015) Towards making systems forget with machine unlearning. In IEEE symposium on security and privacy, Cited by: §1. [8] H. Chen, T. Zhu, X. Yu, and W. Zhou (2024) Machine unlearning via null space calibration. In Proc. IJCAI, Cited by: §6. [9] L. Chen, J. Li, and Q. Liu (2025) Muon optimizes under spectral norm constraints. arXiv preprint arXiv:2506.15054. Cited by: §2. [10] M. Ding, R. Sharma, C. Chen, J. Xu, and K. Ji (2025) Understanding fine-tuning in approximate unlearning: a theoretical perspective. TMLR. Cited by: §6. [11] European Union (2016) Regulation (EU) 2016/679 of the european parliament and of the council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (general data protection regulation). Note: OJ L 119, 4.5.2016, p. 1–88 External Links: Link Cited by: §1. [12] C. Fan, J. Liu, Y. Zhang, E. Wong, D. Wei, and S. Liu (2024) SalUn: empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In Proc. ICLR, Cited by: §1, §1, §1, §3.2, §5.1, §5.1, §5.2, §6, §6, §6. [13] J. Fang, H. Jiang, K. Wang, Y. Ma, J. Shi, X. Wang, X. He, and T. Chua (2025) AlphaEdit: null-space constrained model editing for language models. In Proc. ICLR, Cited by: §6. [14] M. Farajtabar, N. Azizan, A. Mott, and A. Li (2020) Orthogonal gradient descent for continual learning. In Proc. AISTATS, Cited by: §6. [15] J. Foster, S. Schoepf, and A. Brintrup (2024) Fast machine unlearning without retraining through selective synaptic dampening. In Proc. AAAI, Cited by: §5.1. [16] R. Gandikota, S. Feucht, S. Marks, and D. Bau (2025) Erasing conceptual knowledge from language models. Cited by: §2. [17] R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau (2023) Erasing concepts from diffusion models. In Proc. ICCV, Cited by: §2, §5.2, §6. [18] A. Golatkar, A. Achille, and S. Soatto (2020) Eternal sunshine of the spotless net: selective forgetting in deep networks. In Proc. CVPR, Cited by: §6. [19] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proc. CVPR, Cited by: §1, §5.1. [20] A. Heng and H. Soh (2023) Selective amnesia: a continual learning approach to forgetting in deep generative models. In Proc. NeurIPS, Cited by: §5.2, §6. [21] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Proc. NeurIPS, Cited by: §5.2. [22] Imagenette: a smaller subset of 10 easily classified classes from imagenet External Links: Link Cited by: §5.2, §5.2. [23] Z. Huang, X. Cheng, J. Zheng, H. Wang, Z. He, T. Li, and X. Huang (2024) Unified gradient-based machine unlearning with remain geometry enhancement. In Proc. NeurIPS, Cited by: §5.1, §5.2, §6. [24] Z. Izzo, M. A. Smart, K. Chaudhuri, and J. Zou (2021) Approximate data deletion from machine learning models. In Proc. AISTATS, Cited by: §1, §5.1, §6. [25] J. Jia, J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu (2023) Model sparsification can simplify machine unlearning. In Proc. NeurIPS, Cited by: §5.1, §6. [26] K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cecista, L. Newhouse, and J. Bernstein (2024) Muon: an optimizer for hidden layers in neural networks. URL https://kellerjordan. github. io/posts/muon. Cited by: §1, §2. [27] M. Kleinman, A. Achille, and S. Soatto (2024) Critical learning periods emerge even in deep linear networks. In Proc. ICLR, Cited by: §1, §4.1. [28] P. W. Koh and P. Liang (2017) Understanding black-box predictions via influence functions. In Proc. ICML, Cited by: §6. [29] N. Kumari, B. Zhang, S. Wang, E. Shechtman, R. Zhang, and J. Zhu (2023) Ablating concepts in text-to-image diffusion models. In Proc. ICCV, Cited by: §2. [30] M. Kurmanji, P. Triantafillou, J. Hayes, and E. Triantafillou (2023) Towards unbounded machine unlearning. In Proc. NeurIPS, Cited by: §5.1, §6. [31] A. K. Lampinen and S. Ganguli (2019) An analytic theory of generalization dynamics and transfer learning in deep linear networks. In Proc. ICLR, Cited by: §1, §4.1. [32] X. Li, Y. Zhou, T. Wu, R. Socher, and C. Xiong (2019) Learn to grow: a continual structure learning framework for overcoming catastrophic forgetting. In Proc. ICML, Cited by: §6. [33] S. Lin, X. Zhang, W. Susilo, X. Chen, and J. Liu (2024) GDR-GMA: machine unlearning via direction-rectified and magnitude-adjusted gradients. In ACM Multimedia, Cited by: §6. [34] S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao, C. Y. Liu, X. Xu, H. Li, et al. (2025) Rethinking machine unlearning for large language models. Nature Machine Intelligence. Cited by: §6. [35] P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter (2024) Tofu: a task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. Cited by: §5.3. [36] S. Neel, A. Roth, and S. Sharifi-Malvajerdi (2021) Descent-to-delete: gradient-based methods for machine unlearning. In Proc. ALT, Cited by: §2, §6. [37] G. Patel and Q. Qiu (2025) Learning to unlearn while retaining: combating gradient conflicts in machine unlearning. In Proc. CVPR, Cited by: §6. [38] N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville (2019) On the spectral bias of neural networks. In Proc. ICML, Cited by: §1, §4.1. [39] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proc. CVPR, Cited by: §5.2, §5.2. [40] G. Saha, I. Garg, and K. Roy (2021) Gradient projection memory for continual learning. In Proc. ICLR, Cited by: §6. [41] Y. Savani, A. Trockman, Z. Feng, Y. E. Xu, A. Schwarzschild, A. Robey, M. Finzi, and J. Z. Kolter (2025) Antidistillation sampling. Proc. NeurIPS. Cited by: §6. [42] M. Seitzer (2020) pytorch-fid: FID Score for PyTorch. Note: Version 0.3.0https://github.com/mseitzer/pytorch-fid Cited by: §5.2. [43] A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh (2021) Remember what you want to forget: algorithms for machine unlearning. In Proc. NeurIPS, Cited by: §6. [44] M. Sendera, Ł. Struski, K. Książek, K. Musiol, J. Tabor, and D. D. Rymarczyk (2025) SEMU: singular value decomposition for efficient machine unlearning. In Proc. ICML, Cited by: §6. [45] A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot (2022) Unrolling SGD: understanding factors influencing machine unlearning. In Proc. EuroS&P, Cited by: §1, §1, §2, §5.1, §6. [46] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §5.3. [47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Proc. NeurIPS, Cited by: §1. [48] Q. Wang, J. P. Zhou, Z. Zhou, S. Shin, B. Han, and K. Q. Weinberger (2025) Rethinking LLM unlearning objectives: a gradient perspective and go beyond. In Proc. ICLR, Cited by: §6. [49] Y. Wang, Q. Wang, F. Liu, W. Huang, Y. Du, X. Du, and B. Han (2025) GRU: mitigating the trade-off between unlearning and retention for LLMs. In Proc. ICML, Cited by: §6. [50] A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck (2021) Machine unlearning of features and labels. In Proc. NDSS, Cited by: §5.1, §6. [51] J. Wu, T. Le, M. Hayat, and M. Harandi (2025) Erasing undesirable influence in diffusion models. In Proc. CVPR, Cited by: §6. [52] X. Yuan, T. Pang, C. Du, K. Chen, W. Zhang, and M. Lin (2025) A closer look at machine unlearning for large language models. In Proc. ICLR, Cited by: §1, §5.3, §5.3, §6, §6. [53] G. Zhang, K. Wang, X. Xu, Z. Wang, and H. Shi (2024) Forget-me-not: learning to forget in text-to-image diffusion models. In Proc. CVPR, Cited by: §6. [54] R. Zhang, L. Lin, Y. Bai, and S. Mei (2024) Negative preference optimization: from catastrophic collapse to effective unlearning. In Proc. COLM, Cited by: §6. [55] X. Zhong, H. Luo, and C. Liu (2025) DualOptim: enhancing efficacy and stability in machine unlearning with dual optimizers. In Proc. NeurIPS, Cited by: §6. Appendix The appendix is organized as follows: • In Sec. A, we provide the formal assumptions, statements, and complete proofs for the Theorems stated in the main paper. • In Sec. B, we provide additional experiment details. The code will be open-sourced upon the acceptance of this paper. • In Sec. C, we provide additional image generation results after unlearning with SSU. Appendix A Proof of the Theoretical Justification A.1 Formal Statement and Proof of Proposition 4.2 For the unlearning objective ℒu=ℒf+ℒr L_u= L_f+ L_r composed of a forgetting loss ℒf L_f and a retaining loss ℒr L_r, we have for their gradients u=f+r G_u= G_f+ G_r. The SVD of u G_u is given by u=diag()=∑i=1mσiii⊤ G_u= U ( σ ) V= _i=1^m _i u_i v_i . Let re=rank(Gu)r_e=rank (G_u ). Then we have ∀i∈[re]∀ i∈[r_e], σi>0 _i>0 and ∀i∈re+1,⋯,m∀ i∈ \r_e+1,·s,m \, σi=0 _i=0. In practice, in the context of effective rank, we let the former represent significant non-zero singular values and the latter include those that are approximately zero. We have compact SVD u=ediag(e)e⊤ G_u= U_ediag ( σ_e ) V_e where e∈ℝm×re U_e ^m× r_e and e∈ℝn×re V_e ^n× r_e. Assumption A.1 (Smoothness). For constants βf,βr>0 _f, _r>0, the forgetting loss ℒf L_f and the retaining loss ℒr L_r are smooth, i.e., for W, ′ W , |ℒf(′)−ℒf()−⟨f(),′−⟩|≤βf2‖′−‖2, L_f ( W )- L_f ( W )- G_f ( W ),\, W - W ≤ _f2 W - W ^2, |ℒr(′)−ℒr()−⟨r(),′−⟩|≤βr2‖′−‖2. L_r ( W )- L_r ( W )- G_r ( W ),\, W - W ≤ _r2 W - W ^2. See 4.1 Lemma A.2. If a(i)<−min‖fi‖,‖ri‖max‖fi‖,‖ri‖a ( v_i )<- \ G_f v_i , G_r v_i \ \ G_f v_i , G_r v_i \, then for fi=⟨f,ii⊤⟩f_i= G_f,\, u_i v_i , ri=⟨r,ii⊤⟩r_i= G_r,\, u_i v_i , we have firi<0f_ir_i<0. Proof. By definition, fi=⟨f,ii⊤⟩=i⊤fi, f_i= G_f,\, u_i v_i = u_i G_f v_i, ri=⟨r,ii⊤⟩=i⊤ri r_i= G_r,\, u_i v_i = u_i G_r v_i Since fi+ri=ui=∑j=1reσjjj⊤i=σii G_f v_i+ G_r v_i= G_u v_i= _j=1^r_e _j u_j v_j v_i= _i u_i, we know i=fi+riσi u_i= G_f v_i+ G_r v_i _i. Therefore, fi=i⊤fi=(fi+ri)⊤fiσi=‖fi‖22+⟨ri,fi⟩σi. f_i= u_i G_f v_i= ( G_f v_i+ G_r v_i ) G_f v_i _i= G_f v_i _2^2+ G_r v_i,\, G_f v_i _i. Similarly, ri=‖ri‖22+⟨ri,fi⟩σir_i= G_r v_i _2^2+ G_r v_i,\, G_f v_i _i. If a(i)=⟨fi,ri⟩‖fi‖2‖ri‖2<−min‖fi‖,‖ri‖max‖fi‖,‖ri‖a ( v_i )= G_f v_i,\, G_r v_i G_f v_i _2 G_r v_i _2<- \ G_f v_i , G_r v_i \ \ G_f v_i , G_r v_i \, without the loss of generality, we assume ‖fi‖>‖ri‖ G_f v_i > G_r v_i , then ⟨fi,ri⟩‖fi‖2‖ri‖2<−‖ri‖fi‖ G_f v_i,\, G_r v_i G_f v_i _2 G_r v_i _2<- G_r v_i G_f v_i yields ⟨fi,ri⟩<−‖ri‖2 G_f v_i,\, G_r v_i <- G_r v_i ^2. Also, we have by Cauchy-Schwarz inequality, ⟨fi,ri⟩≥−‖fi‖2‖ri‖2>−‖fi‖22. G_f v_i,\, G_r v_i ≥- G_f v_i _2 G_r v_i _2>- G_f v_i _2^2. Therefore, we know ⟨fi,ri⟩+‖ri‖2<0 G_f v_i,\, G_r v_i + G_r v_i ^2<0 and ⟨fi,ri⟩+‖fi‖22>0 G_f v_i,\, G_r v_i + G_f v_i _2^2>0. As a result, we have firi=(‖fi‖22+⟨ri,fi⟩)(‖ri‖22+⟨ri,fi⟩)σi2<0. f_ir_i= ( G_f v_i _2^2+ G_r v_i,\, G_f v_i ) ( G_r v_i _2^2+ G_r v_i,\, G_f v_i ) _i^2<0. ∎ Lemma A.3. For gradient u=η∑i=1reσiii⊤ G_u=η _i=1^r_e _i u_i v_i , the update in direction i v_i is given by ′=−ησiii⊤ W = W-η _i u_i v_i . For fi=⟨f,ii⊤⟩f_i= G_f,\, u_i v_i , ri=⟨r,ii⊤⟩r_i= G_r,\, u_i v_i , η<mini2|fi|βfσi,2|ri|βrσiη< _i \ 2 f_i _f _i, 2 r_i _r _i \, forgetting loss ℒf L_f, and retaining loss ℒr L_r, we have (a) ℒf(′)−ℒf()<0 L_f ( W )- L_f ( W )<0 implies fi>0f_i>0, and fi<0f_i<0 implies ℒf(′)−ℒf()>0 L_f ( W )- L_f ( W )>0. (b) ℒr(′)−ℒr()<0 L_r ( W )- L_r ( W )<0 implies ri>0r_i>0, and ri<0r_i<0 implies ℒr(′)−ℒr()>0 L_r ( W )- L_r ( W )>0. Proof. (a) We prove by contradiction. Suppose when ℒf(′)−ℒf()<0 L_f ( W )- L_f ( W )<0, fi≤0f_i≤ 0. For the update ′=+Δ W = W+ W where Δ=−ησiii⊤ W=-η _i u_i v_i , we have ‖Δ‖=ησi W =η _i. By Assumption A.1, we have ℒf(′)−ℒf() L_f ( W )- L_f ( W ) ≥⟨f,Δ⟩−βf2‖Δ‖2 ≥ G_f,\, W - _f2 W ^2 =−ησi⟨f,ii⊤⟩−βf2η2σi2 =-η _i G_f,\, u_i v_i - _f2η^2 _i^2 =−ησifi−βf2η2σi2 =-η _if_i- _f2η^2 _i^2 >0, >0, where the last inequality follows from η<2|fi|βfσiη< 2 f_i _f _i. This contradicts the condition ℒf(′)−ℒf()<0 L_f ( W )- L_f ( W )<0. Therefore, when ℒf(′)−ℒf()<0 L_f ( W )- L_f ( W )<0, we must have fi>0f_i>0. In addition, from the derivation above, we know fi=⟨f,ii⊤⟩≤0f_i= G_f,\, u_i v_i ≤ 0 implies ℒf(′)−ℒf()>0 L_f ( W )- L_f ( W )>0. (b) Suppose when ℒr(′)−ℒr()<0 L_r ( W )- L_r ( W )<0, ri≤0r_i≤ 0. By Assumption A.1, we have ℒr(′)−ℒr() L_r ( W )- L_r ( W ) ≥⟨r,Δ⟩−βr2‖Δ‖2 ≥ G_r,\, W - _r2 W ^2 =−ησi⟨r,ii⊤⟩−βr2η2σi2 =-η _i G_r,\, u_i v_i - _r2η^2 _i^2 =−ησiri−βr2η2σi2 =-η _ir_i- _r2η^2 _i^2 >0, >0, where the last inequality follows from η<2|ri|βrσiη< 2 r_i _r _i. This contradicts the condition ℒr(′)−ℒr()<0 L_r ( W )- L_r ( W )<0. Therefore, when ℒr(′)−ℒr()<0 L_r ( W )- L_r ( W )<0, we must have ri>0r_i>0. In addition, from the derivation above, we know ri=⟨f,ii⊤⟩≤0r_i= G_f,\, u_i v_i ≤ 0 implies ℒr(′)−ℒr()>0 L_r ( W )- L_r ( W )>0. ∎ Proposition 4.4. For gradient of the unlearning objective u=η∑i=1reσiii⊤ G_u=η _i=1^r_e _i u_i v_i , the update in direction i v_i is given by ′=−ησiii⊤ W = W-η _i u_i v_i . Under Assumption A.1, if a(i)<−min‖fi‖,‖ri‖max‖fi‖,‖ri‖a ( v_i )<- \ G_f v_i , G_r v_i \ \ G_f v_i , G_r v_i \, and η<mini2|fi|βfσi,2|ri|βrσiη< _i \ 2 f_i _f _i, 2 r_i _r _i \ for fi=⟨f,ii⊤⟩f_i= G_f,\, u_i v_i and ri=⟨r,ii⊤⟩r_i= G_r,\, u_i v_i , then we have (a) if Δℒf=ℒf(′)−ℒf()<0 L_f= L_f ( W )- L_f ( W )<0 then Δℒr=ℒr(′)−ℒr()>0 L_r= L_r ( W )- L_r ( W )>0; (b) if Δℒr<0 L_r<0, then Δℒf>0 L_f>0. That is, along any singular direction i v_i where the retain and forget gradients are in significant conflict, any update that improves one objective must necessarily worsen the other. Proof. (a) If the update makes progress in forgetting, that is, Δℒf=ℒf(′)−ℒf()<0 L_f= L_f ( W )- L_f ( W )<0, then by Lemma A.3 (a), we know fi>0f_i>0. Also, from Lemma A.2, we know that if the retain and forget gradients are in significant conflict in the effective space, i.e., a(i)<−min‖fi‖,‖ri‖max‖fi‖,‖ri‖a ( v_i )<- \ G_f v_i , G_r v_i \ \ G_f v_i , G_r v_i \, we have firi<0f_ir_i<0. Since fi>0f_i>0, we must have ri<0r_i<0. Then by Lemma A.3 (b), we have ℒr(′)−ℒr()>0 L_r ( W )- L_r ( W )>0, meaning that the effort for retention is damaged. (b) If the update makes progress in retention, that is, Δℒr=ℒr(′)−ℒr()<0 L_r= L_r ( W )- L_r ( W )<0, then by Lemma A.3 (b), we know ri>0r_i>0. Again, by Lemma A.2, we know if a(i)<−min‖fi‖,‖ri‖max‖fi‖,‖ri‖a ( v_i )<- \ G_f v_i , G_r v_i \ \ G_f v_i , G_r v_i \, we have firi<0f_ir_i<0. Since ri>0r_i>0, we must have fi<0f_i<0. Then by Lemma A.3 (a), we have ℒf(′)−ℒf()>0 L_f ( W )- L_f ( W )>0, meaning that unlearning is not making progress in forgetting. ∎ A.2 Formal Statement and Proof of Proposition 4.3 For compact SVD u=ediag(e)e⊤ G_u= U_ediag ( σ_e ) V_e where e∈ℝm×re U_e ^m× r_e and e∈ℝn×re V_e ^n× r_e, we call the space spanned by e V_e the effective subspace, ℰ=span(e)E=span ( V_e ), and the corresponding projection =ee⊤ P= V_e V_e . Consider the Gram matrix of r G_r and f G_f projected to the effective subspace r=r⊤r∈ℝre×re M_r= P G_r G_r P ^r_e× r_e and f=f⊤f∈ℝre×re M_f= P G_f G_f P ^r_e× r_e, their eigen-decompositions are defined as f=∑j=1reλf,jf,jf,j⊤ M_f= _j=1^r_e _f,j e_f,j e_f,j , r=∑j=1reλr,jr,jr,j⊤ M_r= _j=1^r_e _r,j e_r,j e_r,j . Assumption A.5 (Bounded Spectral Disparity). r M_r and f M_f exhibit bounded spectral disparity. Specifically, there exist constants αf−<1<αf+,αr−<1<αr+ _f^-<1< _f^+, _r^-<1< _r^+ but close enough to 11 such that for the maximum, minimum, and average eigenvalues of r,f M_r, M_f, αf−λf≤λf≤αf+λf, _f^- _f max≤ _f avg≤ _f^+ _f min, αr−λr≤λr≤αr+λr. _r^- _r max≤ _r avg≤ _r^+ _r min. Bounded spectral disparity formalizes that the projected retain and forget signals are bulk-distributed on the effective space ℰE, and the spectrum within ℰE is not so heavy-tailed that ⊤r v M_r v or ⊤f v M_f v varies by orders of magnitude across admissible directions. This is a natural regime for subspace-based unlearning, as ℰE is defined from the joint gradient u G_u, so it filters out directions where the update signal is negligible and where task-specific curvature can be extremely ill-conditioned. Empirically, the projected spectra within ℰE are typically far less heavy-tailed than in the full parameter space. Lemma A.6. Under Assumption A.5, for ϵ=max1αf−1,1−1αf+,1αr−1,1−1αr+ε= \ 1 _f^--1,1- 1 _f^+, 1 _r^--1,1- 1 _r^+ \, we have ∀i∈[re]∀ i∈[r_e], (1−ϵ)λf≤i⊤fi≤(1+ϵ)λf, (1-ε ) _f avg≤ v_i M_f v_i≤ (1+ε ) _f avg, (1−ϵ)λr≤i⊤ri≤(1+ϵ)λr. (1-ε ) _r avg≤ v_i M_r v_i≤ (1+ε ) _r avg. Proof. Given that f=∑j=1reλf,jf,jf,j⊤ M_f= _j=1^r_e _f,j e_f,j e_f,j , we have i⊤fi=∑j=1reλf,j(⟨f,j,i⟩)2≤λf∑j=1re(⟨f,j,i⟩)2≤λf v_i M_f v_i= _j=1^r_e _f,j ( e_f,j,\, v_i )^2≤ _f max _j=1^r_e ( e_f,j,\, v_i )^2≤ _f max as f,j e_f,j forms an orthonormal basis and i v_i is a unit vector so that ∑j=1re(⟨f,j,i⟩)2=1 _j=1^r_e ( e_f,j,\, v_i )^2=1. Similarly, we have λf≤i⊤fi≤λf, _f min≤ v_i M_f v_i≤ _f max, λr≤i⊤ri≤λr. _r min≤ v_i M_r v_i≤ _r max. Given Assumption A.5, we know λr≤1αf−λr _r max≤ 1 _f^- _r avg and by definition, ϵ≥1αf−1ε≥ 1 _f^--1, therefore i⊤fi≤λf≤1αf−λr≤(1+ϵ)λr. v_i M_f v_i≤ _f max≤ 1 _f^- _r avg≤ (1+ε ) _r avg. Similarly, we can show i⊤fi≥(1−ϵ)λf v_i M_f v_i≥ (1-ε ) _f avg, and (1−ϵ)λr≤i⊤ri≤(1+ϵ)λr (1-ε ) _r avg≤ v_i M_r v_i≤ (1+ε ) _r avg. ∎ Proposition 4.7. (a) If the direction carries a nontrivial forget/retain signal, i.e., ‖fi‖+‖ri‖≥ξ G_f v_i + G_r v_i ≥ξ for ξ>0ξ>0 and its singular value is small relative to this signal, i.e., σi≤ρξ _i≤ρξ for ρ∈(0,12)ρ∈ (0, 1 2 ), then its forget-retain alignment satisfies a(i)≤2ρ2−1<0a ( v_i )≤ 2ρ^2-1<0. (b) Under Assumption A.5, for two directions i,j v_i, v_j, i,j∈[re]i,j∈[r_e], with alignment scores a(i)≤−δia ( v_i )≤- _i and a(j)≥δja ( v_j )≥ _j where δi,δj∈(0,1] _i, _j∈(0,1], if the separation (δi+δj)>ϵ1−ϵ⋅s2+1s ( _i+ _j )> ε1-ε· s^2+1s for ϵ=max1αf−1,1−1αf+,1αr−1,1−1αr+ε= \ 1 _f^--1,1- 1 _f^+, 1 _r^--1,1- 1 _r^+ \ where s=‖r‖F‖f‖Fs= G_r P _F G_f P _F, then σi<σj _i< _j. Proof. (a) Given that u=∑i=1reσiii⊤ G_u= _i=1^r_e _i u_i v_i , we have for the singular value σi2 _i^2 =‖ui‖22 = G_u v_i _2^2 =‖(f+r)i‖22 = ( G_f+ G_r ) v_i _2^2 =‖fi‖22+‖ri‖22+2⟨fi,ri⟩ = G_f v_i _2^2+ G_r v_i _2^2+2 G_f v_i,\, G_r v_i =‖fi‖22+‖ri‖22+2‖fi‖2‖ri‖2a(i) = G_f v_i _2^2+ G_r v_i _2^2+2 G_f v_i _2 G_r v_i _2a ( v_i ) where the last equality follows from Definition 4.1. Also, we know from the conditions that σi≤ρξ≤ρ(‖fi‖+‖ri‖). _i≤ρξ≤ρ ( G_f v_i + G_r v_i ). As a result, we have ‖fi‖22+‖ri‖22+2‖fi‖2‖ri‖2a(i)=σi2≤ρ2(‖fi‖+‖ri‖)2. G_f v_i _2^2+ G_r v_i _2^2+2 G_f v_i _2 G_r v_i _2a ( v_i )= _i^2≤ρ^2 ( G_f v_i + G_r v_i )^2. Rearranging the terms yields 2‖fi‖2‖ri‖2a(i)≤(ρ2−1)(‖fi‖2+‖ri‖2)+2ρ2‖fi‖2‖ri‖2.2 G_f v_i _2 G_r v_i _2a ( v_i )≤ (ρ^2-1 ) ( G_f v_i ^2+ G_r v_i ^2 )+2ρ^2 G_f v_i _2 G_r v_i _2. Dividing 2‖fi‖2‖ri‖22 G_f v_i _2 G_r v_i _2 on both sides, a(i)≤ρ2+ρ2−12(‖fi‖2‖ri‖+‖ri‖fi‖2)≤2ρ2−1, a ( v_i )≤ρ^2+ ρ^2-12 ( G_f v_i _2 G_r v_i + G_r v_i G_f v_i _2 )≤ 2ρ^2-1, where the last inequality follows from the facts that (‖fi‖2‖ri‖+‖ri‖fi‖2)≥2 ( G_f v_i _2 G_r v_i + G_r v_i G_f v_i _2 )≥ 2 and ρ2−1≤0ρ^2-1≤ 0. This completes the proof. (b) From the proof of (1), we know σi2=‖fi‖22+‖ri‖22+2‖fi‖2‖ri‖2a(i). _i^2= G_f v_i _2^2+ G_r v_i _2^2+2 G_f v_i _2 G_r v_i _2a ( v_i ). Furthermore, ‖fi‖22 G_f v_i _2^2 =i⊤f⊤fi = v_i G_f G_f v_i =i⊤f⊤fi = v_i P G_f G_f P v_i =i⊤fi = v_i M_f v_i where we used the fact that i=i P v_i= v_i as =ee⊤ P= V_e V_e and i v_i is one column of e V_e. Therefore, by Lemma A.6 we know for ‖fi‖22 G_f v_i _2^2 and similarly for ‖ri‖22 G_r v_i _2^2 (1−ϵ)λf≤‖fi‖22≤(1+ϵ)λf, (1-ε ) _f avg≤ G_f v_i _2^2≤ (1+ε ) _f avg, (1−ϵ)λr≤‖ri‖22≤(1+ϵ)λr. (1-ε ) _r avg≤ G_r v_i _2^2≤ (1+ε ) _r avg. As a result, we can have for direction i v_i with singular value σi _i and conflict a(i)≤−δi<0a ( v_i )≤- _i<0, σi2 _i^2 =‖fi‖22+‖ri‖22+2‖fi‖2‖ri‖2a(i) = G_f v_i _2^2+ G_r v_i _2^2+2 G_f v_i _2 G_r v_i _2a ( v_i ) ≤(1+ϵ)λf+(1+ϵ)λr+(1−ϵ)λf(1−ϵ)λra(i) ≤ (1+ε ) _f avg+ (1+ε ) _r avg+ (1-ε ) _f avg (1-ε ) _r avga ( v_i ) ≤(1+ϵ)(λf+λr)−2(1−ϵ)δiλfλr. ≤ (1+ε ) ( _f avg+ _r avg )-2 (1-ε ) _i _f avg _r avg. And for direction j v_j with singular value σj _j and conflict a(j)≥δj>0a ( v_j )≥ _j>0, σj2 _j^2 =‖fj‖22+‖rj‖22+2‖fj‖2‖rj‖2a(j) = G_f v_j _2^2+ G_r v_j _2^2+2 G_f v_j _2 G_r v_j _2a ( v_j ) ≥(1−ϵ)λf+(1−ϵ)λr+(1−ϵ)λf(1−ϵ)λra(j) ≥ (1-ε ) _f avg+ (1-ε ) _r avg+ (1-ε ) _f avg (1-ε ) _r avga ( v_j ) ≥(1−ϵ)(λf+λr+2δjλfλr). ≥ (1-ε ) ( _f avg+ _r avg+2 _j _f avg _r avg ). Given the condition of a sufficient gap in conflict (δi+δj)>ϵ1−ϵ⋅s2+1s ( _i+ _j )> ε1-ε· s^2+1s for s=‖r‖F‖f‖F=λrλfs= G_r P _F G_f P _F= _r avg _f avg, multiplying both sides by 2(1−ϵ)λfλr2 (1-ε ) _f avg _r avg yields 2(1−ϵ)(δi+δj)λfλr>2ϵ(λf+λr), 2 (1-ε ) ( _i+ _j ) _f avg _r avg>2ε ( _f avg+ _r avg ), which, by rearranging the terms, is equivalent to (1+ϵ)(λf+λr)−2(1−ϵ)δiλfλr<(1−ϵ)(λf+λr+2δjλfλr). (1+ε ) ( _f avg+ _r avg )-2 (1-ε ) _i _f avg _r avg< (1-ε ) ( _f avg+ _r avg+2 _j _f avg _r avg ). Therefore, we have σi _i ≤(1+ϵ)(λf+λr)−2(1−ϵ)δiλfλr ≤ (1+ε ) ( _f avg+ _r avg )-2 (1-ε ) _i _f avg _r avg <(1−ϵ)(λf+λr+2δjλfλr) < (1-ε ) ( _f avg+ _r avg+2 _j _f avg _r avg ) ≤σj. ≤ _j. ∎ A.3 Theoretical Justification for Salun Assumption A.8 (Smoothness). For constants βf,βr>0 _f, _r>0, the forgetting loss ℒf L_f and the retaining loss ℒr L_r are smooth, i.e., for w, ′ w , |ℒf(′)−ℒf()−⟨f(),′−⟩|≤βf2‖′−‖2, L_f ( w )- L_f ( w )- g_f ( w ),\, w - w ≤ _f2 w - w ^2, |ℒr(′)−ℒr()−⟨r(),′−⟩|≤βr2‖′−‖2. L_r ( w )- L_r ( w )- g_r ( w ),\, w - w ≤ _r2 w - w ^2. See 4.4 Proof. We first show that gf,igr,i<0g_f,ig_r,i<0. Assume that gf,igr,i≥0g_f,ig_r,i≥ 0, then we have |gu,i|=|gf,i|+|gr,i|≥ξ g_u,i = g_f,i + g_r,i ≥ξ, which contradicts the condition |gu,i|≤ρξ g_u,i ≤ρξ for ρ<1ρ<1. Therefore, gf,igr,i<0g_f,ig_r,i<0. Next, by Assumption A.8, −ηgf,igu,i−βf2η2gu,i2,≤ℒf(′)−ℒf()≤−ηgf,igu,i+βf2η2gu,i2, -η g_f,ig_u,i- _f2η^2g_u,i^2,≤ L_f ( w )- L_f ( w )≤-η g_f,ig_u,i+ _f2η^2g_u,i^2, If gf,igu,i>0g_f,ig_u,i>0, then for Δℒf=ℒf(′)−ℒf() L_f= L_f ( w )- L_f ( w ), Δℒf≤−ηgf,igu,i+βf2η2gu,i2=η(ηβf2gu,i2−gf,igu,i)<0 L_f≤-η g_f,ig_u,i+ _f2η^2g_u,i^2=η (η _f2g_u,i^2-g_f,ig_u,i )<0 where the last line follows from η<2|gf,i|βf|gu,i|η< 2 g_f,i _f g_u,i . Also, if gf,igu,i<0g_f,ig_u,i<0, then Δℒf≥−ηgf,igu,i−βf2η2gu,i2=η(|gf,i||gu,i|−ηβf2|gu,i|2)>0 L_f≥-η g_f,ig_u,i- _f2η^2g_u,i^2=η ( g_f,i g_u,i -η _f2 g_u,i ^2 )>0 where the last line again follows from η<2|gf,i|βf|gu,i|η< 2 g_f,i _f g_u,i . As a result, we conclude that sign(Δℒf)=−sign(gf,igu,i) ( L_f )=- (g_f,ig_u,i ). Similarly, we can show for Δℒr=ℒr(′)−ℒr() L_r= L_r ( w )- L_r ( w ) that sign(Δℒr)=−sign(gr,igu,i) ( L_r )=- (g_r,ig_u,i ). Therefore, sign(ΔℒfΔℒr)=sign(gf,igu,i)sign(gf,igu,i)=sign(gf,igr,igu,i2)=sign(gf,igr,i), ( L_f L_r )= (g_f,ig_u,i ) (g_f,ig_u,i )= (g_f,ig_r,ig_u,i^2 )= (g_f,ig_r,i ), which completes the proof. ∎ Appendix B Detailed Experiment Setup B.1 Details of unlearning on image classification. We evaluate random sample unlearning where we randomly select 10% of the training data to forget. This corresponds to 5,000 samples that should be unlearned. The same random seed of 42 is used for selecting forget samples to ensure reproducibility. Each experiment is evaluated on three metrics: retain set accuracy with the remaining 45,000 training samples, forget set accuracy with the 5,000 samples to unlearn, and test set accuracy with all 10,000 test samples. During unlearning, we apply SVD on-the-fly. For computational efficiency, we apply SVD only to convolutional layers, while using pre-computed saliency coordinate masks for fully-connected layers. The unlearning process fine-tunes the model on the 45,000-sample retain set for 10 epochs with the learning rate of 0.013 while applying spectral filtering according to the keep ratio γ=0.30γ=0.30, which controls the proportion of singular values retained after reconstruction and was chosen by grid search, selecting the value that yields the best validation performance. B.2 Details of unlearning on image generation. DDPM on CIFAR-10. We evaluate class-conditional unlearning on CIFAR-10 where we aim to remove the model’s ability to generate images from a specific class. The diffusion model uses a U-Net architecture with 128 base channels, channel multipliers of [1,2,2,2][1,2,2,2], 2 residual blocks per resolution, and self-attention at resolution 16. The diffusion process uses a linear noise schedule with 1000 timesteps, βstart=0.0001 _start=0.0001 and βend=0.02 _end=0.02. During unlearning, we fine-tune the diffusion model for 1000 iterations with batch size 128 using gradient ascent-based unlearning loss. For computational efficiency, we apply on-the-fly SVD reconstruction only to convolutional layers, while using pre-computed saliency coordinate masks for non-convolutional layers. For each convolutional layer, we compute its SVD =⊤W=U V during training and apply eigenvalue-based filtering according to the keep ratio γ=0.50γ=0.50, which controls the proportion of eigenvalues retained. The unlearning uses Adam optimizer with learning rate 1×10−51× 10^-5, remain alpha αremain=1×10−3 _remain=1× 10^-3, and forget alpha αforget=1.0 _forget=1.0. We obtained the hyperparameters by grid search, selecting the value that yields the best validation performance. For evaluation, we generate 5,000 images for each class to compute FID and FA. Stable Diffusion on Imagenette. Stable Diffusion uses a latent diffusion model with a U-Net denoising network operating in the latent space of a pre-trained autoencoder. The U-Net has 320 base channels, channel multipliers of [1,2,4,4][1,2,4,4], 2 residual blocks per resolution, 8 attention heads, and transformer depth of 1 with context dimension 768 for CLIP text conditioning. The diffusion process uses a linear noise schedule with 1000 timesteps, βstart=0.00085 _start=0.00085 and βend=0.012 _end=0.012. During unlearning, we fine-tune the model for 5 epochs with batch size 8 using gradient ascent-based unlearning loss with randomly assigned labels for the forget class. For computational efficiency, we apply on-the-fly SVD reconstruction only to cross attention layers, while using pre-computed saliency coordinate masks for the other layers. For each convolutional layer, we compute its SVD =⊤W=U V during training and apply eigenvalue-based filtering according to the keep ratio γ=0.50γ=0.50, which controls the proportion of eigenvalues retained and was chosen by grid search, selecting the value that yields the best validation performance. The unlearning uses Adam optimizer with learning rate 1×10−51× 10^-5 and classifier-free guidance scale of 7.5. For evaluation, we generate 300 images for each class to compute FID and FA. B.3 Details of unlearning on large language models. During unlearning, we fine-tune the model for 5 epochs with batch size 8 and gradient accumulation steps of 4, resulting in an effective batch size of 32. We experiment with two unlearning objectives: IDK+AP (I-Don’t-Know with answer preservation) and ME+GD (mismatch entropy with gradient difference). For computational efficiency, we apply on-the-fly SVD reconstruction only to attention projection layers (Q, K, V), while using pre-computed saliency coordinate masks for feed-forward and other layers. For attention layers, we compute SVD during training and apply eigenvalue-based filtering according to the QKV mask ratio αqkv=0.20 _qkv=0.20, which controls the proportion of eigenvalues retained. For non-attention layers, we use coordinate masks with mask ratio of 0.10. For IDK+AP, we use a learning rate 3×10−53× 10^-5 with forget coefficient 1.0 and regularization coefficient 1.0. For ME+GD, we use a learning rate 2×10−52× 10^-5 with forget coefficient 0.1 and regularization coefficient 1.0. The top eigenvalue ratio for both methods is 0.2, which was chosen by grid search, selecting the value that yields the best validation performance. All experiments use the AdamW optimizer with weight decay 0.01 and are trained with distributed data parallel across 2 GPUs. Appendix C Additional generation results Figure 2: Examples of generated images using SSU. From the rows below, diagonal images represent the forgetting class, while non-diagonal images represent the remaining class. Unlearned Prompt class class Tench springer Cassette Saw Church French horn truck Gas pump Golf ball Parachute Tench English springer Cassette player Chain saw Church French horn Garbage truck Gas pump Golf ball Parachute Figure 3: Examples of generated images using SSU. From the rows below, diagonal images represent the forgetting class, while non-diagonal images represent the remaining class. Unlearned Prompt class class Tench springer Cassette Saw Church French horn truck Gas pump Golf ball Parachute Tench English springer Cassette player Chain saw Church French horn Garbage truck Gas pump Golf ball Parachute Figure 4: Examples of generated images using SSU. From the rows below, diagonal images represent the forgetting class, while non-diagonal images represent the remaining class. Unlearned Prompt class class Tench springer Cassette Saw Church French horn truck Gas pump Golf ball Parachute Tench English springer Cassette player Chain saw Church French horn Garbage truck Gas pump Golf ball Parachute Fig. 2, Fig. 3, and Fig. 4 present class-wise unlearning results on the Imagenette dataset using the SalUn method, evaluated under different random seeds. Each figure is organized as a matrix, where rows indicate the “Unlearned class” and columns indicate the “Prompt class,” clearly separating the intended unlearning target from the generated outputs. Diagonal images correspond to the class being unlearned, highlighting the effectiveness of SSU in removing specific concepts. Off-diagonal images show generations for other classes, demonstrating the model’s ability to generalize and distinguish among the remaining categories.