Paper deep dive
Benchmarking Unlearning for Vision Transformers
Kairan Zhao, Iurie Luca, Peter Triantafillou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 3:08:04 PM
Summary
This paper presents the first comprehensive benchmark for Machine Unlearning (MU) on Vision Transformers (VTs), specifically ViT and Swin-T architectures. It evaluates the performance of MU algorithms (Fine-tune, NegGrad+, SalUn) and their integration with the RUM framework across various datasets (CIFAR-10, CIFAR-100, SVHN, ImageNet-1K) and protocols (single-shot and continual). The study investigates how VTs memorize data compared to CNNs, validates the use of memorization proxies (Confidence, Holdout Retraining), and establishes performance baselines using unified metrics like ToW and ToW-MIA.
Entities (19)
Relation Signals (12)
ViT → issubtypeof → Vision Transformers
confidence 98% · benchmarking MU algorithm performance in different VT families (ViT and Swin-T)
Swin-T → issubtypeof → Vision Transformers
confidence 98% · benchmarking MU algorithm performance in different VT families (ViT and Swin-T)
Vision Transformers → isalternativeto → CNNs
confidence 95% · Vision Transformers (VTs) emerge as strong alternatives to CNNs.
CIFAR-100 → isusedforbenchmarking → Machine Unlearning
confidence 95% · we present the main results, focusing on CIFAR-100 as a representative dataset.
ToW-MIA → measures → Forget Quality
confidence 95% · ToW-MIA uses differences in MIA vulnerability
ToW → measures → Forget Quality
confidence 95% · ToW uses accuracy differences on the forget set
Confidence → isproxyfor → Memorization
confidence 90% · The Confidence proxy consistently achieves the strongest correlations across all models and datasets
Holdout Retraining → isproxyfor → Memorization
confidence 90% · Holdout Retraining shows moderate but significant positive correlations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Research in machine unlearning (MU) has gained strong momentum: MU is now widely regarded as a critical capability for building safe and fair AI. In parallel, research into transformer architectures for computer vision tasks has been highly successful: Increasingly, Vision Transformers (VTs) emerge as strong alternatives to CNNs. Yet, MU research for vision tasks has largely centered on CNNs, not VTs. While benchmarking MU efforts have addressed LLMs, diffusion models, and CNNs, none exist for VTs. This work is the first to attempt this, benchmarking MU algorithm performance in different VT families (ViT and Swin-T) and at different capacities. The work employs (i) different datasets, selected to assess the impacts of dataset scale and complexity; (ii) different MU algorithms, selected to represent fundamentally different approaches for MU; and (iii) both single-shot and continual unlearning protocols. Additionally, it focuses on benchmarking MU algorithms that leverage training data memorization, since leveraging memorization has been recently discovered to significantly improve the performance of previously SOTA algorithms. En route, the work characterizes how VTs memorize training data relative to CNNs, and assesses the impact of different memorization proxies on performance. The benchmark uses unified evaluation metrics that capture two complementary notions of forget quality along with accuracy on unseen (test) data and on retained data. Overall, this work offers a benchmarking basis, enabling reproducible, fair, and comprehensive comparisons of existing (and future) MU algorithms on VTs. And, for the first time, it sheds light on how well existing algorithms work in VT settings, establishing a promising reference performance baseline.
Tags
Links
- Source: https://arxiv.org/abs/2602.20114v1
- Canonical: https://arxiv.org/abs/2602.20114v1
Trouble viewing inline? Open PDF directly →
Full Text
75,781 characters extracted from source content.
Expand or collapse full text
Benchmarking Unlearning for Vision Transformers Kairan Zhao Iurie Luca Peter Triantafillou Abstract Research in machine unlearning (MU) has gained strong momentum: MU is now widely regarded as a critical capability for building safe and fair AI. In parallel, research into transformer architectures for computer vision tasks has been highly successful: Increasingly, Vision Transformers (VTs) emerge as strong alternatives to CNNs. Yet, MU research for vision tasks has largely centered on CNNs, not VTs. While benchmarking MU efforts have addressed LLMs, diffusion models, and CNNs, none exist for VTs. This work is the first to attempt this, benchmarking MU algorithm performance in different VT families (ViT and Swin-T) and at different capacities. The work employs (i) different datasets, selected to assess the impacts of dataset scale and complexity; (i) different MU algorithms, selected to represent fundamentally different approaches for MU; and (i) both single-shot and continual unlearning protocols. Additionally, it focuses on benchmarking MU algorithms that leverage training data memorization, since leveraging memorization has been recently discovered to significantly improve the performance of previously SOTA algorithms. En route, the work characterizes how VTs memorize training data relative to CNNs, and assesses the impact of different memorization proxies on performance. The benchmark uses unified evaluation metrics that capture two complementary notions of forget quality along with accuracy on unseen (test) data and on retained data. Overall, this work offers a benchmarking basis, enabling reproducible, fair, and comprehensive comparisons of existing (and future) MU algorithms on VTs. And, for the first time, it sheds light on how well existing algorithms work in VT settings, establishing a promising reference performance baseline. Machine Learning, ICML 1 Introduction The high success of Vision Transformers (VTs) brings with it new responsibilities regarding responsible/fair/safe AI. Among these, the ability to remove (the influence of) specific “problematic” data from trained models (a.k.a. machine unlearning) (MU) is critical. Said problematic data may include biased, erroneous, poisoned, obsolete, or privacy-sensitive data. Popular VT architectures represent thus an important frontier in this challenge. On the other hand, recently memorization has been identified as playing a fundamental role for MU (and this holds across modalities). For instance, MU research in LLMs (Barbulescu and Triantafillou, 2024; Jang et al., 2022) and in diffusion models (Ren et al., 2024; Wen et al., 2024) explicitly detects and mitigates memorization. In computer-vision tasks, memorization has been shown to be a key factor affecting unlearning performance (Zhao et al., 2024; Torkzadehmahani et al., 2024). Despite the fact that memorization and MU have been extensively studied in LLMs, text-to-image Diffusion Models, and Convolutional Networks (CNNs), it is an open question whether findings transfer to VTs. This uncertainty stems from key differences in architecture, training regimes, and inductive biases. Compared to LLMs (with which they share a transformer backbone), VTs operate on image patches using global self-attention (unlike LLMs which rely on causal masking or token-order constraints and process semantically meaningful tokens) and VTs are typically (pre)trained using supervised classification objectives (e.g., on ImageNet), while LLMs are trained on large corpora using self-supervised objectives, yielding different types of data exposure and memorization. Compared to CNNs, VTs lack strong spatial inductive biases such as locality (semantically related neighbouring pixels), translation equivariance (for positional reasoning). These biases help CNNs localize and isolate memorization effects, whereas VTs must learn such structure from data alone, making VTs more data-hungry, often necessitating a pretrain–then–finetune training regime. Moreover, unlike LLMs, VTs lack the benefit of language’s syntactic and semantic structure. Hence, learned representations are more entangled and spread across layers and attention heads. The Gap. One can thus reasonably expect the above differences to collectively introduce unique challenges for unlearning in VTs. Said potential challenges remain largely unexplored, despite a few related research works on both the algorithmic side (e.g., Cadet et al. (2024); Cho et al. (2024) whose evaluations of CNN-derived unlearning also include some VT, typically, ViT-Tiny) and on the benchmarking side, with recent efforts systematically evaluating MU across tasks and modalities (such as Maini et al. (2024); Li et al. (2024) for LLMs, Ma et al. (2024); Zhang et al. (2024), and Cheng and Amiri (2024) for text-to-image diffusion models). Focusing on vision, image classification tasks, the first comprehensive attempt in this domain, by Triantafillou et al. (2024), was based on the NeurIPS 2023 MU competition, evaluating and ranking the top algorithms on two datasets from the competition on a CNN (ResNet-18) architecture. Interestingly, they report that MU methods can be brittle across architectures/datasets. Grimes et al. (2024) benchmarked MU on the same CNN environment but focused on evaluating more demanding privacy threats and paid attention to continual MU. Cadet et al. (2024) extended the benchmark in Triantafillou et al. (2024), evaluating 18 MU algorithms on four datasets on a ResNet (as well as on ViT-Tiny). Each of the above benchmarks has a different emphasis: Each is typically designed for a specific architecture-modality-task pairing. Some stress new datasets, while others stress comprehensive/exhaustive examinations/rankings of a large set of MU algorithms. This work. Overall, benchmarking for MU has been addressed for LLMs, diffusion models, and CNNs, but not for VTs. We fill this gap by benchmarking MU on VTs, contributing an evaluation basis along the following key axes: 1. Memorization: Do VTs memorize differently to CNNs? How does this affect unlearning? 2. Proxies: Are CNN-based memorization proxies effective in VTs? Can they improve unlearning? 3. Algorithms: How do fundamentally different approaches for CNN-based MU perform on VTs? 4. VT architectures: Do VTs design choices (ViT vs. Swin-T) and capacity impact unlearning? 5. Pretrain–Finetune: How does the VTs’ pretrain–finetune paradigm influence MU? 6. Algorithm-Architecture Pairings: Are certain pairings especially compatible? 7. Continual MU on VTs: How stable is performance under continual unlearning? In contrast to prior “standard” benchmarking research which are leaderboard-style (benchmarking/ranking a large set of algorithms) or proposing new datasets, our work follows a different path: It is centered on VTs, systematically testing representative MU approaches across two VT families (ViT, Swin-T) at different capacities each, over four datasets of varying size/complexity, and under both single-shot and continual unlearning scenarios. We employ unified metrics (ToW, ToW-MIA) that jointly account for retain/test accuracy, forget accuracy, and MIA vulnerability. We contrast results against CNN-derived MU counterpart algorithms. We aim to isolate architecture, capacity, memorization (and proxy) effects to assess how well MU methods perform in VTs and establish a strong performance baseline regarding MU performance in VTs. The code for reproducing the results is available at: https://github.com/kairanzhao/Unlearning_VTs 2 Model Architectures and Algorithms 2.1 Vision Transformers Transformer architectures (Vaswani et al., 2017) leverage self-attention mechanisms instead of recurrence. We focus on, arguably, the two most popular VTs: ViT and Swin-T. ViT (Dosovitskiy et al., 2021) processes images as sequences of flattened, embedded patches, which are (i) projected into an embedding space, (i) combined with positional encodings, and (i) passed through transformer encoder layers comprising multi-headed self-attention (MSA) and multi-layer perceptrons (MLP). Swin-T (Liu et al., 2021) adds hierarchical representations (a la CNNs) by progressively merging patches and reducing spatial resolution at deeper layers, forming multi-scale representations. Additionally, a shifted-window self-attention mechanism is introduced to model both local and global contexts. These allow us to study MU performance on architectures which, due to having/lacking hierarchical representations, are more/less similar to CNNs (Swin-T/ViT). 2.2 Machine Unlearning (MU) MU (Cao and Yang, 2015) aims to remove the influence of specific (“problematic”) training examples from pretrained models. A large body of work has tackled MU, spanning formal definitions and guarantees (Ginart et al., 2019; Sekhari et al., 2021; Neel et al., 2020), empirically effective methods (Goel et al., 2022; Golatkar et al., 2020; Thudi et al., 2022), and other earlier efforts (Xu et al., 2023). Over time, several key baselines have emerged, capturing fundamental components for successful MU. These range from simpler baselines like Fine-tune (FT) (Warnecke et al., 2023; Golatkar et al., 2020), to more sophisticated baselines like NegGrad+ (Kurmanji et al., 2023). Most recent SOTA algorithms include SCRUB (Kurmanji et al., 2023), L1-sparse (Jia et al., 2024), Salun (Fan et al., 2024) and the meta-algorithmic framework RUM (Zhao et al., 2024). Evidently, all of the above methods have focused on CNNs. Our contributions aim to answer whether the above CNN-derived, MU algorithms transfer to VTs across VT families, model capacities, datasets, and protocols. Our benchmark offers a substrate where such existing and new methods and architectures can be integrated and systematically evaluated. Following prior MU work (Kurmanji et al., 2023; Fan et al., 2024), we select three representative families of MU methods, which capture distinct unlearning paradigms and are commonly used as baselines in the literature: (i) FT, a standard baseline that fine-tunes the model on retained data only and is typically effective for low-memorization examples; (i) NegGrad+, a strong gradient-based baseline that explicitly promotes forgetting by pushing model parameters away from the forget set while preserving performance on retained data; and (i) SalUn, a recent SOTA that performs parameter-selective unlearning based on saliency. All MU methods assume a training dataset D partitioned into a forget set DfD_f and a retain set Dr=D∖DfD_r=D D_f. FT continues training exclusively on DrD_r. NegGrad+ combines standard fine-tuning on DrD_r with gradient ascent on DfD_f, aiming to balance retention and forgetting simultaneously during unlearning. SalUn first identifies the key parameters influencing examples in DfD_f using a saliency criterion, and then updates only these parameters while perturbing the associated labels of examples in DfD_f. In addition to comparing raw instantiations of these methods, we instantiate all three within the RUM framework, which has been shown to consistently strengthen a wide range of MU algorithms (Zhao et al., 2024). This allows us to compare method families under their best-performing configurations. 2.3 Leveraging Memorization for Unlearning Memorization quantifies model dependency on specific examples (Feldman, 2019; Feldman and Zhang, 2020). Let D−i:=D∖(xi,yi)D_-i:=D (x_i,y_i) and PrD[⋅]:=Prf∼A(D)[⋅] _D[·]:= _f A(D)[·], the measure, referred to as memorization score, is defined as mem(A,D,i)=PrD[f(xi)=yi]−PrD−i[f(xi)=yi],mem(A,D,i)= _D[f(x_i)=y_i]- _D_-i[f(x_i)=y_i], (1) capturing how predictions change when removing an example from training. Robust computation of this metric necessitates training large number of models (each retrained on subsets excluding specific examples) to achieve stable estimates, which is especially expensive, particularly for VTs. Proxies like the four Learning Events Proxies (Confidence (Conf), Max Confidence (MaxConf), Entropy (Ent), and Binary Accuracy (BA)) (Jiang et al., 2021) and Holdout Retraining (HR) (Carlini et al., 2019) come to the rescue, efficiently estimating memorization scores. The formulas for each of these can be found in the Appendix A.3. RUM leverages memorization to improve unlearning performance. It is structured in three stages: (1) Refinement: Partitions the forget set into homogeneous subsets based on memorization scores of examples in the forget set. Specifically, it creates three partitions consisting of low-, medium- and highly-memorized forget examples. (2)Matching: Selects suitable unlearning methods tailored to each partition. (3)Unlearning: Applies an unlearning algorithm sequentially on the three partitions. We evaluate FT, NegGrad+, and SalUn within the RUM framework. Concretely, RUM applies each base unlearning method sequentially to the three memorization-based partitions (e.g., RUM(SalUn)). For simplicity, we refer to these instantiations as SalUn, NegGrad+, and FT, respectively. We adopt this evaluation protocol because instantiating MU algorithms within RUM has been shown to substantially strengthen their performance (Zhao et al., 2024). For completeness, we additionally demonstrate in Section A.4 in Appendix that this improvement over vanilla instantiations also holds for Vision Transformers. 3 Experimental Setup 3.1 Data Sets We evaluate unlearning methods on four different image classification benchmarks that vary in dataset size, semantic complexity, and number of classes. Specifically, we use CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011), which are standard datasets commonly adopted in prior machine unlearning studies. To assess scalability to larger and more complex settings, we additionally benchmark on ImageNet-1K (Deng et al., 2009) using its validation set. Among these, CIFAR-10 and CIFAR-100 provide controlled settings with increasing class granularity, while SVHN represents a large-scale but semantically simpler dataset. ImageNet-1K serves as the pretraining dataset for VTs and enables evaluation under realistic, high-complexity conditions. 3.2 Transformer Architectures and Implementations We focus on ViT and Swin variants with model sizes comparable to ResNet-50 to enable fair comparisons with CNN-based unlearning results. Our primary benchmarks are ViT-Small and Swin-Tiny, which have similar parameter counts but differ substantially in inductive biases: ViT relies on global self-attention, whereas Swin introduces hierarchical structure and locality through windowed attention. To study the effect of model capacity, we additionally evaluate smaller and larger variants, including ViT-Tiny, ViT-Base, Swin-Small, and Swin-Base. This allows us to examine whether unlearning behavior varies systematically with model scale and architectural bias. 3.3 Evaluation Metrics Unlearning Metrics. The unlearned model θu=U(θo,Df,Dr) _u=U( _o,D_f,D_r) must balance forgetting quality (on DfD_f) while preserving model performance (on DrD_r) and generalizing well to unseen test data DtestD_test. We adopt two primary metrics from Zhao et al. (2024) to assess this balance. ToW(θu,θr,Df,Dr,Dtest)ToW( _u, _r,D_f,D_r,D_test) (ToW for short): ToW =(1−Δa(θu,θr,Df))⋅(1−Δa(θu,θr,Dr)) =(1- a( _u, _r,D_f))·(1- a( _u, _r,D_r)) (2) ⋅(1−Δa(θu,θr,Dtest)) ·(1- a( _u, _r,D_test)) where a(θ,D)=1|D|∑(x,y)∈D[f(x;θ)=y]a(θ,D)= 1|D| _(x,y)∈ D 1[f(x;θ)=y] is the accuracy of a model f parametrized by θ on D, and Δa(θu,θr,D)=|a(θu,D)−a(θr,D)| a( _u, _r,D)=|a( _u,D)-a( _r,D)| is the absolute difference in accuracy between θu _u and the retrained-from-scratch model θr _r on D. And, ToW-MIA(θu,θr,Df,Dr,Dtest)ToW-MIA( _u, _r,D_f,D_r,D_test) (ToW-MIA): ToW-MIA =(1−Δm(θu,θr,Df))⋅(1−Δa(θu,θr,Dr)) =(1- m( _u, _r,D_f))·(1- a( _u, _r,D_r)) (3) ⋅(1−Δa(θu,θr,Dtest)) ·(1- a( _u, _r,D_test)) where m(θ,D)=TND|D|m(θ,D)= TN_D|D| and Δm(θu,θr,D)=|m(θu,D)−m(θr,D)| m( _u, _r,D)=|m( _u,D)-m( _r,D)|. In ToW-MIA, m(θ,D)m(θ,D) accounts for “forget quality” using Membership Inference Attack (MIA) performance. To calculate this, as in Fan et al. (2024); Zhao et al. (2024); Jia et al. (2023), we train a binary classifier C that classifies examples as “in-training” or “out-of-training”. We simplify our notation l(f(x;θ),y)l(f(x;θ),y) (the cross-entropy loss of a model with weights θ on example x with label y) to l(x,y)l(x,y). C is trained on loss values from a balanced dataset Dtb=(l(xi,yi),yib)D_t^b=\(l(x_i,y_i),y_i^b)\, where examples xix_i are drawn equally from the retain set DrD_r (labelled as yib=1y_i^b=1 for “training”) and the test set DtestD_test (labelled as yib=0y_i^b=0 for “non-training”). Once trained, C evaluates loss values for examples in DfD_f. m(θ,D)m(θ,D) measures the proportion of DfD_f examples that C classified as “non-training”, i.e. the true negatives TNDTN_D. The hope is for the unlearning method to cause the model to treat forget set examples as if they were never seen during training, resulting in the classifier categorising them as “non-training.” So similarly to Δa a, Δm(θu,θr,D) m( _u, _r,D) quantifies how closely the unlearned model θu _u resembles the MIA performance of the retrain-from-scratch model θr _r. ToW and ToW-MIA measure “forget quality” differently: ToW uses accuracy differences on the forget set, while ToW-MIA uses differences in MIA vulnerability. Both range from 0 to 1, with higher values indicating better unlearning performance (closer-matching retraining from scratch). Together, ToW and ToW-MIA provide a comprehensive view of unlearning performance. Memorization Proxy Metric. To evaluate how well a proxy captures memorization, we compute Spearman’s rank correlation coefficient between the ground-truth memorization score and the proxy value across training examples. For each example (xi,yi)∈D(x_i,y_i)∈ D, let mi=memm(A,D,i)m_i=mem_m(A,D,i) denote the “Feldman” memorization score (Feldman, 2019) and pi=proxy(xi,yi)p_i=proxy(x_i,y_i) the value of a given proxy metric. We rank mim_i and pip_i independently and compute Spearman’s ρ between their ranks. The resulting coefficient ρ∈[−1,1]ρ∈[-1,1] measures the strength and direction of the monotonic relationship between true memorization and the proxy, with larger absolute values indicating stronger correlation. 4 Results and Analyses In this section we present the main results, focusing on CIFAR-100 as a representative dataset. The Appendix contains results for the other datasets. Also, as we focus on memorization-based MU algorithms, we first established that (i) the memorization patterns in VTs mimic those in CNNs and that (i) memorization proxies can indeed be trustworthy for being leveraged by MU algorithms in VTs. The detailed results can be found in Appendix A.3. 4.1 Memorization and Proxies in VTs Memorization plays a central role in unlearning, so we first ask whether VTs memorize similarly to CNNs, and whether CNN-derived proxies remain valid predictors of memorization in VTs. Key Takeaways–Memorization patterns. We observe (see Figure 3, 4 in Appendix A.3.1) the same long-tailed distributions previously reported for CNNs (Feldman and Zhang, 2020). Thus, VTs and CNNs display fundamentally similar memorization behavior, despite architectural differences. Also, on CIFAR-10, VTs exhibit slightly lower memorization than ResNet-18, reflecting their ability (via pretraining and global attention) to rely less on memorizing individual examples for simpler tasks. Key Takeaways–Proxy validity. Whether CNN-derived proxies remain valid for VTs is unclear due to differences in their inductive biases and training regimes. We evaluated five memorization proxies: Confidence (Conf), Max Confidence (MaxConf), Entropy (Ent), Binary Accuracy (BA) (Jiang et al., 2021), and Holdout Retraining (HR) (Carlini et al., 2019). The results are in Table 1. The Confidence proxy consistently achieves the strongest correlations across all models and datasets, with magnitudes (-0.79 to -0.91) closely resembling those in CNNs. Swin-Tiny shows slightly stronger correlations than ViT-Small, likely due to its hierarchical structure that more closely resembles traditional CNNs. Holdout Retraining shows moderate but significant positive correlations, and is attractive in practice given its large computational advantages vis-a-vis the other proxies. Overall, simple proxies such as Confidence and Holdout Retraining remain predictive for VTs, enabling scalable memorization-based unlearning without expensive Feldman-score computations. Table 1: Spearman Correlation Coefficients Between Memorization and Proxies CIFAR-10 CIFAR-100 Proxy ResNet-18 ViT-Small Swin-Tiny ResNet-50 ViT-Small Swin-Tiny Conf -0.80 -0.79 -0.88 -0.91 -0.85 -0.90 MaxConf -0.76 -0.77 -0.85 -0.87 -0.80 -0.86 Ent -0.75 -0.78 -0.85 -0.80 -0.77 -0.82 BA -0.71 -0.63 -0.79 -0.89 -0.69 -0.78 HR +0.67 +0.45 +0.64 +0.62 +0.50 +0.52 4.2 Unlearning Algorithms Performance on Vision Transformers For these experiments, DfD_f comprises 3,000 examples, divided into M=3M=3 partitions of N=1,000N=1,000 examples each, representing the lowest, medium and highest proxy values. We apply each algorithm unlearning in the order of low → medium → high memorization, using memorization-proxy values. All results are averaged over three runs and we also present 95% confidence intervals. Given that pre-trained transformer models have lower memorization, we anticipated that θr _r might already perform well on DfD_f. To establish a baseline for comparison, we calculated ToW and ToW-MIA parametrized by (θo,θr,Df,Dr,Dtest)( _o, _r,D_f,D_r,D_test) denoted as “Original”. This baseline indicates the performance we would achieve without applying any unlearning algorithm. Hyper-parameters for all algorithms are detailed in Appendix A.2.2. (a) ToW with Confidence (b) ToW with Holdout Retraining (c) ToW-MIA with Confidence (d) ToW-MIA with Holdout Retraining Figure 1: MU performance comparison on CIFAR-100. How do different MU approaches perform on VTs? Figures 1 present unlearning results across both VTs and CIFAR-100 (see Figure 5 and 7 in Appendix for results on CIFAR-10 and SVHN). Our results show that simple approaches like Fine-tune perform surprisingly well, especially on the simpler SVHN dataset. NegGrad+ is consistently strong in all cases, and especially for more complex datasets (CIFAR-100) and outperforms all when paired with Holdout Retraining. Notably, NegGrad+ with Holdout Retraining for Swin-T shows even better performance than in ResNets. SalUn achieves good ToW scores but struggles with ToW-MIA on more complex datasets (CIFAR-100), though it performs competitively on SVHN. From the proxy viewpoint, Holdout Retraining performs excellently compared to Confidence for CIFAR-10/CIFAR-100, while Confidence can match it for the simpler SVHN. More detailed data can be found in Table 13(d). Key Takeaways. SOTA MU algorithms from CNNs can be equally (if not more) effective for VTs. NegGrad+ is the most robust MU method for VTs, while SalUn is vulnerable in ToW-MIA on harder datasets. Proxy choice (Holdout Retraining vs. Confidence) further shapes outcomes. How does unlearning performance in VTs compare to CNNs? We first compare the accuracy of the retrained model θr _r on DfD_f in ResNets versus VTs, since this directly influences ToW and ToW-MIA evaluation for the benchmark model θr _r. Table 2 shows that on CIFAR-10, θr _r achieves much higher DfD_f accuracy in VTs than CNNs (e.g., >90%>90\% for ViT/Swin-T vs. ∼50% 50\% for ResNet-18). This advantage stems from VT pretraining, which enables learning robust feature representations. Thus, after DfD_f examples are removed and the model is retrained, θr _r remains close to the original θo _o for VTs, explaining their high baseline “Original” performance (Table 13(d)). However, this pretraining advantage diminishes on the more complex datasets (CIFAR-100). Figure 1 (and Figure 5 in Appendix) shows how method rankings shift across architectures and proxies. With the Confidence proxy, Fine-tune and NegGrad+ perform best in ResNets, while VTs lag slightly. Under Holdout Retraining, however, VTs close or surpass this gap, particularly for NegGrad+. SalUn behaves differently: it achieves good ToW scores in VTs but underperforms on ToW-MIA compared to CNNs, suggesting that while it effectively adjusts transformer outputs to match θr _r, it struggles to protect against MIAs. Key Takeaways. Pretraining gives VTs an advantage on simpler tasks, but this weakens with increasing complexity. MU method rankings from CNNs do not transfer to VTs; they depend strongly on architectures and proxies. Holdout Retraining narrows or reverses the gap in favor of VTs. Table 2: DfD_f accuracies of model θr _r across architectures. Architecture CIFAR-10 CIFAR-100 Confidence Holdout Retraining Confidence Holdout Retraining ResNet-18 50.433±6.80850.433± 6.808 62.922±4.68162.922± 4.681 – – ResNet-50 – – 64.267±0.50464.267± 0.504 69.856±2.62069.856± 2.620 ViT-Small 94.089±0.45694.089± 0.456 89.244±1.32189.244± 1.321 69.322±1.78869.322± 1.788 68.767±3.82868.767± 3.828 Swin-Tiny 91.867±1.71191.867± 1.711 86.389±1.24686.389± 1.246 69.833±1.24269.833± 1.242 70.267±2.84870.267± 2.848 4.3 How do VT architectures and capacities affect MU performance? We observe systematic differences between ViT and Swin-T, as well as clear effects of model size. Architecture. Swin-T exhibits stronger memorization than ViT (e.g., higher mean Feldman scores, heavier tails on CIFAR-10: μ=0.17 vs. 0.09, see Figures 3 and 4), which likely explains its superior performance with gradient-based methods such as NegGrad+. Swin-T also aligns with CNN-like behavior in SalUn, achieving optimal performance at saliency threshold γ=0.3γ=0.3, consistent with CNNs (Zhao et al., 2024), whereas ViT requires a much lower γ=0.1γ=0.1. This suggests that ViT’s global attention leads to more diffuse parameter involvement, while Swin-T’s local windowed attention allows for more concentrated, targeted unlearning. In terms of algorithm performance, Figure 1 (and Figure 5, Table 13(d) in Appendix) shows that Fine-tune is particularly effective on ViT (e.g., ToW-MIA=0.919 on CIFAR-10, 0.831 on CIFAR-100), while NegGrad+ excels on Swin-T, especially with Holdout Retraining on more complex tasks (e.g., ToW=0.975, ToW-MIA=0.902 on CIFAR-100). SalUn consistently attains good ToW but struggles on ToW-MIA, especially for ViT-Small (e.g., 0.582 on CIFAR-10 with Confidence). Dataset-level effects mirror these trends: ViT has an edge on the smaller, medium-complexity CIFAR-10, Swin-T dominates on the larger, more complex CIFAR-100, and both perform well on the simpler SVHN dataset. Continual unlearning results in Section 4.5 show broadly similar trends across ViT and Swin-T. Key Takeaways. ViT favors fine-tuning-based unlearning, likely due to its global attention. Swin-T better supports gradient-based unlearning (NegGrad+) likely due to its local, hierarchical attention. Dataset complexity further shapes which family is most effective. Capacity. To study the effect of model capacity, we extend our analysis to smaller and larger variants within each family, evaluated on CIFAR-10: ViT-Tiny (∼5.5 5.5M params) vs. ViT-Small (∼21.6 21.6M) and Swin-Tiny (∼27.5 27.5M) vs. Swin-Small (∼48.8 48.8M). We focus on the Holdout Retraining (HR) proxy, which was especially promising in earlier results. We report ToW and ToW-MIA for the three MU algorithms (Fine-tune, NegGrad+, SalUn) alongside the Original baseline in Table 3. The headline findings continue to hold: Fine-tune and NegGrad+ remain consistently strong; SalUn attains high ToW but is often much weaker on ToW-MIA and is notably sensitive to model size, architecture, and proxy choice. We also observe architecture-specific capacity trends: For Swin-T, increasing capacity from Tiny to Small offers little ToW improvement and can reduce ToW-MIA, suggesting Swin-Tiny is already sufficient for CIFAR-10 and that Swin-Small may overfit. For ViT, the opposite holds: ViT-Tiny underperforms ViT-Small on both ToW and ToW-MIA, implying under-capacity limits MU effectiveness. The results reveal a “sweet spot” around ViT-Small and Swin-Tiny, where models are neither under- nor overfitting, yielding balanced unlearning-privacy. Table 3: ToW and ToW-MIA on CIFAR-10 with HR across four VT architectures. Algorithm ViT-Small ViT-Tiny Swin-Small Swin-Tiny Original 0.891±0.0160.891± 0.016 0.773±0.0200.773± 0.020 0.886±0.0130.886± 0.013 0.862±0.0080.862± 0.008 Fine-tune 0.928±0.0130.928± 0.013 0.862±0.0600.862± 0.060 0.921±0.0050.921± 0.005 0.923±0.0210.923± 0.021 NegGrad+ 0.916±0.0160.916± 0.016 0.957±0.0110.957± 0.011 0.944±0.0280.944± 0.028 0.977±0.0230.977± 0.023 SalUn 0.956±0.0030.956± 0.003 0.846±0.0430.846± 0.043 0.950±0.0310.950± 0.031 0.961±0.0480.961± 0.048 (a) ToW Algorithm ViT-Small ViT-Tiny Swin-Small Swin-Tiny Original 0.831±0.0140.831± 0.014 0.699±0.0200.699± 0.020 0.832±0.0200.832± 0.020 0.811±0.0180.811± 0.018 Fine-tune 0.913±0.0080.913± 0.008 0.854±0.0600.854± 0.060 0.924±0.0100.924± 0.010 0.931±0.0210.931± 0.021 NegGrad+ 0.968±0.0150.968± 0.015 0.869±0.0110.869± 0.011 0.888±0.0210.888± 0.021 0.924±0.0280.924± 0.028 SalUn 0.766±0.0390.766± 0.039 0.640±0.0430.640± 0.043 0.568±0.0930.568± 0.093 0.834±0.0220.834± 0.022 (b) ToW-MIA Key Takeaways. Performance trends remain stable across ViT-Tiny/Small and Swin-Tiny/Small. 4.4 Results on Larger/More Complex Data To examine unlearning larger-scale, more complex conditions, we leverage that our Vision Transformers are pretrained on ImageNet-1K and use the validation split (50,000 images unseen during pretraining) as a new evaluation dataset (already 5×5× larger than CIFAR-100’s test set). We form a forget set of |Df|=3,000|D_f|=3,000 (partitioned into three 1k subsets for low/medium/high proxy values, as in Section 4.2), and treat the remaining 47k images as retain/test set. As this setup has only retain/test and forget components, we report ToW and ToW-MIA with two terms (retain/test accuracy and forget accuracy). We evaluate on the (more appropriate) larger Swin-Small (48.8M params) and use the Holdout Retraining proxy since it was previously found to be high performing. Table 4: Unlearning performance with HR proxy for Swin-Small on the 50k ImageNet-1K validation set. Algorithm ToW ToW-MIA Fine-tune 0.780±0.0230.780± 0.023 0.747±0.0090.747± 0.009 NegGrad+ 0.819±0.0180.819± 0.018 0.772±0.0340.772± 0.034 SalUn 0.743±0.0270.743± 0.027 0.647±0.0300.647± 0.030 Table 5: DfD_f accuracies of Retrained θr _r (with HR) across architectures on the 50k ImageNet-1K validation set. Architecture DfD_f accuracy (%\%) ResNet-50 69.139±1.72269.139± 1.722 Swin-Small 69.433±2.24669.433± 2.246 Table 4 shows that the key earlier conclusions continue to hold at this larger scale: NegGrad+ and Fine-tune perform strongly, while SalUn again underperforms, especially on ToW-MIA. This reinforces the conclusion that comparatively simple unlearning strategies can be effective even for larger models and datasets, albeit performance varies with architecture and data complexity. We also note that absolute ToW/ToW-MIA values are lower here than for Swin-Tiny on CIFAR-100, despite Swin-Small’s larger capacity. This is reasonable: (i) retain/test accuracy is harder given ImageNet’s larger scale, diversity, and label space , and (i) forgetting is more challenging on richer ImageNet images (greater embedding entanglement). Consequently, both terms in ToW and ToW-MIA are pressured downward relative to the CIFAR-100 setting, and further gains would likely require even larger-capacity models and/or stronger regularization. Finally, Table 5 reinforces our earlier observation that the benefit of pretraining for forget accuracy diminishes as task complexity increases (compared to Table 2). Key Takeaways. Simple methods (NegGrad+, Fine-tune) remain effective even on ImageNet-scale data, whereas SalUn underperforms, and the benefits of pretraining diminish as complexity grows. Model scaling on ImageNet-1K. To further examine capacity effects under ImageNet-scale complexity, we additionally evaluate larger VT variants, Swin-Base (∼ 88M parameters) and ViT-Base/16 (∼ 85M parameters), under the same ImageNet-1K validation protocol and HR proxy. Table 6 shows that increasing model capacity leads to modest but consistent performance gains within the Swin family. Importantly, the qualitative trends observed for smaller models persist across all architectures at this scale: Fine-tune and NegGrad+ remain strong performers, whereas SalUn continues to underperform. Table 6: Unlearning performance on ImageNet-1K validation set with HR proxy, comparing model scaling within the Swin family, as well as results for ViT-Base/16. Method Swin-Small (49M) Swin-Base (88M) ViT-Base/16 (85M) Fine-tune 0.780±0.0230.780± 0.023 0.808±0.0070.808± 0.007 0.843±0.0160.843± 0.016 NegGrad+ 0.819±0.0180.819± 0.018 0.837±0.0040.837± 0.004 0.842±0.0050.842± 0.005 SalUn 0.743±0.0270.743± 0.027 0.746±0.0130.746± 0.013 0.773±0.0270.773± 0.027 Key Takeaways. Increasing model capacity at ImageNet scale yields small but consistent improvements, and method-level trends remain consistent. 4.5 Does Continual Unlearning Impact Performance in Vision Transformers? In practical deployments, unlearning is often performed continually, through a sequence of smaller forget operations rather than a single large one. We therefore examine whether repeated unlearning leads to cumulative degradation in performance. Based on earlier results, we focus on NegGrad+ with Holdout Retraining, which is the best-performing algorithm-proxy pair. We perform five sequential unlearning steps, unlearning the same total number of examples (|Df|=3,000|D_f|=3,000). At each step, proxy values are recomputed, and the forget set is partitioned as before, with |Df|=600|D_f|=600 per step and N=200N=200 examples per partition. Figure 2 shows unlearning performance across the five steps on CIFAR-100. We observe stable behavior across metrics and architectures. While ViT-Small exhibits a mild downward trend, all metrics remain within the confidence intervals of the initial unlearning step and are comparable to single-step results reported in Section 4.2. Similar trends hold for CIFAR-10 and SVHN (Table 16, Appendix A.5.2 for CIFAR-10 and A.8.2 for SVHN). (a) ViT-Small (b) Swin-Tiny Figure 2: Continual Unlearning Performance Across Five Steps on CIFAR-100, reported with 95% confidence intervals. To further stress-test continual unlearning, we extend this evaluation to ten sequential unlearning steps under the same setting (NegGrad+ with HR), using both ViT-Small and Swin-Tiny. As shown in Table 17, performance remains stable across steps, with no evidence of substantial accumulated degradation in either ToW or ToW-MIA. These results reinforce the earlier observation that suitable method-proxy pairs can support continual unlearning without progressive performance degradation. Key Takeaway. Minimal (if any) degradation occurs under continual unlearning in VTs. 5 Practitioners Takeaways Our results yield several actionable guidelines for applying machine unlearning in VTs. Architecture–method compatibility is critical: ViT models consistently favor Fine-tune, likely due to their global attention and diffuse parameter involvement, while Swin architectures pair best with NegGrad+, which benefits from local and hierarchical inductive biases. SalUn, although competitive in ToW, is unreliable in ToW-MIA for VTs (especially ViT) and should be avoided in privacy-sensitive settings. Memorization proxy choice further shapes performance. Confidence is the highest-fidelity proxy overall and works well on simpler datasets, whereas Holdout Retraining (HR) yields the strongest and most stable unlearning performance on complex datasets and under continual unlearning. In practice: • ViT + Fine-tune + Confidence is effective and efficient for low-to-moderate memorization regimes. • Swin + NegGrad+ + HR is the most robust choice for high-complexity or continual unlearning scenarios. • Stable unlearning is achieved using small learning rates, cosine scheduling, and sequential partitioning from low → high memorization. 6 Conclusions We benchmarked the performance of fundamentally different approaches to MU, ranging from simpler baselines to more advanced SOTA unlearning algorithms in VTs. We focused on algorithms leveraging memorization, as memorization has been found to be key to unlearning (across modalities) and the SOTA MU approaches for vision tasks can be substantially improved by leveraging it. To ensure the practicality of leveraging memorization, we studied memorization patterns in VTs and five different memorization proxies that have been found to offer high memorization-score fidelity in CNNs while being drastically more efficient to compute. We employed different datasets to account for the effects of varying sizes and complexities. We studied two different popular VT architectures (ViT and Swin-T) selected to be closer or further from CNN inductive biases. We also used different model capacities to see the impact of capacities on MU performance (with carefully selected dataset sizes to avoid over- and under-fitting). This work represents the first comprehensive study in this domain and sheds new light on the current state of affairs of MU algorithm performance in VTs. This work contributes the following novel insights: (i) Pretraining of VTs can help improve unlearning for smaller and simpler datasets/tasks. (i) VTs and CNNs largely exhibit the same memorization patterns. (i) Well-known memorization proxies from CNNs can benefit unlearning performance in VTs as they can do in CNNs. (iv) Continual unlearning does not degrade the efficacy of memorization proxies and overall unlearning performance in VTs. (v) VTs can enjoy similar, if not better, unlearning performance to that in CNNs (using CNN-derived algorithms). (vi) Swin-T can outperform ViT on more complex datasets, due to Swin-T’s architectural similarities to CNNs. (vii) NegGrad+, perhaps surprisingly, emerges as a consistent, strong performer for VTs, especially on more complex datasets. (viii) Holdout Retraining emerges as the proxy yielding superior unlearning performance in VTs, whereas Confidence can be competitive on lower-complexity datasets. We hope these insights, as well as the associated publicly available codebase/dataset infrastructure benchmark, will springboard the study of MU in VTs, serving as a basis upon which to add new (future) algorithms, especially developed for VTs. References G. Barbulescu and P. Triantafillou (2024) To each (textual sequence) its own: improving memorized-data unlearning in large language models. In International Conference on Machine Learning (ICML), External Links: 2405.03097, Link Cited by: §1. X. F. Cadet, A. Borovykh, M. Malekzadeh, S. Ahmadi-Abhari, and H. Haddadi (2024) Deep unlearn: benchmarking machine unlearning. External Links: 2410.01276, Link Cited by: §1, §1. Y. Cao and J. Yang (2015) Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, Vol. , p. 463–480. External Links: Document Cited by: §2.2. N. Carlini, Ú. Erlingsson, and N. Papernot (2019) Distribution density, tails, and outliers in machine learning: metrics and applications. External Links: 1910.13427, Link Cited by: §A.3.2, §2.3, §4.1. J. Cheng and H. Amiri (2024) MU-bench: a multitask multimodal benchmark for machine unlearning. External Links: 2406.14796, Link Cited by: §1. I. Cho, C. Park, and J. Hockenmaier (2024) ViT-mul: a baseline study on recent machine unlearning methods applied to vision transformers. External Links: 2403.09681, Link Cited by: §1. J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, p. 248–255. External Links: Document Cited by: §3.1. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. External Links: 2010.11929, Link Cited by: §2.1. C. Fan, J. Liu, Y. Zhang, E. Wong, D. Wei, and S. Liu (2024) SalUn: empowering machine unlearning via gradient-based weight saliency in both image classification and generation. External Links: 2310.12508, Link Cited by: §2.2, §2.2, §3.3. V. Feldman and C. Zhang (2020) What neural networks memorize and why: discovering the long tail via influence estimation. External Links: 2008.03703, Link Cited by: §A.2.1, §A.3.1, §A.3.1, §2.3, §4.1. V. Feldman (2019) Does learning require memorization? a short tale about a long tail. External Links: 1906.05271, Link Cited by: §2.3, §3.3. A. Ginart, M. Y. Guan, G. Valiant, and J. Zou (2019) Making ai forget you: data deletion in machine learning. External Links: 1907.05012, Link Cited by: §2.2. S. Goel, A. Prabhu, and P. Kumaraguru (2022) Evaluating inexact unlearning requires revisiting forgetting. External Links: Document Cited by: §2.2. A. Golatkar, A. Achille, and S. Soatto (2020) Eternal sunshine of the spotless net: selective forgetting in deep networks. External Links: 1911.04933, Link Cited by: §2.2. K. Grimes, C. Abidi, C. Frank, and S. Gallagher (2024) Gone but not forgotten: improved benchmarks for machine unlearning. External Links: 2405.19211, Link Cited by: §1. J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo (2022) Knowledge unlearning for mitigating privacy risks in language models. External Links: 2210.01504, Link Cited by: §1. J. Jia, J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu (2023) Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems 36, p. 51584–51605. External Links: Link Cited by: §3.3. J. Jia, J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu (2024) Model sparsity can simplify machine unlearning. External Links: 2304.04934, Link Cited by: §2.2. Z. Jiang, C. Zhang, K. Talwar, and M. C. Mozer (2021) Characterizing structural regularities of labeled data in overparameterized models. External Links: 2002.03206, Link Cited by: §A.3.2, §2.3, §4.1. A. Krizhevsky (2009) Learning multiple layers of features from tiny images. University of Toronto, p. . External Links: Link Cited by: §3.1. M. Kurmanji, P. Triantafillou, J. Hayes, and E. Triantafillou (2023) Towards unbounded machine unlearning. In 2023 Thirty-Sevent Annual Conference on Neural Information Processing Systems (NeurIPS), External Links: 2302.09880, Link Cited by: §2.2, §2.2. N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-Voss, C. B. Breuer, S. Marks, O. Patel, A. Zou, M. Mazeika, Z. Wang, P. Oswal, W. Lin, A. A. Hunt, J. Tienken-Harder, K. Y. Shih, K. Talley, J. Guan, R. Kaplan, I. Steneker, D. Campbell, B. Jokubaitis, A. Levinson, J. Wang, W. Qian, K. K. Karmakar, S. Basart, S. Fitz, M. Levine, P. Kumaraguru, U. Tupakula, V. Varadharajan, R. Wang, Y. Shoshitaishvili, J. Ba, K. M. Esvelt, A. Wang, and D. Hendrycks (2024) The wmdp benchmark: measuring and reducing malicious use with unlearning. External Links: 2403.03218, Link Cited by: §1. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo (2021) Swin transformer: hierarchical vision transformer using shifted windows. External Links: 2103.14030, Link Cited by: §2.1. R. Ma, Q. Zhou, Y. Jin, D. Zhou, B. Xiao, X. Li, Y. Qu, A. Singh, K. Keutzer, J. Hu, X. Xie, Z. Dong, S. Zhang, and S. Zhou (2024) A dataset and benchmark for copyright infringement unlearning from text-to-image diffusion models. External Links: 2403.12052, Link Cited by: §1. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter (2024) TOFU: a task of fictitious unlearning for llms. External Links: 2401.06121, Link Cited by: §1. S. Neel, A. Roth, and S. Sharifi-Malvajerdi (2020) Descent-to-delete: gradient-based methods for machine unlearning. External Links: 2007.02923, Link Cited by: §2.2. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. N (2011) Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Cited by: §3.1. J. Ren, Y. Li, S. Zeng, H. Xu, L. Lyu, Y. Xing, and J. Tang (2024) Unveiling and mitigating memorization in text-to-image diffusion models through cross attention. External Links: 2403.11052, Link Cited by: §1. A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh (2021) Remember what you want to forget: algorithms for machine unlearning. External Links: 2103.03279, Link Cited by: §2.2. A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot (2022) Unrolling sgd: understanding factors influencing machine unlearning. External Links: 2109.13398, Link Cited by: §2.2. R. Torkzadehmahani, R. Nasirigerdeh, G. Kaissis, D. Rueckert, G. K. Dziugaite, and E. Triantafillou (2024) Improved localized machine unlearning through the lens of memorization. arXiv preprint arXiv:2412.02432. Cited by: §1. E. Triantafillou, P. Kairouz, F. Pedregosa, J. Hayes, M. Kurmanji, K. Zhao, V. Dumoulin, J. J. Junior, I. Mitliagkas, J. Wan, L. S. Hosoya, S. Escalera, G. K. Dziugaite, P. Triantafillou, and I. Guyon (2024) Are we making progress in unlearning? findings from the first neurips unlearning competition. External Links: 2406.09073, Link Cited by: §1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. CoRR abs/1706.03762. External Links: Link, 1706.03762 Cited by: §2.1. A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck (2023) Machine unlearning of features and labels. External Links: 2108.11577, Link Cited by: §2.2. Y. Wen, Y. Liu, C. Chen, and L. Lyu (2024) Detecting, explaining, and mitigating memorization in diffusion models. External Links: 2407.21720, Link Cited by: §1. R. Wightman (2019) PyTorch image models. GitHub. Note: https://github.com/huggingface/pytorch-image-models External Links: Document Cited by: §A.2. H. Xu, T. Zhu, L. Zhang, W. Zhou, and P. S. Yu (2023) Machine unlearning: a survey. External Links: 2306.03558, Link Cited by: §2.2. Y. Zhang, C. Fan, Y. Zhang, Y. Yao, J. Jia, J. Liu, G. Zhang, G. Liu, R. R. Kompella, X. Liu, and S. Liu (2024) UnlearnCanvas: stylized image dataset for enhanced machine unlearning evaluation in diffusion models. External Links: 2402.11846, Link Cited by: §1. K. Zhao, M. Kurmanji, G. Bărbulescu, E. Triantafillou, and P. Triantafillou (2024) What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems 37, p. 12293–12333. Cited by: §1, §2.2, §2.2, §2.3, §3.3, §3.3, §4.3. Appendix A Appendix A.1 Impact Statement & Limitations Impact Statement. This work advances privacy-preserving Vision Transformers by examining how architectural design choices affect the effectiveness of machine unlearning. Through a systematic evaluation across model architectures, scales, and data complexity, we identify key factors that influence reliable data removal. These findings provide practical guidance for developing vision models that better support “right to be forgotten” requirements, contributing to more trustworthy deployment in sensitive real-world settings. Limitations. Our benchmarking setup represents a practical compromise in terms of VT models, their sizes, and the size and complexity of datasets, allowing for insightful comparisons to CNNs, while studying fundamental approaches to MU. Nonetheless, there is much more work to follow, adding more datasets, more models and capacities, more (new and existing) algorithms, and different metrics. A.2 Implementation Details This section outlines the key implementation details for all experiments conducted in this study. For all experiments involving transformer architectures, we fine-tune models initialized with ImageNet-1k pretrained weights provided by the timm library [Wightman, 2019], as described in Section 3.2. All experiments on the CIFAR-10 and CIFAR-100 datasets were conducted using a mix of NVIDIA RTX 2080 Ti, NVIDIA A10 and NVIDIA TITAN Xp GPUs, taking roughly 210 GPU hours. For the experiments on the SVHN dataset, training and evaluation were performed on NVIDIA A5000 GPUs, which consumed approximately 300 GPU hours. The code for reproducing the results is available at: https://github.com/kairanzhao/Unlearning_VTs A.2.1 Memorization Estimation For memorization estimation, we use ResNet-18 for CIFAR-10 and rely on precomputed memorization scores provided by [Feldman and Zhang, 2020] for CIFAR-100. Additionally, we compute memorization scores using ViT-Small and Swin-Tiny architectures for both CIFAR-10 and CIFAR-100. The training configurations and corresponding hyperparameters are summarized in Table 7. Table 7: Training configurations along with hyper-parameters of the models used to compute memorization scores shown in Figures 3 and 4. The “Transformers” column represents the configuration for both ViT-Small and Swin-Tiny across CIFAR-10 and CIFAR-100. ResNet-18 Transformers Optimizer SGD AdamW Base learning rate 0.01 0.0001 Loss Cross-Entropy Cross-Entropy Learning rate scheduler Step decay CosineAnnealingLR Batch size 512 128 Epochs 30 30 Momentum 0.9 - Weight decay 0.00050.0005 0.050.05 Data augmentation None None A.2.2 Unlearning This section presents the configuration and hyper-parameter details used to obtain the necessary models for ToW and ToW-MIA: “Original” θo _o, “Retrained” θr _r and “Unlearned” θu _u. Table 8 shows the hyper-parameter details for training original models & retraining. Table 9 shows the unlearning hyper-parameters, and Table 10 presents the sequential unlearning hyper-parameters. Table 8: Training configurations and hyperparameter used for the original models (θo _o) and the retrained models (θr _r) on the CIFAR-10 and CIFAR-100 datasets. Hyperparameter ViT-Small & Swin-Tiny Optimizer AdamW Base learning rate 0.0001 Loss Cross-Entropy Learning rate scheduler CosineAnnealingLR Batch size 128 Epochs 50 Weight decay 0.050.05 Data augmentation Random Crop + Horizontal Flip Table 9: Hyper-parameters across all unlearn methods within the RUMF meta-algorithm on both Confidence and Holdout Retraining proxies to derive the unlearn models θu _u. “Unlearn Epochs” column refers to the performed number of epochs each algorithm performed for the three memorization partitions (low → medium → high) respectively. Algorithm Architecture Unlearn Epochs Unlearn LR β α γ FineTune ViT-Small 5,5,10 0.0001 – – – Swin-Tiny 5,5,10 0.0001 – – – NegGrad+ ViT-Small 5,5,10 0.00002 0.97 – – Swin-Tiny 5,5,10 0.00002 0.97 – – SalUn ViT-Small 5,5,10 0.00005 – 1 0.1 Swin-Tiny 5,5,10 0.0002 – 1 0.3 Table 10: Hyper-parameters at each sequential unlearning step for the NegGrad+ and Holdout Retraining configuration utilized. “Unlearn Epochs” are reduced to 1 for partition (low → medium → high) due to the smaller forget set size(|Df|=600|D_f|=600) at each step. All other settings remain consistent with the primary experiments. Unlearn Epochs Unlearn LR β 1,1,1 0.00002 0.97 A.3 Memorization and Proxies in VTs A.3.1 Do Vision Transformers Memorize the Same Way as CNNs? As memorization plays a key role in unlearning, this step is critical. For ResNet-18/CIFAR-10 and both VTs for both datasets, the training configuration is detailed in Table 7 (Appendix A.2.1). For ResNet-50/CIFAR-100, scores were already pre-computed [Feldman and Zhang, 2020]. (c) ResNet-50 (d) ViT-Small (e) Swin-Tiny Figure 3: Memorization histograms across different architectures on CIFAR-100 Figures 3 (and 4 present the memorization distributions across different architectures for CIFAR-100 (CIFAR-10). In all cases we see similarly skewed distributions, consistent with the “long-tail” discovery in [Feldman and Zhang, 2020]. For CIFAR-100, in particular, both VTs closely align with ResNet-50. Hence, despite architectural differences between VTs and CNNs, their memorization patterns remain fundamentally similar, especially for more complex tasks. This provides a foundation for studying unlearning a la RUM in VTs. (a) ResNet-18 (b) ViT-Small (c) Swin-Tiny Figure 4: Memorization histograms across different architectures on CIFAR-10 For CIFAR-100, we observe that both VTs closely align with ResNet-50 in terms of memorization. However, on CIFAR-10, VTs exhibit slightly lower memorization compared to ResNet-18. This can be attributed to the fact that pre-trained VTs benefit from a stronger ability to capture global contextual information, which reduces their reliance on memorizing training examples when adapting to simpler tasks. This advantage, however, does not extend to more challenging tasks like CIFAR-100. A.3.2 Are CNN-Derived Memorization Proxies Relevant for VTs? Differences in architecture (e.g., inductive biases) and in training (e.g., finetune-then-retrain in VTs) raise doubts about the appropriateness of CNN memorization proxies for VTs. We first provide the formulas for computing each of the following proxies: Confidence (C), Max Confidence (MaxC), Entropy (E), Binary Accuracy (BA) [Jiang et al., 2021] and Holdout Retraining (HR) [Carlini et al., 2019]. Conf(xi,yi) (x_i,y_i) =1E∑e=1EPθe(y=yi∣xi), = 1E _e=1^EP_ _e(y=y_i x_i), (4a) MaxConf(xi) (x_i) =1E∑e=1EmaxyPθe(y∣xi), = 1E _e=1^E _yP_ _e(y x_i), (4b) Ent(xi) (x_i) =1E∑e=1E∑yPθe(y∣xi)logPθe(y∣xi), = 1E _e=1^E _yP_ _e(y x_i) P_ _e(y x_i), (4c) BA(xi,yi) (x_i,y_i) =1E∑e=1E[argmaxyPθe(y∣xi)=yi], = 1E _e=1^E 1\! [ _yP_ _e(y x_i)=y_i ], (4d) HR(xi) (x_i) =symKL(pθo(xi)∥pθ′(xi)). =symKL\! (p_ _o(x_i)\,\|\,p_θ (x_i) ). (4e) Results in Table 1 answer the question in the affirmative. Proxies were calculated using models trained with identical hyper-parameters as those used to derive memorization scores (see Table 7 in Appendix A.2.1). Table 11: Spearman correlation coefficients between memorization and proxies (duplicate of Table 1, included here for completeness) CIFAR-10 CIFAR-100 Proxy ResNet-18 ViT-Small Swin-Tiny ResNet-50 ViT-Small Swin-Tiny Conf -0.80 -0.79 -0.88 -0.91 -0.85 -0.90 MaxConf -0.76 -0.77 -0.85 -0.87 -0.80 -0.86 Ent -0.75 -0.78 -0.85 -0.80 -0.77 -0.82 BA -0.71 -0.63 -0.79 -0.89 -0.69 -0.78 HR +0.67 +0.45 +0.64 +0.62 +0.50 +0.52 Confidence consistently demonstrates the strongest correlation across all models and datasets. The correlation magnitude for VTs largely resembles that of CNNs – further evidence that memorizations are fundamentally similar across these architectures. Swin-Tiny generally has stronger correlations compared to ViT-Small, possibly due to its hierarchical structure that more closely resembles traditional CNNs. Holdout Retraining shows more moderate positive (albeit still significant) correlations. Based on these findings, we selected Confidence as the best-performing learning event proxy with consistently high absolute correlation values and Holdout Retraining as a representative of a different proxy category. Importantly, Holdout Retraining offers large computational advantages as it does not require monitoring model behaviour during training. A.4 Ablation on Vanilla vs. RUM-Integrated Unlearning To isolate the effect of the RUM framework, we conduct an additional ablation comparing vanilla unlearning algorithms against their RUM-integrated counterparts. Specifically, we evaluate Fine-tuning, NegGrad+, and SalUn, both with and without RUM. Experiments are performed on CIFAR-100 using ViT-Small and Swin-Tiny architectures. Table 12: Comparison of vanilla and RUM-integrated unlearning methods on CIFAR-100. Results are reported using ToW and ToW-MIA (higher is better). Best results in each row are shown in bold. Within RUM Vanilla (w/o RUM) Method ToW ↑ ToW-MIA ↑ ToW ↑ ToW-MIA ↑ Original 0.619 0.538 Same as left Fine-tune 0.855 0.889 0.807 0.844 NegGrad+ 0.931 0.838 0.888 0.845 SalUn 0.903 0.709 0.884 0.686 (a) ViT-Small Within RUM Vanilla (No RUM) Method ToW ↑ ToW-MIA ↑ ToW ↑ ToW-MIA ↑ Original 0.670 0.586 Same as left Fine-tune 0.822 0.839 0.774 0.781 NegGrad+ 0.975 0.902 0.851 0.831 SalUn 0.870 0.751 0.833 0.640 (b) Swin-Tiny The comparison in Table 12 provides a controlled reference point for assessing the benefit of incorporating unlearning algorithms into the RUM framework. Across all methods and architectures, the RUM-based variants consistently outperform their vanilla counterparts in both ToW and ToW-MIA, indicating more effective forgetting with improved resistance to membership inference. These results complement prior findings reported for CNN-based models and further validate the generality of RUM across transformer architectures. Based on this evidence, we adopt the RUM-integrated versions of these algorithms throughout the main experimental evaluation. A.5 Results for CIFAR-10 This section presents the corresponding figures for the CIFAR-10 dataset that mirror the ones of CIFAR-100 found in Section 4. A.5.1 Unlearning Performance for CIFAR-10 (a) ToW with Confidence (b) ToW with Holdout Retraining (c) ToW-MIA with Confidence (d) ToW-MIA with Holdout Retraining Figure 5: Architecture performance comparison on CIFAR-10 Algorithm-Level Comparison on CIFAR-10. Figure 5 shows the unlearning performance results for the CIFAR-10 dataset. First, note that the original VT model (no unlearning) already achieves relatively high scores on CIFAR-10, which is unlike the results in CIFAR-100. This is consistent with our earlier observation that VTs exhibit lower memorization on this simpler dataset, reducing performance differences between θo _o and θr _r on DfD_f. Nevertheless, we see marginal improvements from unlearning algorithms on CIFAR-10, still making a case for their utilization in low-memorized data. Taking a closer look, all methods in VTs appear to show smaller improvements in ToW compared to the original model. Additionally, there is no clear winner for VTs between the two proxy strategies. Again, these results are in contrast to the more substantial improvements observed on the more complex CIFAR-100 dataset. For instance, even a simple approach like Fine-tune significantly outperforms the original model across all configurations on CIFAR-100—an effect not observed here. More advanced algorithms show greater gains in ToW performance on CIFAR-100 than they do on CIFAR-10, compared to the original VT model. Architecture-Level Comparison on CIFAR-10. Figure 5 also presents the architecture-wise comparison on CIFAR-10, analogous to the analysis shown for CIFAR-100 in Figure 1. We observe that VTs generally achieve higher ToW scores than ResNet-18, with NegGrad+ and SalUn performing particularly well on both VTs. However, this performance gap is less evident in the ToW-MIA metrics, where Fine-tune shows comparable performance across all architectures. Notably, NegGrad+ is the only unlearning method that improves ToW-MIA for both proxies. In contrast, SalUn continues to struggle on ToW-MIA within VTs: ResNet-18 significantly outperforms VTs when using the Confidence proxy and matches their performance under Holdout Retraining. A.5.2 Continual Unlearning for CIFAR-10 (a) ViT-Small (b) Swin-Tiny Figure 6: Continual Unlearning Performance Across Multiple Steps on CIFAR-10. Figure 6 and Table 16 show the results in the same fashion as seen in Figure 2 for CIFAR-100. Here we observe an even higher stability in results with even more negligible degradation, further accentuated by the smaller performance gap between ToW and ToW-MIA than those seen for CIFAR-100. This, along with higher average scores, comes as no surprise due to the higher “Original” baseline performance exhibited by the VTs on CIFAR-10. A.6 Detailed ToW and ToW-MIA Results on CIFAR-10 and CIFAR-100 This section presents detailed ToW and ToW-MIA performance results for all evaluated architectures on the CIFAR-10 and CIFAR-100 datasets. These results for ViT-Small, Swin-Tiny (Table 13(d)), and ResNets (Table 14(b)) are used to generate Figures 1 and 5. A.6.1 Results for ViT-Small and Swin-Tiny Across All Algorithms Table 13(d) provides detailed ToW and ToW-MIA results for ViT-Small and Swin-Tiny on CIFAR-10/CIFAR-100. These values are used in the corresponding architecture and method comparison figures. Table 13: Detailed performance results for the two VTs across both the CIFAR datasets. Confidence Holdout Ret Algorithm ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) Original 0.940±0.0050.940± 0.005 0.895±0.0070.895± 0.007 0.891±0.0160.891± 0.016 0.831±0.0140.831± 0.014 Fine-tune 0.934±0.0220.934± 0.022 0.919±0.0400.919± 0.040 0.928±0.0130.928± 0.013 0.913±0.0080.913± 0.008 NegGrad+ 0.969±0.0130.969± 0.013 0.938±0.0150.938± 0.015 0.916±0.0160.916± 0.016 0.968±0.0150.968± 0.015 SalUn 0.975±0.0030.975± 0.003 0.582±0.0490.582± 0.049 0.956±0.0030.956± 0.003 0.766±0.0390.766± 0.039 (a) CIFAR-10 – ViT-Small Confidence Holdout Ret Algorithm ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) Original 0.915±0.0270.915± 0.027 0.866±0.0140.866± 0.014 0.862±0.0080.862± 0.008 0.811±0.0180.811± 0.018 Fine-tune 0.930±0.0190.930± 0.019 0.922±0.0150.922± 0.015 0.923±0.0210.923± 0.021 0.931±0.0210.931± 0.021 NegGrad+ 0.971±0.0120.971± 0.012 0.922±0.0190.922± 0.019 0.977±0.0230.977± 0.023 0.924±0.0280.924± 0.028 SalUn 0.963±0.0480.963± 0.048 0.723±0.1160.723± 0.116 0.961±0.0480.961± 0.048 0.834±0.0220.834± 0.022 (b) CIFAR-10 – Swin-Tiny Confidence Holdout Ret Algorithm ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) Original 0.629±0.0380.629± 0.038 0.521±0.0260.521± 0.026 0.619±0.0400.619± 0.040 0.538±0.0230.538± 0.023 Fine-tune 0.813±0.0080.813± 0.008 0.831±0.0140.831± 0.014 0.855±0.0360.855± 0.036 0.889±0.0080.889± 0.008 NegGrad+ 0.844±0.0340.844± 0.034 0.736±0.0230.736± 0.023 0.931±0.0390.931± 0.039 0.838±0.0240.838± 0.024 SalUn 0.923±0.0190.923± 0.019 0.602±0.0140.602± 0.014 0.903±0.0490.903± 0.049 0.709±0.0850.709± 0.085 (c) CIFAR-100 – ViT-Small Confidence Holdout Ret Algorithm ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) Original 0.665±0.0140.665± 0.014 0.553±0.0150.553± 0.015 0.670±0.0270.670± 0.027 0.586±0.0250.586± 0.025 Fine-tune 0.774±0.0300.774± 0.030 0.758±0.0150.758± 0.015 0.822±0.0340.822± 0.034 0.839±0.0320.839± 0.032 NegGrad+ 0.867±0.0210.867± 0.021 0.767±0.0190.767± 0.019 0.975±0.0270.975± 0.027 0.902±0.0350.902± 0.035 SalUn 0.835±0.0600.835± 0.060 0.557±0.0340.557± 0.034 0.870±0.0390.870± 0.039 0.751±0.0430.751± 0.043 (d) CIFAR-100 – Swin-Tiny A.6.2 Results for ResNet-18 and ResNet-50 Across All Algorithms Table 14(b) reports the detailed results for ResNet-18 (CIFAR-10) and ResNet-50 (CIFAR-100). These metrics support the comparative analysis shown in Figures 1 and 5. Further breakdowns of ToW and ToW-MIA (by individual terms) for Vision Transformer architectures across datasets are reported in Table 15(d). Table 14: Detailed performance result metrics for ResNet-18 and ResNet-50 on the CIFAR-10 and CIFAR-100 datasets respectively. Confidence Holdout Retraining Algorithm ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) Fine-tune 0.919±0.0420.919± 0.042 0.920±0.0590.920± 0.059 0.920±0.0350.920± 0.035 0.865±0.0330.865± 0.033 NegGrad+ 0.880±0.0390.880± 0.039 0.843±0.0740.843± 0.074 0.870±0.0330.870± 0.033 0.791±0.0400.791± 0.040 SalUn 0.859±0.0420.859± 0.042 0.867±0.0920.867± 0.092 0.861±0.0440.861± 0.044 0.783±0.0280.783± 0.028 (a) CIFAR-10 with ResNet-18 Confidence Holdout Retraining Algorithm ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) Fine-tune 0.863±0.0490.863± 0.049 0.857±0.0590.857± 0.059 0.846±0.0320.846± 0.032 0.792±0.0440.792± 0.044 NegGrad+ 0.890±0.0470.890± 0.047 0.922±0.0170.922± 0.017 0.866±0.0310.866± 0.031 0.838±0.0270.838± 0.027 SalUn 0.665±0.0310.665± 0.031 0.636±0.0380.636± 0.038 0.696±0.0320.696± 0.032 0.699±0.0340.699± 0.034 (b) CIFAR-100 with ResNet-50 Table 15: Accuracies on DrD_r, DfD_f and DtestD_test for the “Retrain” models θr _r as well as the respective “Unlearned” models θu _u , evaluated across all unlearning algorithms within the RUMF framework. MIA performance is also reported. Results are averaged over 3 runs, with 95% confidence intervals. Confidence Holdout Retaining Algorithm Retain Acc Forget Acc Test Acc MIA Retain Acc Forget Acc Test Acc MIA Retrain 99.969±0.00699.969± 0.006 94.089±0.45694.089± 0.456 93.703±0.48693.703± 0.486 0.108±0.0070.108± 0.007 99.974±0.03599.974± 0.035 89.244±1.32189.244± 1.321 93.767±0.02993.767± 0.029 0.174±0.0120.174± 0.012 Fine-tune 99.433±0.63999.433± 0.639 94.111±0.91294.111± 0.912 87.947±1.51987.947± 1.519 0.128±0.0140.128± 0.014 99.562±0.34799.562± 0.347 87.889±2.35687.889± 2.356 88.240±0.34888.240± 0.348 0.204±0.0160.204± 0.016 NegGrad+ 99.832±0.14099.832± 0.140 93.411±1.07893.411± 1.078 91.407±0.87291.407± 0.872 0.069±0.0080.069± 0.008 99.814±0.07399.814± 0.073 83.289±2.11583.289± 2.115 91.340±0.41391.340± 0.413 0.167±0.0200.167± 0.020 SalUn 100.000±0.000100.000± 0.000 96.033±0.54396.033± 0.543 93.203±0.61793.203± 0.617 0.523±0.0570.523± 0.057 100.000±0.000100.000± 0.000 93.089±1.97193.089± 1.971 93.180±0.52393.180± 0.523 0.403±0.0420.403± 0.042 (a) CIFAR-10 with ViT-Small Confidence Holdout Retaining Algorithm Retain Acc Forget Acc Test Acc MIA Retain Acc Forget Acc Test Acc MIA Retrain 99.961±0.02199.961± 0.021 91.867±1.71191.867± 1.711 91.607±0.59691.607± 0.596 0.133±0.0020.133± 0.002 99.979±0.02799.979± 0.027 86.389±1.24686.389± 1.246 91.767±0.68991.767± 0.689 0.190±0.0200.190± 0.020 Fine-tune 99.833±0.14199.833± 0.141 96.733±0.84096.733± 0.840 89.463±0.66089.463± 0.660 0.076±0.0050.076± 0.005 99.779±0.01299.779± 0.012 91.478±0.20891.478± 0.208 89.183±0.35289.183± 0.352 0.148±0.0090.148± 0.009 NegGrad+ 99.968±0.01899.968± 0.018 93.833±0.59793.833± 0.597 90.643±1.09690.643± 1.096 0.064±0.0050.064± 0.005 99.903±0.02499.903± 0.024 81.911±0.69481.911± 0.694 90.473±0.25490.473± 0.254 0.181±0.0070.181± 0.007 SalUn 100.000±0.000100.000± 0.000 93.967±8.10393.967± 8.103 90.743±0.54790.743± 0.547 0.403±0.1260.403± 0.126 100.000±0.000100.000± 0.000 88.533±9.39988.533± 9.399 91.160±0.69291.160± 0.692 0.351±0.0430.351± 0.043 (b) CIFAR-10 with Swin-Tiny Confidence Holdout Retaining Algorithm Retain Acc Forget Acc Test Acc MIA Retain Acc Forget Acc Test Acc MIA Retrain 98.457±0.81998.457± 0.819 69.322±1.78869.322± 1.788 69.670±2.44569.670± 2.445 0.432±0.0120.432± 0.012 98.063±0.41398.063± 0.413 68.767±3.82868.767± 3.828 69.467±1.03269.467± 1.032 0.412±0.0240.412± 0.024. Fine-tune 99.325±0.14899.325± 0.148 86.056±1.90786.056± 1.907 68.193±1.57868.193± 1.578 0.283±0.0140.283± 0.014 99.322±0.62299.322± 0.622 81.200±2.76681.200± 2.766 68.350±1.43368.350± 1.433 0.322±0.0250.322± 0.025 NegGrad+ 99.846±0.16699.846± 0.166 82.200±0.29982.200± 0.299 71.440±1.79371.440± 1.793 0.192±0.0040.192± 0.004 99.637±0.17799.637± 0.177 72.722±0.62172.722± 0.621 70.997±0.44870.997± 0.448 0.276±0.0060.276± 0.006 SalUn 99.982±0.00799.982± 0.007 64.756±0.66964.756± 0.669 71.467±0.81671.467± 0.816 0.810±0.0140.810± 0.014 99.981±0.00699.981± 0.006 63.878±2.16163.878± 2.161 72.620±0.48072.620± 0.480 0.666±0.0680.666± 0.068 (c) CIFAR-100 with ViT-Small Confidence Holdout Retaining Algorithm Retain Acc Forget Acc Test Acc MIA Retain Acc Forget Acc Test Acc MIA Retrain 99.220±0.16799.220± 0.167 69.833±1.24269.833± 1.242 68.923±0.29668.923± 0.296 0.430±0.0150.430± 0.015 99.143±0.18599.143± 0.185 70.267±2.84870.267± 2.848 69.057±0.82769.057± 0.827 0.392±0.0240.392± 0.024 Fine-tune 99.638±0.18499.638± 0.184 91.811±3.01991.811± 3.019 68.503±0.40568.503± 0.405 0.194±0.0220.194± 0.022 99.368±0.55799.368± 0.557 87.289±1.30087.289± 1.300 68.377±1.53668.377± 1.536 0.239±0.0160.239± 0.016 NegGrad+ 99.940±0.04799.940± 0.047 81.900±0.95581.900± 0.955 69.410±1.08369.410± 1.083 0.207±0.0020.207± 0.002 99.914±0.04799.914± 0.047 70.111±0.72370.111± 0.723 69.833±0.52169.833± 0.521 0.308±0.0090.308± 0.009 SalUn 99.965±0.00399.965± 0.003 55.578±6.18155.578± 6.181 70.750±0.67670.750± 0.676 0.859±0.0540.859± 0.054 99.966±0.00799.966± 0.007 60.156±1.61560.156± 1.615 71.490±0.69671.490± 0.696 0.616±0.0210.616± 0.021 (d) CIFAR-100 with Swin-Tiny A.7 Continual Unlearning Results in VTs for CIFAR-10, CIFAR-100 Table 16 and Table 17 presents detailed ToW and ToW-MIA results of VTs architectures over CIFAR-10 and CIFAR-100, for five steps and ten steps, respectively. At each sequential unlearning step, we use the strong-performing combination of the NegGrad+ algorithm with the Holdout Retraining proxy. Table 16: Detailed unlearning performance at each of the 5 sequential steps for the NegGrad+ and Holdout Retraining configuration discussed in Section 4.5 and displayed in Figures 2 and 6. CIFAR-10 CIFAR-100 ViT-Small Swin-Tiny ViT-Small Swin-Tiny Step ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) ToW (↑)( ) ToW-MIA (↑)( ) 1 0.907±0.0300.907± 0.030 0.960±0.0170.960± 0.017 0.966±0.0160.966± 0.016 0.933±0.0820.933± 0.082 0.891±0.0650.891± 0.065 0.816±0.0510.816± 0.051 0.906±0.0390.906± 0.039 0.823±0.0230.823± 0.023 2 0.936±0.0630.936± 0.063 0.944±0.0290.944± 0.029 0.957±0.0160.957± 0.016 0.917±0.0300.917± 0.030 0.868±0.0310.868± 0.031 0.770±0.0150.770± 0.015 0.840±0.0640.840± 0.064 0.764±0.0850.764± 0.085 3 0.929±0.0440.929± 0.044 0.948±0.0230.948± 0.023 0.954±0.0540.954± 0.054 0.909±0.0960.909± 0.096 0.857±0.0490.857± 0.049 0.764±0.0470.764± 0.047 0.853±0.0520.853± 0.052 0.776±0.0550.776± 0.055 4 0.943±0.0100.943± 0.010 0.942±0.0280.942± 0.028 0.947±0.0580.947± 0.058 0.899±0.0580.899± 0.058 0.858±0.0530.858± 0.053 0.760±0.0400.760± 0.040 0.843±0.0210.843± 0.021 0.746±0.0350.746± 0.035 5 0.936±0.0470.936± 0.047 0.936±0.0310.936± 0.031 0.944±0.0570.944± 0.057 0.904±0.0800.904± 0.080 0.851±0.0060.851± 0.006 0.754±0.0560.754± 0.056 0.870±0.0700.870± 0.070 0.781±0.0890.781± 0.089 Table 17: Continual unlearning over 10 sequential steps on CIFAR-100 using NegGrad+ with the Holdout Retraining proxy (configuration discussed in Section 4.5). Results are reported as mean ± 95% confidence interval. Step ViT-Small Swin-Tiny ToW (↑ ) ToW-MIA (↑ ) ToW (↑ ) ToW-MIA (↑ ) 1 0.947±0.0430.947± 0.043 0.889±0.1080.889± 0.108 0.947±0.0350.947± 0.035 0.860±0.0820.860± 0.082 2 0.971±0.0330.971± 0.033 0.828±0.0400.828± 0.040 0.900±0.1040.900± 0.104 0.786±0.0640.786± 0.064 3 0.973±0.0190.973± 0.019 0.845±0.0340.845± 0.034 0.953±0.0210.953± 0.021 0.850±0.0240.850± 0.024 4 0.918±0.0530.918± 0.053 0.840±0.2440.840± 0.244 0.940±0.0810.940± 0.081 0.816±0.0940.816± 0.094 5 0.921±0.0460.921± 0.046 0.769±0.0200.769± 0.020 0.958±0.0790.958± 0.079 0.834±0.1380.834± 0.138 6 0.966±0.0290.966± 0.029 0.807±0.1290.807± 0.129 0.921±0.0590.921± 0.059 0.817±0.0250.817± 0.025 7 0.951±0.0400.951± 0.040 0.841±0.1080.841± 0.108 0.908±0.0620.908± 0.062 0.792±0.1110.792± 0.111 8 0.961±0.0240.961± 0.024 0.817±0.0460.817± 0.046 0.939±0.0410.939± 0.041 0.798±0.1260.798± 0.126 9 0.946±0.0490.946± 0.049 0.807±0.0680.807± 0.068 0.936±0.0540.936± 0.054 0.822±0.0890.822± 0.089 10 0.923±0.0800.923± 0.080 0.825±0.1150.825± 0.115 0.921±0.0720.921± 0.072 0.803±0.1000.803± 0.100 A.8 Results for SVHN A.8.1 Detailed ToW and ToW-MIA Results Across VTs for SVHN To extend our study to a different dataset, we repeat the experiments on SVHN using both ViT-Small and Swin-Tiny architectures. Table 18 lists the hyperparameter settings used for the original model training and subsequent retraining. Table 18: Training configurations and hyperparameters used for original models θo _o and retrained models θr _r alike, using ViT-Small and Swin-Tiny architectures on the SVHN dataset. Hyperparameter ViT-Small Swin-Tiny Optimizer AdamW AdamW Base learning rate 0.0001 0.0001 Loss Cross-Entropy Cross-Entropy Learning rate scheduler CosineAnnealingLR CosineAnnealingLR Batch size 256 256 Epochs 30 30 Weight decay 0.01 0.05 Data augmentation - - Following the procedure in Section 4.2, we apply all unlearn methods within the RUMF meta-algorithm to the SVHN dataset. We evaluate both ViT-Small and Swin-Tiny architectures, using Confidence and Holdout Retraining as proxy metrics. Figure 7 shows the comparative unlearning performance of these methods on SVHN. See Section 4.2 for a detailed discussion of the results. (a) ToW scores with Confidence (b) ToW scores with Holdout Retraining (c) ToW-MIA scores with Confidence (d) ToW-MIA scores with Holdout Retraining Figure 7: MU performance comparison on SVHN A.8.2 Continual Unlearning Results in VTs for SVHN We follow the same procedure outlined in Section 4.5 to sequentially apply NegGrad+ and Holdout Retraining on the SVHN dataset over five consecutive steps. The results are presented in Figure 8, evaluated by ToW and ToW-MIA. They further support the observation from Section 4.5: continual unlearning in VTs exhibits high stability across steps, with minimal degradation in performance. (a) ViT-Small (b) Swin-Tiny Figure 8: Continual Unlearning Performance Across Five Steps on SVHN