Paper deep dive
Unmerge: Efficient Machine Unlearning via Task Arithmetic
Haoran Tang, Andrew Tan, Rajiv Khanna
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/4/2026, 3:27:34 AM
Summary
The paper introduces 'Unmerge', an efficient machine unlearning algorithm that recasts unlearning as the inverse of task arithmetic model merging. Instead of gradient-based retraining, Unmerge identifies and subtracts a low-rank 'forget task vector' from the merged model weights, using activation subspace geometry to minimize retain damage and forget leakage. It demonstrates superior performance over existing baselines on ResNet-50, ViT, and Llama models.
Entities (12)
Relation Signals (8)
Unmerge → appliesto → ResNet-50
confidence 95% · on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet
Unmerge → usesmethodology → Task Arithmetic
confidence 95% · We recast unlearning through the lens of task arithmetic... We call this unmerge
Unmerge → reduces → forget leakage
confidence 92% · provably bound forget leakage and retain damage in activation space
Unmerge → reduces → retain damage
confidence 92% · provably bound forget leakage and retain damage in activation space
Unmerge → appliesto → Llama-3.2-3B
confidence 90% · Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B.
Unmerge → outperforms → Tug-of-War
confidence 90% · it improves Tug-of-War by up to ~24% over a baseline of comparable runtime
EU GDPR → motivates → Machine Unlearning
confidence 88% · Regulatory frameworks such as the EU GDPR... codify a right to be forgotten... demand mechanisms that delete the footprint
Task Arithmetic → inspiredby → Cheng et al., 2025
confidence 85% · Building on the activation-subspace assumption of Cheng et al. (2025)...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We recast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector $\tau_m$ that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component $\tau_F$ to recover the retain task vector $\tau_R$. The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by a handful of dominant directions, so we factorize $\tau_F$ in a low-rank forget basis, which is faithful up to a small tail-eigenvalue residual and limits how far the correction can perturb retain. We then optimize three intuitive goals (match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size) that provably bound forget leakage and retain damage in activation space. The resulting algorithm, Unmerge, is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ~24% over a baseline of comparable runtime and by up to ~18% over stronger baselines that run ~5x slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard.
Tags
Links
- Source: https://arxiv.org/abs/2609.38895v1
- Canonical: https://arxiv.org/abs/2609.38895v1
Trouble viewing inline? Open PDF directly →
Full Text
166,802 characters extracted from source content.
Expand or collapse full text
Unmerge: Efficient Machine Unlearning via Task Arithmetic Haoran Tang Andrew Tan Rajiv Khanna Affiliation: Department of Computer Science Affiliation: Purdue University Email: thr, tan434, rajivak@purdue.edu Abstract Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We re-cast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector m τ_m that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component ℱ τ_F to recover the retain task vector ℛ τ_R. The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by only a handful of dominant directions. We exploit this with a low-rank factorization ℱ=Qℱ⊤ τ_F=BQ_F , which is faithful up to a small tail-eigenvalue residual and bounds how far the correction can perturb retain. We then optimize three intuitive goals—match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size—that provably bound forget leakage and retain damage in activation space. The resulting algorithm Unmerge is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ∼ 24% over a baseline of comparable runtime and by up to ∼ 18% over stronger baselines that run ∼5× 5× slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard. 1 Introduction Machine unlearning, which aims to remove the influence of a subset of training data from an already-trained model, has moved from an abstract ideal to an operational requirement. Regulatory frameworks such as the EU GDPR (Voigt and von dem Bussche, 2017) codify a right to be forgotten, and emerging concerns around copyright, privacy, and safety in foundation models (Li et al., 2024) demand mechanisms that delete the footprint of specific data without retraining from scratch. Exact unlearning by partition-and-retrain (Bourtoule et al., 2021) is the gold standard but is computationally prohibitive for modern deep networks, motivating a large body of approximate methods that edit the trained weights post-hoc. We focus on concept-level requests—removing a class, a semantic group or a behavior, of which the forget set is a sample—rather than the deletion of arbitrary individual records, which is a distinct problem with its own reference and metrics (§4.5). The dominant approximate methods can be grouped into four families. Gradient-ascent methods (Kurmanji et al., 2023) reverse-train on the forget set with a retain regularizer; relabeling methods (Graves et al., 2021) replace forget labels with random or noise targets; teacher-student methods (Chundawat et al., 2023) distill an “incompetent teacher” into the forget region; and saliency- or Fisher-based methods (Golatkar et al., 2020; Fan et al., 2024; Foster et al., 2024) dampen weights estimated to encode the forget signal. All four ultimately drive a forget loss to reshape the weights, and trade forget completeness against retain damage through hyperparameter tuning. Despite rapid progress, this line of work shares three limitations: (i) Fragility: gradient-ascent and relabeling variants operate on a knife-edge, where one extra epoch can flip a model from under-forgetting to catastrophic collapse, forcing per-task tuning; (i) Entanglement blindness: scalar forget/retain losses give the optimizer no language for reasoning about where forget and retain knowledge coexist, so when the two sets share features (the common case) retain damage is an unavoidable byproduct; (i) Opacity: methods are evaluated almost entirely through scalar accuracy and privacy numbers and expose nothing about why a layer or a direction was (un)successfully unlearned, leaving diagnosis, comparison, and trust on shaky ground. We aim to address all three by revisiting unlearning from a new angle: task arithmetic (Ilharco et al., 2023). If finetuning on =ℛ∪ℱS=R for retain set ℛR and forget set ℱF is read as implicit multi-task learning that produces a merged task vector m τ_m, then unlearning ℱF becomes the inverse problem: identify the forget component ℱ τ_F inside m τ_m and subtract it. We call this unmerge, and view it as the natural counterpart to recent data-free model merging (Cheng et al., 2025): the activation-subspace geometry that governs cross-task interference when combining experts also governs forget-retain leakage when separating them, with the optimization direction reversed. This single conceptual shift addresses each of the three limitations above. Fragility is replaced with a convex quadratic per layer, solved in a k-dimensional projected space without any gradient on data, so there is no ascent-descent collapse to guard against, and the one remaining scale α can be usually set from forget-set accuracy alone (App. D.1). Entanglement becomes a first-class object: forget and retain subspaces are explicitly constructed and their overlap is measurable per layer and per direction. Opacity dissolves because the algorithm’s internal state—bases, overlaps, per-layer task-vector magnitudes—is directly inspectable. The implications go beyond a single algorithm: unmerging recasts unlearning from a tuning problem on top of opaque weight edits into a structured decomposition problem with a principled subspace vocabulary. That vocabulary predicts unlearning difficulty before any update is applied, localizes where in a network the forget signal lives (late layers are the high-leverage region), and delimits the method’s scope: a forget request with no separable activation subspace—random instance subsets being the extreme case—cannot be unmerged, and Unmerge flags this before editing (§4.5). The diagnostics are method-agnostic, the formulation extends to concept removal and modular unlearning in larger models, and the gradient-free edit is amortizable across forget requests on the same checkpoint. Empirically, our instantiation improves the retain/forget tradeoff over gradient-based and task-vector baselines on ResNet-50 (CIFAR-100, TinyImageNet) and ViT-S (CIFAR-100), avoids the collapse mode of gradient-ascent unlearning, and matches retraining closely under membership-inference attacks on the forget set where relabeling methods leave a larger signature on the forget set. We summarize our contributions as follows: • A new formulation. We recast approximate unlearning as the inverse of model merging and introduce Unmerge: learning a per-layer forget task vector and subtracting it from m τ_m so that the remainder behaves as a retain-only task vector in activation space (§3). • Theoretical motivation. Building on the activation-subspace assumption of Cheng et al. (2025), we give an activation-space analysis that connects each term of our objective to an interpretable quantity: forget leakage, retain damage, and tail-direction over-expansion, positioning unmerging as the counterpart of their merging framework and inviting more rigorous theoretical work (§3.3). • Gradient-free unlearning. We instantiate a low-rank procedure—a convex per-layer objective solved in a k-dimensional projected space—whose only data access is forward passes for activation statistics. It improves the retain/forget tradeoff over gradient-based and task-vector baselines on ResNet-50 (CIFAR-100, TinyImageNet) and ViT-S (CIFAR-100), avoids the collapse failure mode of gradient-ascent methods, and recovers a model closer to retraining in output space (§4). Ablations show that the retain-leakage term carries this gain, and a preliminary study carries the method over to a 3B-parameter language model (App. E.4). • Interpretability tools. We introduce per-layer, per-direction diagnostics—task-vector magnitudes, forget-retain overlap, and their depth-wise profiles—that make unlearning diagnosable layer by layer. A separability score computed before unlearning predicts which forget requests lie beyond the reach of subspace editing; we characterize that boundary with instance-level experiments (§4.5). 2 Related Work Task arithmetic and model merging. Task vectors (Ilharco et al., 2023) are weight-space deltas induced by finetuning that support simple arithmetic for composing or suppressing behaviors without retraining. Subsequent merging work refines this view: weight averaging across finetuned experts (Wortsman et al., 2022; Matena and Raffel, 2022), conflict-aware pruning before averaging (Yadav et al., 2023; Yu et al., 2024), closed-form layer-activation objectives (Jin et al., 2023), and tangent-space analyses of when task-vector linearity holds (Ortiz-Jiménez et al., 2023). Most closely, Cheng et al. (2025) show that task-vector interference is governed by the span of layer inputs and derive a data-free merging objective. Our work builds directly on this view but reverses it: instead of combining task vectors while minimizing cross-interference, we unmerge—decompose an already-merged vector by learning the sub-component that lives in a forget subspace and subtracting it. The handful of papers that compose task vectors for removal (e.g., subtracting a toxicity vector (Ilharco et al., 2023; Ilharco et al., 2022; Zhang et al., 2023)) assume an explicit forget task vector is available, whereas we construct and learn one from data-free bases on the finetuned model itself. Recent task-vector unlearning obtains that vector by finetuning on the forget set and negating it, merging several such vectors under a sign-consensus mask (Kim et al., 2025), or constraining finetuning so that merged models admit exact unlearning at scale (Kuo et al., 2026). §4.2 compares against the negated-vector and NegMerge baselines. Machine unlearning. Motivated by right-to-be-forgotten regulations (Voigt and von dem Bussche, 2017) and the desire to remove poisoned or copyrighted data, machine unlearning (Cao and Yang, 2015; Nguyen et al., 2025; Wang et al., 2024) aims to produce a model indistinguishable from one retrained on ℛ=∖ℱR=S . Exact methods partition training so forgetting a point invalidates one shard (Bourtoule et al., 2021) but require training-time instrumentation. Approximate methods operate post-hoc on FT W_FT: influence- or Fisher-based scrubbing (Golatkar et al., 2020; Golatkar et al., 2021), gradient-ascent variants such as NegGrad+ and SCRUB (Kurmanji et al., 2023), saliency-masked updates (Fan et al., 2024), synaptic dampening (Foster et al., 2024), boundary shifting (Chen et al., 2023), amnesiac relabeling (Graves et al., 2021; Tarun et al., 2024), and distillation-based erasure that separates forgetting from retention through a teacher (Chundawat et al., 2023; Zhou et al., 2025) or adds a masked-distillation penalty against over-unlearning (Ha et al., 2026). Compared to these, Unmerge is data-free at optimization (data enters only when collecting second-moment statistics), layerwise and interpretable, and motivated by a merging-theoretic objective with activation-space bounds (Prop. 3.3); it performs no training on ℱF or ℛR at unlearning time (App. B.3). Subspace projection and representation editing. A separate thread exploits low-dimensional structure to intervene on behaviors: continual-learning subspace methods project gradients orthogonal to previous tasks (Saha et al., 2021; Farajtabar et al., 2020), while concept-erasure and knowledge-editing methods intervene on activation directions or low-rank weight subspaces (Kim et al., 2018; Belrose et al., 2023; Ravfogel et al., 2022; Meng et al., 2022). Closest to our edit are gradient-free class-unlearning methods that project weights away from forget activation subspaces (Kodge et al., 2024; Chen et al., 2024); with init=0 W_init=0 our update is a soft member of this family, from which it differs by acting on the task vector, by solving an explicit forgetting-vs-leakage objective instead of a hard projection, and by its supervised bases and diagnostics (App. A). We adopt Concept Activation Vectors (Kim et al., 2018) as one of our basis constructions but lift them from activation-level probing to weight-level subtraction, and unlike hard orthogonal projection we keep the natural overlap between ℱ Q_F and ℛ Q_R and softly penalize retain leakage via Term 2 of equation 2 (see App. A for method-specific comparisons). 3 Unlearning via Unmerging 3.1 Preliminaries Let init∈0,PT W_init∈\ W_0, W_PT\ be the starting weights: a public pretrained checkpoint PT W_PT, or training from scratch (0 W_0), for which we take init=0 W_init=0 rather than the random draw so that m=FT τ_m= W_FT (ablated in App. D.4). Finetuning on a train set S gives FT W_FT. The forget set ℱ⊂F is revealed after finetuning, and ℛ=∖ℱR=S is the retain set with |ℛ|>|ℱ||R|>|F|. An unlearning algorithm maps FT W_FT to UL W_UL using ℱF and ℛR; the ideal target is the model RT W_RT retrained from init W_init on ℛR only. Following Task Arithmetic Ilharco et al. (2023), a task vector is the weight difference induced by training, and we define m=FT−init,ℛ⋆=RT−init,ℛ≔m−ℱ, τ_m= W_FT- W_init, τ_R = W_RT- W_init, τ_R τ_m- τ_F, (1) where m τ_m is the merged task vector (since =ℛ∪ℱS=R ) and ℛ⋆ τ_R the ideal retrained one. We learn a parameterized forget task vector ℱ τ_F such that the remainder ℛ τ_R behaves like a retain-only task vector in activation space; §4.5 measures how close ℛ τ_R lands to ℛ⋆ τ_R . Why WFT W_FT can be treated as merged. Joint finetuning on ℛ∪ℱR is not additive, and we never assume m=ℱ+ℛ τ_m= τ_F+ τ_R for separately trained experts: ℛ τ_R is defined as the remainder, and the objective selects the ℱ τ_F that makes it approximately orthogonal to the forget span. Assumption 3.1 makes this meaningful for a jointly trained vector, since the rows of m τ_m lie in the span of the joint input activations. “Unmerging” thus names the objective rather than the training provenance, and unlike task arithmetic it needs no separately trained forget expert. Parameterizing forget task vector. Cheng et al. (2025) show that the task vector at layer l of a multi-layer neural network is spanned by l-th layer’s inputs. This motivates us to construct a parameterized forget task vector ℱl∈ℝm×dl τ_F^l ^m× d_l at layer l based on forget inputs ℱl∈ℝNℱ×dl X_F^l ^N_F× d_l at that layer. Denote ℱl=lℱl⊤ τ_F^l= B^l Q_F^l with a learnable matrix l∈ℝm×k B^l ^m× k and a forget basis matrix ℱl∈ℝdl×k Q_F^l ^d_l× k. We can construct ℱl Q_F^l either by singular value decomposition of ℱl X_F^l (unsupervised, high-energy basis) or by ridge-probe concept-activation vectors Kim et al. (2018) (supervised, discriminative basis). The forget projector is ℱl=ℱlℱl⊤ P_F^l= Q_F^l Q_F^l ; the retain basis ℛl∈ℝdl×k Q_R^l ^d_l× k and projector ℛl P_R^l are built likewise from the retain inputs ℛl∈ℝNℛ×dl X_R^l ^N_R× d_l. 3.2 Unmerging The objective decouples across layers. For each layer we learn ℱl τ_F^l such that the remainder ℛl=ml−ℱl τ_R^l= τ_m^l- τ_F^l is approximately orthogonal to the forget span and ℱl τ_F^l approximately orthogonal to the retain span, and we regularize the size of ℱl τ_F^l against over-expansion. The per-layer Unmerge objective, and the update that applies the learned ℱl=lℱl⊤ τ_F^l= B^l Q_F^l with a scaling α>0α>0, are minℱl‖(ml−ℱl)ℱl‖F2+λ||ℱlℛl||F2+γ||ℱl||F2, _ τ_F^l\,|| ( τ_m^l- τ_F^l ) P_F^l||^2_F+λ|| τ_F^l P_R^l||^2_F+γ|| τ_F^l||^2_F, (2) ULl=FTl−αℱl. W_UL^l= W_FT^l-α\, τ_F^l. (3) With orthonormal bases, equation 2 can be reduced to a strictly convex quadratic in l B^l alone, ‖mlℱl−l‖F2+λ‖ll‖F2+γ‖l‖F2\| τ_m^l Q_F^l- B^l\|_F^2+λ\| B^l C^l\|_F^2+γ\| B^l\|_F^2 with l=ℱl⊤ℛl∈ℝk×k C^l= Q_F^l Q_R^l ^k× k: mkmk parameters per layer, no data access, and a retain term that acts only along the directions ℱl Q_F^l shares with ℛl Q_R^l. Why low rank suffices. Replacing ℱl P_F^l and ℛl P_R^l with the empirical activation second-moments 1Nℱℱl⊤ℱl 1N_F X_F^l X_F^l and 1Nℛℛl⊤ℛl 1N_R X_R^l X_R^l recovers the data-space form of the same objective, at mdlmd_l instead of mkmk parameters per layer and with the activations revisited at every step. The low-rank form is both tighter and safer. By Assumption 3.1, the rows of ℱl τ_F^l lie approximately in the span of the forget activations, so the top-k eigenbasis is faithful up to a tail residual bounded by Lemma 3.2, and the forget eigenspectrum decays sharply in the layers we consider. The rank-k form also caps how far ℱl τ_F^l can perturb directions outside the forget subspace, which disproportionately carry retain information: an overly large k hurts retain accuracy at the same α (App. D.1). Implementation. Unmerge runs in three phases—(1) one forward pass over ℱ∪ℛF builds ℱl Q_F^l and ℛl Q_R^l; (2) l B^l is initialized at the projection 0l=mlℱl B^l_0= τ_m^l Q_F^l and moved toward the minimizer of the projected quadratic under a fixed budget of T=100T=100 Adam steps (<1<\!1s); (3) equation 3 is applied, optionally only to deep layers (skip-layers, §4.3)—and is optimization-time data-free: no gradient is ever taken on a training sample (pseudocode and details in App. B.1 and B.3). Solver and step budget. The projected quadratic has the unique minimizer l⋆=mlℱl((1+γ)+λll⊤)−1, B^l = τ_m^l Q_F^l ((1+γ) I+λ\, C^l C^l )^-1, (4) a single k×k× k solve per layer (App. B.2). We do not run Phase 2 to convergence: the fixed budget is part of the estimator, and because Adam’s normalized steps move each entry by at most δ=lr×Tδ=lr× T, this motivates the approximation δl=l⋆+Sδ(0l−l⋆) B^l_δ= B^l +S_δ( B^l_0- B^l ) with SδS_δ the entrywise soft-threshold: every coefficient reaches its regularized optimum except the few largest projection coefficients, which are shrunk only by δ. Early stopping thus acts as an implicit regularizer (Yao et al., 2007); for the low-overlap CAV bases the iterate coincides with the minimizer, for PCA the residual carries part of the edit (see App. D.3 for comparison with actual iterate). Removing terms with α re-swept shows that Term 2 accounts for essentially all of the gain over the plain projection (App. D.2). 3.3 Theoretical Motivation We motivate the objective with activation-space bounds, adapting recent model-merging theory Cheng et al. (2025) to unlearning; the analysis is per layer and explains what each term controls. Assumption 3.1 (Input Subspace Property Cheng et al. (2025)). For a model finetuned with learning rate η under Lipschitz continuity, the task vector ∈ℝm×d τ ^m× d at a given layer satisfies =+ τ= C X+ E where ∈ℝm×N C ^m× N is a coefficient matrix, ∈ℝN×d X ^N× d collects the layer inputs during training, and ‖F=(η2)\| E\|_F=O(η^2). Every SGD update of a linear layer is an outer product with that layer’s inputs, so the span property is exact up to the drift of those inputs during training; the (η2)O(η^2) rate is a finetuning idealization that we use heuristically for from-scratch training (App. B.2). It motivates our low-rank parameterization: Lemma 3.2 (Low-Rank Approximation). Let ℱ=1Nℱℱ⊤ℱ _F= 1N_F X_F X_F be the second-moment matrix of forget inputs with eigenvalues ν1≥⋯≥νd≥0 _1≥·s≥ _d≥ 0, and let ℱ∈ℝd×k Q_F ^d× k be the top-k eigenvectors with projector ℱ=ℱℱ⊤ P_F= Q_F Q_F . Under Assumption 3.1, any task vector τ whose rows lie in span(ℱ⊤)span( X_F ) admits the decomposition =ℱ⊤+ τ= B Q_F + E with =ℱ∈ℝm×k B= τ Q_F ^m× k and ∥F2≤∥op2⋅Nℱ∑i=k+1dνi.\| E\|_F^2≤\| C\|_op^2· N_F _i=k+1^d _i. (5) The approximation error of ℱ=ℱ⊤ τ_F= B Q_F thus decays with the tail eigenvalues of the forget covariance, so rank k suffices when the forget signal is low-dimensional. The three terms of our objective, in turn, control the activation-space quantities that matter for unlearning. Proposition 3.3 (Activation-Space Bounds). Let ℱ∈ℝd×d _F ^d× d be the second-moment matrix of forget inputs with eigenvalues ν1≥⋯≥νd _1≥·s≥ _d and top-k projector ℱ P_F. Similarly, let ℛ _R have eigenvalues μ1≥⋯≥μd _1≥·s≥ _d with top-k projector ℛ P_R. Let α>0α>0 be the scaling used in the update and denote ℛ=m−αℱ τ_R= τ_m-α τ_F. Then: x∼ℱ[‖ℛx‖2]≤ _x \! [\| τ_R\,x\|^2 ]≤ ν1‖(m−αℱ)ℱ‖F2+νk+1‖m−αℱ‖F2, _1 \|( τ_m-α τ_F) P_F \|_F^2+ _k+1 \| τ_m-α τ_F \|_F^2, (6) x∼ℛ[‖αℱx‖2]≤ _x \! [\|α τ_F\,x\|^2 ]≤ α2(μ1‖ℱℛ‖F2+μk+1‖ℱ‖F2). α^2 ( _1 \| τ_F P_R \|_F^2+ _k+1 \| τ_F \|_F^2 ). (7) Proofs are in App. B.2. Each term of the objective controls one component of these bounds: • Term 1 ‖(m−ℱ)ℱ‖F2\|( τ_m- τ_F) P_F\|_F^2 controls the dominant (ν1 _1-weighted) component of forget leakage equation 6—the response of the unlearned model on forget inputs. • Term 2 ‖ℱℛ‖F2\| τ_F P_R\|_F^2 controls the dominant (μ1 _1-weighted) component of retain damage equation 7—how much the forget correction disrupts retain performance. • Term 3 ‖ℱ‖F2\| τ_F\|_F^2 controls the tail (μk+1 _k+1-weighted) component of retain damage, providing uniform protection across all directions beyond the top-k retain subspace. Both bounds hold for any ℱ τ_F, hence for the budgeted iterate we apply, and Terms 2 and 3 carry the factor α2α^2, so α trades forgetting strength against retain damage (§4.3). Corollary 3.4 (The minimizer). Let σ1≥⋯≥σk∈[0,1] _1≥…≥ _k∈[0,1] be the singular values of =ℱ⊤ℛ C= Q_F Q_R, the cosines of the principal angles between the forget and retain spans, with left singular vectors V. Then ⋆=mℱdiag(gi)⊤ B = τ_m Q_F\, V\,diag (g_i ) V with gi=1/(1+γ+λσi2)g_i=1/(1+γ+λ _i^2): the minimizer keeps forget-private directions (σi≈0 _i\!≈\!0) at gain 1/(1+γ)1/(1+γ) and shrinks directions shared with retain (σi≈1 _i\!≈\!1) by up to 1/(1+γ+λ)1/(1+γ+λ), and the leading term of equation 7 shrinks by the factor gi2g_i^2 along each direction. Term 2 therefore acts exactly where the two spans meet, which is the mechanism behind the ablation of App. D.2. The selected α exceeds the unit gain at which the first term of equation 6 is smallest, which we attribute to forget signal in its tail term; the scope of the bounds is discussed in App. B.2. Remark 3.5 (Relation to Model Merging). Our unmerging objective mirrors the model-merging objective of Cheng et al. (2025). For two tasks ℱF and ℛR with task vectors ℱ,ℛ τ_F, τ_R, their merging objective seeks m τ_m minimizing ∑i‖(m−i)i⊤‖F2 _i\|( τ_m- τ_i) τ_i \|_F^2, i.e., bringing the merged vector close to each expert in its own subspace. Substituting m=ℱ+ℛ τ_m= τ_F+ τ_R yields the cross-interference terms ‖ℛℱ⊤‖F2+‖ℱℛ⊤‖F2\| τ_R τ_F \|_F^2+\| τ_F τ_R \|_F^2. By Assumption 3.1, i⊤ τ_i spans approximately the same subspace as i P_i, recovering our Terms 1 and 2. Merging minimizes interference when combining tasks; unmerging maximizes separation when removing one—the same subspace vocabulary with the direction reversed; the correspondence is an analogy at the level of the objectives, not a formal duality. 4 Experiments 4.1 Setup Models, data, and protocol. We use ResNet-50 (He et al., 2016) on CIFAR-100 (Krizhevsky, 2009) and TinyImageNet (Le and Yang, 2015), and ViT-S/16 (Dosovitskiy et al., 2021) on CIFAR-100 to probe architectural generality (App. E.3). Each model is finetuned from both initializations init∈0,PT W_init\!∈\!\ W_0, W_PT\ to obtain FT W_FT, which lets us probe how task-vector magnitude affects unmerging. Forget requests are class-level: a five-class superclass on CIFAR-100 (Trees, Aquatic Mammals, Vehicles) and a ten-class semantic group on TinyImageNet (Dogs & Cats, Arthropods); ℱF is the union of the chosen classes’ training images and ℛ=∖ℱR=S . Two instance-level protocols are introduced in §4.5; class indices, training schedules, and all configurations are in App. C. Baselines. We compare with the retrained model RT W_RT (gold standard), with NegGrad+ (Kurmanji et al., 2023), Random Label (RL) (Graves et al., 2021), SalUn (Fan et al., 2024), the task-vector family (negated task vector (NegTV) (Ilharco et al., 2023) and NegMerge (Kim et al., 2025)), and among gradient-free subspace methods, with the activation-space projection of Kodge et al. (2024). Hyperparameters are tuned per (method, init) on Trees and Dogs & Cats and transferred to the other tasks with minimal changes, which stresses transferability (App. C). Our variants are Unmerge-PCA and Unmerge-CAV in kmeans and class modes (§3, Alg. 3). Metrics. We use three evaluation pipelines, each read as a gap to retraining (the gold standard) rather than in absolute terms. (1) Performance. Forget, retain and held-out test accuracy, and the composite tug-of-war score of Zhao et al. (2024) ToW=(1−|Accℱ−AccℱRT|)⋅(1−|Accℛ−AccℛRT|)⋅(1−|Acctest−AcctestRT|),ToW= (1-|Acc_F-Acc_F^RT| )· (1-|Acc_R-Acc_R^RT| )· (1-|Acc_test-Acc_test^RT| ), (8) which equals 11 when RT W_RT is recovered exactly and penalizes over- and under-unlearning alike. (2) Entanglement (§4.4). Distribution distances between forget and retain features at the penultimate layer, as a relative gap to RT W_RT. (3) Privacy (§4.5). The gap in membership-inference attack accuracy on the forget set between UL W_UL and RT W_RT, following Zhao et al. (2024); Fan et al. (2024); Shokri et al. (2017); Song and Mittal (2021) (App. E.6); a gap far above retraining signals residual memorization, where far below signals over-unlearning (§4.5). App. E.2 reports three-seed μ and σ on Trees. 4.2 Main Unlearning Results Tables 1 (CIFAR-100, mean ± std across three superclasses) and 2 (TinyImageNet, Dogs & Cats) summarize the main comparisons; detailed per-superclass tables and the TinyImageNet Arthropods replication are deferred to App. E.1. Unmerge consistently recovers retain and test accuracy close to retraining while driving forget accuracy toward the retrained level, and does so without any gradient updates on the training data. We highlight several observations: Cross-task consistency. On both datasets and both initializations Unmerge variants lead on tug-of-war (ToW); on CIFAR-100 at least two of the three variants sit above every gradient-based baseline, on TinyImageNet/0 W_0 only CAV-Class does. The dispersion across superclasses (Tab. 1) is small for the leading Unmerge cells (±0.005± 0.005–0.0060.006 ToW) and large for the most disruptive baselines (Random Label forget accuracy ±5.1%9.2\!±\!5.1\% on PT W_PT). Over three seeds on Trees every ordering of the Unmerge variants relative to SalUn and RL is preserved, with 1.71.7–4.5×4.5× smaller ToW standard deviations on 0 W_0 (App. E.2); NegTV and NegMerge trail the best variant by 0.120.12–0.240.24 ToW on every task because the vector they subtract, finetuned on ℱF, overlaps retain directions, which our leakage Term 2 of equation 2 penalizes; the projection of Kodge et al. (2024) is the most competitive baseline on CIFAR-100 (mean ToW 0.9620.962/0.9440.944, within 0.020.02–0.050.05 of our best variant) but trails it by 0.170.17–0.350.35 on TinyImageNet, where it leaves 18%18\% forget accuracy on 0 W_0; and on ViT-S/16 (retrained test 86.1%86.1\%; App. E.3) Unmerge-CAV-KMeans leads every axis (ToW 0.99750.9975 vs. 0.99490.9949 for SalUn at 14×14× lower runtime, MIA gap and entanglement 1.8×1.8× and 7×7× closer to retraining). On 0 W_0 where m τ_m is larger in deeper layers, supervision pays off more: Unmerge-CAV-Class leads on both datasets (0.99090.9909 on CIFAR-100, 0.91520.9152 on TinyImageNet), and PCA is most exposed because its top-k eigenvectors absorb shared forget/retain directions. Efficiency. Unmerge takes 88–105105s on one H100: the PCA variant is about 2×2× faster than NegGrad+, the CAV variants are on par with it, and all are ∼ 5×5× faster than Random Label and SalUn (110110–113113s on CIFAR-100, 666666–679679s on TinyImageNet), without any retain-loop fine-tuning. Combined with the ToW gains this is up to a ∼ 24% relative lift over NegGrad+, the comparable-runtime gradient baseline (the task-vector and projection baselines are as fast or faster but trail by more), and ∼ 18% over Random Label and SalUn at 5×5× their speed. Over 95%95\% of the runtime is the forward pass and basis construction; the speedup is set by the baselines’ epochs, not by model size (App. B.3). Scaling up for Language models. Though not the focus of our study, we conduct preliminary experiments on TOFU with Llama-3.2-3B-Instruct (App. E.4) and find that Unmerge also works on a 3B-parameter LLM: without any training (a retain-deflated PCA basis and the closed-form solver replace the vision configuration), the edited model reaches a forget quality of 0.130.13 at a model utility of 0.6300.630 (retain-only model: 0.6500.650), against 0.090.09 and 0.6330.633 for RMU, the strongest default-setting baseline we ran, indicating promising potential for future extension. Table 1: CIFAR-100 unlearning across three superclasses (Trees, Aquatic Mammals, Vehicles). Cells report mean ± 1 std across superclasses (sample std, N=3N=3; one seed per cell, three-seed variance in App. E.2); RT W_RT shows plain means. Best non-retrained in bold. Unmerge variants achieve best ToW performance with noticeable short runtime. Per-superclass detail tables are in App. E.1. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.00 0.31 ± 0.28 9.24 ± 5.09 6.12 ± 2.16 2.27 ± 2.19 3.43 ± 3.47 0.32 ± 0.14 0.11 ± 0.08 0.08 ± 0.11 0.69 ± 0.65 Retain Acc ↑ 99.98 96.64 ± 1.13 99.94 ± 0.00 99.94 ± 0.04 95.10 ± 4.08 95.83 ± 3.96 99.39 ± 0.49 99.80 ± 0.09 99.63 ± 0.20 99.22 ± 0.36 Test Acc ↑ 80.52 73.82 ± 0.66 80.31 ± 0.29 80.52 ± 0.60 72.81 ± 4.02 73.52 ± 4.09 77.62 ± 1.35 79.31 ± 0.25 78.33 ± 0.74 76.90 ± 0.96 ToW ↑ 1.0000 0.8992 ± 0.0199 0.9053 ± 0.0485 0.9365 ± 0.0226 0.8581 ± 0.0562 0.8604 ± 0.0479 0.9622 ± 0.0150 0.9851 ± 0.0046 0.9739 ± 0.0077 0.9498 ± 0.0070 Runtime ↓ 23 ± 0 110 ± 0 112 ± 0 4 ± 0 11 ± 0 8 ± 0 8 ± 0 26 ± 0 22 ± 0 init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.00 4.53 ± 1.13 9.40 ± 6.02 3.92 ± 0.73 7.32 ± 6.86 7.84 ± 6.77 0.87 ± 0.87 0.51 ± 0.34 0.41 ± 0.36 0.43 ± 0.61 Retain Acc ↑ 99.97 96.89 ± 0.75 99.96 ± 0.01 99.95 ± 0.02 95.73 ± 1.14 96.21 ± 1.15 98.35 ± 1.14 97.05 ± 1.53 99.70 ± 0.11 99.85 ± 0.03 Test Acc ↑ 72.47 67.54 ± 1.49 73.95 ± 0.63 74.18 ± 0.50 66.72 ± 1.89 66.95 ± 1.88 69.19 ± 2.07 68.01 ± 2.00 71.79 ± 0.95 72.66 ± 0.82 ToW ↑ 1.0000 0.8798 ± 0.0235 0.8925 ± 0.0564 0.9443 ± 0.0111 0.8370 ± 0.0722 0.8387 ± 0.0755 0.9435 ± 0.0317 0.9230 ± 0.0347 0.9865 ± 0.0029 0.9909 ± 0.0062 Runtime ↓ 23 ± 1 110 ± 1 113 ± 2 4 ± 0 11 ± 0 8 ± 0 8 ± 2 21 ± 0 18 ± 0 Table 2: TinyImageNet unlearning results (Dogs & Cats). Best non-retrained in bold. Unmerge-CAV-Class achieves the best ToW for both init W_init at a runtime comparable to NegGrad+; the task-vector and projection baselines are faster but trail on ToW. See Arthropods results in Tab. 13. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 1.44 16.26 15.46 0.26 0.86 1.90 0.68 0.00 0.04 Retain Acc ↑ 99.98 96.36 99.98 99.98 89.70 91.62 93.30 99.37 99.96 99.94 Test Acc ↑ 74.46 67.98 74.65 74.62 62.57 64.29 63.91 72.07 73.41 73.58 ToW ↑ 1 0.8883 0.8358 0.8440 0.7885 0.8161 0.8189 0.9635 0.9893 0.9904 Runtime ↓ 93s 666s 671s 18s 54s 22s 71s 105s 99s init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 0.26 9.44 8.92 8.50 9.48 17.74 4.60 2.82 0.52 Retain Acc ↑ 99.98 85.64 99.98 99.97 93.29 94.06 96.39 83.72 97.87 97.88 Test Acc ↑ 61.78 48.08 61.29 61.12 52.18 52.45 54.46 46.10 55.31 55.75 ToW ↑ 1 0.7373 0.9011 0.9047 0.7718 0.7722 0.7350 0.6736 0.8898 0.9152 Runtime ↓ 101s 670s 679s 18s 53s 23s 56s 86s 84s 4.3 Where Does Unlearning Happen? Because Unmerge builds per-layer bases ℱl,ℛl Q_F^l, Q_R^l and extracts ℱl τ_F^l, we can quantify each layer’s contribution. Empirically, the useful unmerging budget concentrates in the deeper layers: skipping the first 35/40 conv layers in ResNet-50 retains the forget drop while improving retain/test accuracy by 2–5 points. This matches the feature hierarchy of vision networks—generic edges and textures early, class-specific parts and objects late (Zeiler and Fergus, 2014; Yosinski et al., 2014; Bau et al., 2017)—and the depth-aware edits of recent class-unlearning methods (Hatami et al., 2026; Chang et al., 2026). It is a property of the request rather than of the method: a class- or concept-level forget set is separable where semantics are encoded, whereas a request defined by a low-level attribute such as a colour or texture would call for earlier layers, which the same diagnostics would indicate. Theory predicts the depth profile: early layers have larger tail eigenvalues ∑i>kνi _i>k _i (Lemma 3.2) and a larger retain-damage constant μ1 _1 (equation 7) because generic edge/texture features dominate their second-moment spectrum, so any forget correction leaks heavily into retain; deep layers encode class-specific representations, μk+1 _k+1 decays, and the forget/retain spectra are more separable. Per-layer geometry. Figs. 1 and 2 (App. D.1) make this visible on Trees. Magnitude (Fig. 2): the cross-init gap is depth-dependent—in layer1–2 ‖m‖\| τ_m\| is slightly larger on PT W_PT (pretrained features must be shifted away from ImageNet), but at layer4 and the classifier it is ∼ 1.51.5–2×2× larger on 0 W_0 (peak 15.415.4/15.815.8 vs. 10.010.0/8.88.8), where the deepest task-specific weights are learned from scratch. Entanglement (Fig. 1): the mean per-direction overlap meani‖ℛli‖mean_i\,\| P_R^l q_i\| of the forget directions with the retain subspace, all bases at rank 1616; it is the quantity that enters the separability score of §4.5. It decays with depth for every basis, from 0.80.8–1.01.0 in layer1–2 to 0.40.4–0.60.6 at layer4; PCA is the most entangled at every depth (0.540.54/0.580.58 at layer4, 0.460.46/0.560.56 at the classifier on PT W_PT/0 W_0), the discriminative CAV probes suppress the shared variance that PCA’s eigenvectors absorb (0.400.40–0.460.46 and 0.080.08–0.200.20), and all three bases are nearly init-invariant (block means within 0.130.13, 0.050.05 at layer4): what changes between regimes is the magnitude of m τ_m, not the entanglement of the bases. From diagnostics to hyperparameters. The two diagnostics fix the hyperparameters used throughout—skip-layers at the overlap elbow, α inversely to the deepest-layer ‖m‖\| τ_m\|, and CAV bases where entanglement is the binding constraint—and explain why Unmerge-PCA leads on PT W_PT yet trails on 0 W_0 at the same entanglement; α itself can be line-searched on forget-set accuracy alone without RT W_RT or test data (App. D.1). (a) init=PT W_init= W_PT (b) init=0 W_init= W_0 Figure 1: Per-layer forget–retain entanglement on CIFAR-100 (Trees), ResNet-50: mean over the forget basis directions i q_i of the overlap ‖ℛli‖\| P_R^l q_i\| with the retain subspace, all bases at rank 1616, seed 1 (the band is the spread over directions). This per-direction overlap is the rank-invariant form of the basis cross-norm ‖ℱl⊤ℛl‖F\| Q_F^l Q_R^l\|_F, the structural analogue of the merging cross-interference term ‖ℱℛ⊤‖F\| τ_F τ_R \|_F (Asm. 3.1, Remark 3.5); smaller means less entangled. 4.4 Feature-Level Entanglement The per-layer overlap of Fig. 1 exists only for basis-constructing methods. To compare all methods on equal footing we measure entanglement in feature space, as the distance between the forget and retain feature distributions at the penultimate layer. Under retraining, forget images are out-of-distribution and project onto retain class manifolds, so the two clouds mix and the distance is small; in a model that has not unlearned they stay separable. An unlearned model should therefore match the retrained distance: Tab. 3 reports the relative gap |dUL−dRT|/dRT|d_UL-d_RT|/d_RT under four distances on a same-size retain subsample—Gaussian-RBF MMD2 (single and multi-bandwidth), the Bures–Wasserstein W22_2^2 between Gaussian fits, and Sliced-W22_2^2 (App. C). Table 3: Feature-level entanglement on CIFAR-100 (Trees): penultimate-layer distribution distance between forget and a same-size retain subsample, reported as relative gap to the retrained reference (lower is better; RT W_RT is 0 by construction). Best non-retrained per row in bold. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class MMD2 0.000 0.612 1.085 0.951 0.702 0.699 0.627 0.319 0.255 0.267 MMD2-multi 0.000 0.638 2.030 1.917 0.735 0.731 0.638 0.309 0.217 0.257 W22_2^2-Bures 0.000 0.313 0.488 0.273 0.522 0.505 0.548 0.553 0.499 0.611 Sliced-W22_2^2 0.000 0.654 1.251 0.916 0.736 0.734 0.722 0.092 0.046 0.125 Mean rel. gap ↓ 0.000 0.554 1.213 1.014 0.674 0.667 0.634 0.318 0.254 0.315 init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class MMD2 0.000 0.271 1.938 1.834 0.512 0.499 0.549 0.219 0.181 0.038 MMD2-multi 0.000 0.340 3.119 3.015 0.546 0.531 0.529 0.252 0.172 0.052 W22_2^2-Bures 0.000 0.078 1.659 1.661 0.233 0.201 0.384 0.438 0.009 0.117 Sliced-W22_2^2 0.000 0.204 3.265 3.226 0.403 0.378 0.535 0.340 0.071 0.181 Mean rel. gap ↓ 0.000 0.223 2.495 2.434 0.423 0.402 0.499 0.312 0.108 0.097 Unmerge is the closest to retraining on both initializations (Tab. 3): the CAV variants land within ∼ 0.100.10 relative gap on 0 W_0 (0.250.25–0.320.32 on PT W_PT), whereas Random Label and SalUn deviate by 1.01.0–2.5×2.5× the retrained reference, 33–25×25× the CAV gap, and Kodge et al. (2024) by 0.500.50–0.630.63. The un-edited FT W_FT calibrates the axis: it sits at 0.670.67 on 0 W_0 and at 0.450.45–0.700.70 on TinyImageNet, and Unmerge moves the features from there toward retraining in all three settings (0.060.06–0.310.31; Tab. 17, App. E.5), whereas relabeling pushes them away from retraining in every setting (on TinyImageNet/0 W_0 the projection of Kodge et al. (2024) reaches the retrained distance too, but while still classifying 18%18\% of the forget set and losing 77 test points, which is why this axis is read together with accuracy and privacy); on CIFAR-100/PT W_PT the pretrained features already sit near retraining for every model (0.220.22 un-edited) and the axis has no headroom. (W22_2^2-Bures on PT W_PT, a mean-shift-dominated distance, favors SalUn.) This exposes a label/feature dissociation: Random Label and SalUn drive forget accuracy down but leave the forget representations cleanly separable from retain, whereas Unmerge lowers it while matching retraining’s forget-retain feature separation. The gap measures distributional fidelity, not decodability (App. F). 4.5 Membership Privacy, Scope and Fidelity Membership privacy. On CIFAR-100 the projection of Kodge et al. (2024) has the smallest mean MIA gap (0.0170.017/0.0280.028 on 0 W_0/PT W_PT), followed on 0 W_0 by Unmerge-CAV-KMeans (0.0260.026) and CAV-Class (0.0290.029) ahead of SalUn (0.0470.047), Random Label (0.0620.062) and NegGrad+ (0.1090.109); on PT W_PT the other methods sit within 0.0670.067–0.0930.093. On TinyImageNet Unmerge sweeps PT W_PT at 0.0090.009–0.0200.020 against 0.0430.043–0.2610.261 for every baseline (full details in App. E.6). Under these attacks, Unmerge remains close to the retrained reference in membership-inference exposure at class level. Concept-level vs. instance-level requests. Unmerge removes what the forget samples share, so we test two instance-level protocols on Trees, each with its own retrained reference (App. E.7). On coherent subsets (half of the images of each Trees class) the aggregate scores favor Unmerge (ToW 0.970.97/0.940.94 vs. 0.940.94 for a re-tuned NegGrad+ and 0.810.81–0.900.90 for relabeling), but a per-class breakdown shows a concept-level edit: the two halves of a class are i.i.d., so the edit lowers both alike, whereas relabeling protects the retained half and under-forgets the other; no method reproduces retraining. On random subsets no method does either: Unmerge and NegGrad+ stay near-inert, and Random Label and SalUn score only by marking the forget set (attack accuracy 0.840.84–0.880.88 vs. 0.630.63–0.680.68 under retraining). The Phase-1 separability score S=0.17S=0.17 is consistent with both outcomes: it tells whether a request carries a concept the edit can act on. Removing the influence of given samples and removing the concept they represent are distinct problems with different references and metrics (Triantafillou et al., 2026; Cooper et al., 2025; Zhao et al., 2024), which coincide only when ℱF covers the concept, and Unmerge targets the latter. Does unmerging recover the retrained task vector? Beyond accuracy proxies, we compare ℛ=m−αℱ τ_R= τ_m-α τ_F with ℛ⋆=RT−init τ_R = W_RT- W_init directly on Trees (App. E.8). In weight space two independent retrains differ by 0.720.72–0.730.73 relative error and the un-edited FT W_FT already sits at 0.750.75; Unmerge stays at that floor (0.7500.750–0.7540.754: the minimal edit does no weight-space harm), whereas Random Label (0.820.82) and SalUn (0.910.91) move away from the retrained solution. Output space discriminates more sharply in the same direction: Unmerge-CAV-KMeans has the smallest forget-set divergence to RT W_RT in both regimes (2.92.9 vs. 4.94.9–5.25.2 on 0 W_0, 5.35.3 vs. 5.95.9–6.66.6 on PT W_PT), because relabeling pushes forget inputs confidently to wrong classes while subspace subtraction lands closer to a model that never saw them. 5 Conclusion We recast approximate machine unlearning as the inverse of model merging: Unmerge learns a low-rank forget component of the merged task vector under a subspace objective whose terms bound forget leakage and retain damage in activation space, and subtracts it without any gradient update on the training data. On class-level forgetting with ResNet-50 and ViT-S/16 it improves the tug-of-war trade-off over gradient-based and task-vector baselines at a fraction of their cost, keeps membership-inference exposure at the level of retraining, and, unlike relabeling methods, moves the forget representations toward the retrained model rather than away from it. The per-layer bases and task-vector magnitudes double as a diagnostic that motivates the depth and scale choices and, computed before any edit, delimits the method’s scope to concept-level requests; instance-level requests are a different problem that no method we tested solves (further limitations in App. F). Promising directions include language models beyond our preliminary study (App. E.4), continual forget requests, and adaptive per-layer rank and depth selection driven by the same diagnostics. Acknowledgement. RK thanks the Central Indiana Corporate Partnership AnalytiXIN Initiative and NSF Award 2543174 for their support. AI use statement In this work, we used a generative AI (Claude) for the following tasks with required disclosure: debugging, providing feedback on experimental design, implementing utility code such as plotting, and interpreting results. We have not used generative AI tools to generate synthetic data sets, for translation, or to clean or reformat data sets; qualitative and thematic data analysis is not applicable to this work. Additionally, we used the assistant to search and summarize related literature. We have reviewed and verified all AI-assisted work. We take responsibility for the final content of this work, including text, claims and artifacts produced with the aid of generative AI. Ethics statement This work uses only public image-classification benchmarks (CIFAR-100, TinyImageNet) and publicly available pretrained weights, and involves no human subjects or personal data. Machine unlearning serves data-protection goals, but an approximate method must not be presented as a guarantee of deletion: we therefore evaluate against retraining on accuracy, feature and membership-inference axes, and we report the forget requests for which our method does not work (§4.5). Our finding that relabeling-based unlearning can make forget samples more identifiable is reported to inform practitioners. We are not aware of other ethical concerns. Reproducibility statement Algorithms 1–3 give complete pseudocode, and App. B.2 contains the assumptions, proofs and the derivation of the closed-form minimizer. App. C lists the forget tasks, training schedules, baseline settings and all Unmerge configurations; App. D.1 documents how they are chosen; App. E.2 reports seed variance; and App. D.4 identifies the pretrained checkpoint by its hash. All datasets are public, and every experiment runs on a single NVIDIA H100 GPU (runtimes in App. B.3). We will release the code, configurations and evaluation pipeline upon acceptance. References Bau et al. (2017) D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba Network dissection: quantifying interpretability of deep visual representations. In IEEE Conference on Computer Vision and Pattern Recognition, p. 3319–3327. External Links: Document, Link Cited by: §4.3. Belrose et al. (2023) N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman LEACE: perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Appendix A, §2. Bourtoule et al. (2021) L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot Machine unlearning. In IEEE Symposium on Security and Privacy, External Links: Link Cited by: §1, §2. Cao and Yang (2015) Y. Cao and J. Yang Towards making systems forget with machine unlearning. In IEEE Symposium on Security and Privacy, External Links: Link Cited by: §2. Chang et al. (2026) W. Chang, T. Zhu, P. Xiong, Y. Wu, F. Guan, and W. Zhou Zero-shot class unlearning via layer-wise relevance analysis and neuronal path perturbation. IEEE Transactions on Dependable and Secure Computing 23 (4), p. 8469–8486. External Links: Document, Link Cited by: §4.3. Chen et al. (2024) H. Chen, T. Zhu, X. Yu, and W. Zhou Machine unlearning via null space calibration. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, p. 358–366. External Links: Document, Link Cited by: Appendix A, §2. Chen et al. (2023) M. Chen, W. Gao, G. Liu, K. Peng, and C. Wang Boundary unlearning: rapid forgetting of deep networks via shifting the decision boundary. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: §2. Cheng et al. (2025) R. Cheng, F. Xiong, Y. Wei, W. Zhu, and C. Yuan Whoever started the interference should end it: guiding data-free model merging via task vectors. In International Conference on Machine Learning, External Links: Link Cited by: 2nd item, §1, §2, §3.1, §3.3, Assumption 3.1, Remark 3.5. Chundawat et al. (2023) V. S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI Conference on Artificial Intelligence, External Links: Link Cited by: §1, §2. Cooper et al. (2025) A. F. Cooper, C. A. Choquette-Choo, M. Bogen, K. Klyman, M. Jagielski, K. Filippova, K. Liu, A. Chouldechova, J. Hayes, Y. Huang, et al. Machine unlearning doesn’t do what you think: lessons for generative AI policy and research. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §E.7, §4.5. Dorna et al. (2025) V. Dorna, A. Mekala, W. Zhao, A. McCallum, Z. C. Lipton, J. Z. Kolter, and P. Maini OpenUnlearning: accelerating LLM unlearning via unified benchmarking of methods and metrics. arXiv preprint arXiv:2506.12618. External Links: Link Cited by: §E.4. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Link Cited by: §E.3, §4.1. Fan et al. (2024) C. Fan, J. Liu, Y. Zhang, E. Wong, D. Wei, and S. Liu SalUn: empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations, External Links: Link Cited by: §D.1, §1, §2, §4.1, §4.1. Farajtabar et al. (2020) M. Farajtabar, N. Azizan, A. Mott, and A. Li Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, External Links: Link Cited by: Appendix A, §2. Foster et al. (2024) J. Foster, S. Schoepf, and A. Brintrup Fast machine unlearning without retraining through selective synaptic dampening. In AAAI Conference on Artificial Intelligence, External Links: Link Cited by: §1, §2. Golatkar et al. (2021) A. Golatkar, A. Achille, A. Ravichandran, M. Polito, and S. Soatto Mixed-privacy forgetting in deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: §2. Golatkar et al. (2020) A. Golatkar, A. Achille, and S. Soatto Eternal sunshine of the spotless net: selective forgetting in deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: §1, §2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Link Cited by: §E.4. Graves et al. (2021) L. Graves, V. Nagisetty, and V. Ganesh Amnesiac machine learning. In AAAI Conference on Artificial Intelligence, External Links: Link Cited by: §1, §2, §4.1. Ha et al. (2026) S. Ha, S. Park, and S. W. Yoon Unlearning’s blind spots: over-unlearning and prototypical relearning attack. In Proceedings of the 43rd International Conference on Machine Learning (ICML), Note: arXiv:2506.01318 (2025) External Links: Link Cited by: §2. Hatami et al. (2026) A. Hatami, R. Aalishah, and I. E. Monosov Class unlearning via depth-aware removal of forget-specific directions. arXiv preprint arXiv:2604.15166. External Links: Link Cited by: Appendix A, §4.3. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: §4.1. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In International Conference on Learning Representations, External Links: Link Cited by: Appendix C, §1, §2, §3.1, §4.1. Ilharco et al. (2022) G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, A. Farhadi, and L. Schmidt Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §2. Jin et al. (2023) X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations, External Links: Link Cited by: §2. Kim et al. (2018) B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). In International Conference on Machine Learning, External Links: Link Cited by: Appendix A, §2, §3.1. Kim et al. (2025) H. S. Kim, D. Han, and J. Choe NegMerge: sign-consensual weight merging for machine unlearning. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Note: PMLR 267:30067–30085; arXiv:2410.05583; OpenReview: https://openreview.net/forum?id=RfiPhUB4wP; code: https://github.com/naver-ai/negmerge External Links: Link Cited by: Appendix C, §2, §4.1. Kodge et al. (2024) S. Kodge, G. Saha, and K. Roy Deep unlearning: fast and efficient gradient-free class forgetting. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: Appendix A, Appendix C, §D.1, §E.1, §E.2, §E.5, §E.6, §E.6, §2, §4.1, §4.2, §4.4, §4.5. Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. Technical report University of Toronto. External Links: Link Cited by: §4.1. Kuo et al. (2026) K. Kuo, A. Setlur, K. Srinivas, A. Raghunathan, and V. Smith Exact unlearning of finetuning data via model merging at scale. In IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), Note: OpenReview: https://openreview.net/forum?id=u89LDBIyDe External Links: Link Cited by: §2. Kurmanji et al. (2023) M. Kurmanji, P. Triantafillou, J. Hayes, and E. Triantafillou Towards unbounded machine unlearning. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §D.1, §1, §2, §4.1. Le and Yang (2015) Y. Le and X. Yang Tiny ImageNet visual recognition challenge. CS 231N, Stanford University. External Links: Link Cited by: §4.1. Li et al. (2024) N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A. Dombrowski, S. Goel, G. Mukobi, et al. The WMDP benchmark: measuring and reducing malicious use with unlearning. In International Conference on Machine Learning, External Links: Link Cited by: §E.4, §1. Maini et al. (2024) P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter TOFU: a task of fictitious unlearning for LLMs. In Conference on Language Modeling, External Links: Link Cited by: §E.4. Matena and Raffel (2022) M. S. Matena and C. A. Raffel Merging models with Fisher-weighted averaging. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §2. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Appendix A, §2. Nguyen et al. (2025) T. T. Nguyen, T. T. Huynh, Z. Ren, P. L. Nguyen, A. W. Liew, H. Yin, and Q. V. H. Nguyen A survey of machine unlearning. ACM Transactions on Intelligent Systems and Technology 16 (5), p. 1–46. External Links: Document, Link Cited by: §2. Ortiz-Jiménez et al. (2023) G. Ortiz-Jiménez, A. Favero, and P. Frossard Task arithmetic in the tangent space: improved editing of pre-trained models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §2. Ravfogel et al. (2022) S. Ravfogel, M. Twiton, Y. Goldberg, and R. Cotterell Linear adversarial concept erasure. In International Conference on Machine Learning, External Links: Link Cited by: Appendix A, §2. Saha et al. (2021) G. Saha, I. Garg, and K. Roy Gradient projection memory for continual learning. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A, §2. Shokri et al. (2017) R. Shokri, M. Stronati, C. Song, and V. Shmatikov Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, External Links: Link Cited by: §4.1. Song and Mittal (2021) L. Song and P. Mittal Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium, External Links: Link Cited by: §4.1. Tarun et al. (2024) A. K. Tarun, V. S. Chundawat, M. Mandal, and M. Kankanhalli Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems 35 (9), p. 13046–13055. External Links: Document, Link Cited by: §2. Triantafillou et al. (2026) E. Triantafillou, A. I. Humayun, M. Ribero, A. M. Turner, M. C. Mozer, and G. Kaissis Is your algorithm unlearning or untraining?. arXiv preprint arXiv:2604.07962. External Links: Link Cited by: §E.7, Appendix F, §4.5. Voigt and von dem Bussche (2017) P. Voigt and A. von dem Bussche The EU general data protection regulation (GDPR). Springer International Publishing, Cham. External Links: Document, Link Cited by: §1, §2. Wang et al. (2024) W. Wang, Z. Tian, C. Zhang, and S. Yu Machine unlearning: a comprehensive survey. arXiv preprint arXiv:2405.07406. External Links: Link Cited by: §2. Wortsman et al. (2022) M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, External Links: Link Cited by: §2. Yadav et al. (2023) P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal TIES-Merging: resolving interference when merging models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §2. Yao et al. (2007) Y. Yao, L. Rosasco, and A. Caponnetto On early stopping in gradient descent learning. Constructive Approximation 26 (2), p. 289–315. External Links: Document, Link Cited by: §3.2. Yosinski et al. (2014) J. Yosinski, J. Clune, Y. Bengio, and H. Lipson How transferable are features in deep neural networks?. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §4.3. Yu et al. (2024) L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li Language models are super Mario: absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning, External Links: Link Cited by: §2. Zeiler and Fergus (2014) M. D. Zeiler and R. Fergus Visualizing and understanding convolutional networks. In European Conference on Computer Vision, p. 818–833. External Links: Document, Link Cited by: §4.3. Zhang et al. (2023) J. Zhang, S. Chen, J. Liu, and J. He Composing parameter-efficient modules with arithmetic operation. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §2. Zhao et al. (2024) K. Zhao, M. Kurmanji, G. Bărbulescu, E. Triantafillou, and P. Triantafillou What makes unlearning hard and what to do about it. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §E.7, §4.1, §4.1, §4.5. Zhou et al. (2025) Y. Zhou, D. Zheng, Q. Mo, R. Lu, K. Lin, and W. Zheng Decoupled distillation to erase: a general unlearning method for any class-centric tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 20350–20359. External Links: Document, Link Cited by: §2. Appendix Appendix A Additional Related Work We expand the comparison to subspace-projection and representation-editing methods that the main paper Related Work (§2) treats only briefly. Continual learning via subspace projection. Gradient Projection Memory (Saha et al., 2021) and Orthogonal Gradient Descent (Farajtabar et al., 2020) project gradient updates onto subspaces orthogonal to previously seen tasks, preventing catastrophic forgetting by hard separation. We share their intuition that forget and retain knowledge should occupy separable subspaces, but rather than enforcing separation by hard orthogonal projection, we let the bases ℱ,ℛ Q_F, Q_R retain their natural overlap and penalize retain leakage softly via Term 2 of equation 2, so that partially shared directions are explicitly traded off rather than discarded by construction. This soft trade-off is what makes Unmerge work in regimes where the forget and retain subspaces are genuinely entangled (e.g., PT W_PT where shared ImageNet-flavor variance lifts ‖ℱ⊤ℛ‖F\| Q_F Q_R\|_F uniformly; see §4.3). Gradient-free activation-subspace class unlearning. Kodge et al. (2024) estimate retain and forget spaces by SVD of layer activations on a few samples, remove the shared part from the forget space, and project the weights so that the remaining class-discriminatory directions are suppressed, selecting two scaling hyperparameters on training accuracy alone. Null-space calibration (Chen et al., 2024) confines the unlearning update to the null space of retain activations, and the concurrent DAMP (Hatami et al., 2026) removes forget directions relative to retain prototypes with a depth-aware scale and argues that class unlearning is often carried by the classifier head. With init=0 W_init=0 our update reads UL=FT(−αℱℱ⊤) W_UL= W_FT( I-α\, Q_F M Q_F ) with the k×k× k filter =((1+γ)+λ⊤)−1 M=((1+γ) I+λ C C )^-1 for the exact minimizer, i.e. a soft input-space projection of the same family; the hard projection ℱ(−ℛ) P_F( I- P_R) of Kodge et al. (2024) corresponds to removing the retain-shared part of the forget directions instead of down-weighting it. Our formulation differs in acting on the task vector, which under PT W_PT leaves the pretrained weights intact and matters for the unsupervised basis (App. D.4); in deriving the filter from an explicit objective that trades forgetting against retain leakage; in offering supervised CAV bases; and in exposing the same quantities as diagnostics. §4 benchmarks Kodge et al. (2024) under the protocol of the other baselines (App. C). Linear concept erasure. LEACE (Belrose et al., 2023) and RLACE (Ravfogel et al., 2022) give closed-form linear concept erasure with activation-distribution guarantees, and Concept Activation Vectors (CAV) (Kim et al., 2018) probe concept directions with linear classifiers trained on labeled positive/negative activation pools. We use CAV as one of our basis constructions (Alg. 3) and inherit its supervised, discriminative flavor, but lift it from representation-level probing/erasure to weight-level subtraction via task-vector decomposition: instead of projecting activations through a learned operator at inference time, we modify the weights themselves so the forget direction is no longer encoded. Knowledge editing. ROME (Meng et al., 2022) and similar locate-and-edit methods modify specific low-rank weight subspaces to update factual associations in language models. They share our weight-level intervention philosophy but require localized (s,r,o)(s,r,o) associations rather than distributional forget/retain sets, and target editing of specific facts rather than removal of broader behaviors. Our setting (class-level forgetting from distributional samples of ℱF and ℛR) is complementary. Unifying view. The retain projector ℛ P_R in our objective plays the role of the “preserve” subspace from continual learning, while ℱ P_F inherits the concept-direction view from CAV/LEACE. Together, the three terms of equation 2 (forget removal, retain leakage, magnitude regularization) map cleanly onto the activation-space bounds of §3.3, giving a single objective that recovers and softens individual instances of these prior approaches. Appendix B Method Details B.1 Algorithm and Implementation Details This subsection expands the implementation sketch in §3 and provides the complete pseudocode. Algorithm 1 realizes equation 2 as a three-phase, optimization-time data-free procedure: data enters only through the second-moment statistics gathered in Phase 1, and no gradient is taken with respect to a training sample. Phase 1: basis construction. A forward pass through ℱ∪ℛF (plus a second pass over ℛR for the CAV variants) collects the layer inputs ℱl,ℛl X_F^l, X_R^l via forward hooks. The retain basis ℛl Q_R^l is built as the top-k eigenbasis of 1Nℛℛl⊤ℛl 1N_R X_R^l X_R^l (PCA, Alg. 2). The forget basis ℱl Q_F^l is built either by the same PCA construction or by the discriminative CAV construction in Alg. 3, which learns a ridge probe per cluster against the retain pool and stacks the orthogonalized probe directions as the basis. CAV admits two cluster definitions: CAV-KMeans partitions forget activations into k unsupervised clusters via k-means; CAV-Class uses the available forget class labels, producing one direction per forget class (so k′=|ℱ|k =|Y_F| regardless of k). The two modes share the ridge-probe and orthogonalization machinery and differ only in line 5 of Alg. 3. KMeans is purely activation-driven and applicable when class labels are unavailable or when the forget set is sub-class; Class exploits supervision when one direction per forget class is the natural granularity, and matches PCA in cost since it skips the clustering step. The projection mlℱl τ_m^l Q_F^l—the minimizer of Term 1 of equation 2 in isolation—and the overlap l=ℱl⊤ℛl C^l= Q_F^l Q_R^l are computed here; they are all Phase 2 needs. Phase 2: optimization in projected space. Substituting ℱl=lℱl⊤ τ_F^l= B^l Q_F^l into equation 2 reduces every term to small matrix products in the projected basis—a strictly convex quadratic in l B^l whose exact minimizer is equation 4 (App. B.2). With mlℱl τ_m^l Q_F^l and l C^l pre-computed once, our procedure initializes l B^l at the projection mlℱl τ_m^l Q_F^l and takes 100 Adam steps at learning rate 10−310^-3 (mkmk parameters per layer, never revisiting the data). This budget does not reach the exact minimizer; App. D.3 measures the gap and shows that applying l⋆ B^l directly at the paper’s (λ,γ)(λ,γ) over-shrinks ℱ τ_F on high-overlap bases, i.e. the fixed budget functions as an early-stopping regularizer. Phase 3: application. We apply ULl=FTl−αlℱl⊤ W_UL^l= W_FT^l-α\, B^l Q_F^l with a scalar α that corrects for the rank-truncation residual ‖F\| E\|_F of Lemma 3.2 and for imperfect rank. The optional skip-layers knob restricts the update to deeper layers, where the retain-damage constant μ1 _1 of equation 7 is smaller and the forget/retain spectra are more separable; §4.3 reads the α and skip-layers choices directly off Figs. 1 and 2. Implementation notes. Five details of the released code differ from, or are not visible in, the description above. (i) Centering. Bases are computed from the centered covariance of the layer inputs rather than from the second moment used in §3.3; this removes the shared mean-activation direction of post-ReLU features, and the bounds of Proposition 3.3 then hold for centered inputs, with the mean response ‖ℛℱ‖2\| τ_R\, μ_F\|^2 as an additional forget term. (i) CAV pools. In Alg. 3 the forget and retain pools are mean-centered separately before the ridge probes are fit, so each probe contrasts one cluster (or class) with the remaining activations in centered coordinates; the common forget-vs-retain mean shift is removed by construction. With |ℱ|=5|Y_F|=5 classes the class probes are nearly linearly dependent (effective rank 44), so the fifth orthogonalized direction of CAV-Class carries almost no signal. (i) Subsampling. PCA statistics use every sample, but the CAV probes are fit on at most max-points activation vectors per layer (81928192 on CIFAR-100, 1638416384 on TinyImageNet), taken from the first shuffled batches with a fixed loader seed; for a convolutional layer with L spatial positions this is max-points/L max-points/L images (e.g. 512512 images at layer4), and CAV makes a second pass over the retain set. (iv) Biases. For an edited layer with a bias, the code shrinks the bias task vector uniformly, bUL=binit+(1−αγ/(1+γ))(bFT−binit)b_UL=b_init+ (1-αγ/(1+γ) )(b_FT-b_init). In ResNet-50 only the classifier has a bias (‖bFT−binit‖≤0.15\|b_FT-b_init\|≤ 0.15, a logit shift below 0.020.02), so the rule is immaterial there; ViT-S/16 has a bias in every edited layer, and App. E.3 reports the ViT cells with and without the rule. (v) Normalization layers. Only convolutional and linear weights are edited. BatchNorm (LayerNorm for ViT) parameters and running statistics are kept from FT W_FT, activations are collected in evaluation mode, and no recalibration pass is run after the edit. Algorithm 1 Unmerging Input: Finetuned model FT W_FT, initial weights init W_init, forget set ℱF, retain set ℛR, rank k, regularization λ,γλ,γ, scaling α, learning rate η, steps T, basis type ∈PCA,CAV∈\PCA,CAV\, CAV mode ∈kmeans,class∈\kmeans,class\ (if basis==CAV) Output: Unlearned model UL W_UL for each target layer l do ml←FTl−initl τ_m^l← W_FT^l- W_init^l Collect layer inputs ℛl,ℱl X_R^l, X_F^l via forward pass ℛl←PcaBasis(ℛl,k) Q_R^l← PcaBasis( X_R^l,k) if basis type is PCA then ℱl←PcaBasis(ℱl,k) Q_F^l← PcaBasis( X_F^l,k) else ℱl←CavBasis(ℱl,ℛl,k,mode) Q_F^l← CavBasis( X_F^l, X_R^l,k,mode) end if l←ℱl⊤ℛl C^l← Q_F^l Q_R^l k×k× k basis overlap end for l←mlℱl B^l← τ_m^l Q_F^l for all l initialize at the projection, the minimizer of Term 1 T Adam steps in projected space: with ℱl=lℱl⊤ τ_F^l= B^l Q_F^l , equation 2 is the strictly convex quadratic below, whose exact minimizer is equation 4; the fixed budget T early-stops (App. D.3) for t=1t=1 to T do ℒ←∑l‖mlℱl−l‖F2+λ‖ll‖F2+γ‖l‖F2L← _l \| τ_m^l Q_F^l- B^l \|_F^2+λ \| B^l C^l \|_F^2+γ \| B^l \|_F^2 l←Adam(l,∇lℒ,η) B^l ( B^l, _ B^lL,η ) for all l end for Recover ℱ τ_F and apply for each target layer l do ℱl←lℱl⊤ τ_F^l← B^l Q_F^l ULl←FTl−αℱl W_UL^l← W_FT^l-α\, τ_F^l end for return UL W_UL Algorithm 2 PCA Basis Construction (PcaBasis) Input: Layer inputs ∈ℝN×d X ^N× d, rank k Output: Orthonormal basis Q∈ℝd×kQ ^d× k ←1N∑i=1Ni μ← 1N _i=1^N x_i ←⊤−N⊤ C← X X-N\, μ μ centered covariance 1,…,k← v_1,…, v_k← top-k eigenvectors of C Q←[1,…,k]Q←[ v_1,…, v_k] return Q Algorithm 3 CAV Basis Construction (CavBasis) Input: Forget inputs ℱ∈ℝNℱ×d X_F ^N_F× d with class labels ℱ y_F, retain inputs ℛ∈ℝNℛ×d X_R ^N_R× d, rank k, ridge penalty μ, mode ∈kmeans,class∈\kmeans,class\ Output: Orthonormal basis Q∈ℝd×k′Q ^d× k where k′≤k ≤ k (kmeans) or k′=|ℱ|k =|Y_F| (class) Mean-center ℱ X_F and ℛ X_R if mode is kmeans then C1,…,Ck←\C_1,…,C_k\← k-means clustering on ℱ X_F else Ccc∈ℱ←\C_c\_c _F← partition ℱ X_F by class label ℱ y_F one cluster per forget class end if Q←[]Q←[] for each cluster CjC_j do +←ℱ[Cj] X^+← X_F[C_j], −←ℛ X^-← X_R ←[+;−]⊤ Z←[ X^+; X^-] , ←[+;−] y←[+ 1;- 1] j←(⊤+μ)−1 w_j←( Z Z +μ I)^-1 Z y ridge regression Orthogonalize j w_j against columns of Q if ‖j‖>ϵ\| w_j\|>ε then Append j/‖j‖ w_j/\| w_j\| to Q end if end for return Q B.2 Proofs and Scope of the Analysis Derivation of the closed form (equation 4). Write ℱ=ℱ⊤ τ_F= B Q_F with ℱ⊤ℱ= Q_F Q_F= I and ℛ⊤ℛ= Q_R Q_R= I, and let =ℱ⊤ℛ C= Q_F Q_R. Right-multiplication by a matrix with orthonormal rows preserves the Frobenius norm, so ‖(m−ℱ⊤)ℱℱ⊤‖F=‖mℱ−‖F\|( τ_m- B Q_F ) Q_F Q_F \|_F=\| τ_m Q_F- B\|_F, ‖ℱ⊤ℛℛ⊤‖F=‖‖F\| B Q_F Q_R Q_R \|_F=\| B C\|_F, and ‖ℱ⊤‖F=‖F\| B Q_F \|_F=\| B\|_F. Equation 2 therefore equals f()=‖mℱ−‖F2+λ‖‖F2+γ‖F2f( B)=\| τ_m Q_F- B\|_F^2+λ\| B C\|_F^2+γ\| B\|_F^2, with gradient ∇f=−2(mℱ−)+2λ⊤+2γ∇ f=-2( τ_m Q_F- B)+2λ B C C +2γ B. Setting it to zero gives ((1+γ)+λ⊤)=mℱ B ((1+γ) I+λ C C )= τ_m Q_F. The matrix in parentheses is symmetric positive definite, so f is strictly convex and its unique minimizer is ⋆=mℱ((1+γ)+λ⊤)−1 B = τ_m Q_F ((1+γ) I+λ C C )^-1. For CAV bases ℱ Q_F has k′≤k ≤ k orthonormal columns and the derivation is unchanged with ∈ℝk′×k C ^k × k. The 100-step Adam iterate used for the reported runs does not reach ⋆ B in general; App. D.3 quantifies the distance and the effect of applying ⋆ B itself. Proof of Lemma 3.2. Let ℱ=1Nℱℱ⊤ℱ _F= 1N_F X_F X_F be the second-moment matrix of forget inputs with eigenvalues ν1≥⋯≥νd≥0 _1≥·s≥ _d≥ 0, and let ℱ∈ℝd×k Q_F ^d× k be the top-k eigenvectors with projector ℱ=ℱℱ⊤ P_F= Q_F Q_F . Under Assumption 3.1, any task vector τ whose rows lie in span(ℱ⊤)span( X_F ) admits the decomposition =ℱ⊤+ τ= B Q_F + E with =ℱ∈ℝm×k B= τ Q_F ^m× k and ∥F2≤∥op2⋅Nℱ∑i=k+1dνi.\| E\|_F^2≤\| C\|_op^2· N_F _i=k+1^d _i. (9) Proof. Write =ℱ τ= C X_F (neglecting (η2)O(η^2) terms). Decompose =ℱℱ⏟ℱ⊤+ℱ(I−ℱ)⏟ τ= C X_F Q_F_ B Q_F + C X_F(I- P_F)_ E. By sub-multiplicativity, ‖F≤‖op‖ℱ(I−ℱ)‖F\| E\|_F≤\| C\|_op\| X_F(I- P_F)\|_F, and ‖ℱ(I−ℱ)‖F2=tr((I−ℱ)ℱ⊤ℱ(I−ℱ))=Nℱ∑i>kνi\| X_F(I- P_F)\|_F^2=tr\! ((I- P_F) X_F X_F(I- P_F) )=N_F _i>k _i. ∎ Proof of Proposition 3.3. Let ℱ∈ℝd×d _F ^d× d be the second-moment matrix of forget inputs with eigenvalues ν1≥⋯≥νd _1≥·s≥ _d and top-k projector ℱ P_F. Similarly, let ℛ _R have eigenvalues μ1≥⋯≥μd _1≥·s≥ _d with top-k projector ℛ P_R. Let α>0α>0 be the scaling used in the update and denote ℛ=m−αℱ τ_R= τ_m-α τ_F. Then: x∼ℱ[‖ℛx‖2]≤ _x \! [\| τ_R\,x\|^2 ]≤ ν1‖(m−αℱ)ℱ‖F2+νk+1‖m−αℱ‖F2, _1 \|( τ_m-α τ_F) P_F \|_F^2+ _k+1 \| τ_m-α τ_F \|_F^2, (10) x∼ℛ[‖αℱx‖2]≤ _x \! [\|α τ_F\,x\|^2 ]≤ α2(μ1‖ℱℛ‖F2+μk+1‖ℱ‖F2). α^2 ( _1 \| τ_F P_R \|_F^2+ _k+1 \| τ_F \|_F^2 ). (11) Proof. For any ∈ℝm×d A ^m× d and random vector x with second-moment =∑iσivivi⊤ = _i _iv_iv_i : [‖x‖2] \! [\| Ax\|^2 ] =tr(⊤)=∑i=1dσi‖vi‖2 =tr( A A )= _i=1^d _i\| Av_i\|^2 (12) =∑i=1kσi‖vi‖2⏟≤σ1‖‖F2+∑i=k+1dσi‖vi‖2⏟≤σk+1‖F2, = _i=1^k _i\| Av_i\|^2_≤\; _1\| A Q\|_F^2+ _i=k+1^d _i\| Av_i\|^2_≤\; _k+1\| A\|_F^2, (13) where =[v1,…,vk] Q=[v_1,…,v_k] and the tail bound uses ∑i>k‖vi‖2≤‖F2 _i>k\| Av_i\|^2≤\| A\|_F^2. Since ⊤=k Q Q= I_k, we have ‖‖F=‖⊤‖F=‖‖F\| A Q\|_F=\| A Q Q \|_F=\| A P\|_F. Setting =ℛ=m−αℱ A= τ_R= τ_m-α τ_F with ℱ _F gives equation 10; setting =αℱ A=α τ_F with ℛ _R and factoring α2α^2 out of both norms gives equation 11. ∎ Remark B.1 (Bases that are not eigenbases). Proposition 3.3 takes ℱ P_F and ℛ P_R to be top-k eigenprojectors. The retain basis is always built by PCA, so equation 7 applies to every variant. The CAV forget bases are orthonormal but are not eigenvectors of ℱ _F; for any orthonormal ℱ Q_F with projector ℱ P_F, splitting x=ℱx+(−ℱ)x= P_Fx+( I- P_F)x and using ‖a+b‖2≤2‖a‖2+2‖b‖2\|a+b\|^2≤ 2\|a\|^2+2\|b\|^2 gives x∼ℱ[‖ℛx‖2]≤2ν1‖ℛℱ‖F2+2‖(−ℱ)ℱ(−ℱ)‖op‖ℛ‖F2,E_x \! [\| τ_R\,x\|^2 ]≤ 2 _1 \| τ_R P_F \|_F^2+2\, \|( I- P_F) _F( I- P_F) \|_op \| τ_R \|_F^2, (14) which reduces to equation 6 without the factor 22 when ℱ P_F commutes with ℱ _F. Term 1 therefore still controls the dominant component of forget leakage, and the tail constant νk+1 _k+1 is replaced by the largest forget variance left outside the span of ℱ Q_F, which the captured-variance diagnostic of §4.5 reports. Proof of Corollary 3.4. Write the singular value decomposition =⊤ C= V U with =diag(σi) =diag( _i); since ℱ Q_F and ℛ Q_R are orthonormal, σi∈[0,1] _i∈[0,1] are the cosines of the principal angles between their spans. Then ⊤=2⊤ C C = V ^2 V and ((1+γ)+λ⊤)−1=diag(gi)⊤ ((1+γ) I+λ C C )^-1= V\,diag(g_i)\, V , which gives the form of ⋆ B from equation 4. For the retain term, ‖⋆‖F2=∑igi2σi2‖i‖2\| B C\|_F^2= _ig_i^2 _i^2\| a_i\|^2 with i=mℱi a_i= τ_m Q_F v_i, against ∑iσi2‖i‖2 _i _i^2\| a_i\|^2 for the plain projection 0 B_0. ∎ Scope of the assumption and of the bounds. Assumption 3.1. Each SGD update of a linear (or unfolded convolutional) layer is an outer product between a backpropagated signal and that layer’s inputs, so the rows of the accumulated update lie in the span of the inputs seen during training. The error term accounts for the drift of those inputs while earlier layers change, and its (η2)O(η^2) rate is a small-learning-rate idealization suited to finetuning from PT W_PT. For from-scratch training with init=0 W_init=0 we use the property heuristically: weight decay discounts the early updates, whose inputs differ most from the final activations, and we check the resulting low-rank structure empirically (§4.3). What equation 6 measures. Its left-hand side is the response of ℛ τ_R on forget inputs, a proxy for forgetting rather than a distance to RT W_RT: a retrained model still responds to the features that forget inputs share with retain inputs, which is the part Term 2 protects. Gain. The first term of equation 6 is smallest at unit gain, α/(1+γ+λσi2)=1α/(1+γ+λ _i^2)=1, whereas the selected α exceeds 1+γ1+γ (e.g. α/(1+γ)≈1.7α/(1+γ)≈ 1.7 for the CAV bases on PT W_PT): the edit over-corrects the k captured directions. We attribute this to forget signal left outside the rank-k span (the tail term of equation 6), which a unit-gain edit cannot out-vote at the classifier; the bounds motivate the three terms and are not minimized by the tuned edit. Layers. The bounds are per layer, with bases computed on the un-edited network, and do not track how upstream edits shift downstream activations; skip-layers confines this interaction to the last 88–1818 layers, and we did not test sequential re-estimation of the bases. B.3 Requirements, Runtime and Cost Requirements. Tab. 4 makes precise what each method consumes at unlearning time. No method in our comparison is data-free in the sense of touching no data; Unmerge is optimization-time data-free: ℱF and ℛR are read in forward passes only (one for PCA, two over ℛR for CAV), to form second-moment statistics, and no gradient is ever taken with respect to a training sample. The retain statistics can be cached and reused across forget requests on the same checkpoint. Every method uses the retrained model for evaluation only. Table 4: Resources consumed at unlearning time (CIFAR-100 Trees, ResNet-50). init W_init: whether the pre-finetuning checkpoint is used (init=0 W_init=0 is the convention in the 0 W_0 regime, App. D.4). Runtimes: PT W_PT cells of Tab. 10. Method Forget data Retain data Forget labels Gradient updates init W_init Runtime NegGrad+ yes yes yes 10 epochs (ascent on ℱF, descent on ℛR) no 24s Random Label yes yes replaced by random 10 epochs no 110s SalUn yes yes replaced by random 10 epochs + saliency pass no 111s NegTV yes no yes 5 epochs on ℱF no 4s NegMerge yes no yes 3×3× 5 epochs on ℱF no 11s Kodge et al. 1 forward pass (55k images) 1 forward pass (55k images) no none no 8s Unmerge-PCA / CAV-KMeans 1 forward pass 1 forward pass no none PT W_PT regime only 8s / 26s Unmerge-CAV-Class 1 forward pass 1 forward pass yes none PT W_PT regime only 22s Runtime and cost model. Tab. 5 times the three phases of Alg. 1 on the official Trees cells. The pipeline is data-bound: more than 95%95\% of the wall-clock is the single forward pass over ℱ∪ℛF plus basis construction, the optimization of l\ B^l\ takes under one second. Per layer, the costs are (Ndl)O(Nd_l) to collect activations, (Ndl2)O(Nd_l^2) to accumulate the second-moment matrix, (dl3)O(d_l^3) for its eigendecomposition (only over non-skipped layers; a randomized top-k solver reduces this to (dl2k)O(d_l^2k)), and (Tmk2)O(Tmk^2) for the T Adam steps ((mk2+k3)O(mk^2+k^3) for the exact solve of equation 4), so neither solver affects how the method scales. A gradient baseline that trains for E epochs costs E forward and backward passes over the same data, so the runtime ratio is governed by E rather than by model size as long as the per-layer eigendecomposition stays cheaper than one forward pass over N samples—true by orders of magnitude for dld_l up to ∼104 \!10^4. The measured ratios against Random Label and SalUn (44–23×23× on ResNet-50 and ViT-S/16 at E=10E=10) vary with how much of Unmerge’s time goes to basis construction, which is larger for the CAV variants. One detail of the timing should be kept in mind: the baselines’ training loops evaluate on the test set after every epoch to log progress, and their timers include it (0.720.72s per epoch on CIFAR-100, 7.27.2s over E=10E=10 epochs), whereas Unmerge’s timer stops before evaluation. This is under 7%7\% of Random Label’s and SalUn’s runtime but about a third of NegGrad+’s, whose epochs are short because they only traverse ℱF with as many retain samples (2020 steps). Net of that logging pass NegGrad+ takes 16.716.7s instead of 23.923.9s on Trees (PT W_PT), so Unmerge-PCA (8.68.6s) remains about 2×2× faster than NegGrad+ and the CAV variants (1818–2626s) are on par with it; the ∼ 5×5× advantage over Random Label and SalUn is unaffected. Beyond the preliminary 3B study of App. E.4, language-model scale is scoped as future work. Table 5: Phase-wise runtime of Unmerge on CIFAR-100 (Trees), ResNet-50, single H100. Cell Phase 1 (activations + bases + m τ_m) Phase 2 (100 Adam steps) Phase 3 (apply) Total 0 W_0, CAV-Class 17.1s 0.5s 0.0s 17.6s PT W_PT, PCA (k=64k=64) 8.2s 0.35s 0.0s 8.6s Appendix C Experimental Details Forget tasks. CIFAR-100 superclasses (five classes, |ℱ|=2500|F|=2500): Trees [47,52,56,59,96], Aquatic Mammals [4,30,55,72,95], Vehicles-1 [8,13,48,58,90]. TinyImageNet semantic groups (ten classes): Dogs & Cats [24--33] and Arthropods [7,9,10,36,38,39,41,42,43,45]. Within each task ℱF is the union of the chosen classes’ training images and ℛ=∖ℱR=S ; the instance-level protocols are described in App. E.7. Training. All models are trained with SGD (momentum 0.90.9, cosine schedule, batch size 128128). ResNet-50 on CIFAR-100: 100100 epochs at learning rate 0.10.1 from 0 W_0 and 5050 epochs at 0.010.01 from PT W_PT (torchvision IMAGENET1K_V1; hash in App. D.4), weight decay ×10−45\!×\!10^-4; TinyImageNet uses the same schedules with weight decay 10−410^-4; ViT-S/16 is finetuned for 1010 epochs at 0.010.01. Each retrained reference RT W_RT uses the schedule of its FT W_FT on ℛR only. All experiments run on a single NVIDIA H100. Baselines. All gradient baselines run for E=10E=10 epochs. CIFAR-100: NegGrad+ (α=0.5α=0.5, learning rate 10−410^-4), Random Label (learning rate 0.010.01), SalUn (saliency threshold 0.50.5, learning rate 0.010.01–0.020.02). TinyImageNet: NegGrad+ (α∈0.5,0.8α∈\0.5,0.8\, learning rate 10−410^-4), Random Label and SalUn at learning rate 10−310^-3. The task-vector baselines subtract a vector obtained by gradient training on ℱF: NegTV (Ilharco et al., 2023) finetunes FT W_FT on ℱF for 55 epochs to ℱ W_F and applies UL=FT−α(ℱ−FT) W_UL= W_FT-α( W_F- W_FT); NegMerge (Kim et al., 2025) obtains K=3K=3 such vectors at 0.5,1,2×\0.5,1,2\× the base learning rate, keeps the entries whose sign agrees across all K, averages them, and subtracts. Both were given an equal-budget grid on Trees (learning rate 10−3,10−2\10^-3,10^-2\, α∈[0.25,4]α∈[0.25,4], 2828 runs plus an α refinement), and the Trees configuration (learning rate 10−310^-3, 55 epochs) is transferred to the other tasks with only α re-swept over 0.25,0.35,0.5,0.75,1.0\0.25,0.35,0.5,0.75,1.0\; every reported optimum is interior. Their failure mode is retain collapse: the finetuned vector overlaps retain directions, so the best cells already lose 11–1313 points of retain accuracy or leave up to 26%26\% forget accuracy and every stronger α pushes retain toward chance, and the two methods are within 0.040.04 ToW of each other in every cell. The activation-subspace baseline of Kodge et al. (2024) is run as published, on every convolutional and linear layer, from the same forward-pass statistics as Unmerge (50005000 images per set): its retain and forget scalings are tuned on Trees and Dogs & Cats by ToW over αr∈100,300,1000,3000 _r∈\100,300,1000,3000\ and αf∈3,10,30,100 _f∈\3,10,30,100\, and αf _f is re-swept on the transfer tasks; the grids were extended at the edges until every reported optimum is interior. Unmerge configurations. Phase 2 always uses T=100T=100 Adam steps at learning rate 10−310^-3, never tuned. Tab. 6 lists the Trees configurations. They are tuned once per (variant, init) on Trees (CIFAR-100) and Dogs & Cats (TinyImageNet) and transferred to the other tasks changing at most α (by ≤0.5≤\!0.5; PCA on PT W_PT transfers with α unchanged) and, in three 0 W_0 cells, skip-layers by one step of five layers. The three metrics are defined in §4.1; attack details are in App. E.6. The entanglement distances of §4.4 are computed on penultimate-layer features (the classifier input) of ℱF and a same-size retain subsample (|ℛ|=|ℱ||R|=|F|): the unbiased Gaussian-RBF MMD2 at the median bandwidth σmed _med and averaged over σ∈0.5,1,2⋅σmedσ\!∈\!\0.5,1,2\\!·\! _med (multi), the closed-form Bures–Wasserstein W22_2^2 between Gaussian fits (mean-shift dominated when the feature dimension exceeds the sample size), and the Sliced-W22_2^2 over 10001000 random one-dimensional projections. Table 6: Unmerge configurations on CIFAR-100 (Trees), ResNet-50. Cell rank k skip-layers α λ γ PT W_PT, PCA 64 45 1.6 5 0.5 PT W_PT, CAV-KMeans 16 35 2.5 5 0.5 PT W_PT, CAV-Class 16 35 2.5 5 0.5 0 W_0, PCA 128 45 1.25 50 0.5 0 W_0, CAV-KMeans 16 40 1.0 50 0.5 0 W_0, CAV-Class 16 40 1.3 50 0.5 Appendix D Ablations of Unmerge D.1 Per-Layer Diagnostics and Hyperparameters (a) init=PT W_init= W_PT (b) init=0 W_init= W_0 Figure 2: Per-layer merged-task-vector magnitude ‖ml‖F\| τ_m^l\|_F on CIFAR-100 (Trees), ResNet-50, log-y. The three variant curves coincide exactly because m τ_m is basis-independent; the shaded band is the range over three finetuning seeds. The cross-init gap is depth-dependent: shallow blocks (layer1–2) carry slightly larger magnitudes on PT W_PT (pretrained features must be shifted away from ImageNet), while at layer4 and the classifier ‖m‖\| τ_m\| is ∼ 1.51.5–2×2× larger on 0 W_0, where the deepest task-specific weights are encoded from scratch. This subsection covers the design choices that determine Unmerge performance—basis construction (PCA vs. CAV), rank k, the scaling α, skip-layers, and the regularizers λ,γλ,γ—and how α is selected without the retrained model; the objective terms and the solver are ablated in App. D.2 and D.3. The per-layer figures of §4.3 (Figs. 1, 2) drive the recipe; the paragraphs below summarize what each knob controls and the ranges that worked across all reported runs. Recipe in brief. The two diagnostics read off the hyperparameters used throughout: skip-layers at the overlap elbow (layer3→ 4; CAV 3535/4040, PCA 4545–5050), α inversely to the deepest-layer ‖m‖\| τ_m\| ([1.3,2.5][1.3,2.5] on PT W_PT, [0.5,1.3][0.5,1.3] on 0 W_0), and CAV where entanglement is the binding constraint (0 W_0). They also explain why Unmerge-PCA wins ToW on PT W_PT yet loses on 0 W_0 at the same entanglement (next paragraph). PCA vs. CAV. PCA gives a strong, unsupervised baseline but its top-k eigenvectors absorb dataset-generic variance that overlaps with retain—visible as the highest per-direction overlap at every depth in Fig. 1 (0.540.54–0.580.58 at layer4 and 0.460.46–0.560.56 at the classifier, all bases at rank 1616). CAV’s discriminative ridge probes target only forget-distinguishing directions (0.400.40–0.460.46 at layer4, 0.080.08–0.200.20 at the classifier); the two CAV modes are essentially tied per direction, so the gap between them in the main tables comes from the number of directions (|ℱ||Y_F| vs. k), not from entanglement. The empirical consequence is the regime-split observed in Tab. 1, which the two diagnostics of §4.3 explain jointly: by equation 7, retain damage scales as α2μ1‖ℱ⊤ℛ‖F2α^2 _1\| Q_F Q_R\|_F^2 times the basis mass ‖2\| B\|^2, so entanglement and ‖m‖\| τ_m\| enter the retain budget together, and since the entanglement of each basis is nearly the same in both regimes it is the deep ‖m‖\| τ_m\| that decides. PCA wins ToW on PT W_PT (0.98510.9851), where moderate deep ‖m‖\| τ_m\| tolerates the higher overlap at larger α; on 0 W_0, deep ‖m‖\| τ_m\| peaks at ∼ 1515 and forces a smaller α to protect retain, which under-drives PCA (0.92300.9230), while CAV-Class’s less entangled basis (0.450.45 at layer4, 0.170.17 at the classifier) lets α climb back and forget cleanly (0.99090.9909). All three bases are nearly init-invariant per direction (≤0.13≤\!0.13 change in any block between PT W_PT and 0 W_0), and CAV-Class remains the safer default when the init is unknown. Rank k. Per Lemma 3.2, the residual ‖F2\| E\|_F^2 is bounded by Nℱ∑i>kνiN_F _i>k _i, so accuracy saturates once k covers the effective dimensionality of the per-layer forget covariance. We sweep k∈8,16,32,64,128k\!∈\!\8,16,32,64,128\; for CAV bases the elbow lies near k=16k\!=\!16 on ResNet-50 and k=16k\!=\!16–3232 on ViT-S, beyond which retain accuracy degrades by 0.50.5–2%2\% at fixed α as the rank-k form picks up more retain-aligned directions. PCA prefers larger ranks (k=64k\!=\!64–128128): its variance-ordered eigenvectors need more directions to cover the forget signal, and on 0 W_0—where deep task vectors are large—k=128k\!=\!128 captures more forget energy per unit α and clearly pays off. CAV-Class is invariant to k by construction (one direction per forget class, k′=|ℱ|k \!=\!|Y_F|), which removes this knob in the supervised regime. Scale α. α∈[0.5,2.5]α\!∈\![0.5,2.5] tracks the deepest-layer ‖m‖\| τ_m\| (Fig. 2) inversely: PT W_PT keeps α∈[1.3,2.5]α\!∈\![1.3,2.5] where deep ‖m‖≤10\| τ_m\|\!≤\!10, 0 W_0 uses α∈[0.5,1.3]α\!∈\![0.5,1.3] to cap retain damage when deep ‖m‖≈15\| τ_m\|\!≈\!15. The product α‖m‖α\| τ_m\| is the operative quantity in equation 7. Skip-layers. Restricting unmerging to deeper layers (skip-layers= 35\,=\,35–5050 of the 5353 convolutions of ResNet-50; the classifier is always edited: 3535 edits the last 1818 convolutions, from layer3.3.conv2 on, 4040 the last 1313, from layer3.5.conv1 on, 4545 the last 88, all in layer4, and 5050 the last 33) preserves the forget drop while improving retain/test by 22–55 points relative to all-layers updates. The cutoff is read directly off the overlap elbow in Fig. 1 (layer3→ 4 transition) and is pushed deeper the more entangled the basis: CAV uses 3535/4040, while PCA—whose variance-driven eigenvectors overlap retain most—works best at 4545–5050. Shallower blocks have larger retain-damage constant μ1 _1 and more entangled bases, so their unmerging budget costs more retain than it buys in forget. Regularizers λ, γ. λ∈5,20,50λ\!∈\!\5,20,50\ scales the retain-leakage penalty (Term 2 of equation 2), chosen per initialization regime: λ=5λ\!=\!5–2020 on PT W_PT and λ=50λ\!=\!50 on 0 W_0, where larger deep task vectors demand stronger retain protection. The penalty is most binding on PCA bases, whose overlap with retain is highest; CAV-Class runs are insensitive to λ across an order of magnitude because the basis itself already separates forget from retain. γ∈0.25,0.5γ\!∈\!\0.25,0.5\ acts as a uniform magnitude prior on l B^l (shrinking the closed-form solution of Term 1 by 1/(1+γ)1/(1+γ)) and is most helpful on 0 W_0 where ℱl τ_F^l would otherwise over-expand along tail directions of the forget covariance. Both regularizers transfer across superclasses without retuning (§4.1), which we view as part of the unlearning quality of Unmerge. Selecting α without the retrained model. All configurations in this paper, ours and the baselines’, are selected by ToW, which uses RT W_RT and the test set, as is common practice (Fan et al., 2024; Kurmanji et al., 2023). For a class-level request a practitioner has neither, but knows the target forget accuracy (zero). Because l B^l does not depend on α, the update of equation 3 can be line-searched after a single solve, at the cost of one forward pass over ℱF per candidate. We swept α on a uniform grid for the six Trees configurations. Forget and retain accuracy are monotone in α, so we test the rule take the smallest α whose forget-set accuracy is at most 1%1\%. It selects the ToW-optimal grid point in five of the six cells and loses 0.0030.003 ToW in the sixth; the training-only score Accℛ(1−Accℱ)Acc_R(1-Acc_F) of Kodge et al. (2024) loses at most 0.0060.006. On the coarser grids of the transfer superclasses the rule selects the ToW-optimal cell or loses at most 0.0120.012 (nine cells). On TinyImageNet/0 W_0 it applies only to CAV-Class (loss ≤0.007≤\!0.007): for CAV-KMeans no α on our grid reaches 1%1\% forget accuracy, and for PCA on Arthropods the rule over-shoots (−0.15-0.15 ToW), so the hardest regime still needs a reference. Rank, skip-layers, λ and γ are transferred from the tuning task unchanged (App. C). D.2 Objective Terms Tab. 7 removes the terms of equation 2 one at a time with the reported estimator and re-sweeps α for every variant. With Term 1 alone the objective is minimized at the initialization, so the edit is α times the projection mℱ τ_m Q_F: it collapses retain for the PCA basis (ToW 0.460.46–0.480.48, retain 7171–94%94\%) and trails the full objective by 0.040.04–0.350.35 for the CAV bases, whose directions overlap retain less. Term 2 accounts for essentially all of the gain: with Terms 1 and 2 every cell is within 0.020.02 of the full objective, and above it in one. Term 3 is a uniform shrinkage that α already provides: on its own it closes at most part of the gap (Terms 1+3: 0.500.50–0.960.96, below Terms 1+2 in every cell), and at the reported λ its effect is within 0.020.02 everywhere. The optimization of Phase 2 is therefore meaningful precisely as the solver of the Term-2-regularized problem—the iterate ends at relative distance 0.70.7–1.91.9 from its initialization (App. D.3)—and not because the initialization is already a good edit. The task-vector baselines corroborate this from outside the method: NegTV and NegMerge subtract a vector finetuned on ℱF with no retain penalty, and they lose retain accuracy in the same way as Term 1 alone (App. C). Table 7: Term ablation on CIFAR-100 (Trees) with the reported estimator (100100 Adam steps): best ToW over an α sweep for each variant, with the selected α in parentheses. Term 1 only: λ=γ=0λ=γ=0 (the edit is αmℱα\, τ_m Q_F). Terms 1+3: λ=0λ=0, γ=0.5γ=0.5. Terms 1+2: λ as reported, γ=0γ=0. All three: λ and γ as reported. Every optimum is interior to its grid. Cell Term 1 only Terms 1+3 Terms 1+2 All three 0 W_0, PCA 0.4622 (0.5) 0.4978 (0.75) 0.9608 (1.25) 0.9609 (1.25) 0 W_0, CAV-KMeans 0.9419 (0.3) 0.9433 (0.4) 0.9819 (1) 0.9834 (1) 0 W_0, CAV-Class 0.9530 (0.5) 0.9553 (0.6) 0.9863 (1.5) 0.9895 (1.4) PT W_PT, PCA 0.4802 (1) 0.8174 (1.5) 0.9843 (1.6) 0.9868 (1.6) PT W_PT, CAV-KMeans 0.6162 (1.5) 0.6897 (1.875) 0.9775 (1.6) 0.9699 (2.4) PT W_PT, CAV-Class 0.8089 (1.5) 0.8600 (2.25) 0.9294 (2) 0.9459 (2.4) D.3 The Budgeted Solver and the Exact Minimizer Phase 2 of Alg. 1 moves l B^l from the projection 0l=mlℱl B^l_0= τ_m^l Q_F^l toward the exact minimizer l⋆ B^l of equation 4 under a fixed budget (T=100T=100 Adam steps, learning rate 10−310^-3). This subsection characterizes the estimator that results, compares it with the exact minimizer, and states what depends on the budget, on the six official Trees cells. (Implementation note: the code averages each loss term over its entries rather than summing, so its exact minimizer uses λeff=λkF/kR _eff=λ\,k_F/k_R; this differs from λ only for CAV-Class, where kF=|ℱ|=5k_F=|Y_F|=5 and kR=16k_R=16.) What the budgeted iterate computes. Adam’s normalized updates move every entry of l B^l by at most the step budget δ=lr×T=0.1δ=lr× T=0.1, several times the typical entry of the projection (rms 0.0120.012–0.0250.025). Entries closer than δ to the minimizer therefore converge, and the others move by about δ, so the iterate should be close to δl=l⋆+Sδ(0l−l⋆), B^l_δ= B^l +S_δ ( B^l_0- B^l ), (15) with SδS_δ the entrywise soft-threshold: the exact minimizer plus a sparse residual of the largest projection coefficients, interpolating between the projection (δ=0δ=0) and the minimizer (δ→∞δ→∞). Tab. 8 tests this model, which has no free parameter. Entrywise, δ B_δ is the closest of the three candidates to the Adam iterate in every cell (relative distance 0.010.01–0.320.32, against 0.050.05–0.960.96 for ⋆ B and 0.740.74–1.851.85 for 0 B_0), and applying 0.1 B_0.1 in place of the iterate, at the reported hyperparameters, reproduces the reported ToW within 0.0050.005 in five of the six cells (0.0230.023 in the sixth). The model also explains the split between bases. For the CAV bases the residual is nearly empty (at most 0.15%0.15\% of the entries survive the threshold) and ToW moves by at most 0.020.02 over δ∈[0.05,0.1]δ∈[0.05,0.1]: the solver hardly matters, which is why the exact minimizer can replace the iterate there. For PCA the residual carries the edit. On 0 W_0 the minimizer is a no-op at the reported λ and 0.36%0.36\% of the coefficients do the forgetting; this is a selection over (output unit, direction) pairs—the iterate places 37%37\% of its energy in the top 5%5\% of output units, against 22%22\% for ⋆ B —that no ridge solution, which filters all output units identically, can express. The price is a dependence on the step budget that the CAV variants do not have (last columns of Tab. 8). All our experiments use one budget (T=100T=100, learning rate 10−310^-3), never tuned per cell. Table 8: The early-stopped Adam iterate vs. the closed-form model δ B_δ of equation 15 on the six official CIFAR-100 (Trees) cells, at the reported (λ,γ,α)(λ,γ,α), seed 1. Rel. dist.: distance of the Adam iterate to the projection 0 B_0, to the minimizer ⋆ B , and to δ B_δ at the best δ∈0.02,…,0.12δ∈\0.02,…,0.12\, relative to ‖Adam‖F\| B_Adam\|_F and aggregated over edited layers. Resid.: share of entries with |0−⋆|>0.1| B_0- B |>0.1. ToW: the reported cell, and δ B_δ applied in its place. Rel. dist. of the iterate to Resid. ToW ToW of δ B_δ Cell 0 B_0 ⋆ B δ B_δ (%) Adam δ=0.1δ=0.1 0.0750.075 0.050.05 0 W_0, CAV-Class 1.46 0.35 0.09 0.07 0.9851 0.9804 0.9864 0.9860 0 W_0, CAV-KMeans 1.64 0.49 0.32 0.15 0.9834 0.9604 0.9784 0.9795 0 W_0, PCA (k=128k=128) 1.68 0.96 0.31 0.36 0.9609 0.9610 0.9088 0.5926 PT W_PT, CAV-Class 0.74 0.07 0.01 0.01 0.9418 0.9380 0.9418 0.9429 PT W_PT, CAV-KMeans 1.31 0.05 0.01 0.00 0.9652 0.9651 0.9654 0.9648 PT W_PT, PCA (k=64k=64) 1.85 0.45 0.16 0.10 0.9868 0.9867 0.9779 0.9452 The exact minimizer, tuned for itself. Hyperparameters do not transfer between the two estimators. At the (λ,γ,α)(λ,γ,α) tuned for the budgeted iterate, l⋆ B^l reproduces the CAV cells on PT W_PT (ToW 0.9320.932/0.9650.965 vs. 0.9420.942/0.9650.965; the iterate is within relative distance 0.050.05–0.070.07 of it there), under-forgets the CAV cells on 0 W_0 (0.890.89/0.680.68) and is a no-op for both PCA cells, because at that λ the retain penalty shrinks ℱ τ_F to 33–10%10\% of m τ_m along PCA’s heavily overlapping directions; the plain projection 0l B^l_0 at the same α collapses retain instead (1.1%1.1\%/14.6%14.6\%). Tuned for itself over λ∈0,…,200λ∈\0,…,200\, γ∈0,0.5γ∈\0,0.5\ and α (3434–5050 runs per cell, all optima interior), the exact minimizer prefers γ=0γ=0 and a larger λ (1010–100100) with a correspondingly larger α—it scales the component of mlℱl τ_m^l Q_F^l along the i-th principal direction between the forget and retain spans by 1/(1+γ+λσi2)1/(1+γ+λ _i^2), so a larger λ shrinks only the shared directions—and is then within 0.010.01 of the reported iterate or above it in five of six cells (0.9900.990/0.9750.975 for CAV-Class/KMeans on 0 W_0; 0.9990.999/0.9960.996/0.9910.991 for CAV-Class/KMeans/PCA on PT W_PT), while it fails for PCA on 0 W_0 (0.650.65). The reported estimator run at those hyperparameters reproduces the closed form for the CAV bases (0.9990.999, 0.9950.995, 0.9910.991 and 0.9790.979), so the gap to the reported CAV cells reflects the λ range of the original Trees grid (up to 55 on PT W_PT) rather than the solver, and those cells are conservative; for PCA the two estimators remain different objects (0.400.40 and 0.00010.0001 at the closed form’s optimum). Scope. Both solvers are negligible in cost (0.20.2–0.80.8s for the iterate and 0.010.01–0.030.03s for the solve, against 88–2121s for Phase 1), so the choice is about universality, not efficiency. We report the budgeted iterate throughout because, at one fixed budget, it is the only estimator that works in all six cells. For the CAV variants the exact minimizer is an equivalent, budget-free alternative, and equation 15 gives the iterate an explicit form in which the threshold δ replaces the step count; the PCA variants, in contrast, depend on the budget (last columns of Tab. 8), which is a limitation of those variants. Adopting an explicit form as the default changes the scale of (λ,α)(λ,α) and needs a multi-seed, multi-dataset validation that we leave to future work. All cells of this subsection are single-seed ablations on Trees. D.4 init W_init Ablation Unmerge forms m=FT−init τ_m= W_FT- W_init, so we ask how much the pre-finetuning checkpoint matters (Tab. 9; Trees, PT W_PT regime). Setting init=0 W_init=0—our convention throughout the 0 W_0 regime—makes m τ_m larger (it now contains the pretrained weights), so the paper’s α over-shoots; with α retuned on the same grid (0.90.9 for CAV and 1.21.2 for PCA, against 2.52.5 and 1.61.6 at the true init) zero-init matches the CAV cells and trails only on PCA, whose variance basis otherwise absorbs the shared pretrained content that the true init subtracts. Substituting a different pretrained checkpoint (torchvision IMAGENET1K_V2 in place of V1) is catastrophic for every variant: the subtracted vector then contains the difference between two unrelated pretrainings. The exact checkpoint is therefore not required—but a wrong one must not be used, and init=0 W_init=0 is the safe fallback. The PT W_PT checkpoint is part of the method’s input and we release it: torchvision ResNet-50 IMAGENET1K_V1 with CIFAR-adapted first convolution and classifier (SHA-256 prefix 2a54223ea63a790f; full hash in the code release). The adapted layers are seeded, so the file, not the recipe, is the artifact. Table 9: init W_init ablation on CIFAR-100 (Trees), PT W_PT regime: ToW per Unmerge variant (retrained reference F/R/T 0.000.00/99.9899.98/80.6780.67). init W_init CAV-Class CAV-KMeans PCA True init (V1, paper cells) 0.9418 0.9652 0.9868 Zero, paper α 0.7249 0.6036 0.8291 Zero, α retuned 0.9629 0.9639 0.9422 Wrong checkpoint (V2) 0.4742 0.0745 0.0628 Appendix E Additional Results E.1 Detailed Per-Task Results The summary in Tab. 1 averages over three CIFAR-100 superclasses (Trees, Aquatic Mammals, Vehicles); we expose the per-superclass breakdown here. Tabs. 10, 11, and 12 report the three CIFAR-100 superclasses individually, and Tab. 13 reports the second TinyImageNet superclass (Arthropods). The hyperparameter set is tuned once per (method, init) on Trees and Dogs & Cats and transferred to Aquatic Mammals, Vehicles, and Arthropods with at most a small change of α (App. C), so these tables also stress-test transferability. The pattern from the main paper is preserved: in every (superclass, init) cell at least one Unmerge variant leads every baseline on ToW, including the task-vector baselines and the projection of Kodge et al. (2024); on the CIFAR-100 superclasses at least two of three lead every gradient-based baseline, whereas on TinyImageNet/0 W_0 (Dogs & Cats and Arthropods) only Unmerge-CAV-Class leads and PCA/KMeans trail SalUn. Unmerge-CAV-Class is consistently strongest on 0 W_0; on PT W_PT the lead goes to Unmerge-PCA on the CIFAR-100 superclasses, to Unmerge-CAV-KMeans on Arthropods and to Unmerge-CAV-Class on Dogs & Cats. Table 10: CIFAR-100 unlearning results (Trees). Rows: evaluation metric (Forget/Retain/Test accuracy, %) and tug-of-war score (equation 8; higher is better). Columns: unlearning method. FT W_FT and RT W_RT are the finetuned upper bound and retrained gold standard. Best non-retrained in bold. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 0.36 12.24 4.32 1.72 2.60 0.16 0.08 0.00 0.12 Retain Acc ↑ 99.98 97.21 99.94 99.92 96.87 97.50 99.67 99.80 99.40 98.82 Test Acc ↑ 80.67 74.48 80.62 80.64 74.59 75.24 78.25 79.60 77.75 76.07 ToW ↑ 1 0.9089 0.8768 0.9560 0.8943 0.8983 0.9712 0.9868 0.9652 0.9418 Runtime ↓ 24s 110s 111s 4s 11s 8s 8s 26s 22s init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 4.40 15.44 3.08 1.40 1.96 0.08 0.12 0.68 1.12 Retain Acc ↑ 99.97 96.04 99.96 99.94 95.60 96.50 99.01 98.75 99.83 99.87 Test Acc ↑ 72.92 66.05 74.00 74.14 66.19 66.91 70.05 70.31 72.07 72.65 ToW ↑ 1 0.8553 0.8364 0.9571 0.8795 0.8895 0.9613 0.9609 0.9834 0.9851 Runtime ↓ 24s 111s 115s 4s 12s 9s 9s 21s 18s Table 11: CIFAR-100 unlearning results (Aquatic Mammals). Best non-retrained in bold. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 0.56 3.36 5.52 4.68 7.24 0.44 0.20 0.20 1.40 Retain Acc ↑ 99.99 95.34 99.94 99.98 98.00 98.68 99.67 99.72 99.71 99.52 Test Acc ↑ 80.74 73.16 80.26 81.05 75.64 76.46 78.54 79.20 79.16 77.95 ToW ↑ 1 0.8763 0.9613 0.9417 0.8866 0.8763 0.9705 0.9799 0.9795 0.9540 Runtime ↓ 23s 110s 112s 4s 11s 8s 9s 27s 22s init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 5.72 3.40 4.32 5.72 6.32 0.72 0.72 0.56 0.16 Retain Acc ↑ 99.97 97.18 99.97 99.97 96.93 97.18 99.00 95.80 99.66 99.81 Test Acc ↑ 72.8 69.02 74.55 74.69 68.82 68.85 70.68 66.66 72.57 73.48 ToW ↑ 1 0.8819 0.9491 0.9387 0.8778 0.8747 0.9623 0.8929 0.9891 0.9900 Runtime ↓ 23s 109s 112s 4s 11s 8s 6s 21s 18s Table 12: CIFAR-100 unlearning results (Vehicles). Best non-retrained in bold. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 0.00 12.12 8.52 0.40 0.44 0.36 0.04 0.04 0.56 Retain Acc ↑ 99.97 97.38 99.94 99.91 90.43 91.31 98.82 99.89 99.78 99.32 Test Acc ↑ 80.15 73.82 80.05 79.87 68.21 68.85 76.07 79.14 78.09 76.68 ToW ↑ 1 0.9125 0.8777 0.9117 0.7934 0.8066 0.9448 0.9887 0.9771 0.9537 Runtime ↓ 23s 110s 112s 4s 11s 8s 8s 26s 22s init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 3.48 9.36 4.36 14.84 15.24 1.80 0.68 0.00 0.00 Retain Acc ↑ 99.96 97.46 99.96 99.93 94.66 94.94 97.04 96.59 99.62 99.86 Test Acc ↑ 71.7 67.56 73.29 73.70 65.15 65.09 66.83 67.07 70.74 71.85 ToW ↑ 1 0.9021 0.8920 0.9370 0.7536 0.7519 0.9069 0.9153 0.9870 0.9975 Runtime ↓ 23s 110s 111s 4s 11s 8s 9s 21s 18s Table 13: TinyImageNet unlearning results (Arthropods). Best non-retrained in bold. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 4.42 15.12 14.70 1.30 2.04 23.68 1.84 0.00 0.00 Retain Acc ↑ 99.98 94.33 99.98 99.98 87.49 90.45 98.83 99.04 99.96 99.95 Test Acc ↑ 73.62 65.38 74.17 73.91 60.69 62.65 69.68 71.44 73.20 72.72 ToW ↑ 1 0.8275 0.8441 0.8505 0.7520 0.7890 0.7247 0.9512 0.9956 0.9907 Runtime ↓ 94s 671s 674s 18s 53s 22s 71s 104s 100s init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Forget Acc ↓ 0.0 0.70 10.02 9.56 23.22 25.86 12.42 4.12 9.30 0.12 Retain Acc ↑ 99.98 80.14 99.98 99.97 96.34 96.42 85.35 94.91 98.85 98.18 Test Acc ↑ 61.86 44.78 60.97 60.78 53.28 53.75 47.46 52.26 56.51 55.76 ToW ↑ 1 0.6600 0.8918 0.8946 0.6764 0.6570 0.6400 0.8228 0.8488 0.9210 Runtime ↓ 93s 666s 673s 18s 53s 22s 38s 85s 61s E.2 Seed Variance Tab. 14 repeats the Trees experiments for three seeds with frozen configurations. The seed controls model initialization only: the data order and augmentation streams are fixed across seeds. In the 0 W_0 regime it therefore changes the initialization of the finetuned and the retrained model; in the PT W_PT regime retraining from the fixed checkpoint is deterministic given the data order, so all seeds share one retrained reference and the variance reflects finetuning and method stochasticity (init W_init is held fixed across seeds, including its seeded stem and classifier entries). Every ordering between the Unmerge variants and SalUn/Random Label in Tab. 10 is preserved in the means; NegGrad+ and Random Label swap on 0 W_0 within noise. On 0 W_0 the Unmerge standard deviations (0.0080.008–0.0210.021) are 1.71.7–4.5×4.5× smaller than those of Random Label and SalUn (0.0360.036–0.0370.037), and Unmerge-PCA’s mean (0.9370.937) sits above SalUn’s (0.9260.926) although the single-seed cells are close. On PT W_PT SalUn is the most stable method (0.0030.003) but trails Unmerge-PCA and CAV-KMeans by 1.01.0–1.31.3 points. The projection of Kodge et al. (2024) is on par with Unmerge-PCA on PT W_PT (±0.0050.974\!±\!0.005 vs. ±0.0150.972\!±\!0.015) and below the CAV variants on 0 W_0 (±0.0160.944\!±\!0.016 vs. 0.9690.969–0.9760.976); the task-vector baselines stay at 0.860.86–0.900.90. Table 14: ToW on CIFAR-100 (Trees) over three seeds (seeds 11–33 control the finetuning run, the from-scratch retraining on 0 W_0, and the unlearning method; init W_init and the data order are fixed), mean ± sample std. Seed 1 is the cell reported in Tab. 10. Best per column in bold. Method init=PT W_init= W_PT init=0 W_init= W_0 NegGrad+ 0.9142 ± 0.0099 0.8280 ± 0.0237 Rand. Label 0.8803 ± 0.0179 0.8391 ± 0.0374 SalUn 0.9587 ± 0.0025 0.9263 ± 0.0357 NegTV 0.8947 ± 0.0054 0.8571 ± 0.0220 NegMerge 0.9010 ± 0.0097 0.8633 ± 0.0237 Kodge et al. 0.9744 ± 0.0046 0.9439 ± 0.0156 Unmerge-PCA 0.9716 ± 0.0146 0.9370 ± 0.0208 Unmerge-CAV-KMeans 0.9688 ± 0.0103 0.9688 ± 0.0127 Unmerge-CAV-Class 0.9408 ± 0.0102 0.9756 ± 0.0083 E.3 Unlearning with ViT We evaluate Unmerge on ViT-S/16 Dosovitskiy et al. (2021) (ImageNet-1k weights, 224×224224× 224 inputs) finetuned on CIFAR-100 (test 90.2%90.2\%) with the Trees superclass as forget set, in the PT W_PT regime. The retrained reference uses the same schedule and reaches 86.1%86.1\% test accuracy, so equation 8 is well calibrated. The per-layer pipeline (Alg. 1) is applied to the linear projections of the attention and MLP blocks; the skip-layers recipe of §4.3 carries over to the transformer’s stack of 49 linear layers and, as in ResNet-50, deep-block editing wins: the best cells skip the first 32 (PCA) and 40 (CAV-KMeans) linear layers and edit only the last blocks and the head. Tab. 15 reports all three axes of §4. Result. Unmerge-CAV-KMeans leads every baseline on every axis simultaneously: ToW 0.99750.9975 vs. 0.99490.9949 (SalUn) and 0.99250.9925 (Random Label), at 3434s vs. ∼ 474474s (14×14×), with the MIA gap to retraining 1.71.7–1.8×1.8× smaller (0.0610.061 vs. 0.1050.105–0.1090.109) and the feature-entanglement gap 6.56.5–7.4×7.4× smaller (0.1010.101 vs. 0.6530.653–0.7430.743). Unmerge-PCA has the smallest MIA gap (0.0560.056) and runs in 2121s; NegGrad+ under-forgets (20%20\% forget accuracy) and damages retain. The retrained reference is strong (test 86.1%86.1\%), and the deep-layer locality finding of §4.3 transfers to attention-based backbones. Bias rule. ViT-S/16 has a bias in every edited layer, so the bias treatment of App. B.1 (implementation note iv) could matter here. It does not: with biases left at their finetuned values the two cells move from 0.99750.9975 to 0.99770.9977 (CAV-KMeans) and from 0.98100.9810 to 0.97920.9792 (PCA), and the α profiles are unchanged to within 0.0020.002. Table 15: ViT-S/16 on CIFAR-100 (Trees), init=PT W_init= W_PT: accuracies (%), ToW (equation 8), runtime, mean MIA gap to retraining over 9 attacks (§4.5), and mean relative feature-entanglement gap (§4.4). Best non-retrained in bold. NegGrad+ was not evaluated on the MIA/entanglement axes. RT W_RT NegGrad+ Rand. Label SalUn Unmerge-PCA Unmerge-CAV KMeans Forget Acc ↓ 0.00 20.00 0.12 0.24 0.92 0.16 Retain Acc ↑ 99.97 86.03 99.98 99.98 99.64 99.94 Test Acc ↑ 86.11 73.43 85.49 85.85 85.44 86.05 ToW ↑ 1 0.6012 0.9925 0.9949 0.9810 0.9975 Runtime ↓ 88s 472s 474s 21s 34s MIA gap ↓ 0 – 0.105 0.109 0.056 0.061 Entanglement gap ↓ 0 – 0.653 0.743 0.171 0.101 E.4 Preliminary Study on a Language Model We test whether unmerging carries over to language models on TOFU Maini et al. (2024) (forget10: the 400400 question–answer pairs of 2020 fictitious authors; 36003600 retain pairs) with Llama-3.2-3B-Instruct Grattafiori et al. (2024), using the checkpoints and the evaluation harness of OpenUnlearning Dorna et al. (2025): init W_init is the public instruction-tuned model, FT W_FT the released model finetuned on all of TOFU, and RT W_RT the released model finetuned on the retain split, so nothing is trained. We apply the per-layer objective to the linear projections of the last 1212–1616 of 2828 decoder layers, with activation statistics taken at question and answer tokens, and use the closed-form solver of equation 4 (the entries of mℱ τ_m Q_F are about two orders of magnitude smaller here than in the vision models, so the fixed-budget iterate would simply converge to the minimizer). This is a preliminary exploration: each variant receives only light tuning of α, the rank and the edited layers on a proxy that needs no generation (TOFU’s normalized answer probability on ℱF and on held-out retain pairs, relative to RT W_RT), and the baselines run at OpenUnlearning’s default settings. Results. With a retain-deflated PCA forget basis—the forget covariance after projecting out the top 10241024 retain directions, i.e. the hard orthogonal projection that our vision runs leave switched off—and λ=0λ=0, k=64k=64, α=64α=64, the edit is selective (Tab. 16). Under the official evaluation its forget quality is 0.1310.131, i.e. the Kolmogorov–Smirnov test no longer separates the edited model from RT W_RT at the 5%5\% level, at a model utility of 0.6300.630 with the MLP projections of the last 1616 layers edited (RT W_RT: 0.6500.650). RMU Li et al. (2024), the strongest of three baselines, reaches 0.0940.094 and 0.6330.633 (GradDiff: ×10−271.6\!×\!10^-27 and 0.6180.618; NPO: ×10−141.8\!×\!10^-14 and 0.6010.601): Unmerge attains a higher forget quality at a comparable utility (0.0030.003 lower) and lands closer to RT W_RT on the forget set, where RMU over-forgets (answer probability 0.040.04 against 0.120.12 for RT W_RT and 0.200.20 for Unmerge). The edit needs one pass for activation statistics (4343s) and no training, whereas the baselines train for 7070–290290s. A training-free, closed-form edit is thus already competitive with a dedicated language-model unlearning method, which makes scaling Unmerge a promising extension. What changes from vision. Two observations guide that extension. The basis matters more. With the plain PCA forget basis the forget and retain bases overlap almost completely (mean per-direction overlap 0.870.87–0.910.91) and the edit lowers the answer probability on ℱF and on ℛR in lockstep, the signature of the random subsets of §4.5; deflating the retain directions restores selectivity. The calibrated separability ratio reads 2.82.8–3.43.4 even for the plain basis, so on this model the small absolute score (S=0.03S=0.03–0.050.05) is the more informative signal. The CAV bases trail. Token-level ridge probes, one per forget author (CAV-Class) or per k-means cluster of forget tokens (CAV-KMeans), are clean (overlap with the retain basis below 0.130.13) but capture only 11–8%8\% of the forget-token variance, so retain degrades before forgetting completes (best proxy score 0.550.55–0.590.59 against 0.850.85 for the deflated PCA basis and 0.180.18 for the un-edited model). Concept probes better matched to language, e.g. on subject tokens, are a natural next step. Scope. The study uses one split, one model and one seed. Unmerge is tuned lightly on a proxy that uses RT W_RT, whereas the baselines are untuned. The scale α=64α=64 is far above the vision range: the deflated directions carry little of m τ_m, so the edit extrapolates along forget-private input directions rather than removing a finetuning delta. Table 16: TOFU forget10 with Llama-3.2-3B-Instruct under the official OpenUnlearning evaluation. Forget quality is the p-value of a two-sample test against RT W_RT (higher is better; 11 for RT W_RT by definition). Unmerge: closed form, retain-deflated PCA forget basis, MLP projections of the last 1616 layers, lightly tuned. RMU: OpenUnlearning default settings. Model Forget quality ↑ Model utility ↑ Forget answer prob. Forget ROUGE FT W_FT (finetuned) ×10−273.6\!×\!10^-27 0.666 0.951 0.926 RT W_RT (retain-only) 1 0.650 0.124 0.386 RMU 0.094 0.633 0.037 0.225 Unmerge 0.131 0.630 0.200 0.364 E.5 Feature-Level Entanglement on TinyImageNet The CIFAR-100 picture in Tab. 3 is sharpened on TinyImageNet (Tab. 17), measured on the Dogs & Cats split with the same penultimate-feature protocol and same-size retain subsample. Unmerge-CAV-Class/0 W_0 matches the retrained reference to within ∼ 6% relative gap (mean 0.0620.062, with MMD2 at 0.0000.000 indistinguishable from retrained), while Random Label and SalUn deviate by 3.973.97–3.99×3.99×, roughly 64× the unmerge gap. The projection of Kodge et al. (2024) posts a smaller mean gap on 0 W_0 (0.0360.036), but the two cells are close on this axis: both match the retrained distance to within 2.4%2.4\% on the two MMD2 variants and deviate by the same 13%13\% on W22_2^2-Bures (in opposite directions), and the difference in the mean comes from Sliced-W22_2^2, where CAV-Class lands 9%9\% below the retrained distance (forget and retain features slightly more mixed than under retraining) and the projection 0.3%0.3\% above. What distinguishes them is how they get there: the projection reaches the retrained feature distance while still classifying 18%18\% of the forget set correctly and losing 77 test points (Tab. 2), i.e. by degrading features overall, which ToW and the MIA gap (0.1570.157 vs. 0.0580.058) expose; the entanglement axis alone cannot. On PT W_PT the spread compresses but Unmerge still leads: CAV-Class (0.0820.082) is ∼ 6×6× tighter than NegGrad+, ∼ 50×50× tighter than RL/SalUn, and slightly tighter than the projection (0.0990.099). The W22_2^2-Bures column is mean-shift dominated in the D≫ND N regime; the MMD2 and Sliced-W22_2^2 metrics, sensitive to higher moments and full distribution shape, rank the best Unmerge variant ahead of every gradient and task-vector baseline on both initializations. Table 17: Feature-level entanglement on TinyImageNet (Dogs & Cats), same protocol as Tab. 3. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class MMD2 0.000 0.514 4.531 4.368 0.508 0.542 0.103 0.178 0.090 0.058 MMD2-multi 0.000 0.545 5.373 5.179 0.562 0.579 0.188 0.076 0.070 0.074 W22_2^2-Bures 0.000 0.419 2.174 2.084 0.477 0.453 0.031 0.756 0.176 0.064 Sliced-W22_2^2 0.000 0.545 4.668 4.458 0.562 0.564 0.074 0.221 0.083 0.132 Mean rel. gap ↓ 0.000 0.506 4.187 4.022 0.527 0.534 0.099 0.308 0.105 0.082 init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class MMD2 0.000 0.439 4.187 4.186 0.352 0.342 0.012 0.098 0.268 0.000 MMD2-multi 0.000 0.469 4.968 4.978 0.371 0.359 0.003 0.209 0.246 0.024 W22_2^2-Bures 0.000 0.331 2.302 2.317 0.136 0.123 0.125 0.490 0.095 0.129 Sliced-W22_2^2 0.000 0.474 4.441 4.469 0.315 0.301 0.003 0.433 0.194 0.093 Mean rel. gap ↓ 0.000 0.428 3.974 3.988 0.293 0.281 0.036 0.308 0.201 0.062 E.6 Membership Inference: Detailed Results We provide per-attack MIA tables for both datasets, expanding the summary in §4.5. Both protocols are identical (RUM/SalUn-style shadow/target split, nine attack signals, gap-to-retrained metric). Three details matter when reading the numbers. The attack accuracy on ℱF averages the attacker’s true-positive rate on retain samples and its true-negative rate on forget samples, so damage to retain accuracy also registers as a privacy gap. The SVC attackers are trained on an unbalanced shadow set (∼ 83%83\% members) without class weighting, and the correctness signal appears twice (threshold and SVC give the same attacker). Finally, the class-level tables report the mean over signals of the absolute gap, whereas the instance-level table (Tab. 20) reports the signed gap of the signal-averaged attack accuracy, in which per-signal errors of opposite sign can cancel; App. E.7 gives both aggregates for the coherent protocol. CIFAR-100 (Trees). Tab. 18 reports per-attack gaps. On 0 W_0 the projection of Kodge et al. (2024) has the smallest mean gap (0.0170.017), followed by Unmerge-CAV-KMeans (0.0260.026) and Unmerge-CAV-Class (0.0290.029), 1.61.6–1.8×1.8× tighter than SalUn (0.0470.047), 2.12.1–4.2×4.2× tighter than Random Label (0.0620.062) and NegGrad+ (0.1090.109), and 2.82.8–3.5×3.5× tighter than NegTV and NegMerge (0.0820.082–0.0920.092); the SVC-probs and SVC-entropy attacks (full softmax-vector attackers, the strongest in our set) are where the gap closes most, with the CAV variants cutting both by 1.51.5–2.3×2.3× over the saliency baselines. On PT W_PT the projection leads again (0.0280.028) and the spread among the other methods is tight: NegGrad+ 0.0670.067, Unmerge-CAV-Class 0.0700.070, NegMerge 0.0720.072, NegTV 0.0730.073, Unmerge-CAV-KMeans 0.0750.075, Unmerge-PCA 0.0780.078, SalUn 0.0820.082, Random Label 0.0930.093. NegGrad+’s small edge over Unmerge comes at a 2.82.8-point retain and 6.26.2-point test drop (Tab. 10), so the joint utility+\,+\,privacy ranking still favors Unmerge. Table 18: MIA gap to the retrained reference on CIFAR-100 (Trees): |attℱunl−attℱRT||att_F^unl-att_F^RT| per attack signal (lower is better; RT W_RT is 0 by construction). init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Threshold-Conf. 0.000 0.034 0.017 0.011 0.036 0.035 0.015 0.024 0.028 0.033 Threshold-Ent. 0.000 0.033 0.171 0.173 0.027 0.048 0.104 0.160 0.154 0.144 Threshold-M-Ent. 0.000 0.035 0.021 0.007 0.036 0.035 0.018 0.024 0.027 0.040 Threshold-Corr. 0.000 0.016 0.061 0.022 0.025 0.025 0.002 0.001 0.003 0.007 SVC-Conf. 0.000 0.009 0.002 0.003 0.020 0.021 0.004 0.003 0.002 0.006 SVC-Ent. 0.000 0.211 0.209 0.207 0.261 0.242 0.065 0.197 0.194 0.154 SVC-M-Ent. 0.000 0.049 0.001 0.002 0.024 0.027 0.003 0.002 0.003 0.006 SVC-Corr. 0.000 0.015 0.061 0.022 0.023 0.025 0.002 0.001 0.003 0.007 SVC-Probs. 0.000 0.203 0.291 0.290 0.201 0.190 0.038 0.288 0.258 0.233 Mean||gap|| ↓ 0.000 0.067 0.093 0.082 0.073 0.072 0.028 0.078 0.075 0.070 init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Threshold-Conf. 0.000 0.037 0.010 0.003 0.022 0.026 0.007 0.008 0.001 0.001 Threshold-Ent. 0.000 0.103 0.063 0.065 0.072 0.057 0.017 0.040 0.046 0.054 Threshold-M-Ent. 0.000 0.040 0.007 0.002 0.025 0.032 0.004 0.005 0.006 0.001 Threshold-Corr. 0.000 0.042 0.077 0.016 0.029 0.027 0.005 0.006 0.004 0.006 SVC-Conf. 0.000 0.041 0.000 0.000 0.019 0.021 0.003 0.003 0.003 0.002 SVC-Ent. 0.000 0.281 0.143 0.139 0.295 0.231 0.017 0.354 0.084 0.093 SVC-M-Ent. 0.000 0.079 0.000 0.001 0.023 0.025 0.004 0.005 0.002 0.001 SVC-Corr. 0.000 0.041 0.077 0.016 0.030 0.027 0.005 0.006 0.004 0.006 SVC-Probs. 0.000 0.314 0.184 0.183 0.309 0.294 0.094 0.316 0.080 0.094 Mean||gap|| ↓ 0.000 0.109 0.062 0.047 0.092 0.082 0.017 0.083 0.026 0.029 TinyImageNet (Dogs & Cats). On PT W_PT, all three Unmerge variants sweep the table (Tab. 19): Unmerge-PCA (0.0090.009) and Unmerge-CAV-Class/Unmerge-CAV-KMeans (0.0180.018/0.0200.020) are 22–5×5× tighter than the saliency baselines (Random Label/SalUn ∼ 0.0430.043), 99–20×20× tighter than NegGrad+ (0.1770.177), and 1313–29×29× tighter than the task-vector baselines and the projection of Kodge et al. (2024) (0.2540.254–0.2610.261). On 0 W_0, Random Label (0.0340.034) and SalUn (0.0400.040) appear to lead the CAV variants (0.0580.058). This is an artifact of under-unlearning: the baseline configurations used on this split (Random Label lr==10−310^-3, SalUn τ=0.5τ=0.5/lr==10−310^-3) leave about 9%9\% forget accuracy (Tab. 2), so their predictions on ℱF have barely moved from the finetuned model and the MIA gap closes by inaction rather than by erasure. The entanglement axis exposes it: the same checkpoints deviate from the retrained reference by ∼ 4×4× its value, while Unmerge-CAV-Class/0 W_0 matches retraining to within 6%6\% (Tab. 17). Neither axis suffices alone: the MIA gap can be closed by leaving the model unchanged, and the feature distance can be matched by degrading the features overall, as the projection does on this split (App. E.5). Read together with accuracy, the two axes single out the methods that forget without either failure. Table 19: MIA gap to the retrained reference on TinyImageNet (Dogs & Cats), same protocol as Tab. 18. init=PT W_init= W_PT RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Threshold-Conf. 0.000 0.044 0.007 0.009 0.051 0.044 0.046 0.016 0.008 0.012 Threshold-Ent. 0.000 0.108 0.012 0.011 0.113 0.095 0.076 0.005 0.012 0.006 Threshold-M-Ent. 0.000 0.048 0.006 0.011 0.051 0.048 0.048 0.018 0.008 0.009 Threshold-Corr. 0.000 0.025 0.081 0.077 0.052 0.046 0.042 0.007 0.000 0.000 SVC-Conf. 0.000 0.028 0.001 0.001 0.500 0.500 0.500 0.002 0.001 0.001 SVC-Ent. 0.000 0.431 0.066 0.066 0.431 0.431 0.413 0.021 0.055 0.043 SVC-M-Ent. 0.000 0.048 0.000 0.000 0.287 0.278 0.301 0.002 0.000 0.001 SVC-Corr. 0.000 0.500 0.081 0.077 0.500 0.500 0.500 0.007 0.000 0.000 SVC-Probs. 0.000 0.360 0.137 0.137 0.363 0.362 0.362 0.007 0.099 0.092 Mean||gap|| ↓ 0.000 0.177 0.044 0.043 0.261 0.256 0.254 0.009 0.020 0.018 init=0 W_init= W_0 RT W_RT NegGrad+ Rand. Label SalUn NegTV NegMerge Kodge et al. Unmerge-PCA Unmerge-CAV KMeans Unmerge-CAV Class Threshold-Conf. 0.000 0.064 0.005 0.009 0.064 0.063 0.063 0.047 0.028 0.025 Threshold-Ent. 0.000 0.171 0.018 0.017 0.092 0.089 0.061 0.107 0.039 0.040 Threshold-M-Ent. 0.000 0.067 0.003 0.008 0.065 0.064 0.065 0.052 0.029 0.025 Threshold-Corr. 0.000 0.072 0.047 0.045 0.076 0.076 0.107 0.104 0.025 0.013 SVC-Conf. 0.000 0.500 0.001 0.001 0.111 0.117 0.170 0.500 0.028 0.006 SVC-Ent. 0.000 0.435 0.083 0.137 0.435 0.435 0.304 0.435 0.128 0.171 SVC-M-Ent. 0.000 0.137 0.001 0.002 0.214 0.200 0.191 0.311 0.052 0.010 SVC-Corr. 0.000 0.500 0.047 0.045 0.500 0.500 0.106 0.500 0.024 0.011 SVC-Probs. 0.000 0.397 0.101 0.101 0.383 0.380 0.349 0.397 0.172 0.220 Mean||gap|| ↓ 0.000 0.260 0.034 0.040 0.215 0.214 0.157 0.272 0.058 0.058 E.7 Instance-Level Forgetting: Details Protocols and references. Coherent: a fixed random half of the 25002500 Trees training images forms ℱF (|ℱ|=1250|F|=1250; the draw is not stratified, giving 237237–262262 images per class) and the other half stays in ℛR. Random: 25002500 training images drawn uniformly from all 100 classes. Each protocol and initialization has its own retrained reference (coherent: 0 W_0 F/R/T 70.6470.64/99.9699.96/76.2776.27, PT W_PT 74.2474.24/99.9899.98/84.9384.93; random: 0 W_0 74.9274.92/99.9799.97/74.7374.73, PT W_PT 84.5684.56/99.9899.98/84.0284.02). Baselines use the class-level configurations of §4.1, except NegGrad+ on the coherent protocol: its class-level setting over-forgets there (gradient ascent on half a class erases the class; forget accuracy 26%26\%/11%11\%, ToW 0.49700.4970/0.32510.3251), so we tune it on a 40-run grid over its forget/retain weight 0.3,…,0.99\0.3,…,0.99\, learning rate 10−5,…,10−4\10^-5,…,10^-4\ and epochs 5,…,12\5,…,12\. Unmerge uses, on the random protocol, the protocol grid of App. D.1 (skip-layers 35,…,50\35,…,50\, λ∈5,20,50λ∈\5,20,50\, γ∈0.25,0.5γ∈\0.25,0.5\, PCA k∈64,128k∈\64,128\, CAV k=16k=16, fine α), and on the coherent protocol its class-level configuration with only α swept (five values per basis). All cells report the best final ToW of their grid, and every reported optimum is interior in the swept quantity. Table 20: Instance-level forgetting on CIFAR-100 (Trees), ResNet-50. Coherent: half of the training images of each Trees class (|ℱ|=1250|F|=1250); Random: 25002500 images drawn uniformly from all 100 classes. Accuracies in %. attℱatt_F: mean attack accuracy on ℱF over the 9 MIA signals of §4.5, with the signed gap to RT W_RT in parentheses (positive = forget samples more identifiable than under retraining). Ent.: mean relative entanglement gap of Tab. 3. Best non-retrained ToW and entanglement per protocol in bold. †On the coherent protocol NegGrad+ is tuned on a 40-run grid (forget/retain weight, learning rate, epochs), because its class-level setting over-forgets there (ToW 0.49700.4970/0.32510.3251); Random Label and SalUn keep their class-level settings, and the Unmerge cells sweep only α (five values per basis). The identical forget/retain accuracies of Unmerge-PCA on 0 W_0 and Unmerge-CAV-KMeans on PT W_PT are genuine (901 of 1250 forget images correct in both; test accuracies and losses differ). init=0 W_init= W_0 Coherent (|ℱ|=1250|F|=1250) Random (|ℱ|=2500|F|=2500) Forget Retain ToW ↑ attℱatt_F (gap) Ent. ↓ Forget Retain ToW ↑ attℱatt_F (gap) Ent. ↓ RT W_RT 70.64 99.96 1 0.700 0 74.92 99.97 1 0.681 0 NegGrad+† 71.60 98.62 0.9368 0.624 (−-0.076) 0.46 99.92 99.97 0.7302 0.502 (−-0.180) 0.62 Rand. Label 88.72 99.96 0.8137 0.839 (++0.140) 0.41 69.80 99.96 0.9405 0.872 (++0.191) 21.9 SalUn 87.44 99.93 0.8243 0.825 (++0.125) 0.29 72.20 99.89 0.9595 0.839 (++0.158) 15.5 Unmerge-PCA 72.08 99.15 0.9665 0.712 (++0.013) 0.15 99.64 99.77 0.7509 0.502 (−-0.180) 0.67 Unmerge-CAV-KMeans 73.60 99.25 0.9534 0.691 (−-0.009) 0.27 97.32 98.19 0.7515 0.506 (−-0.176) 0.60 init=PT W_init= W_PT Coherent (|ℱ|=1250|F|=1250) Random (|ℱ|=2500|F|=2500) Forget Retain ToW ↑ attℱatt_F (gap) Ent. ↓ Forget Retain ToW ↑ attℱatt_F (gap) Ent. ↓ RT W_RT 74.24 99.98 1 0.686 0 84.56 99.98 1 0.630 0 NegGrad+† 74.96 99.08 0.9427 0.620 (−-0.066) 0.48 100.0 99.97 0.8406 0.500 (−-0.130) 0.55 Rand. Label 85.84 99.92 0.8639 0.852 (++0.165) 0.55 69.04 99.93 0.8328 0.879 (++0.249) 81.7 SalUn 83.12 99.92 0.8970 0.841 (++0.155) 0.41 73.68 99.88 0.8788 0.850 (++0.221) 50.1 Unmerge-PCA 67.12 99.10 0.9002 0.642 (−-0.044) 0.16 87.40 90.89 0.7804 0.519 (−-0.111) 0.50 Unmerge-CAV-KMeans 72.08 99.15 0.9417 0.677 (−-0.010) 0.11 98.64 99.09 0.8423 0.505 (−-0.125) 0.54 Coherent protocol, per group: the aggregate hides a concept-level edit. The forget half and the retained half of a Trees class are i.i.d. draws, so no activation subspace separates them. Tab. 21 splits the retain and test accuracies of Tab. 20 accordingly. Unmerge matches the retrained accuracy on the forget half by lowering the whole class: the retained half falls to the same level (6767–73%73\%, against ≥99.9%≥\!99.9\% under retraining) and the Trees test accuracy falls 1818–2626 points below the retrained model’s, while the other 9595 classes stay within 2.22.2 test points of retraining. The tuned NegGrad+ does the same. The aggregate retain accuracy moves by at most 1.31.3 points because the retained half is 2.6%2.6\% of ℛR, which is why ToW ranks these cells first. Random Label and SalUn are the mirror image: they keep the retained half (≥99%≥\!99\%) and most of the Trees test accuracy, and leave the forget half 99–1818 points above the retrained level while marking it (attack accuracy above retraining, Tab. 20). No method reproduces retraining on this protocol; Unmerge treats the request as a partial removal of the concept, which is the behavior its construction implies and the reason we scope it to concept-level requests (§4.5). Table 21: Coherent protocol, per-group accuracy (%) of the cells of Tab. 20. Forget half: the 12501250 forget images; retained half: the 12501250 Trees training images that stay in ℛR; other: the 47,50047,500 retain images of the other 9595 classes; Trees test / other test: the 500500 / 95009500 test images. Retraining keeps the retained half at ≥99.9%≥\!99.9\% while the forget half drops to test level. Unmerge and the tuned NegGrad+ (†) lower both halves alike and the Trees test accuracy with them; Random Label and SalUn keep the retained half and under-forget the forget half. init=0 W_init= W_0 init=PT W_init= W_PT Forget half Retained half Other retain Trees test Other test Forget half Retained half Other retain Trees test Other test RT W_RT 70.64 99.92 99.96 66.40 76.79 74.24 100.00 99.97 74.80 85.46 FT W_FT (un-edited) 99.92 100.00 99.96 73.80 77.69 99.92 100.00 99.97 74.00 85.27 NegGrad+† 71.60 71.12 99.35 44.20 73.61 74.96 75.44 99.70 47.20 82.51 Rand. Label 88.72 99.92 99.96 67.40 77.44 85.84 99.76 99.93 67.80 83.49 SalUn 87.44 99.36 99.95 68.40 77.63 83.12 99.04 99.94 69.20 84.19 Unmerge-PCA 72.08 72.40 99.85 45.80 76.65 67.12 67.12 99.94 48.40 84.53 Unmerge-CAV-KMeans 73.60 73.36 99.93 44.20 76.85 72.08 73.20 99.83 56.80 83.32 Random protocol: the lockstep ceiling. The retrained model itself barely forgets a random subset (forget accuracy 7575/85%85\%, close to its test accuracy), so the untouched FT W_FT already scores ToW 0.730.73/0.840.84, and a 60-configuration sweep confirms that the ceiling is not under-tuning: forget and retain accuracy move together at every α, and matching the retrained forget accuracy on PT W_PT costs 1313 points of retain. Random Label and SalUn reach ToW up to 0.960.96 only by marking the forget samples: attack accuracy on ℱF rises to 0.840.84–0.880.88, above the retrained model’s 0.630.63–0.680.68, and their forget features become separable from retain features (relative entanglement gap 1616–8282 against ≤0.7≤\!0.7 for Unmerge). The cells in Tab. 20 are the ToW maxima over genuine edits (no-op edits, whose forget accuracy stays at the un-edited level, excluded; ties within 0.0050.005 broken by entanglement, then signed MIA gap). Instance-level and concept-level forgetting are different problems. Triantafillou et al. (2026) distinguish untraining—reversing the effect of having trained on the given forget samples, whose reference is the model retrained on ∖ℱS —from unlearning the distribution or concept that the samples represent, and note that the two coincide when ℱF contains every training instance of the concept, which is our class-level protocol. Otherwise they pull in opposite directions: while the concept stays represented in ℛR, the untraining target still predicts the forget samples correctly and the right edit is close to none, whereas removing the concept calls for a large one; Cooper et al. (2025) describe the same mismatch between removing observed information and suppressing what a model has generalized. Our random protocol is the extreme case: its forget samples are i.i.d. draws from the training distribution, so the retrained forget accuracy matches test accuracy and the un-edited FT W_FT already scores ToW 0.730.73/0.840.84, in line with Zhao et al. (2024), who find that for forget sets that are not memorized the original model is already close to retraining. What remains is per-sample memorization, on which an edit built to generalize over what the forget samples share has nothing to act—which the separability score reports beforehand. Separability diagnostics. The separability score is S=fvar⋅(1−overlap)S=fvar·(1-overlap), the fraction of forget variance captured by ℱ Q_F times one minus the mean per-direction overlap of ℱ Q_F with ℛ Q_R; both factors are available after Phase 1, before any edit. Tab. 22 reports them. Within a dataset S separates the regimes cleanly (2.42.4–2.5×2.5× between class-level and random CIFAR-100 in both initializations, through different factors: 0 W_0 loses the private subspace through overlap, →0.830.61\!→\!0.83, and PT W_PT through captured variance, →0.280.63\!→\!0.28). Across datasets it does not: TinyImageNet class-level runs measure S≈0.07S≈ 0.07–0.100.10, the CIFAR-100 “hard” range, yet unlearn well, because from-scratch TinyImageNet features are less structured and S is basis-dependent. The ratio Stask/SrandS_task/S_rand against a same-size random subset of the same dataset removes both dependencies at the cost of one extra forward pass. Table 22: Phase-1 separability diagnostics over edited layers: fraction of forget variance captured by ℱ Q_F (fvar), mean per-direction overlap between ℱ Q_F and ℛ Q_R, and S=fvar⋅(1−overlap)S=fvar·(1-overlap). CIFAR-100: CAV-KMeans basis (k=16k=16); the coherent row uses the PCA basis of its winning cell. TinyImageNet: bases of the official 0 W_0 cells (the diagnostic was not logged for the PT W_PT TinyImageNet runs). Last column: S relative to the same-dataset random-subset value. Dataset Forget request fvar overlap S S/SrandS/S_rand CIFAR-100 (0 W_0) Trees, class-level 0.438 0.611 0.170 2.4 Trees, coherent half-classes – – 0.17 2.4 Random subset (25002500) 0.421 0.832 0.071 1 CIFAR-100 (PT W_PT) Trees, class-level 0.629 0.729 0.170 2.5 Random subset (25002500) 0.281 0.755 0.069 1 TinyImageNet (0 W_0) Dogs & Cats, CAV-KMeans 0.252 0.628 0.094 – Arthropods, CAV-Class 0.135–0.143 0.347–0.460 0.073–0.093 – Arthropods, CAV-KMeans 0.259 0.611 0.101 – Arthropods, PCA 0.526 0.864 0.072 – E.8 Decomposition Validation: Details Accuracy-based scores are proxies; the quantity the method acts on, ℛ=m−αℱ τ_R= τ_m-α τ_F, can be compared with ℛ⋆=RT−init τ_R = W_RT- W_init directly. For every checkpoint on Trees we compute, over all weight tensors with at least two dimensions (5353–5454 layers, not only the edited ones), the relative Frobenius error ‖ℛ−ℛ⋆‖F/‖ℛ⋆‖F\| τ_R- τ_R \|_F/\| τ_R \|_F averaged over layers and the size-weighted cosine between ℛ τ_R and ℛ⋆ τ_R (for baselines ℛ=UL−init τ_R= W_UL- W_init), and the predictive divergence DKL(pUL∥pRT)D_KL(p_UL\|p_RT) on the forget, retain, and test splits (Tab. 23). Two references anchor the weight-space columns: the un-edited FT W_FT, and the retrain-to-retrain noise floor obtained by retraining twice more from the same init W_init with different data orders. On 0 W_0 every checkpoint, including independent retrains, has cosine ≈0.005≈\!0.005 and relative error 1.51.5–1.71.7 to the reference: from-scratch runs converge to unrelated basins, so only the output-space columns are meaningful there. On PT W_PT the floor is 0.7180.718–0.7310.731 and FT W_FT sits at 0.7510.751; all Unmerge variants stay at the un-edited level (0.7500.750–0.7540.754: the minimal edit does no weight-space harm), whereas Random Label (0.8230.823) and SalUn (0.9080.908) move away from the retrained solution. Output space discriminates more sharply in the same direction: Unmerge-CAV-KMeans has the smallest forget-set divergence in both regimes (2.892.89 on 0 W_0 vs. 4.94.9–5.25.2 for the baselines; 5.295.29 vs. 5.95.9–6.66.6 on PT W_PT), because relabeling pushes forget inputs confidently to wrong classes—far from a model that never saw them—while subspace subtraction lands closer (cf. §4.4). Caveats. (a) CAV-Class at α=2.5α=2.5 on PT W_PT matches the retrained forget accuracy to within 0.120.12 points but has the largest forget-set divergence of any cell (7.157.15): an aggressive α over-shoots distributionally while matching accuracy, so α should be selected on a distributional criterion when one is available. (b) Unmerge-PCA on 0 W_0 has large divergence on every split (3.873.87 on retain despite 98.75%98.75\% retain accuracy): the argmax survives but calibration does not, consistent with its entanglement outlier in Tab. 3 and with the regime split of §4.2. Table 23: Decomposition validation on CIFAR-100 (Trees), ResNet-50. Weight space: layer-averaged relative Frobenius error and size-weighted cosine of ℛ τ_R vs. ℛ⋆ τ_R over all weight matrices. Output space: DKL(pUL∥pRT)D_KL(p_UL\|p_RT) per split (lower is closer to retraining). Floor: two independent retrains from the same init W_init. Best non-reference per KL column in bold; the 0 W_0 weight-space columns are uninformative (see text). init=PT W_init= W_PT init=0 W_init= W_0 Method rel. err cos KLF KLR KLtest_test rel. err cos KLF KLR KLtest_test Retrain floor (2 runs) 0.718–0.731 0.684–0.688 – – – 1.539–1.669 0.005 – – – FT W_FT (no unlearning) 0.751 0.691 – – – 1.579 0.006 – – – NegGrad+ 0.751 0.691 5.89 0.397 1.151 1.579 0.006 4.89 0.500 1.293 Rand. Label 0.823 0.682 6.62 0.014 0.704 1.515 0.006 5.20 0.011 0.755 SalUn 0.908 0.684 6.62 0.040 0.698 1.547 0.006 5.21 0.024 0.736 Unmerge-PCA 0.750 0.688 6.03 0.172 0.864 1.572 0.006 4.99 3.873 3.471 Unmerge-CAV-KMeans 0.754 0.683 5.29 0.107 0.825 1.577 0.006 2.89 0.043 0.717 Unmerge-CAV-Class 0.754 0.685 7.15 0.192 1.049 1.577 0.006 3.31 0.029 0.708 Appendix F Limitations and Future Work Scope. Unmerge is a training-free edit for concept-level forget requests, i.e. requests that occupy a separable activation subspace, such as whole classes and semantic groups (§4.5). On a coherent instance subset it lowers the forget and retained halves of a class alike (Tab. 21). Random instance subsets have none—they pose a per-sample de-memorization problem rather than a concept-removal one (Triantafillou et al., 2026) (App. E.7)—and no subspace edit can move their forget accuracy without moving retain; gradient-on-retain methods can, at 44–23×23× the cost and at the price of a membership signature on the forget set. The Phase-1 separability ratio flags the regime beforehand but needs a same-size random-subset control for calibration (App. E.7). Requirements. The method is optimization-time data-free, not data-free: it needs forward passes over ℱ∪ℛF , CAV-Class needs forget labels (App. B.3), and the PT W_PT regime needs the same pretraining checkpoint—substituting a different one is catastrophic, while init=0 W_init=0 is a safe fallback (App. D.4). Fidelity and solver. Unmerging does no weight-space harm and has the smallest forget-set divergence to RT W_RT among the methods of Tab. 23, but does not reconstruct ℛ⋆ τ_R in weight space beyond the un-edited model (§4.5). At class level a logit mask on the un-edited model already matches retraining on ToW and the MIA gap, and a linear probe on frozen features still recovers the forget classes from every approximate method we tested, ours included; the feature-entanglement axis separates Unmerge from such a mask on 0 W_0 and on TinyImageNet, but not on CIFAR-100/PT W_PT, where the un-edited features already sit near retraining. We therefore claim fidelity to retraining in outputs and feature distribution, not that class information becomes undecodable. The reported cells use an early-stopped Adam iterate rather than the exact minimizer, so their λ is nominal and the PCA variants depend on the step budget, which we fix throughout (App. D.3). Breadth and future work. The instance-level, task-vector-validation and seed-variance studies use one CIFAR-100 superclass, and the cost model is validated on ResNet-50 and ViT-S/16; at language-model scale we only have the preliminary study of App. E.4. The framework itself is general: the CAV construction accepts arbitrary positive/negative pools, so concept labels (toxicity, bias, copyrighted style) lift Unmerge to language and vision-language models without changing the objective, and the layerwise structure fits LoRA-merged and mixture-of-experts checkpoints; explicit solvers (App. D.3), continual forget requests, and adaptive per-layer rank and skip selection are further directions.