Paper deep dive

MPU: Towards Secure and Privacy-Preserving Knowledge Unlearning for Large Language Models

Tiantong Wang, Xinyu Yan, Tiantong Wu, Yurong Hao, Pengjun Xie, Wei Yang Bryan Lim

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 134

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/20/2026, 7:35:01 AM

Summary

The paper introduces MPU (Multiple Perturbed Copies Unlearning), a privacy-preserving framework for machine unlearning in large language models. It addresses the dual non-disclosure constraint where neither the server's parameters nor the client's forget set can be shared. MPU uses server-side Pre-Process (generating perturbed, reparameterized model copies) and Post-Process (aggregating updates via harmonic denoising) to allow clients to perform local unlearning without exposing sensitive data or model weights.

Entities (8)

MPU · framework · 98%Large Language Models · model-type · 95%Llama-3.2-1B-Instruct · model · 90%Post-Process · module · 90%Pre-Process · module · 90%Qwen2.5-1.5B-Instruct · model · 90%Harmonic Denoising · technique · 85%Reparameterization · technique · 85%

Relation Signals (9)

MPU → containsmodule → Pre-Process

confidence 95% · MPU... primarily introduces two server-side modules: Pre-Process... and Post-Process

MPU → containsmodule → Post-Process

confidence 95% · MPU... primarily introduces two server-side modules: Pre-Process... and Post-Process

MPU → targets → Large Language Models

confidence 95% · MPU: Towards Secure and Privacy-Preserving Knowledge Unlearning for Large Language Models

MPU → evaluatedon → Llama-3.2-1B-Instruct

confidence 90% · We conduct experiments using five representative base models: Llama-3.2-1B-Instruct

MPU → evaluatedon → Qwen2.5-1.5B-Instruct

confidence 90% · Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct

Post-Process → performsaction → Update Aggregation

confidence 90% · Post-Process for update aggregation

Pre-Process → performsaction → Perturbed Copy Generation

confidence 90% · Pre-Process for randomized copy generation

MPU → usestechnique → Harmonic Denoising

confidence 85% · aggregating updates with a harmonic denoising procedure

MPU → usestechnique → Reparameterization

confidence 85% · transformed by an invertible, data-independent, function-preserving reparameterization

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Machine unlearning for large language models often faces a privacy dilemma in which strict constraints prohibit sharing either the server's parameters or the client's forget set. To address this dual non-disclosure constraint, we propose MPU, an algorithm-agnostic privacy-preserving Multiple Perturbed Copies Unlearning framework that primarily introduces two server-side modules: Pre-Process for randomized copy generation and Post-Process for update aggregation. In Pre-Process, the server distributes multiple perturbed and reparameterized model instances, allowing the client to execute unlearning locally on its private forget set without accessing the server's exact original parameters. After local unlearning, the server performs Post-Process by inverting the reparameterization and aggregating updates with a harmonic denoising procedure to alleviate the impact of perturbation. Experiments with seven unlearning algorithms show that MPU achieves comparable unlearning performance to noise-free baselines, with most algorithms' average degradation well below 1% up to 10% noise, and can even outperform the noise-free baseline for some algorithms under 1% noise. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

133,090 characters extracted from source content.

Expand or collapse full text

MPU: Towards Secure and Privacy-Preserving Knowledge Unlearning for Large Language Models Tiantong Wang 1,2 Xinyu Yan 1,2 Tiantong Wu 1,2, ∗ Yurong Hao 1 Pengjun Xie 3 Wei Yang Bryan Lim 1 1 College of Computing and Data Science, Nanyang Technological University 2 Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) 3 Tongyi Lab, Alibaba Group Abstract Machine unlearning for large language models often faces a privacy dilemma in which strict constraints prohibit sharing either the server’s parameters or the client’s forget set. To address this dual non-disclosure constraint, we propose MPU, an algorithm-agnostic privacy-preservingMultiplePerturbed CopiesUnlearning framework that primarily introduces two server-side modules: Pre-Process for ran- domized copy generation and Post-Process for update aggregation. In Pre-Process, the server distributes multiple perturbed and reparameterized model instances, allowing the client to execute unlearning locally on its private forget set with- out accessing the server’s exact original parameters. After local unlearning, the server performs Post-Process by inverting the reparameterization and aggregating updates with a harmonic denoising procedure to alleviate the impact of pertur- bation. Experiments with seven unlearning algorithms show that MPU achieves comparable unlearning performance to noise-free baselines, with most algorithms’ average degradation well below 1% up to 10% noise, and can even outperform the noise-free baseline for some algorithms under 1% noise. Code is available at https://github.com/Tristan0318/MPU. 1 Introduction As large language models (LLMs) continue to advance, their tendency to memorize and reproduce training data has raised serious concerns about privacy, safety, and intellectual property. These concerns motivate an urgent need for machine unlearning [Cao and Yang, 2015], whose goal is to selectively remove undesired data, knowledge, or behaviors from a trained model while preserving its general utility for normal tasks. Unlearning is particularly challenging for modern LLMs, as full retraining is prohibitively expensive, and deletion requests may arrive continuously over the model’s lifecycle. As a result, a growing body of research has investigated approaches ranging from training-time strategies such as sharding or slicing [Bourtoule et al., 2021], post-hoc model editing and selective forgetting [Golatkar et al., 2020], influence-based approximations [Koh and Liang, 2017], and foundational formulations of deletion guarantees [Ginart et al., 2019]. However, many real-world deployments impose an additional constraint that is often overlooked: the data to be forgotten may belong confidentially to a client and must remain local, while the deployed model is proprietary to the server. This creates a central tension in server-client unlearning: the server requires an update that removes the effect of a client-local forget set, but (i) the client should not disclose raw data (or fine-grained sufficient statistics) to the server, and (i) the server may prefer not to reveal its exact current model parameters to the client. Consequently, this setting calls for a ∗ Corresponding author. Preprint. arXiv:2602.23798v2 [cs.LG] 14 May 2026 restricted server-client unlearning framework that enables effective forgetting without direct data sharing, and without exposing the server’s exact parameters. Existing unlearning approaches do not directly address this challenge. Training-time methods based on SISA-style sharding or slicing can reduce the deletion cost by retraining only affected subsets, but require maintaining specific training structures and retaining per-shard state to support subsequent retraining upon deletion requests [Bourtoule et al., 2021]. Post-hoc techniques such as selective forgetting can lower the cost of removing particular classes or examples, but typically assume that the entity performing unlearning has direct access to the model and relevant data distributions [Go- latkar et al., 2020]. Influence-function-based approximations offer a principled lens on example influence. However, they can be computationally demanding for LLMs and often rely on second- order information that is difficult to obtain robustly at scale [Koh and Liang, 2017]. In federated settings, the right to be forgotten has been explored through reconstructing an unlearned model using server-side training histories [Liu et al., 2020] or by coordinating efficient retraining while keeping data local [Liu et al., 2022b]. Recent work further highlights that federated unlearning methods can exhibit substantial trade-offs between effectiveness and efficiency across scenarios [Zhang et al., 2025]. Overall, many existing solutions rely on substantial server-side state or centralized access to training records, exposing the server’s exact current model to clients. In this paper, we propose MPU, a privacy-preservingMultiplePerturbed CopiesUnlearning frame- work tailored to server–client deployments. Our key idea is to let the server publish perturbed model instances to clients, with the perturbation designed to be self-canceling during server-side aggregation. Specifically, at each communication round, instead of broadcasting the exact model parameters, the server releasesm ≥ 2copies that are (i) perturbed by structured noise and (i) transformed by an invertible, data-independent, function-preserving reparameterization sampled from parameter symme- tries. Starting from each published copy, the client runs a local unlearning routine on its private forget set and returns copy-wise updates. The server then inverts the reparameterizations and aggregates the returned updates using harmonic weights, which cancel the first-order error term introduced by noise. As a result, MPU yields a server-side update that matches the noise-free unlearning step, while keeping forget set local and obscuring the server’s exact model parameters by communicating only perturbed, symmetry-transformed copies. The key contributions are: • Dual Non-Disclosure Unlearning Framework. We propose a server-client parameter- unlearning framework where the client keeps the forget set local (sharing neither raw data nor fine-grained sufficient statistics/distribution), while the server avoids disclosing its exact current parameters by communicating perturbed model copies. To our knowledge, this is the first solution to the dual non-disclosure setting without relying on auxiliary statistics, such as surrogate data. •Invertible and Secure Function-Preserving Reparameterizations. We generalize in- vertible, data-independent, function-preserving reparameterizations to modern Transformer architectures, including RoPE-style positional mechanisms, enabling symmetry-based repa- rameterizations for LLMs (e.g., Meta’s Llama family of models). We furthermore prove the security level in theory that breaking the reparameterization key is NP-hard. •Theoretical Guarantees for First-Order Noise Cancellation. We provide theoretical guarantees that, under our structured noise injection and harmonic aggregation, the first- order error induced by noise is eliminated after aggregation, resulting in a server update that is consistent with the noise-free unlearning step. 2 Related Work LLM Unlearning. LLMs acquire vast amounts of knowledge during pre-training, which may include sensitive or otherwise undesired information [Qiu et al., 2025]. Machine unlearning aims to enable models to “forget” specific pieces of knowledge while maintaining performance on remaining data. Recent work has therefore focused on selective unlearning, i.e., suppressing undesired outputs for a designated forget set. Text-based strategies include Gradient Ascent fine-tuning that maximizes cross-entropy loss on forget samples [Jang et al., 2023, Yao et al., 2024], preference-inspired objectives (e.g., NPO and SimNPO) that constrain updates with a reference model and length-normalized rewards [Zhang et al., 2024, Fan et al., 2024], and substitute-response training that learns safe alternative answers to forget queries [Maini et al., 2024, Mekala et al., 2025]. 2 Beyond text-level objectives, distribution-based approaches drive the model’s output distribution toward a target distribution aligned with unlearning goals [Liu et al., 2025, Wang et al., 2025]. Meanwhile, activation-based methods intervene on internal representations rather than only on outputs, for example, by perturbing hidden states for harmful inputs toward random or refusal-like directions [Shen et al., 2025]. Rather than weighting multiple losses, recent works such as NGDiff and MolLM [Jin et al., 2025, Pan et al., 2025] formulate the combination as a multi-task problem, normalizing gradients and computing common descent directions to better trade off forgetting target knowledge against retaining overall utility. However, many existing LLM unlearning algorithms are studied in settings where the unlearning party can access the model parameters and optimize using the forget set. In this work, we focus on a more constrained server-client interaction rule, in which the forget set remains client-local, and the server does not expose its exact parameters. Direct Model Merging. Our framework aggregates multi-copy updates computed from multiple models, which relates to direct model merging and the empirical linearity of parameter updates in the fine-tuning paradigm. In this paradigm, models adapted to different tasks from a shared pretrained checkpoint often exhibit cross-task linearity, whereby their weights and feature spaces can be combined through linear interpolation, enabling direct model merging without additional retraining and substantially reducing computational overhead [Zhou et al., 2024b]. A widely used strategy is weight averaging, where parameters of similarly initialized fine-tuned models are averaged. Model Soups [Wortsman et al., 2022] further demonstrated that such averaging can improve accuracy and out-of-distribution robustness compared to individual models. Alternatively, task arithmetic [Ilharco et al., 2022] operates on task vectors to compose or edit specific model capabilities. Recently, Zhou et al. [2024a] extended task arithmetic to LLMs, formulating it as an optimization problem that exploits local linearity and near-orthogonality of task updates. Wang et al. [2026] quantified the merging capability of multiple models by output discrepancy, and theoretically proved the plausibility of merging task-vectors given the same pre-trained model. The above works motivate our design of aggregating different perturbed copies by weighted merging, to obtain efficient privacy- aware unlearning. 3 Proposed Method: MPU Forget Set DA4344 Fine- Tuned Model Noised Models Noise Generate Repara- meterize Local Unlearn Perturbed Copies Generation Step1: Pre-ProcessStep2: Unlearning Server-SideClient-Side Reparameterized Noised Models Aggregate & Update Server-Side Step3: Post-Process Unlearned Model Denoise Reparam Invert Aggregate Unlearned Noised Models Inverted Noised Models Figure 1: Overview of the proposed MPU framework across communication rounds. The server generates perturbed, reparameterized model copies fromθ r−1 , clients unlearn onD f , and the server inverts the reparameterization and aggregates updates to obtain θ r . 3.1 Algorithm Overview We consider a server-client unlearning framework operating overRcommunication rounds. Each round consists of three sequential stages. First, the server generates and distributesmperturbed 3 copies of the current global model to the client. Second, the client performs local unlearning on these perturbed models using its private dataset. Third, the server collects the resulting local unlearning updates and aggregates them via a harmonic denoising mechanism, producing the updated global model for the next round. Remark 1. In Algorithm 1, only the results of the following two operations are public θ (k,r) pub ← T k,r θ r−1 + ε (r) k (1) ∆ (k,r) ← UNLEARN θ (k,r) pub ,D f (2) and all other operations are kept private to the server. 3.2 Pre-Process: Perturbed Copies Generation Algorithm 1 MPU Require: θ 0 ,R,L,m ≥ 2,σ ℓ L ℓ=1 ,α k > 0 m k=1 ,η srv , s r R r=1 ,t r R r=1 1: for r ∈ [R] do 2:for all k ∈ [m] and ℓ∈ [L] do 3:Generateε 0,(r) k,ℓ using Eq. (4) 4:end for 5:for k ∈ [m] do 6: ε (r) k,ℓ ← α k ε 0,(r) k,ℓ 7: ε (r) k := stack ℓ ε (r) k,ℓ 8:end for 9: S 0 ← 0; S 1 ← 0∈ R d 10:for k ∈ [m] do 11: T k,r ← SAMPLEREPARAM(t r ,k) 12:Client Receives: θ (k,r) pub ← T k,r θ r−1 + ε (r) k 13:Client Updates: ∆ (k,r) ← UNLEARN θ (k,r) pub ,D f 14: b ∆ (k,r) ← T −1 k,r ∆ (k,r) 15: S 0 ← S 0 + α −1 k ; S 1 ← S 1 + α −1 k b ∆ (k,r) 16:end for 17: ̄ ∆ (r) ← S 1 /S 0 18: θ r ← θ r−1 + η srv ̄ ∆ (r) 19: end for 20: Return θ R In each round, the server perturbs the current model before transmitting it to the client. The generation of per- turbed copies consists of two com- ponents: (i) structured noise injec- tion and (i) an invertible, function- preserving reparameterization. Noise injection mitigates privacy leakage risks during inference. Meanwhile, the reparameterization shifts the orig- inal parameter space, preventing the client from reconstructing the original parameters even when multiple per- turbed copies are accessed. 3.2.1 Privacy Guarantee: Noise Generation Noise injection is a common and effec- tive defense against inference attacks. In MPU, noise is generated indepen- dently per blockℓwith scaleσ ℓ . We setσ ℓ based on a reference task vec- torv ℓ , defined as the difference be- tween the current model parameters and a public reference model param- eter. Typically, the public reference model refers to a released pretrained model before fine-tuning. This design is motivated by the fact that the task vector captures the parameter update induced by fine-tuning, which encodes sensitive information about the underlying private data, which requires protection. We define σ ℓ as: σ ℓ = κ· RMS(v ℓ ),RMS(v ℓ ) := v u u t 1 d ℓ d ℓ X t=1 v ℓ (t) 2 ,(3) where κ > 0 is a controllable noise-level hyperparameter. For each roundr ∈ [R]and blockℓ, the server draws i.i.d. Gaussian vectorsz (r) k,ℓ ∼N 0,σ 2 ℓ I d ℓ for k ∈ [m]from seedss r andℓ, subtracts their mean, and rescales the resulting vectors so that each perturbed copy maintains the prescribed marginal variance. The resulting base noise vectors satisfy a block-wise zero-sum constraint: ε 0,(r) k,ℓ = r m m− 1 z (r) k,ℓ − ̄z (r) ℓ ⇒ m X k=1 ε 0,(r) k,ℓ ≡ 0,(4) 4 where ̄z (r) ℓ = 1 m P m k=1 z (r) k,ℓ . Intuitively, the zero-sum structure forces themcopy noises to lie in an(m− 1)-dimensional subspace, which is the key algebraic property enabling noise cancellation during the server’s aggregation. We then apply a per-copy positive scaling α k > 0: ε (r) k,ℓ = α k ε 0,(r) k,ℓ , ε (r) k := stack ℓ ε (r) k,ℓ ∈ R d .(5) where the stack operator combines the layer-level noisesε (r) k,ℓ together to a model-level noiseε (r) k . Theα k ’s introduce heterogeneous noise magnitudes across copies, work as a secondary protection, avoiding parameter reconstruction even if reparameterization information is leaked. Finally, we add the noiseε (r) k to the original model parameters to obtainmfirst-stage perturbed models prior to reparameterization. Formal Privacy Guarantee by Differential Privacy In standard noise-injection formal privacy mechanisms such as Differential Privacy (DP) [Dwork et al., 2006b], one typically specifies a privacy budget first (e.g.,(ε,δ)), and then computes the noise level according to a chosen DP mechanism and its accounting method. In practice, the resulting noise scale can vary across different DP mechanisms, and is influenced by hyperparameters used in DP mechanisms (i.e., sensitivity, clipping bound in Gaussian Mechanism [Dwork et al., 2006a], search range in Renyi-DP Mechanism [Mironov, 2017]). For consistency and avoiding additional hyperparameters’ influence, we therefore use a fixed noise level directly in MPU. See Appendix A.2 for a detailed discussion on possible DP combinations to achieve a formal privacy guarantee. 3.2.2 Security Guarantee: Reparameterization To prevent client-side reconstruction of the original model from multiple copies when scalarsα k m k=1 are leaked, since a small number of scalar keys alone provides insufficient security, the server applies a reparameterization to each block prior to releasing the perturbed models. We extend previous work on neural network functional invariance [K ̊urková and Kainen, 1994] to derive functionally invariant reparameterization on modern Transformer model architectures. LetΘdenote the parameter tuple of an attention or feed-forward network (FFN) block, and let f Θ : R d model → R d model denote the forward function induced byΘ. We consider a family of reparameterizations acting on the parameter space, defined as invertible mappingsT : Θ7→ T (Θ) equipped with explicit inverses T −1 . For each copy k and round r, the server samples a distinct data-independent reparameterization T k,r ← SAMPLEREPARAM(t r ,k),(6) where T k,r is dependent on solely seed t r and k, which is function-preserving, i.e., f T k,r (Θ) (x) = f Θ (x), ∀x∈ R d model .(7) Equivalently, lettingGdenote the set of all such invertible function-preserving maps,Gforms a parameter-symmetry group under composition. EachT k,r ∈Gacts onΘwhile leaving the realized function invariant.(See Eq. 51, 64 in Appendix for the detailed operation rule of T ) Feed-Forward Network Reparameterization For feed-forward network reparameterization, we adopt the conventional MLP channel permutation reparameterization. We give the formal reparame- terization steps in Appendix A.4.3 Attention ReparameterizationMore generally, Attention blocks admit a rich parameter-symmetry group that includes continuous attention symmetries that leave the attention computation invariant. We derive detailed reparameterization constructions and invariance proofs for the full Transformer architecture in Appendix A.4. Specifically, for RoPE-based [Su et al., 2024] models such as Llama-3 [Grattafiori et al., 2024] series models, we restrict attention transformations to those that commute with the RoPE operators, ensuring that the reparameterization preserves functional equivalence for both RoPE and non-RoPE network structures. Concrete RoPE-aware reparameterization is presented in Appendix. A.4.2. 5 Optimization Trajectory InvarianceThe reparameterization is not only function-preserving, but also induces an invariant optimization (learning) trajectory. Specifically, supposeAis a deterministic learning algorithm (e.g., stochastic gradient descent with fixed mini-batch ordering). Then the learning dynamics satisfy Eq. 8: A(θ) = T −1 (A(T (θ))),(8) indicating that optimization commutes with the reparameterization, so the local unlearning process will not be affected by reparameterization. Meanwhile, we also provide the detailed analysis of trajectory invariance in Appendix A.5. Theoretical Security Guarantee of Reparameterization We formally show that the alignment task for breaking the reparameterization key is mathematically equivalent to the Multi-Dimensional Assignment Problem (MDAP) for discrete Feed-Forward Network permutations, and the Generalized Orthogonal Procrustes Problem (GOPP) for continuous Attention symmetries. Both alignment problems are computationally prohibitive: they requireO(d 3 )operations form = 2and are strictly NP-hard form≥ 3. Consequently, this spatial misalignment mathematically prevents the client from discovering cross-copy correlations, effectively blocking reconstruction of the server’s parameters. We provide formal theoretical analysis in Appendix A.6. 3.3 Client-Side Local Unlearning Upon receiving the reparameterized model parameters, the client performs local unlearning using its private data, employing standard unlearning algorithms such as GradAscent, NPO, and DPO. Notably, MPU is algorithm-agnostic and can be integrated with any parameter-based unlearning methods. 3.4 Performance Guarantee: Update Aggregation After the client returns the local unlearning update∆ (k,r) , the server maps it back to the original parameter coordinates using the explicit inverse reparameterization: b ∆ (k,r) ← T −1 k,r ∆ (k,r) ,(9) so that all copy-wise updates are expressed in a common parameterization prior to aggregation. Note thatT −1 k,r is efficiently computable by transpose due to the property of orthogonal reparameterization matrices. After inversion, the server aggregates themreturned updates using harmonic aggregation to cancel the first-order noise: ̄ ∆ (r) = P m k=1 α −1 k b ∆ (k,r) P m k=1 α −1 k .(10) Here, we briefly explain why the aggregation cancels the first-order noise. Let∆ ⋆ (θ)denote the ideal (noise-free) unlearning displacement, and letJbe the Jacobian of∆ ⋆ evaluated atθ r−1 . Under a local linearization assumption, the inverted update first-order approximately satisfies b ∆ (k,r) ≈ ∆ ⋆ (θ r−1 ) + J ε (r) k = ∆ ⋆ (θ r−1 ) + J α k ε 0,(r) k ,(11) whereε 0,(r) k is the stacked noise before scaling. Substituting this expression into the harmonic average, the injected term becomes P m k=1 α −1 k J α k ε 0,(r) k P m k=1 α −1 k = J P m k=1 ε 0,(r) k P m k=1 α −1 k = 0,(12) where the last equality follows from the block-wise zero-sum property in Eq. 4. Therefore, the aggregation eliminates the correlated first-order noise error, while requiring only the scalar coefficients α k rather than storing the full noise parameters. A more detailed error analysis for higher orders is provided in Appendix A.3. At the end of each round, the server applies the aggregated update with step size η srv : θ r ← θ r−1 + η srv ̄ ∆ (r) .(13) 6 3.5 Memory-Efficiency Although Algorithm 1 conceptually adoptsmpublished copies per round, the server and client do not need to store all m perturbed models (nor the m returned updates) in memory. This observation follows from the fact that the harmonic aggregation coefficient depends solely onα k ’s. Consequently, the sufficient statistics for each round reduce to the two accumulators: S 0 = m X k=1 α −1 k , S 1 = m X k=1 α −1 k b ∆ (k,r) ∈ R d ,(14) after which the update is given by ̄ ∆ (r) = S 1 /S 0 . Concretely, the server can implement each round in a streaming manner. Fork = 1,...,m, the server publishes only a single perturbed copyθ (k,r) pub = T k,r (θ r−1 + ε (r) k ), receives the corresponding client update ∆ (k,r) , inverts it as b ∆ (k,r) = T −1 k,r (∆ (k,r) ), and updates the accumulators: S 0 ← S 0 + α −1 k , S 1 ← S 1 + α −1 k b ∆ (k,r) .(15) At no point does the server need to store allmmodels or allmupdates. The peak server-side memory footprint (beyond the base parametersθ r−1 ) is therefore dominated by a singled-dimensional accumulatorS 1 and the currently processed copy, yielding anO(d)memory requirement rather than O(md). The same streaming procedure applies on the client side, where the client processes one published copy at a time, avoiding the need to store m models simultaneously. 3.6 Computational Overhead We provide the derivation of computational overhead in Appendix A.7. The arithmetic intensity is A = 6N + 4d h N attn 9Nb (16) which is only 4% for a modern A100 GPU’s arithmetic intensity ridge point. The server latency is memory-bound as T server = 9mNb C mem (17) which takes around 0.13 seconds for an A100 GPU. Thus, the computational overhead is negligible. 4 Experiments 4.1 Experimental Setup To evaluate the effectiveness of the proposed MPU framework when coupled with diverse unlearning algorithms, we design and conduct a comprehensive set of experiments. 4.1.1 Models and Benchmark We conduct experiments using five representative base models: Llama-3.2-1B-Instruct, Llama- 3.2-3B-Instruct, Llama-3.1-8B-Instruct [Grattafiori et al., 2024], Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct [Yang et al., 2024]. Models are evaluated on the widely adopted TOFU [Maini et al., 2024], and MUSE [Shi et al., 2024] benchmarks, following established experimental setups in prior works [Wang et al., 2024, Dorna et al., 2025], using full-finetuning by default and LoRA [Hu et al., 2022] for Llama-3.1-8B model. The evaluation metrics are detailed in Appendix B.2. 4.1.2 Unlearning Algorithms To contextualize our results, we benchmark MPU against representative unlearning objectives spanning distinct algorithmic paradigms, organized by the primary mechanism used to suppress information associated with the forget set (Appendix B.3): (i) Loss-Reversal, First-Order Unlearning: GradAscent [Jang et al., 2023] and GradDiff [Liu et al., 2022a]. (i) Preference-Style, Bounded Objectives: DPO [Rafailov et al., 2023], NPO [Zhang et al., 2024], and SimNPO [Fan et al., 2024]. (i) Distribution Shaping via Self-Distillation: UnDIAL [Dong et al., 2025]. (iv) Loss Reweighting for Targeted Forgetting: SatImp [Yang et al., 2025]. 7 Table 1: Performance comparison of different unlearning algorithms using the Llama-3.2-1B model on the TOFU benchmark (Split99). Results are reported under three settings: CLEAN, a noise-free baseline; NOISED, a single-copy noise baseline with the same noise magnitude but without denoising; and MPU, usingm=2copies with noise levelκ=0.01. Higher values indicate better performance for Forget Quality, Forget Truth Ratio, and Model Utility, while values ofPrivLeakcloser to0are preferred. Unlearning Algorithms Forget Quality↑Forget Truth Ratio↑Model Utility↑PrivLeak CLEANNOISEDMPUCLEANNOISEDMPUCLEANNOISEDMPUCLEANNOISEDMPU GRADASCENT [ACL 2023]6.58e-52.81e-85.41e-20.3550.2460.4680.0000.0002.31e-465.858.969.6 GRADDIFF [PMLR 2022]0.4050.2660.4050.5350.5330.5470.4610.4610.46477.173.377.2 DPO [NEURIPS 2023]0.1650.1650.2660.6370.6200.6410.5910.5950.591-25.5-19.8-28.9 NPO [COLM 2024]0.9190.7660.9190.6240.6400.6280.5990.6000.59730.632.928.2 SIMNPO [NEURIPS 2025]5.41e-25.41e-29.71e-20.5260.5220.5250.5980.5920.598-68.4-70.2-71.8 UNDIAL [NAACL 2025]1.43e-21.43e-21.43e-20.5300.5270.5290.6130.6140.615-76.4-77.4-78.0 SATIMP [ICML 2025]3.02e-36.76e-36.76e-30.4740.4700.4760.6000.5970.601-98.9-99.1-98.9 4.1.3 Baselines Since MPU is designed to perform denoising after noisy perturbation, we compare it against two baselines. (i) CLEAN: a noise-free, centralized single-copy unlearning setting, corresponding to the standard unlearning framework with full access to the data and model and without any noise injection. The CLEAN baseline serves as an approximate upper bound on unlearning performance. (i) NOISED (Appendix A.8): a single-copy baseline in which noise is directly injected before model publication, and the server adopts the client’s returned update without any denoising. NOISED provides a lower-bound reference for isolating the effect of denoising. 4.2 Experimental Results For the complete experiment results and analysis, please refer to Appendix B.5 and B.6. We provide a brief analysis of the comparison of the Llama-3.2-1B model on the TOFU benchmark here. Privacy Privacy measures whether sensitive data in the forget set can still be retrieved from the model. As shown in Table 1, we find that MPU under a low noise level consistently outperforms single- copy noised/noise-free unlearning in Forget Quality (FQ). Specifically, for unlearning algorithms with high FQ (GradDiff, NPO), MPU matches the FQ of noise-free unlearning while substantially outperforming single-copy noisy unlearning, with0.405vs.0.266for GradDiff, and0.919vs.0.766 for NPO. For other low-FQ unlearning algorithms (GradAscent, SimNPO, and DPO), we observe that the single-copy noisy and noise-free frameworks yield similar FQ scores, whereas MPU improves over both baselines significantly:0.054vs. near zero for GradAscent,0.097vs.0.054for SimNPO, and0.266vs.0.165for DPO. The increase in FQ relative to noise-free unlearning can be attributed to the multi-copy stability effect. For more details, please refer to Appendix A.9. For Forget Truth Ratio (FTR) and PrivLeak (PL), we do not observe significant differences across the compared settings. This suggests that, under these two privacy-related metrics, all unlearning algorithms except GradAscent exhibit stable behavior. Comparing across unlearning algorithms, NPO performs best overall, achieving the highest average FQ together with the smallest absolute value of PL. Conversely, GradAscent exhibits almost zero FQ and MU under both the noise-free and noised unlearning settings, indicating a complete breakdown of the model. UtilityUtility measures model performance on general tasks, reflecting whether general capability is preserved after unlearning. Table 1 shows that single-copy noise, noise-free unlearning, and MPU achieve very similar MU under each unlearning algorithm, with variations below0.01. This indicates that utility preservation is not particularly sensitive to noise injection in our setup. Memorization Memorization measures the extent to which the model retains information from the training data. From Figure 2, the Forget QA Probability of MPU is higher than the single- copy noise-free/no-denoise frameworks, except for the SimNPO algorithm (while only0.002lower compared with the noise-free one). The Forget QA ROUGE results show that MPU and the noise- free framework attain similar scores, while both outperform the no-denoise framework. Moreover, MPU achieves the best ROUGE for the unlearning algorithms like GradAscent, DPO, and SatImp, whereas the noise-free framework yields the best ROUGE for NPO, SimNPO, and UnDIAL. Overall, 8 Figure 2: Performance comparison of different unlearning algorithms using the Llama-3.2-1B model. Results are reported under three settings: Clean, a noise-free baseline; Noised, a single-copy noise baseline with the same noise magnitude but without denoising; and MPU, usingm=2copies with noise levelκ=0.01. Higher values indicate better performance for Forget QA Probability and ROUGE. these memorization results suggest that direct noise injection can introduce undesirable memorization artifacts, while MPU mitigates this effect via harmonic denoising aggregation. Beyond the low-noise comparison in Table 1, we further evaluate MPU with extensive experiments on the number of published noisy copiesm, which measures the computational overhead, and noise levelκ. Overall, MPU is robust under moderate choices ofm, andκ(Appendix B.6 and Table 4, 6), indicating that MPU is tolerable for large noise levels, and few copies is the best choice for resource consideration. 5 Conclusion We propose MPU, a server-client privacy-preserving unlearning framework under a dual non- disclosure constraint: the server does not reveal its exact model parameters, and the client does not share its data. MPU is the first, to our knowledge, to enable unlearning under this strict setting without additional assumptions (e.g., distributional constraints or surrogate models/data), while achieving performance comparable to, and sometimes better than, no-privacy noise-free unlearn- ing. We provide theoretical guarantees that harmonic aggregation eliminates first-order noise error, multi-copy unlearning improves stability, and the reparameterization preserves both functionality and optimization trajectories, while ensuring the breaking of the security key is NP-hard. Empirically, MPU consistently outperforms privacy baselines and can even surpass the noise-free baseline due to multi-copy stability. Although local computation grows linearly with the copy number, two copies suffice in all our experiments. Future work may design improved communication protocols and further reduce computational cost. 9 References Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pages 141–159. IEEE, 2021. Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder–decoder approaches. In Dekai Wu, Marine Carpuat, Xavier Carreras, and Eva Maria Vecchi, editors, Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-4012. URL https://aclanthology.org/W14-4012/. Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli ́ c. Undial: Self-distillation with adjusted logits for robust unlearning in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8827–8840, 2025. Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C Lipton, J Zico Kolter, and Pratyush Maini. OpenUnlearning: Accelerating LLM unlearning via unified benchmarking of methods and metrics. arXiv preprint arXiv:2506.12618, 2025. Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Annual international conference on the theory and applications of cryptographic techniques, pages 486–503. Springer, 2006a. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer, 2006b. Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163, 2024. Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9304–9312, 2020. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408, 2023. Xiaomeng Jin, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Mingyi Hong. Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 11278–11294, 2025. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017. V ˇ era K ̊urková and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6 (3):543–558, 1994. 10 Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR, 2022a. Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu. Federated unlearning. arXiv preprint arXiv:2012.13891, 2020. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pages 1–14, 2025. Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. The right to be forgotten in federated learning: An efficient realization with rapid retraining. In IEEE INFOCOM 2022-IEEE conference on computer communications, pages 1749–1758. IEEE, 2022b. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121, 2024. Anmol Reddy Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid A Hasan, and Elita A Lobo. Alternate preference optimization for unlearning factual knowledge in large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3732–3752, 2025. Ilya Mironov. Rényi differential privacy. In IEEE Computer Security Foundations Symposium (CSF), 2017. Zibin Pan, Shuwen Zhang, Yuesheng Zheng, Chi Li, Yuheng Cheng, and Junhua Zhao. Multi-objective large language model unlearning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. Ruichen Qiu, Jiajun Tan, Jiayue Pu, Honglin Wang, Xiao-Shan Gao, and Fei Sun. A survey on unlearning in large language models. arXiv preprint arXiv:2510.25117, 2025. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. William F Shen, Xinchi Qiu, Meghdad Kurmanji, Alex Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, and Nicholas D Lane. Lunar: Llm unlearning via neural activation redirection. arXiv preprint arXiv:2502.07218, 2025. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Balancing forget quality and model utility: A reverse kl-divergence knowledge distillation approach for better unlearning in llms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1306–1321, 2025. Tiantong Wang, Yiyang Duan, Haoyu Chen, Tiantong Wu, and Wei Yang Bryan Lim. M-loss: Quantifying model merging compatibility with limited unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 26471–26479, 2026. Yaxuan Wang, Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Parag Shah, Yujia Bao, Yang Liu, and Wei Wei. LLM unlearning via loss adjustment with only forget data. arXiv preprint arXiv:2410.11143, 2024. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965–23998. PMLR, 2022. An Yang, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, F Huang, H Wei, et al. Qwen2. 5 technical report. arXiv preprint, 2024. 11 Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. Exploring criteria of loss reweighting to enhance llm unlearning. arXiv preprint arXiv:2505.11953, 2025. Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 37:105425–105475, 2024. Fuyao Zhang, Xinyu Yan, Tiantong Wu, Wenjie Li, Tianxiang Chen, Yang Cao, Ran Yan, Longtao Huang, Wei Yang Bryan Lim, and Qiang Yang. Oblivionis: A lightweight learning and unlearning framework for federated large language models. arXiv preprint arXiv:2508.08875, 2025. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024. Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. Metagpt: Merging large language models using model exclusive task arithmetic. arXiv preprint arXiv:2406.11385, 2024a. Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence of cross-task linearity in the pretraining-finetuning paradigm. arXiv preprint arXiv:2402.03660, 2024b. 12 Appendix Overview This appendix provides supplementary materials for MPU that expand the main paper along two axes: (i) mathematical derivations and analysis supporting the core design (noise construction, denoising aggregation, and function-preserving reparameterizations), and (i) experimental details, evaluation, additional results, and ablations. To keep the appendix navigable, we organize it into two parts. Part A: Mathematical Details and Analysis This part provides the mathematical foundations behind MPU, including the structured zero-sum noise, harmonic denoising aggregation, Transformer reparameterization symmetries, and a formal comparison against a naive single-copy “noise-only” baseline. • Appendix A.1: Structured zero-sum noise construction and statistics, including marginal variance, cross-covariance, and the stacked covariance form. • Appendix A.2: How the DP mechanism can be integrated in MPU to ensure a formal privacy guarantee. • Appendix A.3: Harmonic denoising aggregation, including first-order exact cancellation, second-order remainder bounds (Appendix A.3.2), and the uniqueness/optimality of har- monic weights (Appendix A.3.3). •Appendix A.4: Function-preserving reparameterizations for Transformers, including at- tention head basis transforms (Appendix A.4.1), RoPE-aware commutant restrictions (Appendix A.4.2), Feed-Forward Neural Network hidden-channel permutations (Ap- pendix A.4.3), and linearity of the reparameterization map (Appendix A.4.4). •Appendix A.5: Discussion of whether reparameterization affects client-side optimization trajectories and loss smoothness (equivariance and Euclidean smoothness invariance). •Appendix A.6: Discuss the hardness level of breaking the reparameterization. Provide a theoretical security guarantee. •Appendix A.7: Discuss the overhead of MPU. Showing that MPU only adds a negligible computation burden. • Appendix A.8: Comparison of update error between MPU and single-copy noisy unlearning, including one-round bias and variance analysis (Appendix A.8.1), multi-round implications (Appendix A.8.2), and an SNR interpretation (Appendix A.8.3). • Appendix A.9: Method discussion on stability enhancement via multi-copy learning. Part B: Implementation Details and Supplementary ExperimentsThis part provides benchmark descriptions, evaluation metrics, unlearning algorithm formulations, implementation details, and extensive supplementary experiments and ablations. • Appendix B.1: Benchmark description. • Appendix B.2: Evaluation metrics, including memorization (Appendix B.2.1), privacy (Appendix B.2.2), and utility (Appendix B.2.3). • Appendix B.3: Unlearning algorithms instantiated within MPU. •Appendix B.4: Implementation details, including testbed configuration (Appendix B.4.1), default hyperparameters (Appendix B.4.2), and the prompt template used for training and evaluation (Appendix B.4.3). • Appendix B.5: Additional experimental results. •Appendix B.6: Detailed supplementary analysis and ablations, including the effect of copy numberm(Appendix B.6.2), noise levelκ(Appendix B.6.3), round–epoch allocation (Appendix B.6.4), the no-denoise ablation (Appendix B.6.5), robustness to larger forget splits (Appendix B.6.6), and scaling across model sizes (Appendix B.6.7). 13 A Mathematical Details and Analysis This appendix provides the mathematical details supporting the method described in Section 3, including: (i) the structured zero-sum noise construction (Section 3.2.1), (i) the harmonic denoising aggregation and its associated error terms (Section 3.4), and (i) the function-preserving reparameter- ization family used to obfuscate the original parameter space (Section 3.2.2). We additionally present a comparison with a naive single-copy “noise-only” baseline, which injects noise but performs no denoising. A.1 Structured Zero-Sum Noise Construction and Statistics Fix a communication roundrand a parameter block (e.g., a layer)ℓwith dimensiond ℓ . Recall the construction in Eq.(4): draw i.i.d.z (r) k,ℓ ∼N (0,σ 2 ℓ I d ℓ )fork ∈ [m], define ̄z (r) ℓ = 1 m P m k=1 z (r) k,ℓ , and set ε 0,(r) k,ℓ = r m m− 1 z (r) k,ℓ − ̄z (r) ℓ , ε (r) k,ℓ = α k ε 0,(r) k,ℓ .(18) By construction, P m k=1 ε 0,(r) k,ℓ ≡ 0 for every blockℓ, and hence P k ε 0,(r) k ≡ 0 for the stacked vector. We define ε 0,(r) k = stack ℓ (ε 0,(r) k,ℓ ),(19) where the stack operation means to stack the block-wise noises together to obtain a model-level noise. Marginal Distribution and Cross-Covariance of Zero-Sum Base Letz k := z (r) k,ℓ ∈ R d ℓ and ̄z = 1 m P m k=1 z k . Since z k ∼N (0,σ 2 ℓ I d ℓ ) are i.i.d., we have Var(z k − ̄z) = 1− 1 m σ 2 ℓ I d ℓ .(20) The scaling factor q m m−1 therefore restores the desired marginal variance: Var ε 0,(r) k,ℓ = m m− 1 Var(z k − ̄z) = σ 2 ℓ I d ℓ ⇒ ε 0,(r) k,ℓ ∼N (0,σ 2 ℓ I d ℓ ) marginally.(21) For k ̸= j, using Cov(z k , ̄z) = 1 m σ 2 ℓ I d ℓ and Var( ̄z) = 1 m σ 2 ℓ I d ℓ , we obtain Cov(z k − ̄z, z j − ̄z) =− 1 m σ 2 ℓ I d ℓ ,(22) and hence, after scaling, Cov ε 0,(r) k,ℓ ,ε 0,(r) j,ℓ =− 1 m− 1 σ 2 ℓ I d ℓ , k ̸= j.(23) After applying the per-copy scaling, this becomes Cov ε (r) k,ℓ ,ε (r) j,ℓ =− α k α j m− 1 σ 2 ℓ I d ℓ , k ̸= j.(24) Scaled-Copy Covariance (Matrix Form)Letε ℓ := [ε (r) 1,ℓ ;... ;ε (r) m,ℓ ]∈ R md ℓ be the stacked noise vector for block ℓ. Then Cov ε ℓ = σ 2 ℓ (DBD)⊗ I d ℓ ,(25) where D = diag(α 1 ,...,α m ), B = m m− 1 I m − 1 m− 1 11 ⊤ .(26) SinceBhas eigenvalues m m−1 (multiplicitym− 1) and0(multiplicity1),Cov[ε ℓ ]has rank(m− 1)d ℓ . This expresses the core property used by harmonic denoising: themcopy noises live in an(m− 1)- dimensional subspace (per block), and the “missing” direction is precisely the all-ones direction that harmonic aggregation cancels. 14 A.2 Possible Combination of MPU and Differential Privacy While the noise injection serves as an effective defense, certain deployments may require formal, theoretical privacy guarantees. In this section, we provide the preliminaries of Differential Privacy (DP) and detail how MPU’s noise injection mechanism can be naturally combined with DP to offer rigorous mathematical privacy guarantees for the server’s parameters. A.2.1 Preliminaries: Differential Privacy DP is a rigorous mathematical framework for privacy-preserving data analysis that quantifies and strictly bounds the privacy risks associated with algorithmic outputs. Formally, a randomized algorithmMsatisfies(ε,δ)-DP if, for all adjacent datasetsDandD ′ (which differ by exactly one data record) and for all sets of possible outputsS ⊆ Range(M), the following inequality holds: P[M(D)∈S]≤ e ε P[M(D ′ )∈S] + δ(27) whereε > 0is the privacy budget andδ ∈ [0, 1)is a cryptographically small probability of a strict privacy breach. A smaller ε corresponds to a stronger privacy guarantee. DP provides formal privacy guarantee DP ensures that the probability distribution of the algorithm’s output remains almost indistinguishable whether any single individual’s data is included in the dataset or not. This provides plausible deniability to all individuals in the training data. Because the guarantee formally bounds the worst- case likelihood ratio of the outputs, an adversary cannot reliably infer the presence, absence, or specific content of any individual training sample. This holds true regardless of how much arbitrary side information or computational power the adversary possesses, thereby provably preventing attacks such as membership inference and data reconstruction. The Gaussian Mechanism and Post-Processing To achieve(ε,δ)-DP for a deterministic vector-valued functionf (D) ∈ R d (e.g., releasing model parameters), the standard approach is the Gaussian mechanism. This involves adding independent noise drawn fromN (0,σ 2 DP ) to each coordinate. The required noise scaleσ DP is determined by the L 2 -sensitivity of the function, defined as∆f = max D,D ′ ∥f (D)− f (D ′ )∥ 2 . Specifically, returning f (D) +N (0,σ 2 DP I) guarantees (ε,δ)-DP if: σ DP ≥ ∆f p 2 ln(1.25/δ) ε (28) A crucial property of DP is its immunity to post-processing: if a randomized outputM(D)satisfies DP, then for any arbitrary data-independent functiong, the compositiong(M(D))also satisfies the exact same DP guarantees. A.2.2 Combining MPU with Computational Differential Privacy In the standard DP framework, privacy guarantees are typically established against computationally unbounded adversaries (i.e., information-theoretic DP). In the context of MPU, relying on information- theoretic DP would be problematic: because the zero-sum noise vectors are algebraically constrained ( P m k=1 ε 0,(r) k ≡ 0 ), an unbounded adversary could hypothetically break the reparameterizationsT k,r , align themcopies, average them, and perfectly eliminate the noise, thereby bypassing the privacy guarantee. To prevent this, one would traditionally be forced to inject an additional, independent shared Gaussian noise that cannot be canceled, which inherently degrades the unlearning utility. However, as formally established in Appendix A.6, attempting to reverse the reparameterizations to align the mismatched coordinate spaces is NP-hard (equivalent to the Multi-Dimensional Assignment Problem and the Generalized Orthogonal Procrustes Problem). Therefore, against any realistic, probabilistic polynomial-time (PPT) adversary, we can safely establish our guarantees under the framework of Computational DP (CDP). CDP provides the same robust mathematical privacy guarantees as standard DP, but relaxes the adversary assumption to computationally bounded entities. Because the computationally bounded adversary cannot solve the NP-hard alignment problem, the zero-sum structural dependency between the copies remains cryptographically inaccessible. Consequently, we can use the zero-sum noise itself to provide the formal privacy guarantee. The integration of MPU with CDP operates as follows: 15 Bounding Sensitivity and Calibrating Zero-Sum Noise. The server first bounds the globalL 2 - sensitivity∆θof its model parametersθ r−1 via standard DP-training techniques (e.g., DP-SGD, gradient clipping). To achieve a target computational privacy budget(ε srv ,δ srv ), the server ensures that the marginal variance of each ε (r) k is σ 2 DP I d , where σ DP is determined strictly according to the Gaussian mechanism: σ DP = ∆θ p 2 ln(1.25/δ srv ) ε srv (29) The server scales the base zero-sum noise vectors (Eq. 4) such that their marginal distributions match N (0,σ 2 DP I d ), and adds them to the base parameters: ̃ θ (r) k = θ r−1 + ε (r) k . Reparameterization and Computational DP Guarantee The server then applies the data- independent reparameterizationT k,r ← SAMPLEREPARAM(t r ,k)to yield the published models distributed to the clients: θ (k,r) pub = T k,r ( ̃ θ (r) k )(30) To a PPT adversary, the transformed copiesθ (k,r) pub m k=1 are computationally indistinguishable from copies perturbed by truly independent Gaussian noise, because discovering the zero-sum correlation requires solving an NP-hard alignment problem. Since independent Gaussian noise of scaleσ DP satisfies (ε srv ,δ srv )-DP, the published copies in MPU satisfy (ε srv ,δ srv )-CDP. Perfect Privacy-Utility Decoupling (Zero First-Order Utility Cost) This formulation yields a profound advantage for the MPU framework: it completely bypasses the fundamental privacy-utility trade-off inherent in standard DP. During the Post-Process update aggregation (Section 3.4), the server uses its secret keyst r to invert the reparameterizations inO(d)time and applies harmonic denoising. Because all the injected DP noise is constructed with the zero-sum constraint ( P m k=1 ε 0,(r) k ≡ 0 ), the server’s harmonic aggregation perfectly eliminates the first-order noise error. Consequently, the server can inject an arbitrarily massive noise scaleσ DP to achieve an exception- ally small privacy budgetε srv (extremely strong privacy) without suffering the catastrophic utility degradation that typically plagues DP frameworks. The formal privacy guarantee is satisfied during client-side exposure, while the noise is mathematically canceled upon server-side aggregation. Remark: Client-Side Differential Privacy.While the above details the protection of the server’s model, MPU also naturally complements client-side DP. If the client wishes to formally protect its local forget set, it can perform local unlearning using standard DP-SGD. As proved in Appendix A.3.3, the server’s harmonic aggregation inherently provides a variance reduction effect for any independent client-side noise. Thus, MPU not only guarantees server privacy without first-order cost, but also actively suppresses the utility degradation associated with the client’s local DP mechanism. A.3 Harmonic Denoising Aggregation: Cancellation, Remainder, and Streaming We (i) formalize the first-order cancellation argument, (i) derive a clean second-order remainder bound under a local smoothness assumption, (i) establish the uniqueness and optimality of harmonic weights for zero-sum cancellation, and (iv) justify the streaming implementation used by MPU (Section 3.5). Local Linearization Model Let∆ ⋆ (θ)denote the ideal (noise-free) unlearning displacement produced by the chosen unlearning trainer when initialized at parametersθ. For analytical clarity, we assume fixed client data and fixed algorithmic randomness. Define the Jacobian of∆ ⋆ at the current iterate as J = ∂∆ ⋆ (θ) ∂θ θ r−1 .(31) After inverting the reparameterization (Section 3.4), we model each aligned update via a first-order expansion in the injected perturbation: b ∆ (k,r) = ∆ ⋆ (θ r−1 ) + J ε (r) k + ρ (r) k , ε (r) k = α k ε 0,(r) k ,(32) where the remainder termρ (r) k captures second-order (and higher-order) effects, as well as any deviation from modeling the client-side unlearning routine as a deterministic map. 16 A.3.1 First-Order Exact Cancellation via Harmonic Weights The server aggregates the aligned updates using harmonic weights w k ∝ α −1 k : ̄ ∆ (r) = m X k=1 w k b ∆ (k,r) , w k := α −1 k P m j=1 α −1 j .(33) Substituting the first-order model in Eq. (32) yields ̄ ∆ (r) = ∆ ⋆ (θ r−1 ) + J m X k=1 w k ε (r) k + m X k=1 w k ρ (r) k .(34) Since w k ε (r) k = α −1 k S 0 α k ε 0,(r) k = 1 S 0 ε 0,(r) k , S 0 := m X j=1 α −1 j ,(35) we obtain m X k=1 w k ε (r) k = 1 S 0 m X k=1 ε 0,(r) k = 0,(36) where the last equality follows from the zero-sum property in Eq. (4). Therefore, ̄ ∆ (r) = ∆ ⋆ (θ r−1 ) + m X k=1 w k ρ (r) k ,(37) which formally establishes that the injected noise is canceled exactly to first order. A.3.2 Second-Order Remainder under Lipschitz Jacobian A standard approach to control the remainder termρ (r) k is to assume local smoothness of the Jacobian. Assumption (Local Lipschitz Jacobian) Assume that∆ ⋆ (θ)is Fréchet differentiable in a neigh- borhood of θ r−1 , and that its Jacobian J (θ) is L J -Lipschitz in this neighborhood: ∥J (θ)− J (θ ′ )∥ ≤ L J ∥θ− θ ′ ∥.(38) Remainder BoundLetδ (r) k := ε (r) k . By the integral remainder form of Taylor’s theorem, we have ρ (r) k = ∆ ⋆ (θ r−1 + δ (r) k )− ∆ ⋆ (θ r−1 )− J δ (r) k = Z 1 0 J (θ r−1 + tδ (r) k )− J (θ r−1 ) δ (r) k dt. (39) Taking norms and applying the Lipschitz condition yields ∥ρ (r) k ∥ ≤ Z 1 0 L J t∥δ (r) k ∥ 2 dt = L J 2 ∥ε (r) k ∥ 2 .(40) Consequently, ̄ ∆ (r) − ∆ ⋆ (θ r−1 ) = m X k=1 w k ρ (r) k ≤ L J 2 m X k=1 w k ∥ε (r) k ∥ 2 ,(41) which matches the O(∥ε∥ 2 ) behavior stated in Section 3.4. A.3.3 Uniqueness and Optimality of Harmonic Weights In this section, we show that harmonic weights are the unique linear weights that cancel the first-order perturbation for all zero-sum noise realizations. 17 Proposition 1 (Uniqueness of Harmonic Weights for Zero-Sum Cancellation). Consider a linear estimator of the form P m k=1 w k b ∆ (k,r) with P m k=1 w k = 1. Assume the linear response model b ∆ (k,r) = ∆ ⋆ (θ r−1 ) + J α k ε 0,(r) k + ρ (r) k ,(42) where the base noises satisfy P m k=1 ε 0,(r) k ≡ 0. If the first-order term cancels for all such zero-sum realizations, i.e., m X k=1 w k α k ε 0,(r) k ≡ 0 for allε 0,(r) k m k=1 with m X k=1 ε 0,(r) k = 0,(43) then necessarily w k = α −1 k P m j=1 α −1 j , k = 1,...,m.(44) Proof. Cancellation for all zero-sum realizations is equivalent to requiring that the vector(w k α k ) m k=1 is orthogonal to the entire subspace(ε k ) ∈ R m : P m k=1 ε k = 0 . This subspace is spanned by pairwise difference vectors of the forme i − e j . Hence, orthogonality impliesw i α i = w j α j for alli,j, i.e.,w k α k = cfor some constantc. Imposing the normalization P m k=1 w k = 1yields c = 1/ P m j=1 α −1 j , which gives the harmonic form. Variance Reduction of Client-Side RandomnessSuppose the aligned updates additionally include zero-mean stochasticity from local training, b ∆ (k,r) = ∆ ⋆ (θ r−1 ) + J ε (r) k + η (r) k + ρ (r) k ,(45) whereE[η (r) k ] = 0andCov(η (r) k ) = Σ η , independently acrossk. Ignoring theO(∥ε∥ 2 )remainder terms for clarity, the covariance of the aggregated update satisfies Cov( ̄ ∆ (r) ) = m X k=1 w 2 k Σ η = P m k=1 α −2 k ( P m k=1 α −1 k ) 2 ! Σ η .(46) When the scalesα k are of comparable magnitude, the prefactor scales as≍ 1/m, formalizing the 1/m variance-reduction intuition discussed in Section A.9. A.4 Function-Preserving Reparameterizations This section expands Section 3.2.2 by providing full Transformer-level details. We describe the reparameterizations used by MPU from a group-theoretic perspective and present explicit forward and inverse formulas together with invariance proofs, including the RoPE-commutation constraint. Parameter-Symmetry Group Viewpoint LetΘdenote the parameter tuple of a Transformer block, and letf Θ : R L×d model → R L×d model denote the forward function induced byΘon a length-L sequence. Define the set of function-preserving reparameterizations as G = n T : Θ7→ T (Θ) T is invertible and f T (Θ) (X) = f Θ (X)∀X o .(47) Under composition,Gforms a group, and eachT ∈ Gconstitutes a parameter-symmetry of the model. In MPU, we takeL = d f and sampleT k,r data-independently from a structured subsetT ⊆ G constructed from: (i) discrete channel permutations within FFN submodules, and (i) continuous orthogonal basis changes within attention head subspaces. For RoPE-based models, we further restrict the attention basis changes to lie in the commutant of the RoPE operators, ensuring functional invariance. 18 A.4.1 Attention Head Reparameterization Letd model denote the model width,d h the head dimension,H Q the number of query heads, andH KV the number of key/value heads (GQA/MQA are covered byH Q ≥ H KV ). We use the following parameters: W Q ∈ R d model ×(H Q d h ) ,(48a) W K ,W V ∈ R d model ×(H KV d h ) ,(48b) W O ∈ R (H Q d h )×d model .(48c) with parameter tuple Θ = (W Q ,W K ,W V ,W O )(49) Group Element and Lifted ActionLetπ : [H Q ]→ [H KV ]denote the fixed assignment mapping each query head to its corresponding key/value head. Sample per-KV orthogonal blocksS j ∈ O(d h ) for j = 1,...,H KV , and define S KV := diag(S 1 ,...,S H KV )∈ O(H KV d h ),(50a) U (π,S KV ) := blkdiag S π(1) ,...,S π(H Q ) ∈ O(H Q d h ).(50b) Both S KV and U (π,S KV ) are orthogonal matrices. Forward and Inverse Formulas The reparameterization acts on the attention weights as W ′ Q = W Q U (π,S KV ),W ′ K = W K S KV ,(51a) W ′ V = W V S KV ,W ′ O = U (π,S KV ) ⊤ W O .(51b) T (Θ) = (W ′ Q ,W ′ K ,W ′ V ,W ′ O )(51c) If biases are present, apply the same right-multiplication:b ′ Q = b Q U (π,S KV ) ,b ′ K = b K S KV , and b ′ V = b V S KV . The inverse transformation is given by transposition: W Q = W ′ Q U (π,S KV ) ⊤ ,W K = W ′ K S ⊤ KV ,(52a) W V = W ′ V S ⊤ KV ,W O = U (π,S KV )W ′ O .(52b) Function Invariance (without RoPE) LetX ∈ R L×d model and defineQ = XW Q ,K = XW K , andV = XW V , with head-wise partitionsQ = [Q 1 | · | Q H Q ] ,K = [K 1 | · | K H KV ] , and V = [V 1 |·| V H KV ] , where eachQ i ,K j ,V j ∈ R L×d h . For GQA attention, the outputs are given by A i = softmax 1 √ d h Q i K ⊤ π(i) , O i = A i V π(i) , Y = [O 1 |·| O H Q ]W O .(53) Claim Under the reparameterization in Eq.(51), the realized attention function is invariant, i.e., Y ′ = Y . Proof.From Eq.(51), we haveQ ′ = QU (π,S KV )andK ′ = K S KV ,V ′ = V S KV . Consequently, Q ′ i = Q i S π(i) and K ′ π(i) = K π(i) S π(i) . Since S ⊤ π(i) S π(i) = I , it follows that Q ′ i K ′ π(i) ⊤ = Q i S π(i) S ⊤ π(i) K ⊤ π(i) = Q i K ⊤ π(i) ,(54) and hence the attention logits, and therefore A ′ i , are unchanged. Moreover, O ′ i = A ′ i V ′ π(i) = A i V π(i) S π(i) = (O i )S π(i) .(55) Concatenation yields h O ′ 1 |·| O ′ H Q i = O 1 |·| O H Q U (π,S KV ). Finally, usingW ′ O = U (π,S KV ) ⊤ W O , we obtain Y ′ = [O ′ ]W ′ O = ([O]U ) U ⊤ W O = [O]W O = Y,(56) which completes the proof. 19 A.4.2 RoPE-Aware Specialization Llama-type decoders apply rotary positional embeddings (RoPE) to queries and keys via right- multiplication by a position-dependent orthogonal operator Φ(p)∈ R d h ×d h : e Q i [p, :] = Q i [p, :] Φ(p) ⊤ , e K j [p, :] = K j [p, :] Φ(p) ⊤ .(57) RoPE acts as independent 2× 2 rotations on d h /2 disjoint planes, Φ(p) = blkdiag R(ω 1 p),...,R(ω d h /2 p) , R(φ) = cosφ − sinφ sinφcosφ ∈ SO(2).(58) To preserve the attention logits, we require the head-basis transform to commute with all Φ(p), i.e., S j Φ(p) = Φ(p)S j , ∀p.(59) Commutant under Distinct FrequenciesWhen the RoPE frequenciesω r are distinct across the d h /2planes, the orthogonal transformations that commute with allΦ(p)are exactly the per-plane rotations: S j = blkdiag R (φ j,1 ),...,R φ j,d h /2 ∈ SO(2) d h /2 , j = 1,...,H KV .(60) Reflections inO(2) d h /2 \ SO(2) d h /2 generally fail to commute withΦ(p), except at degenerate angles. The rotation anglesφ j,r may be sampled deterministically from the round seed and indices (t r ,k, layer,j,r). Function Invariance with RoPE Under the commutation conditionS π(i) Φ(p) = Φ(p)S π(i) , the proof of function invariance follows identically to the non-RoPE case. In particular, commutativity implies S π(i) Φ(p) ⊤ Φ(p ′ )S ⊤ π(i) = Φ(p) ⊤ Φ(p ′ ),(61) inside the attention logits, so the attention weights remain unchanged. The output invariance then follows as before via cancellation with W ′ O = U (π,S KV ) ⊤ W O . A.4.3 Feed-Forward Blocks: Hidden-Channel Permutations Standard Two-Layer Feed-Forward Neural Network Consider an FFN with parametersΘ = (W 1 ,b 1 ,W 2 ,b 2 ) and an element-wise nonlinearity σ: f Θ (x) = W 2 σ(W 1 x + b 1 ) + b 2 .(62) LetP ∈ R d f ×d f be a randomly drawn permutation matrix (P −1 = P ⊤ ) acting on hidden channels. Define the reparameterization W ′ 1 = PW 1 , b ′ 1 = Pb 1 , W ′ 2 = W 2 P ⊤ , b ′ 2 = b 2 .(63) so that T (Θ) = (W ′ 1 ,b ′ 1 ,W ′ 2 ,b ′ 2 )(64) Since element-wise activations are permutation-equivariant, i.e., σ(Pz) = Pσ(z), we have f T (Θ) (x) = W ′ 2 σ(W ′ 1 x + b ′ 1 ) + b ′ 2 = W 2 P ⊤ σ P (W 1 x + b 1 ) + b 2 = W 2 P ⊤ Pσ(W 1 x + b 1 ) + b 2 = W 2 σ(W 1 x + b 1 ) + b 2 = f Θ (x). (65) Thus, hidden-channel permutations are function-preserving. SwiGLU/GEGLU-Style Gated Feed-Forward Neural NetworksModern LLMs (e.g., the Llama 3.2 family) employ gated FFNs such as SwiGLU. For gated architectures with gate and up branches combined via an element-wise product, the same permutation must be applied to both branches to preserve consistency. Concretely, for parameters (W 1,gate ,W 1,up ,b 1 ,W 2 ,b 2 ), define W ′ 1,gate = PW 1,gate , W ′ 1,up = PW 1,up , b ′ 1 = Pb 1 , W ′ 2 = W 2 P ⊤ , b ′ 2 = b 2 .(66) Permutation-equivariance together with P ⊤ (Pa)⊙ (Pb) = a⊙ b ensures functional invariance. 20 A.4.4 Linearity of Reparameterization Map In MPU, eachT k,r is implemented via multiplication by orthogonal or permutation matrices acting on parameter tensors (through left/right multiplication and block-diagonal composition). Consequently, T k,r is a linear isomorphism on the parameter space. For conformable parameter collectionsa,band any scalar c, T (a + b) = T (a) + T (b), T (ca) = cT (a), T −1 exists and is linear.(67) This property justifies applyingT −1 k,r directly to the returned parameter updates in Algorithm 1. Specifically, if the client returns a displacement in the transformed coordinates, mapping it back via T −1 yields the corresponding displacement in the canonical coordinates: T −1 T (θ) + η ∆(T (θ)) = θ + η T −1 ∆(T (θ)) .(68) A.5 Discussion: Effects of Reparameterization on Optimization Trajectory and Smoothness A natural concern is whether the client-side reparameterization introduced in Section 3.2.2 (and Appendix A.4) alters the optimization dynamics of local unlearning. We address two questions: (i) when the optimization trajectory is equivariant under the reparameterization, and (i) whether the reparameterization can change the apparent smoothness of the loss landscape. Setup LetU (θ)denote the client-side unlearning objective (e.g., the loss used by GradAscent, NPO, or GradDiff) as a function of the model parametersθ ∈ R d . For a function-preserving reparameterizationT ∈T(Section 3.2.2), we havef T (θ) (·)≡ f θ (·)and therefore, for any objective depending only on model outputs, U (T (θ)) = U (θ).(69) In MPU, eachT k,r is implemented via orthogonal or permutation actions on weight matrices (Appendix A.4). Viewed as a linear map on the vectorized parameter space, this implies thatTis an isometry: ⟨Ta, Tb⟩ = ⟨a, b⟩, ∥Ta∥ 2 = ∥a∥ 2 , T −1 = T ⊤ .(70) A.5.1 Equivariance of Gradient Descent under Orthogonal or Permutation Reparameterizations Define the reparameterized objectiveU T (θ) := U (T −1 θ)in the published coordinatesθ = T (θ). By the chain rule together with Eq. (70), ∇ θ U T (θ) = T ∇ θ U (θ), θ = T −1 θ.(71) Consider t steps of (stochastic) gradient descent in the published coordinates: θ s+1 = θ s − η g s (θ s ),(72) whereg s (θ s )is an unbiased estimator of∇ θ U T (θ s )(e.g., mini-batch SGD). Mapping back via θ s := T −1 θ s and using Eq. (71) yields θ s+1 = T −1 θ s+1 = θ s − η T −1 g s (θ s ) ≈ θ s − η∇ θ U (θ s ),(73) i.e., the mapped-back iterates follow the same optimization trajectory as if the client had optimized directly in the canonical coordinates from the corresponding initialization. This equivariance holds exactly for deterministic gradient descent, and holds in distribution for SGD under the natural coupling in which the same mini-batches are used and the stochastic gradients are transformed consistently by T. The argument also extends to common modifications such as momentum andℓ 2 weight decay, since Eq. (70) preserves the Euclidean norm. A.5.2 Invariance of Euclidean Smoothness BecauseTis an orthogonal or permutation change of coordinates, it does not alter curvature measured under the Euclidean metric. LetH(θ) :=∇ 2 θ U (θ)andH T (θ) :=∇ 2 θ U T (θ). Differentiating Eq.(71) yields the similarity relation H T (θ) = T H(θ)T ⊤ , θ = T −1 θ.(74) 21 Consequently, the Hessian spectrum is preserved:λ max (H T (θ)) = λ max (H(θ)), and likewise for the operator norm∥H T ∥ 2 . Equivalently, ifUisL-smooth inθ(i.e., its gradient isL-Lipschitz), then U T is alsoL-smooth inθ. Thus, under Euclidean geometry, the reparameterization does not make the loss landscape appear smoother or sharper; it merely rotates or permutes the coordinate axes within a functionally equivalent parameter orbit. A.5.3 Influence of Nonlinear Optimizers on Trajectory While optimization trajectory invariance rigorously holds for linear optimizers like SGD, it is theoretically violated by adaptive optimizers such as AdamW in the Attention blocks. Specifically, AdamW scales updates using an element-wise moving average of squared gradients (v t ). For Attention blocks reparameterized by dense continuous orthogonal matricesU ∈ O(d h ), the non-commutativity of element-wise squaring and matrix multiplication—that is,(gU ) ⊙2 ̸= g ⊙2 U—breaks the exact equivariance conditionA(θ) = T −1 (A(T (θ))). Consequently, the adaptive preconditioning becomes copy-dependent, slightly altering the local unlearning Jacobians and preventing the mathematically perfect cancellation of first-order injected noise. Nevertheless, this deviation remains highly bounded in practice, allowing the empirical results to remain robust due to three key mitigating factors. First, Feed-Forward Networks (FFNs), which account for the vast majority (roughly 65%) of standard LLM parameters, are reparameterized via discrete permutations. Because permutations strictly commute with element-wise operations, exact trajectory invariance and noise cancellation are perfectly preserved for the bulk of the model. Second, since orthogonal transformations are isometric, they preserve the Euclidean geometry and overall gradientL 2 norms, ensuring AdamW still optimizes over a functionally equivalent and well-conditioned landscape. Finally, because unlearning operates over very few epochs with exceedingly small learning rates (e.g.,1× 10 −5 ), the accumulated coordinate- wise divergence in the second-moment estimatorv t remains small. Any residual trajectory variations introduced by this mismatch act as mild, independent stochastic noise that is effectively smoothed out by the variance-reducing properties of the server’s multi-copy harmonic aggregation. A.6 Security Analysis: Theoretical Hardness of Reversing Reparameterization In this section, we formally analyze the computational security guarantees of the MPU reparameteri- zation module. We demonstrate that client-side reconstruction of the exact server parameters from multiple published copies reduces to intractable alignment optimization problems. A.6.1 Double Protection Against Known-Scaling Attacks A naive multi-copy noise scheme without reparameterization relies entirely on keeping the noise scaling factorsα k m k=1 secret. Because the base noise satisfies a zero-sum constraint ( P m k=1 ε 0,(r) k ≡ 0 ), an attacker who compromises the exact values ofα k could theoretically recover the server’s exact original parameters θ r−1 by performing a harmonic aggregation: P m k=1 α −1 k θ (k,r) pub ∝ θ r−1 . The MPU reparameterization introduces a second, mathematically robust layer of protection. The client observes the published copies: θ (k,r) pub = T k,r (θ r−1 + α k ε 0,(r) k ) for k ∈ [m],(75) whereT k,r ∈ Gdenotes the data-independent, function-preserving transformations. Because the transformations (permutations and orthogonal matrices) act as coordinate basis changes, they do not commute with cross-copy addition. Consequently, attempting to perform harmonic aggregation on the mismatched spaces yields a completely scrambled and meaningless result. Therefore, even if the scaling factorsα k are completely leaked, the information is mathematically useless unless the attacker can simultaneously break the reparameterization keys T k,r to realign the parameter spaces. A.6.2 Alignment as NP-Hard Optimization To bypass the double protection, the attacker must blindly find the inverse transformationsT −1 k,r strictly from the noisy, transformed observations. Without a clean template model to anchor the search, the attacker is forced to find a set of candidate transformations ˆ T k ∈Gthat mutually align 22 all m copies to minimize their structural discrepancy. Formally, this alignment objective is: min ˆ T 1 ,..., ˆ T m ∈G X 1≤i<j≤m ˆ T −1 i θ (i,r) pub − ˆ T −1 j θ (j,r) pub 2 F .(76) We show that for both FFN and Attention blocks, this translates to well-known computationally hard optimization problems. Discrete Symmetries (FFN) and the Multi-Dimensional Assignment Problem (MDAP). For FFN blocks (Appendix A.4.3), the transformations are hidden-channel permutation matricesP k ∈ S d f . The attacker’s objective is to findmpermutations to align the row/column spaces of the noisy weight matrices: min P 1 ,...,P m ∈S d f X 1≤i<j≤m P ⊤ i W (i) pub − P ⊤ j W (j) pub 2 F .(77) Whenm = 2, finding the optimal matching between two sets of noisy features is equivalent to the Linear Assignment Problem (or Bipartite Graph Matching), which requiresO(d 3 f )time using the Hungarian algorithm. However, form≥ 3, enforcing transitive consistency to simultaneously align three or more sets of items constitutes the Multi-Dimensional Assignment Problem (MDAP). MDAP is famously NP-hard (acting as a generalization of the 3-Dimensional Matching problem, one of Karp’s 21 NP-complete problems). The discrete combinatorial search space expands factorially as(d f !) m−1 , effectively neutralizing any exact alignment strategy for modern LLM architectures where d f is massively large. Continuous Symmetries (Attention) and the Generalized Orthogonal Procrustes Problem (GOPP).For Attention blocks (Appendix A.4.1), the transformations are orthogonal basis changes U k ∈ O(d h ). The attacker must align m continuous spaces: min U 1 ,...,U m ∈O(d h ) X 1≤i<j≤m W (i) pub U i − W (j) pub U j 2 F .(78) Whenm = 2, this reduces to the classical Orthogonal Procrustes Problem, which admits a closed- form global optimum computed via Singular Value Decomposition (SVD) inO(d 3 h )time. However, form ≥ 3, this formulation is exactly the Generalized Orthogonal Procrustes Problem (GOPP). GOPP requires finding mutually consistent orthogonal rotations to centralize multiple matrices. Unlike them = 2case, GOPP lacks a closed-form solution and has been rigorously proven to be NP-hard. The simultaneous constraints across multiple non-convex orthogonal manifolds prevent exact polynomial-time alignment. In modern LLMs, the hidden dimensions governing these transformations are massive. Furthermore, the presence of large independent obfuscation noise severely perturbs the distance metrics, heavily degrading the reliability of analytical solvers like SVD or the Hungarian algorithm even in the m = 2case. In summary, the space misalignment mathematically prevents the client from exploiting cross-copy correlations. The fundamental parameter alignment tasks map directly to non-convex, NP-hard continuous and discrete optimization problems, ensuring the theoretical security of the MPU framework. Remark 2. In our experiments, as empirically verified in Appendix B.6.2, varying copy numberm has no typical trend in performance, we usem = 2by default to save computational resources. While in real-world applications, we recommend at least m = 3 for strong security. A.7 Overhead Analysis To model the computational overhead, we theoretically analyze the exact Memory Reads and Writes (MRW) and FLOPs. By evaluating Arithmetic Intensity (A = FLOPs / MRW bytes), we can determine whether operations are compute-bound or memory-bound. LetNbe the total parameter count,N attn the attention parameters,d h the head dimension,bthe bytes per parameter (b = 2 for 16-bit precision), and m the number of perturbed copies. 23 1. Profiling Server-Side Operations (per copy): • Pre-Process: – Noise injection requires 2Nb MRW and 4N FLOPs. –Reparameterization takes2N attn bMRW alongside2d h N attn FLOPs for Attention (block-diagonal orthogonal matmuls), and2N ffn bMRW alongside0FLOPs for FFN (pure memory routing). – Total: 4Nb MRW, (4N + 2d h N attn ) FLOPs. • Post-Process: – Inverse reparameterization takes 2Nb MRW alongside 2d h N attn FLOPs. –Streaming harmonic aggregation takes3NbMRW (read copy, read accumulator, write accumulator) alongside 2N FLOPs. – Total: 5Nb MRW, (2N + 2d h N attn ) FLOPs. 2. Latency Modeling: Formcopies, the total MRW is9mNband the total FLOPs ism(6N + 4d h N attn ). The server’s Arithmetic Intensity is: A = 6N + 4d h N attn 9Nb (79) For a typical Llama architecture,A ≈ 6.0FLOPs/Byte. Because this is vastly lower than the operational ridge point of modern GPUs (e.g.,∼ 150FLOPs/Byte for an A100), MPU server operations are strictly memory-bound. Thus, the server latency T server is modeled entirely by memory bandwidth (C mem ): T server = 9mNb C mem (80) For a 7B model (m = 2) on an A100 GPU, sweeping∼ 252GB of memory traffic takes merely ∼ 0.13 seconds, mathematically proving the server overhead is negligible. A.8 Comparison of Update Error: MPU vs. Single-Copy Noisy Unlearning In this section, we compare MPU with a noise-injection baseline that performs no denoising, which serves as a controlled reference for isolating the effect of our denoising mechanism. To ensure a fair comparison between MPU and the NOISED baseline, we scale the noise parameter κof the NOISED baseline by the expected value of the scaling factor in MPU. As a result, the noise level of the NOISED baseline matches the average noise level across multiple copies in MPU. Baseline (Noise-Only, No Denoising) Define a single-copy procedure that, in each roundr: (i) samples a noise vectorε (r) with the same block-wise scales as MPU (e.g.,ε (r) ℓ ∼N (0,σ 2 ℓ I d ℓ )) and overall scale matched toE k (α k ); (i) publishesθ r−1 + ε (r) to the client; (i) runs local unlearning from this noisy initialization; and (iv) uses the resulting parameters as the next-round model. Let the unlearning routine induce a displacement map∆(θ)so that the updated parameters are θ + ∆(θ). The baseline state evolution is θ NO r = θ NO r−1 + ε (r) + η srv ∆ θ NO r−1 + ε (r) ,(81) where η srv matches the server step size in Algorithm 1. Linear Response Model Fix an anchor pointθ(e.g.,θ = θ r−1 ) and write the perturbation asu. Under the same local linearization used in Section 3.4, ∆(θ + u) = ∆ ⋆ (θ) + Ju + ρ(u), ∥ρ(u)∥≤ L J 2 ∥u∥ 2 .(82) 24 A.8.1 One-Round Comparison: Bias and Variance from Injected Noise Noise-Only Baseline Error Substituting the linear response into Eq. (81) yields θ NO r = θ + η srv ∆ ⋆ (θ) + I + η srv J ε (r) | z First-Order Injected-Noise Term +η srv ρ ε (r) .(83) Thus, even if E[ε (r) ] = 0, the state θ NO r inherits a first-order random perturbation I + η srv J ε (r) . MPU ErrorIn contrast, MPU updates the clean anchorθusing the harmonically denoised estimate ̄ ∆ (r) : θ MPU r = θ + η srv ̄ ∆ (r) = θ + η srv ∆ ⋆ (θ) + η srv m X k=1 w k ρ ε (r) k ,(84) where the first-order injected-noise term cancels exactly (Section A.3). Consequently, the dominant injected-noise contribution is second order in the noise scale (cf. Eq. (41)). Variance ComparisonLetΣ ε denote the covariance of the single-copy baseline noise (e.g., block- diagonal with blocks σ 2 ℓ I d ℓ ). Ignoring the O(∥ε∥ 2 ) remainder for clarity, Eq. (83) implies Cov θ NO r − θ− η srv ∆ ⋆ (θ) ≈ I + η srv J Σ ε I + η srv J ⊤ ,(85) whereas MPU exhibits no first-order injected-noise covariance term. A.8.2 Multi-Round Implications: Noise Accumulation vs. Denoised Tracking The contrast becomes more pronounced over multiple rounds. The recursion in Eq.(81)injects fresh noise into the model state at each round, leading—under the linearized view—to a random-walk-like accumulation whose variance grows approximately linearly withR(modulated byI + η srv J). This accumulation can degrade both model utility and unlearning stability. By contrast, MPU never commits the injected noise to the global model state (Algorithm 1). Instead, noise is used solely as a private publishing perturbation and its first-order effect is canceled via multi-copy harmonic denoising. As a result, the global iterate tracks the intended unlearning trajectory up to the second-order remainder and reduced client-side randomness. A.8.3 SNR View: Task-Relevant Subspace If the desired unlearning displacement lies approximately in a low-dimensional task-relevant subspace Swith projectorP S , a convenient summary metric is the signal-to-noise ratio (SNR) of the projected estimate. Let b ∆ be an estimator of ∆ ⋆ (θ) and define SNR = ∥P S ∆ ⋆ (θ)∥ 2 F tr P S Cov( b ∆) .(86) In the noise-only baseline,Cov( b ∆)contains the first-order termJ Σ ε J ⊤ , whereas in MPU this term is eliminated by design (up to second-order effects). Any independent client-side randomness is further averaged down by P k w 2 k ≍ 1/m. This explains why MPU more reliably preserves the task- relevant update direction than a single-copy noise-only approach, particularly when rank(S)≪ d. A.9 Stability Benefits of Multi-Copy Aggregation A practical challenge in LLM unlearning is that local optimization can be unstable: the forget set may be small and high-variance, the unlearning objective may be sharp near the current iterate, and common unlearning trainers can produce oscillatory updates when executed from a single initialization. Multi-copy unlearning improves stability by averaging local updates over a small neighborhood around the clean server iterate. Consider the common case where the local unlearning routine corresponds to (approximately) a gradient step on an unlearning objectiveU (θ), so that∆ ⋆ (θ)≈−η∇U (θ)for some local step sizeη. 25 Then the aggregated update becomes −η m X k=1 w k ∇U θ r−1 + ε (r) k ≈−η E ε∼Q [∇U (θ r−1 + ε)],(87) whereQis the discrete distribution placing massw k onε (r) k . Equivalently, this is the gradient of a locally averaged objective e U (θ) := E ε∼Q [U (θ + ε)]. Such local averaging reduces sensitivity to sharp directions nearθ r−1 and makes the effective update direction less erratic, improving stability across rounds. Importantly, averaging is performed with a centered, symmetric stencil of perturbations: our base noises satisfy P m k=1 ε 0,(r) k,ℓ ≡ 0for every blockℓ(Eq.(4)), and the harmonic weights are chosen to cancel the resulting first-order perturbation in the local linearization (Section 3.4). Expanding∆ ⋆ around θ r−1 yields ∆ ⋆ θ r−1 + ε (r) k = ∆ ⋆ (θ r−1 ) + J ε (r) k + O ε (r) k 2 ,(88) so harmonic aggregation cancels the J ε (r) k term to first order, leaving ̄ ∆ (r) = ∆ ⋆ (θ r−1 ) + O m X k=1 w k ε (r) k 2 ! .(89) Thus, for small perturbation scales, MPU tracks the intended (noise-free) unlearning direction while still benefiting from the stabilizing effect of local averaging (the remaining higher-order term acts as a mild, local regularization). In summary, beyond privacy, the multi-copy mechanism can be viewed as estimating a locally averaged unlearning update aroundθ r−1 , which empirically improves stability in settings where single-start unlearning is brittle. A.10 Limitations and Broader Impacts The MPU is discussed within the field of model unlearning, as the gradients in the unlearning process are typically small, and result in the first-order noise dominance, so that the harmonic denoising can perfectly alleviate the noise influence. Future works can discuss whether higher-order noise influences can be alleviated so that MPU can be generalized to general collaborative learning cases. 26 B Implementation Details and Supplementary Experiments This section provides supplementary materials to support a deeper understanding of MPU. It includes detailed descriptions of the benchmarks, evaluation metrics, models, and unlearning algorithms used in our framework, as well as comprehensive implementation details covering the experimental setup, hyperparameters, and prompt templates. We further report additional experimental results and discuss the limitations of MPU. B.1 Benchmarks TOFU (Task of Fictitious Unlearning), introduced by Maini et al. [2024], is a question–answer (QA) benchmark specifically designed to evaluate the unlearning capabilities of large language models. The dataset comprises QA pairs derived from synthetic autobiographies of 200 fictitious authors, with all content generated by GPT-4 to ensure exclusion from the pretraining corpora of existing LLMs. Each author profile contains 20 QA pairs covering biographical attributes such as name, birthplace, gender, birth year, literary genre, awards, and parental occupations, with book titles seeded from the Goodreads Books dataset to enhance topical diversity. MUSE (Machine Unlearning Six-Way Evaluation) [Shi et al., 2024] is a comprehensive benchmark designed to assess the effectiveness and practicality of machine unlearning algorithms for language models. Moving beyond narrow, task-specific evaluations, MUSE incorporates six distinct criteria reflecting the needs of both data owners and model deployers. From the data owner’s perspective, it measures verbatim memorization, knowledge memorization, and privacy leakage (assessed via membership inference attacks). For deployers, it evaluates the algorithm’s ability to preserve general model utility, scale to large forget sets, and sustainably handle sequential unlearning requests. Evaluated on unlearning Harry Potter books and news articles, MUSE reveals that while current algorithms can reduce memorization, they struggle significantly with preventing privacy leakage, maintaining model utility, and scaling effectively. B.2 Metrics For clarity, we categorize the evaluation metrics into three types, summarized below. B.2.1 Memorization Metrics These metrics measure the degree to which information from the training data remains encoded in the model after unlearning. Probability We quantify the model’s confidence in generating correct answers by measuring the conditional probabilityP (a| q)assigned to the ground-truth answeragiven the questionq, evaluated on Retain Set. The resulting score is reported as a normalized probability in[0, 1]. Following standard practice [Cho et al., 2014], we normalize for answer length by exponentiating the sequence probability by 1/|a|, as formalized in Equation 90: P retain (x) = P (a| q) 1/|a| .(90) For the Real Authors and World Facts subsets, we evaluate model performance using a relative probability formulation in a multiple-choice setting: P real/world (x) = P (a 1 | q) P n i=1 P (a i | q) ,(91) whereqdenotes a multiple-choice question with candidate answersa 1 ,...,a n , anda 1 corresponds to the ground-truth correct answer. This formulation measures the probability mass assigned to the correct option relative to all candidate choices. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) We adopt ROUGE to quantify the overlap between model-generated answers and the ground-truth responses. In particular, we report ROUGE-L recall [Lin, 2004], which measures similarity via the length of the longest common subsequence (LCS). This metric provides a robust estimate of answer correctness in QA settings by tolerating minor paraphrasing and surface-form variations. 27 ROUGE L (x) = LCS(a, ˆa) |a| ,(92) where ˆa is the generated answer, and LCS(a, ˆa) is the LCS length between a and ˆa. Truth Ratio For a given question, we define the truth ratioR Truth to approximate the relative likelihood assigned by the model to correct versus incorrect answers. Since the model is fine-tuned on a specific canonical phrasing of the ground-truth answer, the corresponding probability may be artificially inflated compared to alternative but semantically equivalent formulations. To mitigate this bias, we evaluate the probability of a paraphrased version of the correct answer rather than the original ground truth. Likewise, instead of contrasting against a single incorrect response, we consider a set of syntactically similar but factually incorrect answers and compute their average probability. This design yields a more stable and representative estimate of the model’s preference for incorrect information. Formally, let ̃adenote a paraphrased correct answer to questionq, and let ̃x = [q, ̃a]. LetA err be a set of incorrect answersa err , constructed by preserving the general textual structure of ̃awhile introducing factual errors. The truth ratio R Truth is then defined as: R truth (x) = 1 |A err | P a err ∈A err P (a err | q) 1/|a err | P ( ̃a| q) 1/| ̃a| .(93) Additionally, as shown in Equation 94, we normalize and rescale the metric to ensure that all values lie within the interval [0, 1], with larger values indicating better model performance. R adjusted (x) = max(0, 1− R Truth (x)).(94) Verbatim Memorization (Data Extraction)To evaluate whether a model inappropriately extracts or replicates exact details from a removed dataset, the MUSE benchmark utilizes the Verbatim Memorization (V erbMem) metric. This is measured by prompting the modelfwith the firstltokens of a sequence from the forget set, denoted asx [:l] . The model’s generated continuation,f (x [:l] ), is then compared to the actual ground-truth continuation,x [l+1:] , using the ROUGE-L F1 score. The overall metric is calculated by averaging these scores across the entire forget set, formalized as: V erbMem(f,D) := 1 |D forget | X x∈D forget ROUGE(f (x [:l] ),x [l+1:] ) B.2.2 Privacy Metrics These metrics evaluate whether sensitive information from the forget set can still be inferred or extracted from the model after unlearning. We note that such metrics often rely on idealized assumptions, such as access to perfectly i.i.d. holdout samples or an oracle retain model, which may limit their applicability in practical deployment scenarios. Membership Inference LeakageWe could evaluate privacy leakage through membership inference attacks (MIAs), which assess a model’s tendency to memorize training data. Specifically, MIAs test whether an adversary can distinguish between examples drawn from the forget setD forget (members) and unseen examples from a holdout setD holdout (non-members), based on model confidence or loss statistics. Ideally, a model that has not been trained onD forget should yield an AUC of0.5, indicating indistin- guishability between member and non-member samples. In practice, however, constructing perfectly i.i.d. holdout splits is challenging, and even retrained models may exhibit nontrivial membership signals. Accordingly, following prior benchmarks such as MUSE, we calibrate membership inference results using the AUC score of a retrained reference model, rather than relying on the absolute AUC value alone. Unlearning generally increases the loss on forgotten samples, but privacy leakage may still arise in two failure modes: (i) under-unlearning, where the loss increase is insufficient, and membership 28 information remains detectable; and (i) over-unlearning, where the loss becomes abnormally large, again yielding a distinguishable signal. Both cases induce separable loss distributions betweenD forget andD holdout . To quantify this effect, we compare the AUC-ROC achieved by the unlearned modelf unlearn against that of a retrained reference model f retrain , and define the relative privacy leakage as PrivLeak : = AUC(f Unlearn ;D Forget ,D Holdout )− AUC(f Retrain ;D Forget ,D Holdout ) AUC(f Retrain ;D Forget ,D Holdout ) .(95) A well-behaved unlearning algorithm should yield aPrivLeakvalue close to zero, indicating privacy leakage comparable to retraining. In contrast, under-unlearning and over-unlearning result in large negative and positive deviations, respectively, reflecting increased membership distinguishability. Forget Quality In our setting, letF U (x)andF R (x)denote the empirical cumulative distribution functions (CDFs) of a chosen privacy-related statistic (e.g., Truth Ratio) computed from the unlearned and retained models, based onnandmsamples, respectively. We employ the Kolmogorov–Smirnov (KS) test to quantify the discrepancy between these two distributions, with the test statistic defined as D n,m = sup x |F U (x)− F R (x)|.(96) This statistic measures the maximum deviation between the two empirical CDFs, providing a non- parametric assessment of the distributional shift induced by the unlearning procedure. Under the null hypothesis that the two sample sets are drawn from the same underlying distribution, the hypothesis is rejected at significance level α if D n,m > c(α) r n + m nm ,(97) where c(α) is the KS critical value given by c(α) = r − 1 2 ln α 2 .(98) We define the correspondingp-value as the smallest significance levelαat which the null hypothesis can be rejected: Q forget = p = min ( α D n,m > c(α) r n + m nm ) .(99) Consequently, Forget Quality quantifies the statistical confidence with which we can assert that the distributions of Truth Ratio values over the forget set differ between the unlearned and retained models. B.2.3 Utility Metrics The objective of unlearning is to effectively remove the influence of targeted data while preserving the model’s performance on non-forget data. Utility metrics evaluate whether the unlearned model maintains its capabilities on tasks beyond the retain set, thereby ensuring that unlearning does not degrade general performance on real-world data distributions. Model UtilityModel Utility (MU) measures the retained performance of a model after unlearning, covering both the closely related retain set and broader general-knowledge tasks. Following the TOFU benchmark protocol, MU is computed as the harmonic mean of nine metrics spanning three data levels: Retain Set, Real Authors, and World Facts. At each level, three metrics are evaluated: Probability, ROUGE, and Truth Ratio, to ensure balanced assessment across memorization, semantic accuracy, and factual correctness. U model = 9 P m∈M 1 m ,(100) where M denotes the set of all evaluated metrics. 29 B.3 Unlearning Algorithms In this section, we introduce the seven unlearning algorithms instantiated within MPU as modular optimization objectives. Notation LetD forget andD retain denote the forget and retain sets, respectively, and letf θ be the model being updated during unlearning. For an input–output (prompt–completion) pair(x,y)under a causal LM, we define logp θ (y | x) = |y| X t=1 logp θ (y t | y <t ,x), ℓ CE (y | x;f θ ) =− logp θ (y | x).(101) Unless otherwise stated, we optionally include a retain regularizer: L retain (θ) = E (x,y)∼D retain ℓ CE (y | x;f θ ),(102) and optimizemin θ L forget (θ) +αL retain (θ), whereα≥ 0controls the forgetting–retention trade-off. B.3.1 GradAscent [Jang et al., 2023] Gradient Ascent (GradAscent) directly “reverses” standard likelihood training on the forget set by maximizing the cross-entropy (equivalently, minimizing the log-likelihood). Under the minimization convention, the forget objective is L GA forget (θ) =−E (x,y f )∼D forget ℓ CE (y f | x;f θ ) = E (x,y f )∼D forget logp θ (y f | x).(103) This objective is effective at reducing the model likelihood on targeted samples, but without an explicit retention constraint, it may cause collateral degradation on non-forgotten behavior. B.3.2 GradDiff [Liu et al., 2022a] GradDiff augments GradAscent with a retain loss to explicitly preserve utility on non-forgotten data. A common instantiation writes the full objective as min θ L GradDiff (θ) =−λ f E (x,y f )∼D forget ℓ CE (y f | x;f θ ) + λ r E (x,y)∼D retain ℓ CE (y | x;f θ ), (104) where λ f ,λ r > 0 trade off forgetting strength and retention fidelity. B.3.3 DPO [Rafailov et al., 2023] Direct Preference Optimization (DPO) is a preference-learning objective originally proposed for alignment. Given a preference datasetD pref of triples(x,y w ,y l )(preferredy w vs. dispreferredy l ) and a reference model f ref , DPO optimizes L DPO (θ) =−E (x,y w ,y l )∼D pref logσ β log p(y w | x;f θ ) p(y w | x;f ref ) − β log p(y l | x;f θ ) p(y l | x;f ref ) ,(105) whereβ > 0controls the sharpness of preference separation. In unlearning-style instantiations, one can construct preference pairs so that generations containing forget targets are treated as dispreferred. B.3.4 NPO [Zhang et al., 2024] Negative Preference Optimization (NPO) reformulates unlearning as a bounded, alignment-inspired objective that discourages the forget targets relative to a frozen reference model. Letf ref denote the reference model (typically the pre-unlearning checkpoint) andσ(·)the sigmoid. The NPO’s forget loss is L NPO forget (θ) =− 2 β E (x,y f )∼D forget logσ −β log p(y f | x;f θ ) p(y f | x;f ref ) ,(106) whereβ > 0is a temperature parameter. In practice, NPO is often combined with the retain regularizer αL retain (θ) to preserve utility. 30 B.3.5 SimNPO [Fan et al., 2024] SimNPO removes the explicit reference model and introduces a margin/offset term (denotedγin the original paper) while retaining the stabilized log-sigmoid form: L SimNPO forget (θ) =− 2 β E (x,y f )∼D forget logσ − β |y f | logp(y f | x;f θ )− γ ,(107) where|y f |is the target sequence length (used for normalization), andγ ≥ 0calibrates the separation threshold. As with other unlearning objectives, SimNPO is commonly paired with αL retain (θ). B.3.6 UnDIAL [Dong et al., 2025] UnDIAL (Unlearning via Self-Distillation on Adjusted Logits) stabilizes unlearning by defining an explicit, fixed target distribution and distilling the model toward it. Letf orig be the frozen pre- unlearning model. For a forget example(x,y f )and each positiont, letz orig t be the teacher logits ande y f,t the one-hot vector of the target token. UnDIAL constructs adjusted logits and a target distribution: z adj t = z orig t − γ UD e y f,t , p adj t = softmax z adj t ,(108) whereγ UD > 0controls the strength of demoting the memorized token. The self-distillation (cross-entropy) unlearning objective is L UnDIAL forget (θ) = E (x,y f )∼D forget   |y f | X t=1 H p adj t , p(·| x,y f,<t ;f θ )   ,(109) whereH(·,·)is the cross-entropy between the fixed adjusted distribution and the student model’s predictive distribution. Optionally, UnDIAL can also be combined withαL retain (θ)for utility preservation. B.3.7 SatImp [Yang et al., 2025] Saturated Importance (SatImp) is a token-wise soft reweighting strategy for enhancing unlearning. It fits into the general token-wise reweighted objective L forget (θ) = E (x,y f )∼D forget |y f | X k=1 w x,y f ,k logp(y f,k | y f,<k ,x;f θ ).(110) SatImp defines the weight function as w satimp x,y f ,k = p(y f,k | y f,<k ,x;f θ ) β 1 · 1− p(y f,k | y f,<k ,x;f θ ) β 2 ,(111) whereβ 1 ,β 2 ≥ 0control the smoothness and shape of the weight distribution. As with other objectives, SatImp is commonly paired with αL retain (θ). B.4 Experimental Settings B.4.1 Testbed Hardware Configuration All experiments are conducted on two Elastic Compute Service (ECS) instances. For experiments using Llama-3.2-1B-Instruct and Qwen2.5-1.5B-Instruct, we employ an instance equipped with an Intel Xeon Gold 6462C CPU (16 available cores), 128 GB RAM, 512 GB of available disk space, and an NVIDIA L20 GPU with 48 GB memory. For experiments using Llama- 3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-3B-Instruct, we use an instance equipped with an Intel Xeon Platinum 8469C CPU (24 available cores), 128 GB RAM, 512 GB of available disk space, and an NVIDIA HGX H20 GPU with 96 GB memory. For reproducibility and consistent performance, we recommend theecs.gn8is.4xlargeand ecs.gn8v.6xlarge instance types on Alibaba Cloud. 31 Software Environment All experiments are performed on Ubuntu 22.04.5 LTS with NVIDIA driver version 570.195.03 and CUDA 12.8. MPU framework is implemented in Python 3.12.12 using PyTorch 2.9.1, and is developed on top of OpenUnlearning framework. All unlearning algorithms are instantiated using the implementations integrated within OpenUnlearning. The baseline unlearning models are initialized from the publicly released checkpoints provided by OpenUnlearning on Hugging Face: https://huggingface.co/open-unlearning. Table 2: Default configurations and hyperparameters used in our experiments. NotationHyperparameterValueExplanation General Training Configuration –Random Seed0Seed for reproducibility –Precision bfloat16Numerical precision used for training –Attention BackendFlashAttention-2Optimized attention implementation –Optimizer paged_adamw_32bitMemory-efficient AdamW variant λ wd Weight Decay0.01ℓ 2 regularization coefficient BBatch Size32Number of samples per batch Server-Side Configuration RGlobal Communication RoundsSee Table 7Total number of server–client rounds mCopy Number2Number of perturbed model copies per round κNoise Level0.01Global multiplier for block-wise noise scalesσ ℓ α k Noise Scaling1 + k−1 m−1 Used to match noise magnitude in the single-copy Noised baseline and MPU η srv Server Step Size1.0Step size for applying the aggregated update ̄ ∆ (r) Client-Side Local Unlearning Configuration –Unlearning AlgorithmsSee Appendix B.3Unlearning trainer executed on each published copy E cli Local Client EpochSee Table 7Local unlearning epochs per copy per round η cli Client Learning Rate1× 10 −5 Learning rate used by the chosen client unlearning trainer B.4.2 Hyperparameters The key hyperparameters used in our experiments across different domains are summarized in Table 2. We configure the training schedule by selecting the number of global communication rounds (R) and local client epochs per round (E) based on the client-side unlearning method. Specifically, GradAscent, GradDiff, and UnDIAL use a single-round schedule with(R,E) = (1, 10); SimNPO, DPO, and SatImp use a more communication-intensive schedule with(R,E) = (10, 1); and NPO adopts an intermediate configuration with(R,E) = (2, 5). For any unspecified method, we default to (R,E) = (1, 10). B.4.3 Prompt Template The chat template is enabled during training. User queries and assistant responses follow the format, as illustrated in Figure 3 and Figure 4. Prompt: Llama-3.2 Series Template System Prompt: You are a helpful assistant. System Prompt with Special Tokens: <|begin_of_text|><|start_header_id|> system<|end_header_id|> are a helpful assistant.<|eot_id|> User Start Tag: <|start_header_id|>user<|end_header_id|> User End Tag: <|eot_id|> Asst Start Tag: <|start_header_id|>assistant<|end_header_id|> Asst End Tag: <|eot_id|> Data String: 10 Apr 2025 Figure 3: Prompt template for Llama-3.2 series. 32 Prompt: Qwen Series Template System Prompt: You are a helpful assistant. System Prompt with Special Tokens:<|im_start|>system are a helpful assistant.<|im_end|> User Start Tag: <|im_start|>user User End Tag: <|im_end|> Asst Start Tag: <|im_start|>assistant Asst End Tag: <|im_end|> Figure 4: Prompt template for Qwen2.5 series. B.5 Supplementary Experimental Results Table 3: Performance comparison of different unlearning algorithms using the Qwen2.5-1.5B, Llama-3.2-3B, and Qwen2.5-3B models on the TOFU benchmark (Split99). Results are reported under three settings: Clean, a noise-free baseline; Noised, a single-copy noise baseline; and MPU, usingm=2copies with noise levelκ=0.01. Higher values indicate better performance for Forget Quality, Forget Truth Ratio, and Model Utility, while values of PrivLeak closer to 0 are preferred. Unlearning Algorithms Forget Quality↑Forget Truth Ratio↑Model Utility↑PrivLeak CLEANNOISEDMPUCLEANNOISEDMPUCLEANNOISEDMPUCLEANNOISEDMPU Qwen2.5-1.5B-Instruct GRADASCENT [ACL 2023]1.43e-25.41e-20.9900.5160.5360.7409.75e-46.76e-40.34613.762.1-21.7 GRADDIFF [PMLR 2022]6.58e-56.61e-60.9900.6220.5860.7400.2970.3220.34680.976.7-21.6 DPO [NEURIPS 2023]0.2660.405 0.9900.7570.7500.7380.3050.3350.34786.087.5-21.7 NPO [COLM 2024]1.43e-22.86e-20.9190.7080.7090.7390.3050.3180.34689.490.2-21.9 SIMNPO [NEURIPS 2025]0.9900.9900.9900.7380.7420.7380.3490.3500.347-18.7-17.4-22.0 UNDIAL [NAACL 2025]1.0001.0000.9900.7690.7670.7400.3380.3330.34612.314.1-21.8 SATIMP [ICML2025]0.9190.9190.9900.7380.7400.7400.3490.3480.346-21.8-20.5-21.5 Llama-3.2-3B-Instruct GRADASCENT [ACL 2023]3.06e-115.91e-144.47e-160.1870.1570.1428.23e-57.71e-52.28e-550.653.045.9 GRADDIFF [PMLR 2022]5.41e-22.86e-25.41e-20.5040.4760.4850.4190.3840.409108.0106.0107.0 DPO [NEURIPS 2023]0.5790.7660.5790.6850.6910.6830.6340.6340.633-10.2-15.3-3.95 NPO [COLM 2024]0.9900.9900.9900.6350.6360.6340.6500.6530.65161.262.157.5 SIMNPO [NEURIPS 2025]9.71e-29.71e-29.71e-20.5500.5500.5540.6580.6620.656-62.4-72.3-64.3 UNDIAL [NAACL 2025]2.86e-25.41e-22.86e-20.5670.5660.5630.6930.6920.693-72.0-71.8-73.3 SATIMP [ICML2025]1.43e-21.43e-22.86e-20.5100.5070.5120.6590.6610.659-99.7-100.0-99.9 Qwen2.5-3B-Instruct GRADASCENT [ACL 2023]0.5791.86e-230.5790.7096.4e-200.7090.3920.0000.391-28.78.57-28.1 GRADDIFF [PMLR 2022]0.7663.02e-30.7660.7120.4630.7100.3940.2370.395-28.067.7-27.1 DPO [NEURIPS 2023]0.7660.9190.7660.7110.7330.7090.3970.4120.397-28.984.4-28.8 NPO [COLM 2024]0.7660.2660.7660.7120.7240.7110.3980.4200.397-28.590.5-28.5 SIMNPO [NEURIPS 2025]0.7660.9190.7660.7120.7260.7100.3980.3880.397-28.5-18.9-28.6 UNDIAL [NAACL 2025]0.7660.9190.7660.7100.7340.7110.3980.4050.396-28.14.17-28.8 SATIMP [ICML2025]0.7660.7660.7660.7110.7140.7120.3970.3870.397-28.9-24.9-28.7 Table 4: Performance comparison of different unlearning algorithms using the Llama-3.2-1B model under MPU on the TOFU benchmark (Split99), with varying noise levelsκ ∈ 0, 0.05, 0.1and fixed perturbed copiesm=2. Higher values indicate better performance for Forget Quality, Forget Truth Ratio, and Model Utility, while values of PrivLeak closer to 0 are preferred. Unlearning Algorithms Forget Quality↑Forget Truth Ratio↑Model Utility↑PrivLeak κ=0 κ=0.01 κ=0.05 κ=0.1 κ=0 κ=0.01 κ=0.05 κ=0.1 κ=0 κ=0.01 κ=0.05 κ=0.1 κ=0 κ=0.01 κ=0.05 κ=0.1 GRADASCENT6.76e-35.41e-20.5791.43e-20.4400.4680.5340.4301.39e-42.31e-43.06e-41.49e-467.969.668.069.1 GRADDIFF0.4050.4050.2660.4000.5460.5470.5440.5530.4680.4640.4740.47275.377.276.576.7 DPO0.2660.2660.2660.1650.6400.6410.6410.6370.5920.5910.5940.594-30.1-28.9-29.2-28.7 NPO0.9190.9190.9190.9190.6210.6280.6240.6180.5950.5970.5990.59627.728.226.828.1 SIMNPO5.41e-2 9.71e-29.71e-25.41e-20.5200.5250.5260.5250.5960.5980.5980.597-72.4-71.8-72.1-71.7 UNDIAL1.43e-21.43e-21.43e-21.43e-20.5280.5290.5270.5260.6140.6150.6150.613-78.3-78.0-78.2-78.0 SATIMP6.76e-36.76e-33.02e-36.76e-30.4740.4760.4750.4750.6010.6010.6000.600-99.3-98.9-99.2-99.2 33 Table 5:Performance comparison of different unlearning algorithms using the Llama-3.2-1B model under no-denoise setting on the TOFU benchmark (Split99), with varying noise levelsκ ∈ 0, 0.05, 0.1and fixed perturbed copiesm=2. Additionally, we multiplyκby the expected scaling factor in MPU (E k (α k ) = 1.5). Higher values indicate better performance for Forget Quality, Forget Truth Ratio, and Model Utility, while values of PrivLeak closer to 0 are preferred. Unlearning Algorithms Forget Quality↑Forget Truth Ratio↑Model Utility↑PrivLeak κ=0.015 κ=0.075 κ=0.15 κ=0.015 κ=0.075 κ=0.15 κ=0.015 κ=0.075 κ=0.15 κ=0.015 κ=0.075 κ=0.15 GRADASCENT2.81e-81.12e-91.23e-70.2460.1900.2520.0000.0004.11e-558.957.466.4 GRADDIFF0.2660.2660.4050.5330.5340.5460.4610.4660.46373.375.472.3 DPO0.1650.0970.1650.6200.6180.6200.5950.5950.595-19.8-20.0-20.0 NPO0.7660.7660.7660.6400.6230.6220.6000.5990.59732.935.433.3 SIMNPO5.41e-25.41e-25.41e-20.5220.5220.5230.5920.5930.597-70.2-71.8-72.1 UNDIAL1.43e-21.43e-21.43e-20.5270.5260.5270.6140.6140.616-77.4-77.3-77.2 SATIMP6.76e-36.76e-36.76e-30.4700.4710.4710.5970.5970.598-99.1-99.2-99.2 Table 6:Performance comparison of different unlearning algorithms using the Llama-3.2-1B model under MPU on the TOFU benchmark (Split99), with varying numbers of perturbed copies m∈2, 3, 4and noise levelκ=0.01. Higher values indicate better performance for Forget Quality, Forget Truth Ratio, and Model Utility, while values of PrivLeak closer to 0 are preferred. Unlearning Algorithms Forget Quality↑Forget Truth Ratio↑Model Utility↑PrivLeak m=2m=3 m=4 m=2 m=3 m=4 m=2 m=3 m=4 m=2 m=3 m=4 GRADASCENT5.41e-20.1652.16e-50.4680.5000.3342.31e-41.68e-40.00069.664.063.8 GRADDIFF0.4050.4050.5790.5470.5550.5700.4640.4670.46777.277.777.4 DPO0.2660.5790.1650.6410.6310.6150.5910.5920.594-28.9-22.0-22.2 NPO0.9190.7660.7660.6280.6230.6200.5970.5900.59428.225.325.1 SIMNPO9.71e-25.41e-25.41e-20.5250.5180.5190.5980.5970.599-71.8-74.6-74.1 UNDIAL1.43e-21.43e-26.76e-30.5290.5280.5280.6150.6130.615-78.0-78.0-78.4 SATIMP6.76e-3 1.43e-23.02e-30.4760.4730.4710.6010.5980.599-98.9-99.2-99.8 Table 7: Performance comparison of different unlearning algorithms using the Llama-3.2-1B model under MPU on the TOFU benchmark, across varying global communication rounds (R) and local client epochs (E), with fixed perturbed copies m=2 and noise level κ=0.01. Higher values indicate better performance for Forget Quality and Model Utility, while values of PrivLeak closer to0are preferred. Unlearning Algorithms Forget Quality↑Model Utility↑PrivLeak R1E10R2E5R5E2R10E1R1E10R2E5R5E2R10E1R1E10R2E5R5E2R10E1 GRADASCENT5.41e-21.86e-231.86e-231.86e-232.31e-40.000.0000.00069.635.2-2.2425.6 GRADDIFF0.4051.43e-25.04e-41.95e-100.4640.4840.4940.57877.286.888.988.9 DPO5.41e-20.1650.1650.2660.5840.5910.5920.591-49.5-42.1-17.9-28.9 NPO0.766 0.9190.9190.9190.5840.5970.5920.59451.828.234.740.7 SIMNPO2.86e-22.86e-2 9.71e-29.71e-20.5990.6010.5980.598-78.0-80.5-75.6-71.8 UNDIAL1.43e-26.76e-31.43e-26.76e-30.6150.6170.6120.615-78.0-82.8-77.6-84.2 SATIMP6.76e-33.02e-36.76e-36.76e-30.6000.5990.5980.601-99.9-99.8-99.3-98.9 Table 8: Performance comparison of different unlearning algorithms using the Llama-3.2-1B model under MPU on the TOFU benchmark, across varying split strategies (FORGET01, FORGET05, and FORGET10), with fixed perturbed copiesm=2and noise levelκ=0.01. Higher values indicate better performance for Forget Quality, Forget Truth Ratio, and Model Utility, while values of PrivLeak closer to 0 are preferred. Unlearning Algorithms Forget Quality↑Forget Truth Ratio↑Model Utility↑PrivLeak FORGET01FORGET05FORGET10FORGET01FORGET05FORGET10FORGET01FORGET05FORGET10FORGET01FORGET05FORGET10 GRADASCENT5.41e-21.94e-1191.06e-2390.4681.71e-231.74e-222.31e-40.0000.00069.6-28.0-21.6 GRADDIFF0.4055.99e-1058.51e-2370.5470.0023.09e-100.4640.4210.26477.256.861.8 DPO0.2664.30e-39.07e-80.6410.5740.5450.5910.5950.598-28.9-43.7-59.0 NPO0.9190.1124.46e-60.6280.6030.5550.5970.6140.62328.2-4.00-0.946 SIMNPO9.71e-24.61e-75.42e-130.5250.5140.4980.5980.5950.598-71.8-77.7-82.0 UNDIAL1.43e-22.44e-107.98e-170.5290.5210.5310.6150.6150.616-78.0-91.7-94.9 SATIMP6.76e-31.33e-132.81e-200.4760.4710.4690.6010.5930.597-98.9-99.9-99.2 34 Table 9: Baseline performance of the OpenUnlearning Framework without MPU, using the Llama- 3.2-1B and Llama-3.2-3B models under MPU on the TOFU benchmark (Split99). Higher values indicate better performance for Forget Quality, Forget Truth Ratio, and Model Utility, while values of PrivLeak closer to 0 are preferred. Unlearning Algorithms Forget Quality↑Forget Truth Ratio↑Model Utility↑PrivLeak FORGET01FORGET05FORGET10FORGET01FORGET05FORGET10FORGET01FORGET05FORGET10FORGET01FORGET05FORGET10 Llama-3.2-1B-Instruct GRADASCENT6.58e-51.94e-1191.06e-2390.3552.12e-231.27e-220.0000.0000.00065.8-28.4-21.1 GRADDIFF0.4055.99e-1058.51e-2370.5350.0047.22e-100.4610.3930.24777.156.861.3 DPO0.1652.08e-33.08e-70.6370.5730.5450.5910.5940.598-25.5-44.5-57.8 NPO0.9190.1124.58e-70.6240.5950.5510.5990.6140.61930.6-2.113.78 SIMNPO5.41e-21.46e-72.73e-120.5260.5100.4980.5980.5930.597-68.4-76.6-82.4 UNDIAL1.43e-24.87e-104.24e-170.5300.5240.5310.6130.6150.618-76.4-91.6-94.8 SATIMP3.02e-32.96e-131.12e-190.4740.4710.4690.6000.5940.600-98.9-99.9-99.2 Llama-3.2-3B-Instruct GRADASCENT3.06e-111.94e-1191.06e-2390.1876.63e-84.79e-208.23e-50.0000.00050.6-22.5-19.2 GRADDIFF5.41e-21.94e-1198.51e-2370.5042.01e-91.20e-100.4190.5930.34510856.364.6 DPO0.5791.18e-22.13e-60.6850.5880.5940.6340.6350.634-10.2-62.0-69.1 NPO0.9900.7932.99e-20.6350.6170.5810.6500.6750.67361.29.83-24.0 SIMNPO9.71e-23.60e-91.03e-140.5500.4990.5010.6580.6520.659-62.4-81.0-86.3 UNDIAL2.86e-23.08e-128.08e-220.5670.5070.5150.6930.7010.696-72.0-92.1-94.3 SATIMP1.43e-22.96e-132.05e-240.5100.4630.4690.6590.6490.659-99.7-100-99.5 Table 10: Results of Attack Tests among Six Methods with Qwen2.5-1.5B-Instruct on the TOFU Benchmark(Split99). The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsLOSSZLibGradNormMinKMinK++Reference GRADASCENT0.39/0.5870.388/0.6080.565/0.7270.385/0.590.431/0.6320.28/0.993 GRADDIFF0.383/0.5890.416/0.6090.479/0.7340.378/0.5890.388/0.6360.128/0.992 NPO0.363/0.5890.359/0.6110.546/0.7310.354/0.5910.39/0.640.246/0.999 SIMNPO0.584/0.5890.604/0.6080.727/0.7340.586/0.5910.634/0.6360.961/0.998 UNDIAL0.46/0.5890.47/0.6080.379/0.7330.512/0.5910.609/0.6390.463/0.999 SATIMP0.589/0.5880.606/0.6110.732/0.7340.585/0.5890.641/0.6420.977/0.995 Table 11: Results of LoRA compatibility with Llama-3.1-8B-Instruct on the TOFU Bench- mark(Split99). The results are shown by comparison of CLEAN/MPU, with hyperparameterm = 2 and κ = 0.01 by default. AlgorithmsForget QualityForget Truth RatioModel UtilityPrivLeak GRADASCENT0.919/0.9190.721/0.7210.427/0.426-11.3/-10.9 GRADDIFF0.919/0.9190.721/0.7240.427/0.426-9.78/-8.85 NPO0.919/0.990.723/0.7240.429/0.429-9.95/-9.55 SIMNPO1.0/1.00.735/0.7330.396/0.411-12.0/-11.6 DPO1.0/0.990.731/0.7320.426/0.426-8.03/-6.98 UNDIAL0.99/0.990.733/0.7310.376/0.381-12.8/-12.8 SATIMP0.99/1.00.733/0.7340.394/0.411-11.9/-12.0 Table 12: Results of Qwen2.5-1.5B-Instruct on the TOFU(Split99) Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsForget QualityForget Truth RatioModel UtilityPrivLeak GRADASCENT1.43e-2/0.9900.516/0.7409.75e-4/0.34613.7/-21.7 GRADDIFF6.58e-5/0.9900.622/0.7400.297/0.34680.9/-21.6 DPO0.266/0.9900.757/0.7380.305/0.34786.0/-21.7 NPO1.43e-2/0.9190.708/0.7390.305/0.34689.4/-21.9 SIMNPO0.990/0.9900.738/0.7380.349/0.347-18.7/-22.0 UNDIAL1.000/0.9900.769/0.7400.338/0.34612.3/-21.8 SATIMP0.919/0.9900.738/0.7400.349/0.346-21.8/-21.5 35 Table 13: Results of Qwen2.5-3B-Instruct on the TOFU(Split99) Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsForget QualityForget Truth RatioModel UtilityPrivLeak GRADASCENT0.579/0.5790.709/0.7090.392/0.391-28.7/-28.1 GRADDIFF0.766/0.7660.712/0.7100.394/0.395-28.0/-27.1 DPO0.766/0.7660.711/0.7090.397/0.397-28.9/-28.8 NPO0.766/0.7660.712/0.7110.398/0.397-28.5/-28.5 SIMNPO0.766/0.7660.712/0.7100.398/0.397-28.5/-28.6 UNDIAL0.766/0.7660.710/0.7110.398/0.396-28.1/-28.8 SATIMP0.766/0.7660.711/0.7120.397/0.397-28.9/-28.7 Table 14: Results of Llama-3.2-1B-Instruct on the MUSE-Books Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.00915/0.009150/00/0-32.0/-34.5 GRADDIFF0.00915/0.009150.0455/0.03310.146/0.14128.9/31.7 NPO0.0125/0.01270.172/0.1810.188/0.191-23.7/-24.0 SIMNPO0.0122/0.01250.161/0.1520.206/0.218-30.1/-28.5 UNDIAL0.0112/0.01120.167/0.1830.204/0.195-56.9/-57.2 SATIMP0.00924/0.009240.152/0.1470.202/0.198-2.22/-1.87 Table 15: Results of Llama-3.2-3B-Instruct on the MUSE-Books Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.00915/0.009150/00/052.3/56.6 GRADDIFF0.00915/0.009150/00.115/0.1467.42/35.4 NPO0.0155/0.0150.234/0.2260.313/0.333-71.9/-73.2 SIMNPO0.0143/0.01410.237/0.2270.32/0.316-84.2/-83.6 UNDIAL0.0109/0.01060.243/0.2290.359/0.316-89.6/-89.6 SATIMP0.00915/0.009150.221/0.2390.315/0.3-28.8/-29.8 Table 16: Results of Qwen2.5-1.5B-Instruct on the MUSE-Books Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.0148/0.01490.22/0.220.24/0.242-29.7/-29.7 GRADDIFF0.0143/0.0140.218/0.2140.281/0.277-29.3/-28.9 NPO0.0147/0.01440.221/0.2230.306/0.304-26.0/-25.8 SIMNPO0.0144/0.01470.221/0.2210.294/0.297-25.9/-26.0 UNDIAL0.0146/0.01420.226/0.2150.273/0.282-27.8/-27.8 SATIMP0.0147/0.01460.227/0.2180.297/0.303-25.9/-25.9 Table 17: Results of Qwen2.5-3B-Instruct on the MUSE-Books Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.0172/0.01750.214/0.2180.238/0.247-56.1/-55.9 GRADDIFF0.0159/0.01550.191/0.2130.229/0.201-57.9/-57.1 NPO0.0145/0.01480.206/0.1860.194/0.198-53.1/-52.9 SIMNPO0.0147/0.01460.182/0.1860.205/0.2-52.9/-53.0 UNDIAL0.0136/0.01360.151/0.1350.129/0.13-48.2/-48.6 SATIMP0.0147/0.01470.193/0.2080.192/0.219-52.6/-53.1 36 Table 18: Results of Llama-3.2-1B-Instruct on the MUSE-News Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.00975/0.009750/00/019.3/19.2 GRADDIFF0.00985/0.010.0381/0.03460.0292/0.012379.7/82.9 NPO0.0279/0.0280.163/0.1670.178/0.178-28.0/-27.5 SIMNPO0.0282/0.02790.155/0.1690.181/0.175-52.0/-52.0 UNDIAL0.016/0.0160.136/0.1370.21/0.207-49.0/-49.0 SATIMP0.0281/0.0280.173/0.170.179/0.176-52.2/-52.1 Table 19: Results of Llama-3.2-3B-Instruct on the MUSE-News Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.00975/0.009750/00/014.2/15.4 GRADDIFF0.0142/0.01320/00/0.02664.1/68.3 NPO0.0271/0.02830.234/0.2130.255/0.233-64.5/-64.9 SIMNPO0.0282/0.02880.236/0.2310.213/0.216-82.7/-82.6 UNDIAL0.0136/0.01340.236/0.2380.276/0.256-81.6/-81.6 SATIMP0.0288/0.02780.224/0.2320.213/0.212-82.5/-82.8 Table 20: Results of Qwen2.5-1.5B-Instruct on the MUSE-News Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.0242/0.02460.196/0.1960.168/0.159-18.7/-18.8 GRADDIFF0.0279/0.02790.149/0.1470.2/0.193-19.0/-18.9 NPO0.0278/0.02590.123/0.1250.149/0.159-15.7/-15.8 SIMNPO0.0263/0.0280.129/0.1290.159/0.156-15.7/-15.6 UNDIAL0.0259/0.02630.107/0.1030.119/0.123-15.3/-15.3 SATIMP0.0261/0.0280.125/0.1260.15/0.157-15.6/-15.8 Table 21: Results of Qwen2.5-3B-Instruct on the MUSE-News Benchmark. The results are shown by comparison of CLEAN/MPU, with hyperparameter m = 2 and κ = 0.01 by default. AlgorithmsExtraction StrengthForget KnowMem ROUGERetain KnowMem ROUGEPrivLeak GRADASCENT0.0254/0.02540.042/0.04130.0738/0.067-39.5/-39.4 GRADDIFF0.031/0.03090.0658/0.0760.0636/0.0588-41.3/-41.0 NPO0.0244/0.02780.063/0.06390.0694/0.0727-28.5/-28.6 SIMNPO0.0244/0.02450.0644/0.06020.0702/0.0703-28.6/-28.7 UNDIAL0.023/0.02340.0653/0.06410.0713/0.0758-27.7/-28.1 SATIMP0.0238/0.02790.0621/0.06340.0689/0.0664-28.3/-28.5 B.6 Supplementary Experimental Analysis B.6.1 Qwen Result As shown in Table 3, MPU consistently outperforms all baseline methods across the four evaluation metrics, demonstrating strong stability under different unlearning settings. Across various unlearning algorithms, MPU exhibits only minor performance variations and achieves superior results compared to the CLEAN and NOISED baselines in most cases. In particular, MPU maintains highly stable Forget Quality scores, all exceeding 0.9. In contrast, GradAscent, GradDiff, and NPO yield FQ values below 0.01, indicating ineffective unlearning. Moreover, MPU achieves higher Model Utility than the baseline methods in five out of seven unlearning algorithms, suggesting that it better preserves model performance while avoiding excessive unlearning. B.6.2 Effect of Copy Number m In Table 6, we study the sensitivity of MPU to the number of published perturbed copies by varying m ∈ 2, 3, 4while keeping the noise level and all other hyperparameters identical to Table 1. 37 Overall, increasingmcan improve stability for high-variance or unstable unlearning routines, but the gains are not necessarily monotonic and may saturate (or regress) for certain algorithms. For relatively stable algorithms (e.g., UnDIAL and SatImp), metrics are largely invariant tom. Specifically, UnDIAL maintains FQ≈ 0.014, and MU≈ 0.615form=2andm=3, with only a minor FQ drop atm=4. Similarly, SatImp remains within a narrow band for FTR/MU and exhibits only small FQ variation. This suggests that when the underlying unlearning update is already stable, multi-copy aggregation mainly provides robustness rather than large performance shifts. For more hyperparameter-sensitive unlearning algorithms, moderate increases inmcan substantially improve forgetting. For example, GradAscent improves from FQ0.054atm=2to0.165atm=3, alongside a higher FTR (0.468→ 0.500). Likewise, DPO achieves its best FQ atm=3(FQ0.579 versus0.266atm=2), and GradDiff benefits from a larger copy number, reaching FQ0.579at m=4. However, these gains are not uniformly monotonic: GradAscent collapses atm=4(FQ ≈ 2.16 × 10 −5 ), and DPO also drops atm=4. These patterns indicate that while multi-copy aggregation can stabilize updates, overly aggressive averaging can interact unfavorably with certain training dynamics. Finally, privacy leakage is generally comparable acrossm, with modest improvements for some methods. For instance, NPO reduces|PrivLeak|from28.2atm=2to25.3and25.1atm=3and m=4. Taken together,m=2provides a strong and efficient default, whilem=3can be a favorable choice for stabilizing unstable unlearning algorithms (e.g., GradDiff) without materially sacrificing utility. B.6.3 Effect of Noise Level κ In Table 4, we next vary the noise levelκ ∈ 0, 0.01, 0.05, 0.1while keeping all other settings fixed (Table 4). Across most algorithms, MPU is robust to a broad range ofκ, with FTR and MU remaining largely stable and only modest variations in privacy leakage. For strong and stable unlearning methods, performance is nearly invariant toκ. Most notably, NPO maintains identical Forget Quality (FQ= 0.920) across all tested noise levels, while FTR and MU remain in a narrow range. Similarly, UnDIAL keeps FQ= 0.014and FTR≈ 0.53across allκ, and SatImp shows only a small dip in FQ at κ=0.05. For instability-prone methods, moderate noise can act as a stabilizer. GradAscent exhibits a clear non-monotonic trend: FQ increases from0.007atκ=0to0.054atκ=0.01, peaks at0.580for κ=0.05, and then drops again atκ=0.1. This suggests that intermediate noise can regularize unstable unlearning dynamics, whereas overly large noise begins to erode the usefulness of the client update. A similar (but weaker) effect appears for SimNPO, where small-to-moderate noise improves FQ from 0.054 to 0.097. Overall, these results indicate that MPU tolerates noise injection well, and that small-to-moderate κcan improve stability for certain algorithms without harming model utility. In our default setting, κ=0.01 provides a balanced operating point that performs competitively across methods. B.6.4 Round–Epoch Allocation Under Fixed Total Local Compute Table 7 studies how the training schedule affects unlearning when the total number of local epochs is fixed to10(consistent with the default setting in OpenUnlearning), but allocated differently across global roundsRand local epochs per roundE. Overall, we observe a round–epoch trade-off: increasing the number of rounds (hence smallerE) often improves MU but can weaken FQ, while concentrating training into fewer rounds with largerEcan strengthen forgetting for some algorithms, but may risk over-updating in others. Different unlearning algorithms exhibit distinct sensitivity to the schedule. GradAscent performs best with R1E10, while increasing the number of rounds causes both FQ and MU to collapse to nearly zero, indicating severe instability under repeated round-wise aggregation. In contrast, NPO benefits from R2E5, achieving substantially higher FQ than R1E10 while also improving MU. Similarly, SimNPO prefers more rounds for stronger forgetting, whereas DPO improves forgetting monotonically as the number of rounds increases. 38 B.6.5 No-Denoising Ablation Under Matched Effective Noise To isolate the importance of MPU’s denoising aggregation, we evaluate a no-denoise baseline in which the client trains on a single noisy model and no inverse reparameterization or aggregation is applied (Table 5). To ensure a fair comparison, we scale the baseline noise such that the effective noise magnitude matches MPU by multiplying κ by E k (α k )=1.5. No-denoise training is markedly less reliable for unstable unlearning algorithms. In particular, GradAscent essentially fails under no-denoise: its FQ remains extremely close to zero (on the order of10 −9 to10 −7 ), and MU is approximately zero across all tested noise levels. This contrasts sharply with MPU (Table 4), where GradAscent can achieve strong forgetting with FQ 0.579 at κ=0.05. For moderately stable methods, the no-denoise baseline often requires substantially larger noise to approach comparable forgetting. For example, GradDiff reaches its best FQ (0.405) only at the largest tested noise (κ=0.15), whereas MPU attains comparable FQ at much smaller noise (e.g.,κ=0.01or evenκ=0). By contrast, for already-stable methods such as NPO and UnDIAL, no-denoise does not provide consistent improvements and often matches (but does not surpass) MPU. Overall, this ablation supports that multi-copy denoising and harmonic aggregation are key contribu- tors to MPU’s stability—especially for high-variance unlearning updates—enabling strong forgetting at lower noise without sacrificing utility. B.6.6 Robustness to Forget-Split Strategies We evaluate MPU under different forget ratios by increasing the amount of data to be forgotten from1%(FORGET01) to5%(FORGET05) and10%(FORGET10), while fixingκ=0.01andm=2 (Table 8). A consistent trend emerges: larger forget splits substantially degrade Forget Quality across nearly all algorithms, indicating that unlearning becomes significantly more challenging as the forget request scales up. Many methods exhibit dramatic FQ collapse when moving from FORGET01 to FORGET05/10. For instance, GradDiff drops from FQ0.405at FORGET01 to effectively zero (e.g.,5.99× 10 −105 and 8.51× 10 −237 ) at FORGET05 and FORGET10. Even the strongest unlearning algorithm in our comparison, NPO, decreases from FQ0.919at FORGET01 to0.112at FORGET05 and4.46× 10 −6 at FORGET10. This suggests that scaling to larger forget partitions likely requires additional tuning (e.g., more rounds, different learning rates, or algorithm-specific regularization) beyond the default hyperparameters tuned for the FORGET01 setting. Interestingly, model utility is comparatively more robust for several algorithms. SimNPO, DPO, and UnDIAL maintain MU around0.59–0.62across all splits, even when FQ collapses. Privacy leakage exhibits mixed behavior: NPO reduces|PrivLeak|strongly as the forget split increases (from 28.2to−4.0to−0.946), whereas DPO and UnDIAL show increased leakage in magnitude (e.g., −28.9→−43.7→−59.0and−78.0→−91.7→−94.9). These results highlight that scaling the forget ratio can introduce new privacy–forgetting trade-offs that vary across algorithms. B.6.7 Scaling MPU across Model Sizes Finally, we test whether MPU scales to larger base models by comparing results on Llama-3.2-1B- Instruct and Llama-3.2-3B-Instruct under multiple forget ratios (Table 9). Overall, MPU remains effective at larger scales, and larger models often provide improved headroom for strong unlearning algorithms—especially under more challenging forget ratios. Under FORGET01, the 3B model generally attains equal or better forgetting quality and higher utility for several algorithms. For example, NPO improves from FQ0.919(1B) to0.990(3B) while increasing MU from roughly0.599to0.650. Similarly, preference-based unlearning (DPO) improves substantially with scale, rising FQ from0.165(1B) to0.579(3B) and MU from0.591to0.634. These patterns suggest that higher-capacity models can better accommodate targeted forgetting updates while preserving general capabilities. The benefit of scale is even clearer for larger forget splits. For NPO, the 3B model retains strong FQ at FORGET05 (FQ0.793), whereas the 1B model drops to FQ0.112. At FORGET10, the 3B model still achieves non-trivial forgetting (FQ0.030), while the 1B model collapses toward zero. In 39 contrast, some algorithms remain fragile regardless of scale: GradAscent yields near-zero FQ even at FORGET01 for both 1B and 3B, and GradDiff degrades sharply for FORGET05/10 on both models. For privacy leakage, the sign ofPrivLeakcan vary, and thus the magnitude (distance to zero) is the relevant indicator. In many cases (e.g., NPO), the magnitude decreases as the forget ratio increases, whereas other methods exhibit large-magnitude values that worsen under harder forget splits. Overall, Table 9 indicates that MPU scales favorably with model size for the most effective unlearning algorithms, and that larger models can better sustain non-trivial forgetting under more demanding forget requests. B.6.8 Robustness Against Extraction and Inference Attacks To rigorously evaluate the privacy-preserving capabilities of MPU, we subject the unlearned models to a suite of membership inference and extraction attacks, including LOSS, ZLib, GradNorm, MinK, and MinK++ (Table 10). Across all evaluated unlearning algorithms on the Qwen2.5-1.5B- Instruct model, MPU elevates the attack metrics compared to the CLEAN baseline. For instance, under GRADASCENT, the MinK metric rises from0.385to0.590, and MinK++ increases from 0.431to0.632. Higher metric values under MPU indicate that the model’s responses to forget-set probes are statistically indistinguishable from holdout data, effectively neutralizing the attacks. This demonstrates that communicating reparameterized, perturbed copies successfully obscures fine- grained parameter updates. Because all injected noise is strictly designed to be cancellable during the server-side harmonic aggregation, the model maintains robust defenses against gradient-based extraction that an attacker might exploit. B.6.9 Compatibility with Parameter-Efficient Fine-Tuning (LoRA) To assess the scalability of MPU to larger architectures under strict memory constraints, we integrate it with LoRA on the Llama-3.1-8B-Instruct model (Table 11). The results indicate exceptional architectural compatibility. MPU achieves near-perfect Forget Quality (FQ ≥ 0.919, frequently reaching the maximum of1.0) across all tested algorithms while maintaining stable Forget Truth Ratios (∼ 0.72–0.73) and reliable Model Utility (∼ 0.38–0.42). Furthermore, the magnitude of PrivLeak remains tightly bounded within an acceptable range (e.g.,−6.98to−12.8). This establishes that MPU is highly adaptable to Parameter-Efficient Fine-Tuning (PEFT) paradigms, enabling secure, multi-copy unlearning on massive LLMs without necessitating full-parameter updates. B.6.10 Comprehensive Evaluation on the MUSE Benchmark Beyond the synthetic TOFU dataset, we extend our evaluation to the highly challenging MUSE benchmark, which evaluates unlearning across real-world continuous corpora (News and Books) while strictly monitoring verbatim extraction and knowledge retention. Tables 14 through 21 present the comprehensive results across both the Llama-3.2 and Qwen-2.5 model families at the 1.5B and 3B parameter scales. Across both the MUSE-News and MUSE-Books subsets, MPU consistently matches or exceeds the CLEAN baseline in suppressing Extraction Strength and Forget KnowMem ROUGE. For instance, leveraging stable methods like GRADASCENT and GRADDIFF under MPU successfully drives Forget KnowMem ROUGE to exactly0or near-0in the Llama-3.2-1B and 3B models, fulfilling the data owner’s expectation of absolute forgetting. At the same time, Retain KnowMem ROUGE is largely preserved, maintaining parity with the CLEAN baseline and satisfying the deployer’s requirement for general utility. The fact that MPU maintains this delicate balance across diverse domains (News and Books) and varying model architectures (Llama and Qwen) highlights the generalizability of our dual non- disclosure constraint. By ensuring that the localized noise is completely cancelled during the Post- Process aggregation phase, MPU achieves state-of-the-art unlearning efficacy without the persistent structural degradation typical of naive noise injection. 40

Instant research discovery