Paper deep dive
A Model Merging Approach for Continual MLLM Unlearning
Yuhang Wang, Linlin Zhang, Haoxuan Ji, Xianmin Ye, Zhenxing Niu, Haichang Gao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language model (MLLM) unlearning methods have been proposed to remove private, sensitive, or proprietary information from well-trained models. However, most existing MLLM unlearning methods are designed for one-shot requests and fail to adequately address continual scenarios, as repeatedly applying one-shot operations leads to cumulative utility degradation, unlearning rebound, and retention drift. We introduce Merging for Continual Unlearning (MCU), an approach that dynamically merges multiple one-shot unlearning adapters into a unified adapter upon receiving each new unlearning this http URL a leave-one-out merging analysis, we reveal that these unlearning adapters exhibit strong cross-task dependencies. Such dependencies have two contrasting effects: they can facilitate cross-task unlearning transferability, but they can also introduce severe interference that degrades unlearning effectiveness and compromises retained knowledge. To address this challenge, MCU projects the adapters into a shared representation space, preserves their dominant directions, suppresses over-concentrated coordinates, and reconfigures cross-task dependencies to mitigate interference while enhancing transferability. Experiments on ICU-Bench and MLLMU-Bench demonstrate that MCU achieves superior unlearning effectiveness while preserving both retained knowledge and general multimodal utility.
Tags
Links
- Source: https://arxiv.org/abs/2608.04548v1
- Canonical: https://arxiv.org/abs/2608.04548v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
90,799 characters extracted from source content.
Expand or collapse full text
A Model Merging Approach for Continual MLLM Unlearning Yuhang Wang1, Linlin Zhang1, Haoxuan Ji2, Xianmin Ye1, Zhenxing Niu1, Haichang Gao1 Abstract Multimodal large language model (MLLM) unlearning methods have been proposed to remove private, sensitive, or proprietary information from well-trained models. However, most existing MLLM unlearning methods are designed for one-shot requests and fail to adequately address continual scenarios, as repeatedly applying one-shot operations leads to cumulative utility degradation, unlearning rebound, and retention drift. We introduce Merging for Continual Unlearning (MCU), an approach that dynamically merges multiple one-shot unlearning adapters into a unified adapter upon receiving each new unlearning request. Through a leave-one-out merging analysis, we reveal that these unlearning adapters exhibit strong cross-task dependencies. Such dependencies have two contrasting effects: they can facilitate cross-task unlearning transferability, but they can also introduce severe interference that degrades unlearning effectiveness and compromises retained knowledge. To address this challenge, MCU projects the adapters into a shared representation space, preserves their dominant directions, suppresses over-concentrated coordinates, and reconfigures cross-task dependencies to mitigate interference while enhancing transferability. Experiments on ICU-Bench and MLLMU-Bench demonstrate that MCU achieves superior unlearning effectiveness while preserving both retained knowledge and general multimodal utility. Introduction Multimodal large language models (MLLMs) may encode private, proprietary, copyrighted, or outdated multimodal information from their training data (Pi et al. 2024; Li et al. 2024a; Cohen et al. 2025). Machine unlearning seeks to remove specified information while preserving unrelated capabilities (Garg et al. 2020; Gupta et al. 2021; Sekhari et al. 2021). In practice, deletion requests often arrive sequentially, and existing continual unlearning methods typically process each incoming request by applying a new one-shot model or adapter update to the current model (Gao et al. 2025; Shi et al. 2025; Kawakami et al. 2025). As illustrated in Fig. 1, this sequential paradigm couples each request to preceding model states, which can overwrite earlier unlearning effects and accumulate parameter drift, resulting in unlearning rebound, retention drift, and utility degradation (Li et al. 2026; Wang et al. 2026b). The key challenge is therefore to accommodate an expanding sequence of deletion requests while preserving both historical unlearning effects and general multimodal capabilities. Figure 1: Comparison between sequential continual unlearning and the proposed MCU. Existing methods repeatedly update the current model, coupling new requests to preceding model states and accumulating unlearning rebound, retention drift, and utility degradation. To address this challenge, we formulate continual multimodal unlearning as a model-merging problem. Specifically, we regard each task-specific unlearning adapter as a task vector and achieve continual unlearning by merging these task vectors into a unified adapter. Since all one-shot adapters are independently derived from the same base model, their updates can be represented and jointly processed in a shared parameter space without repeatedly modifying the current model. Through a leave-one-out merging study, we observe that the knowledge associated with a specific target remains partially forgotten even when its corresponding adapter is excluded from the merge, as shown in Fig. 3. This indicates that adapters trained for other requests can also contribute to forgetting that target, revealing shared or overlapping unlearning directions across tasks. We refer to this phenomenon as cross-task unlearning transfer. Recent studies (Lin et al. 2026) show that unlearning adapters exhibit structures distinct from ordinary capability adaptation, often concentrating their effects within a limited set of parameters or update directions. To characterize the structures underlying cross-task unlearning transfer, we analyze the task singular directions of different adapters in a shared space, following structural analyses of model merging (Gargiulo et al. 2025). We find substantial dependencies among their dominant singular directions. These dependencies may be synergistic, reinforcing unlearning across requests, or interfering, weakening target unlearning or damaging retained knowledge. Under direct merging, synergistic dependencies can transfer and reinforce unlearning effects across requests, whereas interference can cancel target updates or broaden suppression to retained knowledge. Indiscriminately removing all cross-task dependencies is also undesirable because it may eliminate useful synergy. Continual unlearning through adapter merging therefore poses three coupled objectives: preserving retained knowledge, exploiting cross-task synergy, and reducing interference. To address this challenge, we propose Merging for Continual Unlearning (MCU), a model-merging framework that consolidates accumulated one-shot unlearning adapters into a unified model update. We treat each adapter-induced update as an unlearning task vector and jointly merge the vectors derived from the same base model. MCU first maps these task vectors into a shared space, where it preserves their dominant directions and controls over-concentrated coordinates. It then performs dependency-aware direction reconfiguration to suppress antagonistic cross-task interactions while retaining beneficial shared structures that support cross-task unlearning transfer. Finally, the reconfigured task vectors are merged and reconstructed as a unified adapter. Extensive experiments demonstrate that MCU enables one-shot unlearning adapters to support long-horizon continual unlearning, reducing unlearning rebound and retention drift while preserving effective unlearning and retained knowledge. Our contributions are summarized as follows: • We introduce a merging-based framework for continual unlearning that replaces repeated model modification with the dynamic merging of one-shot unlearning adapters, and uncover cross-task unlearning transfer among these adapters. • We develop MCU, which preserves dominant task structures, controls over-concentrated coordinates, and reconfigures cross-task dependencies in a shared space to suppress antagonistic interactions while preserving beneficial unlearning transfer. • Extensive experiments on ICU-Bench and MLLMU-Bench demonstrate that MCU enables one-shot unlearning adapters to support long-horizon continual unlearning, achieving superior unlearning effectiveness while preserving retained knowledge. Related Work Machine Unlearning. Machine unlearning removes designated training influence without full retraining. Representative approximate approaches use gradient-based objectives, distribution-preserving regularization, or preference optimization (Thudi et al. 2022; Liu et al. 2022; Maini et al. 2024; Rafailov et al. 2023; Zhang et al. 2024). Recent work extends unlearning to MLLMs by editing modality-specific neurons, visual modules, or cross-modal pathways (Huo et al. 2025; Liu et al. 2025b; Li et al. 2024b; Wang et al. 2026c, 2025). Static MLLM unlearning is evaluated by benchmarks such as MU-Bench, PEBench, MLLMU-Bench, CLEAR, UMU-Bench, and ForgetMe (Cheng and Amiri 2024; Xu et al. 2025; Liu et al. 2025a; Dontsov et al. 2025; Wang et al. 2026a; Yu et al. 2025), while MLUBench and ICU-Bench study sequential deletion requests (Li et al. 2026; Wang et al. 2026b). Unlike methods that update the model separately or sequentially for each request, MCU merges independently obtained unlearning adapters and explicitly models their cross-request dependencies. Model Merging. Model merging combines multiple models or task-specific adapters into a single model. Representative approaches include Fisher-weighted merging, Model Soups, and Task Arithmetic (Matena and Raffel 2022; Wortsman et al. 2022; Ilharco et al. 2022). To mitigate task interference, TIES-Merging resolves parameter-level conflicts, while DARE reduces adapter redundancy through random dropping and rescaling (Yadav et al. 2023; Yu et al. 2024). Other studies investigate merge scaling and importance-aware weighting (Yadav et al. 2024; Lee et al. 2025). Core Space merging aligns LoRA adapters in a shared low-rank basis (Panariello et al. 2026), whereas TSV uses task singular directions to characterize and reduce cross-task interference (Gargiulo et al. 2025). Figure 2: Overall framework of MCU. (1) One-shot unlearning LoRA adapters derived from the same base model are projected into a shared space and decomposed into task singular directions. (2) Dominant direction selection and channel capacity control preserve the principal task structure while discarding low-contribution tail directions and rescaling over-concentrated core-space coordinates. (3) The processed task directions undergo dependency-aware Gram reconfiguration and Procrustes recovery before being merged in the core space and mapped back to the original parameter space. Method Adapter Merging and Cross-Task Dependencies Following the merging-based perspective introduced above, we first formalize the continual adapter-merging setting and characterize the cross-task dependencies that motivate the design of MCU. Dynamic adapter merging. Let ℳ0M_0 denote a pretrained MLLM with parameters θ0 _0, and let =(ft,rt)t=1T.T=\(D_f^t,D_r^t)\_t=1^T. denote a sequence of unlearning requests. For each request t, an existing one-shot unlearning method independently trains a LoRA adapter from the same base model ℳ0M_0. For the l-th linear layer, the resulting parameter update is ΔWtl=BtlAtl,Wtl=W0l+ΔWtl, W_t^l=B_t^lA_t^l, W_t^l=W_0^l+ W_t^l, (1) where AtlA_t^l and BtlB_t^l are the low-rank LoRA factors, with the standard LoRA scaling absorbed into the update (Hu et al. 2022). Because all adapters are learned relative to the same initialization, their updates can be directly compared and merged. At continual step s, after receiving the first s requests, MCU dynamically merges the corresponding one-shot adapters into a unified adapter: ΔWMCU,sl W_MCU,s^l =Merge(ΔW1l,…,ΔWsl), =Merge ( W_1^l,…, W_s^l ), (2) W⋆,sl W_ ,s^l =W0l+αΔWMCU,sl. =W_0^l+α W_MCU,s^l. (3) where α is a global merging coefficient. The objective is to preserve the unlearning effects of the available one-shot adapters while maintaining retained knowledge and general multimodal utility. Shared core-space representation. Directly manipulating full parameter adapters is inefficient and obscures their low-rank structures. Following Core Space Merging (Panariello et al. 2026), at step s we construct orthonormal bases PslP_s^l and QslQ_s^l spanning the joint column and row spaces of ΔWtlt=1s.\ W_t^l\_t=1^s. Each adapter is represented in this shared coordinate system as Mtl=(Psl)⊤ΔWtlQsl,ΔWtl=PslMtl(Qsl)⊤.M_t^l= (P_s^l ) W_t^lQ_s^l, W_t^l=P_s^lM_t^l (Q_s^l ) . (4) Since the bases span the joint update subspaces, the representation is reversible up to numerical precision. The core matrices Mtlt=1s\M_t^l\_t=1^s therefore retain the original LoRA adapters in a compact and aligned space, where their spectral structures and cross-task dependencies can be analyzed efficiently. For readability, we omit the continual-step subscript of PslP_s^l and QslQ_s^l in the remainder of the paper. Cross-task unlearning transfer. Although the one-shot adapters are trained independently, their unlearning effects need not be isolated across requests. To examine their interactions, we conduct a leave-one-out merging study: for each target request t, we exclude its corresponding adapter, merge the remaining adapters, and evaluate the resulting model on ftD_f^t. If the adapters encoded disjoint unlearning effects, excluding the target adapter would largely restore the base-model behavior on the held-out target. Instead, the merged model still exhibits a non-negligible unlearning effect, as shown in Fig. 3. We refer to this phenomenon as cross-task unlearning transfer. It indicates that independently trained one-shot adapters exhibit cross-task dependencies whose effects can transfer across unlearning requests. Task-direction dependencies. We characterize these shared structures through the singular directions of the aligned core updates. For request t at layer l, we decompose Mtl=UtlΣtl(Vtl)⊤,M_t^l=U_t^l _t^l (V_t^l ) , (5) where UtlU_t^l and VtlV_t^l contain the left and right singular directions, respectively. Because all MtlM_t^l are expressed in the same core bases, these directions are comparable across requests. At continual step s, we concatenate them as SUl=[U1l,…,Usl],SVl=[V1l,…,Vsl],S_U^l= [U_1^l,…,U_s^l ], S_V^l= [V_1^l,…,V_s^l ], (6) and define the corresponding Gram matrices GUl=(SUl)⊤SUl,GVl=(SVl)⊤SVl.G_U^l= (S_U^l ) S_U^l, G_V^l= (S_V^l ) S_V^l. (7) Their off-diagonal task blocks encode cross-request directional dependencies. Importantly, a cross-task dependency may be synergistic or interfering, and its magnitude alone does not distinguish between the two. For component a of request i and component b of request j, their normalized matrix interaction is χij,abl=[(Uil)⊤Ujl]ab[(Vil)⊤Vjl]ab. _ij,ab^l= [ (U_i^l ) U_j^l ]_ab [ (V_i^l ) V_j^l ]_ab. (8) Indeed, the Frobenius inner product between the corresponding rank-one update components equals σi,alσj,blχij,abl _i,a^l _j,b^l _ij,ab^l. Positive values indicate geometrically aligned rank-one updates, whereas negative values indicate parameter-space interference that may cause cancellation during merging. Since directly constraining χij,abl _ij,ab^l couples the left and right geometries, MCU adopts a separable sufficient surrogate that targets non-negative cross-request similarities on both sides while remaining close to the original Gram geometry. Unlike either singular-vector similarity alone, χij,abl _ij,ab^l is invariant to the paired sign ambiguity of the singular value decomposition. Merging for Continual Unlearning Overview. Figure 2 illustrates the overall procedure of MCU. Given the one-shot unlearning adapters available at continual step s, MCU first preserves the dominant structure of each core update, then controls over-concentrated core-space coordinates, and finally reconfigures their cross-request directional dependencies before merging. The first two operations produce structured updates that retain the principal information of the individual adapters while reducing unnecessary complexity for the subsequent joint reconfiguration. Dominant direction selection. The singular spectra of unlearning adapters are typically concentrated, with low-energy tail components contributing limited update magnitude while increasing the number of directions involved in cross-request reconfiguration. MCU therefore retains only the leading k singular components of each core update. Using the decomposition in Eq. (5), we define M¯tl=Ut,klΣt,kl(Vt,kl)⊤, M_t^l=U_t,k^l _t,k^l (V_t,k^l ) , (9) where Ut,klU_t,k^l and Vt,klV_t,k^l contain the leading k left and right singular directions, respectively, and Σt,kl _t,k^l contains their associated singular values. By the optimality of truncated SVD, M¯tl M_t^l is the closest rank-k approximation to MtlM_t^l under the Frobenius norm, reducing the number of low-contribution directions involved in joint reconfiguration. Figure 3: Leave-one-task-out VQA analysis before and after dependency-aware reconfiguration. Columns indicate the excluded adapters, and rows indicate the evaluation targets. Each cell reports the accuracy change relative to the corresponding one-shot unlearned model. Channel capacity control. Dominant direction selection controls spectral complexity but does not regulate how the retained update is distributed across core-space coordinates. In practice, the row norms of M¯tl M_t^l exhibit a long-tailed distribution, such that a small number of coordinates can dominate the geometry of the update and its subsequent merging behavior. MCU applies a soft row-capacity constraint to reduce this over-concentration without removing the corresponding coordinates. For row i of M¯tl M_t^l, we define ρt,il=‖M¯tl[i,:]‖2,τt,ql=Quantileq(ρt,ili). _t,i^l= \| M_t^l[i,:] \|_2, _t,q^l=Quantile_q ( \ _t,i^l \_i ). (10) where q∈(0,1)q∈(0,1) determines the capacity threshold. The structured update is then obtained by M~tl[i,:]=min(1,τt,qlρt,il+ϵ)M¯tl[i,:], M_t^l[i,:]= (1, _t,q^l _t,i^l+ε ) M_t^l[i,:], (11) where ϵε is a small constant for numerical stability. Rows below the threshold remain unchanged, whereas rows above it are rescaled to the threshold. Dominant direction selection and channel capacity control therefore operate on complementary structures: the former limits spectral complexity, while the latter prevents a few core-space coordinates from disproportionately dominating the retained update. Dependency-aware direction reconfiguration. Although the structured updates preserve their dominant task-specific information, dependencies remain among their cross-request singular directions. Rather than enforcing complete orthogonality, MCU suppresses antagonistic interactions while preserving the original geometry and beneficial dependencies that support cross-task unlearning transfer. We first decompose each structured update as M~tl=U~tlΣ~tl(V~tl)⊤. M_t^l= U_t^l _t^l ( V_t^l ) . (12) After fixing the paired signs of the singular components using a deterministic orientation rule solely for reproducibility, we concatenate the left and right directions as S~Ul=[U~1l,…,U~sl],S~Vl=[V~1l,…,V~sl]. S_U^l= [ U_1^l,…, U_s^l ], S_V^l= [ V_1^l,…, V_s^l ]. (13) For X∈U,VX∈\U,V\, the corresponding Gram matrix is GX,0l=(S~Xl)⊤S~Xl.G_X,0^l= ( S_X^l ) S_X^l. (14) Directly constraining the sign-invariant interaction in Eq. (8) jointly couples the left and right geometries. We therefore adopt a tractable separable surrogate that requires non-negative cross-request similarities on both sides. This condition is sufficient, but not necessary, for a non-interfering rank-one interaction. Let ℐtlI_t^l denote the indices of the directions belonging to request t at layer l, and define the set of cross-request direction pairs as l=(p,q)|p∈ℐil,q∈ℐjl,i≠j.C^l= \(p,q)\; |\;p _i^l,\;q _j^l,\;i≠ j \. (15) For X∈U,VX∈\U,V\, MCU solves G^Xl=argminG G_X^l= _G ‖G−GX,0l‖F2, \|G-G_X,0^l \|_F^2, (16) s.t. .t. G⪰0,rank(G)≤dXl,Gpq≥0,(p,q)∈l, G 0,\;rank(G)≤ d_X^l,\;G_pq≥ 0,\;(p,q) ^l, G[ℐtl,ℐtl]=I|ℐtl|,t=1,…,s. G\! [I_t^l,I_t^l ]=I_ _t^l , t=1,…,s. Here, dUld_U^l and dVld_V^l denote the ambient dimensions of the left and right direction spaces, respectively. The proximity objective limits unnecessary changes introduced by the conservative surrogate, while the identity-block constraints preserve within-request orthonormality. Because the rank-constrained feasible set is non-convex, we approximately solve Eq. (16) by alternating projections onto the rank-constrained positive-semidefinite set, the non-negative cross-request entries, and the within-request identity blocks. The sign invariance of the original interaction, the sufficient nature and limitations of the separable surrogate, and the detailed optimization procedure are provided in the supplementary material. Method Eval. Qwen2-VL-7B LLaVA-1.5-7B Task 10 Task 20 Task 50 Task 100 Task 10 Task 20 Task 50 Task 100 Forget ↓ Retain ↑ Forget ↓ Retain ↑ Forget ↓ Retain ↑ Forget ↓ Retain ↑ Forget ↓ Retain ↑ Forget ↓ Retain ↑ Forget ↓ Retain ↑ Forget ↓ Retain ↑ VQA 88.1 88.9 87.9 88.4 89.0 88.6 89.1 88.1 66.8 66.2 64.4 67.1 67.4 68.0 65.8 67.2 Vanilla QA 95.5 94.6 96.5 94.8 96.1 95.2 93.1 94.2 87.8 81.1 85.3 82.5 85.1 81.9 86.8 82.6 GA VQA — — — — — — — — 0.0 17.0 0.3 0.1 — — — — QA — — — — — — — — 5.9 13.2 — — — — — — GA-Diff VQA 69.3 71.6 21.8 60.0 15.2 47.8 25.3 45.5 49.7 46.5 42.7 42.1 38.5 42.7 34.2 41.2 QA 78.1 86.9 52.8 84.1 45.8 46.3 66.7 67.5 75.6 71.5 70.0 69.4 65.2 68.4 65.2 66.8 KL-Min VQA 46.6 79.3 24.3 43.8 — 13.8 — — 58.8 63.3 63.2 60.9 57.4 59.0 46.6 53.6 QA 44.3 74.3 14.3 28.6 3.8 11.8 — — 77.3 78.7 81.3 80.8 78.1 77.4 73.7 71.5 NPO VQA — — — — — — — — — — — — — — — — QA — — — — — — — — — — — — — — — — MANU VQA 71.9 71.5 60.0 59.8 23.9 25.5 21.8 25.2 56.0 60.3 54.7 57.1 24.4 24.1 24.2 23.8 QA 80.4 93.2 75.6 93.4 48.3 45.0 45.4 44.3 84.1 80.4 83.2 81.2 38.7 37.1 32.3 34.3 MMU VQA 45.8 67.6 48.5 66.1 39.0 46.6 33.3 42.7 38.9 48.2 40.6 51.4 35.7 54.0 32.5 57.3 QA 72.7 79.3 60.6 81.8 52.3 42.7 38.5 46.5 54.8 68.8 50.3 56.0 43.8 68.4 33.3 58.8 VQA 23.7 91.6 29.4 86.1 31.4 85.4 28.7 75.6 37.3 67.6 35.3 64.5 25.3 67.4 24.4 60.6 MCU QA 34.5 80.3 33.4 73.0 36.5 70.6 30.9 75.2 41.3 71.0 39.7 68.4 23.9 67.8 23.7 66.8 Table 1: Current-batch unlearning and retention results on ICU-Bench after 10, 20, 50, and 100 unlearning requests. Each method is evaluated using VQA and QA accuracy. Lower Forget and higher Retain indicate better performance. — denotes unavailable or invalid results due to unstable optimization or model collapse. Figure 4: (a) Off-diagonal task-direction overlap of the singular subspaces. unlearning adapters exhibit substantially stronger overlap on the U-side than on the V-side, with OU/OV=6.6×O_U/O_V=6.6×, indicating stronger output-side cross-task dependencies. (b) Cumulative spectral energy of one-shot unlearning adapters. The curve reports the mean cumulative energy across all task–module updates, while the shaded region denotes one standard deviation. The leading singular components capture most of the update energy: the top-4 directions retain 74.1%74.1\% energy on average, while the top-6 directions retain 89.6%89.6\%. (c) Row-norm concentration after spectral selection. The retained updates show a long-tailed coordinate distribution, where the 99th-percentile row norm is 3.5×3.5× the median and the maximum row norm reaches 12.9×12.9× the median. Direction recovery and merging. The optimized Gram matrices specify the desired pairwise geometry but do not directly provide direction matrices in the original core space. For X∈U,VX∈\U,V\, we factor G^Xl=(RXl)⊤RXl G_X^l= (R_X^l ) R_X^l (17) and recover the closest realization to S~Xl S_X^l through an orthogonal Procrustes problem: OXl,⋆=argminO⊤O=I‖ORXl−S~Xl‖F2,S^Xl=OXl,⋆RXl.O_X^l, = _O O=I \|OR_X^l- S_X^l \|_F^2, S_X^l=O_X^l, R_X^l. (18) The recovered directions satisfy (S^Xl)⊤S^Xl=G^Xl ( S_X^l ) S_X^l= G_X^l while remaining as close as possible to their structured counterparts under the chosen factorization. Because the within-request Gram blocks are fixed to identity in Eq. (16), the recovered direction blocks remain orthonormal within each request. We partition S^Ul S_U^l and S^Vl S_V^l according to their original request blocks, obtaining U^tlt=1s\ U_t^l\_t=1^s and V^tlt=1s\ V_t^l\_t=1^s. Each processed core update is reconstructed using the structured singular-value coefficients: M^tl=U^tlΣ~tl(V^tl)⊤. M_t^l= U_t^l _t^l ( V_t^l ) . (19) Finally, the request-specific updates are merged in the shared core space and mapped back to the original parameter space: MMCU,sl=∑t=1sM^tl,ΔWMCU,sl=PlMMCU,sl(Ql)⊤.M_MCU,s^l= _t=1^s M_t^l, W_MCU,s^l=P^lM_MCU,s^l (Q^l ) . (20) Applying the procedure to all LoRA-enabled layers yields the unified model ℳθs⋆M_ _s at continual step s. Given the one-shot adapters, MCU is a training-free, post-hoc merging procedure and requires no additional gradient-based optimization. Experiments Figure 5: Stage-wise VQA performance under the five-stage continual unlearning protocol on MLLMU-Bench. Columns denote unlearning stages, while rows denote the fixed Retain set and task-specific unlearning targets. Lower target accuracy indicates stronger unlearning, whereas stable Retain accuracy reflects better utility preservation. Blank cells correspond to stages before a target is introduced. Experimental Setup Benchmarks and protocols. We evaluate MCU on ICU-Bench (Wang et al. 2026b) and MLLMU-Bench (Liu et al. 2025a). For ICU-Bench, we follow the official 100-task protocol, with every ten tasks forming one batch; Forget is evaluated after each task, while Retain and in-domain utility are evaluated after each batch. For MLLMU-Bench, we partition the 15% Forget Set into five disjoint stages of 15 target profiles and evaluate the same retain subset throughout the sequence. The complete split and evaluation protocol are provided in the supplementary material. Method Qwen2-VL-7B LLaVA-1.5-7B VQA-F ↓ VQA-R ↑ QA-F ↓ QA-R ↑ VQA-F ↓ VQA-R ↑ QA-F ↓ QA-R ↑ Task Arithmetic 42.8 76.5 47.9 61.3 45.6 35.2 55.8 42.1 TIES 39.5 78.8 44.2 64.7 43.8 36.9 53.4 43.6 DARE 36.7 80.1 41.8 66.5 42.2 37.5 52.1 44.0 TSV 34.9 81.7 39.9 67.2 40.8 38.2 50.3 45.1 Core-space 34.1 72.5 39.6 58.6 35.6 35.2 44.8 44.7 MCU 31.4 85.4 36.5 70.6 25.3 67.4 23.9 67.8 Table 2: Comparison with representative merging methods on ICU-Bench after 50 unlearning tasks. F and R denote Forget and Retain accuracy, respectively. Models and baselines. We evaluate LLaVA-1.5-7B (Liu et al. 2024) and Qwen2-VL-7B (Wang et al. 2024). MCU independently trains one-shot GA-Diff LoRA adapters from the same base checkpoint (Liu et al. 2022). Sequential baselines include GA (Thudi et al. 2022), GA-Diff (Liu et al. 2022), KL-Min (Maini et al. 2024), NPO (Zhang et al. 2024), MANU (Liu et al. 2025b), and MMUnlearner(MMU) (Huo et al. 2025). Merging baselines include Task Arithmetic (Ilharco et al. 2022), TIES-Merging (Yadav et al. 2023), DARE (Yu et al. 2024), TSV (Gargiulo et al. 2025), and Core Space merging (Panariello et al. 2026); all use the same adapter bank. Metrics. We report VQA and QA accuracy on both benchmarks. For ICU-Bench, we additionally report Current/Historical Forget and Retain, in-domain utility, Generation Quality (GQ), Retain Stability Rate (RSR), and Forgetting Rebound (FR). For MLLMU-Bench, we report Current/Historical Forget and Retain performance. Lower Forget, RSR, and FR are preferred, while higher values are better for the remaining metrics. Full configurations are provided in the supplementary material. Main Results Continual unlearning on ICU-Bench. Table 1 reports Current Forget and Current Retain performance at representative checkpoints of the 100-task ICU-Bench sequence. Existing methods exhibit an increasingly unstable unlearning–retention trade-off as the sequence grows. Gradient-based methods often achieve low Forget accuracy at the cost of severe utility degradation, whereas preference-based and multimodal-specific methods struggle to maintain consistent unlearning and retention over long sequences. MCU consistently establishes a stronger balance across both backbones and all sequence lengths. At Task 100 on Qwen2-VL-7B, MCU achieves Forget VQA/QA scores of 28.7/30.9 while preserving Retain VQA/QA scores of 75.6/75.2. Compared with MMUnlearner, this reduces Forget accuracy by 4.6/7.6 points and improves Retain accuracy by 32.9/28.7 points. On LLaVA-1.5-7B, MCU obtains Forget scores of 24.4/23.7 together with Retain scores of 60.6/66.8, outperforming MMUnlearner by 8.1/9.6 points on Forget and 3.3/8.0 points on Retain. The consistent advantage from short to long sequences shows that MCU remains effective as the number of accumulated one-shot adapters increases. Importantly, low Forget accuracy does not always correspond to successful unlearning. MANU obtains competitive Forget scores at later checkpoints, but its retained performance and generation quality collapse, showing that its apparent unlearning efficacy is largely caused by broad model degradation. MCU instead maintains effective target removal together with substantially stronger retained performance. Method 50 Tasks 100 Tasks RSR ↓ FR ↓ GQ-F ↑ GQ-R ↑ RSR ↓ FR ↓ GQ-F ↑ GQ-R ↑ GA – – – – – – – – GA-Diff 9.23 0.51 1.340 1.390 6.46 0.28 0.900 1.240 NPO – – – – – – – – KL-Min 4.72 0.87 0.450 1.910 4.20 2.61 0.370 1.840 MANU 11.42 – 0.776 0.771 5.12 0.34 0.447 0.448 MMU 2.62 1.95 1.832 1.827 6.82 5.45 1.804 1.826 MCU 1.02 1.86 1.989 1.994 1.10 1.12 1.990 1.993 Table 3: Sequence-level evaluation on Qwen2-VL-7B after 50 and 100 unlearning requests. Continual unlearning on MLLMU-Bench. Figure 5 evaluates all methods under the five-stage continual protocol constructed from MLLMU-Bench. Because the Retain Set remains fixed, its stage-wise accuracy directly reflects retention drift. Existing methods either suppress new targets by progressively degrading the retained sets or fail to preserve unlearning on historical targets. In particular, MMUnlearner and NPO undergo substantial retention degradation over the sequence, whereas GA and MANU exhibit unstable responses on historical targets. MCU instead maintains stable Retain performance while consistently suppressing both current and previously introduced target knowledge. Together with the ICU-Bench results, this demonstrates the effectiveness of MCU across different sequence lengths and target knowledge types. Variant Sel. Cap. Orth. Reconf. Forget ↓ Retain ↑ VQA QA VQA QA Direct Addition 47.9 51.6 80.7 70.5 Reconfiguration Only ✓ 43.5 46.7 78.6 68.3 Selection + Reconf. ✓ ✓ 33.5 38.9 83.1 68.9 Capacity + Reconf. ✓ ✓ 34.3 39.8 82.2 68.1 Shaping Only ✓ ✓ 36.0 41.2 84.2 69.8 Shaping + Orth. ✓ ✓ ✓ 32.8 37.9 83.1 69.1 MCU ✓ ✓ ✓ 31.4 36.5 85.4 70.6 Table 4: Component and direction-handling ablation on ICU-Bench with Qwen2-VL-7B after 50 unlearning tasks. Sel., Cap., Orth., and Reconf. denote direction selection, capacity control, strict orthogonalization, and dependency-aware reconfiguration, respectively. Sequence-level stability. While Table 1 evaluates individual checkpoints, Table 3 summarizes stability over the complete ICU-Bench sequence. At Task 100, MCU obtains an FR of 1.12 and an RSR of 1.10, reducing historical rebound and retention drift by 4.33 and 5.72 points relative to MMUnlearner, respectively. MCU also maintains GQ-F/GQ-R scores of 1.990/1.993, confirming that its low Forget accuracy is not caused by invalid or collapsed generation. Thus, MCU preserves historical unlearning, retained performance, and normal response quality throughout long request sequences. Ablation and Analysis Cross-task unlearning transfer and update geometry. Figure 3 reports the leave-one-task-out analysis in the 20-task setting. For each column, we remove the adapter of the corresponding task, merge the remaining adapters, and evaluate the resulting model on all targets shown by the rows. Each cell reports the VQA accuracy change relative to the corresponding one-shot unlearned model. Before direction reconfiguration, removing a target adapter often produces only limited recovery on the diagonal, indicating that the remaining adapters still transfer non-negligible unlearning effects to the held-out target. Complete results and additional analysis are provided in the supplementary material. Figure 4(a) provides a geometric explanation: their singular directions exhibit non-negligible cross-task dependencies, with U=0.081O_U=0.081 and V=0.012O_V=0.012. After dependency-aware direction reconfiguration, most tasks exhibit stronger unlearning performance, while removing a target adapter produces a clearer recovery on its corresponding target. This indicates that the reconfigured geometry reduces harmful cross-task interference while limiting disruption to synergistic dependencies. Figure 4(b) further shows that the leading four and six singular components preserve 74.1% and 89.6% of the update energy, respectively, supporting the removal of low-contribution tail directions. Figure 4(c) reveals a strongly long-tailed row-norm distribution: the 99th-percentile and maximum row norms are 3.5× and 12.9× the median, respectively. These results motivate dominant direction selection and channel capacity control before joint reconfiguration. Complete leave-one-out results and geometric statistics are reported in supplementary material. Comparison with model merging methods. As shown in Table 2, MCU achieves stronger unlearning while preserving more retained knowledge across both backbones. By reconfiguring cross-task singular directions in the shared core space, MCU reduces cross-task interference that weakens unlearning while limiting disruption to synergistic dependencies across requests. Conventional merging methods instead treat the parameter updates induced by unlearning adapters as ordinary task vectors and fail to account for their distinct suppressive geometry. Component ablation. Table 4 evaluates the contribution of update shaping and cross-task direction handling after 50 unlearning tasks. Compared with Direct Addition, dependency-aware reconfiguration lowers Forget VQA/QA from 47.9/51.6 to 43.5/46.7, but also reduces Retain from 80.7/70.5 to 78.6/68.3. Thus, directly reconfiguring the complete direction space strengthens unlearning but can disrupt the original update structures. Adding direction selection substantially improves both sides of the trade-off, achieving Forget VQA/QA scores of 33.5/38.9 and Retain scores of 83.1/68.9. Capacity control similarly improves Forget to 34.3/39.8 while recovering Retain VQA to 82.2. These results show that removing low-contribution directions and limiting over-concentrated coordinates provide more suitable updates for subsequent joint reconfiguration. The remaining variants clarify the role of cross-task direction handling. Shaping Only preserves strong Retain performance but leaves cross-task interactions insufficiently resolved, resulting in weaker Forget scores of 36.0/41.2. Strict orthogonalization improves unlearning, but indiscriminately removes both interfering and synergistic dependencies. Under the same shaping operations, MCU improves over strict orthogonalization by 1.4/1.4 points on Forget and 2.3/1.5 points on Retain. The complete MCU therefore achieves the best Forget VQA/QA scores of 31.4/36.5 and the best Retain scores of 85.4/70.6, confirming the complementary roles of direction selection, capacity control, and dependency-aware reconfiguration. Conclusion In this work, we introduced MCU, a model-merging framework for continual multimodal unlearning that consolidates accumulated one-shot unlearning adapters into a unified model update. Our analysis reveals cross-task dependencies that can either support beneficial unlearning transfer or induce antagonistic interactions, motivating MCU to preserve useful shared structures while suppressing harmful interference. Extensive experiments demonstrate that MCU achieves effective current and historical unlearning while preserving retained knowledge and general multimodal utility. Given the one-shot adapters, MCU requires no additional gradient-based optimization during merging and directly produces a unified update for deployment. References J. Cheng and H. Amiri (2024) Mu-bench: a multitask multimodal benchmark for machine unlearning. arXiv preprint arXiv:2406.14796. Cited by: Machine Unlearning.. I. Cohen, D. Gottesman, M. Geva, and R. Giryes (2025) Performance gap in entity knowledge extraction across modalities in vision language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 29095–29108. Cited by: Introduction. A. Dontsov, D. Korzh, A. Zhavoronkin, B. Mikheev, D. Bobkov, A. Alanov, O. Rogov, I. Oseledets, and E. Tutubalina (2025) Clear: character unlearning in textual and visual modalities. In Findings of the Association for Computational Linguistics: ACL 2025, p. 20582–20603. Cited by: Machine Unlearning.. C. Gao, L. Wang, K. Ding, C. Weng, X. Wang, and Q. Zhu (2025) On large language model continual unlearning. In International Conference on Learning Representations, Vol. 2025, p. 101772–101801. Cited by: Introduction. S. Garg, S. Goldwasser, and P. N. Vasudevan (2020) Formalizing data deletion in the context of the right to be forgotten. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, p. 373–402. Cited by: Introduction. A. A. Gargiulo, D. Crisostomi, M. S. Bucarelli, S. Scardapane, F. Silvestri, and E. Rodola (2025) Task singular vectors: reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 18695–18705. Cited by: Introduction, Model Merging., Experimental Setup. V. Gupta, C. Jung, S. Neel, A. Roth, S. Sharifi-Malvajerdi, and C. Waites (2021) Adaptive machine unlearning. Advances in Neural Information Processing Systems 34, p. 16319–16330. Cited by: Introduction. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. Iclr 1 (2), p. 3. Cited by: Dynamic adapter merging.. J. Huo, Y. Yan, X. Zheng, Y. Lyu, X. Zou, Z. Wei, and X. Hu (2025) Mmunlearner: reformulating multimodal machine unlearning in the era of multimodal large language models. In Findings of the Association for Computational Linguistics: ACL 2025, p. 7190–7206. Cited by: Machine Unlearning., Experimental Setup, Baseline methods.. G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi (2022) Editing models with task arithmetic. arXiv preprint arXiv:2212.04089. Cited by: Model Merging., Experimental Setup. T. Kawakami, K. Egashira, A. Miyai, G. Irie, and K. Aizawa (2025) Pulse: practical evaluation scenarios for large multimodal model unlearning. arXiv preprint arXiv:2507.01271. Cited by: Introduction. S. Lee, J. Liu, Q. Wang, J. Wang, X. Cai, and Y. Wu (2025) Dynamic fisher-weighted model merging via bayesian optimization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4923–4935. Cited by: Model Merging.. H. Li, G. Deng, Y. Liu, K. Wang, Y. Li, T. Zhang, Y. Liu, G. Xu, G. Xu, and H. Wang (2024a) Digger: detecting copyright content mis-usage in large language model training. arXiv preprint arXiv:2401.00676. Cited by: Introduction. H. Li, H. Chi, Q. Wang, Y. Mao, Z. Zhang, J. Tan, T. Liu, W. Yang, and B. Han (2026) MLUBench: a benchmark for lifelong unlearning evaluation in mllms. arXiv preprint arXiv:2606.12809. Cited by: Introduction, Machine Unlearning.. J. Li, Q. Wei, C. Zhang, G. Qi, M. Du, Y. Chen, S. Bi, and F. Liu (2024b) Single image unlearning: efficient machine unlearning in multimodal large language models. Advances in Neural Information Processing Systems 37, p. 35414–35453. Cited by: Machine Unlearning.. S. Lin, J. Dong, R. Chen, X. Zhang, L. Xu, and X. Chen (2026) CATA: continual machine unlearning via conflict-averse task arithmetic. arXiv preprint arXiv:2605.18610. Cited by: Introduction. B. Liu, Q. Liu, and P. Stone (2022) Continual learning and private unlearning. In Conference on Lifelong Learning Agents, p. 243–254. Cited by: Machine Unlearning., Experimental Setup, Baseline methods.. H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26296–26306. Cited by: Experimental Setup, Vanilla models.. Z. Liu, G. Dou, M. Jia, Z. Tan, Q. Zeng, Y. Yuan, and M. Jiang (2025a) Protecting privacy in multimodal large language models with mllmu-bench. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4105–4135. Cited by: Machine Unlearning., Experimental Setup, MLLMU-Bench.. Z. Liu, G. Dou, X. Yuan, C. Zhang, Z. Tan, and M. Jiang (2025b) Modality-aware neuron pruning for unlearning in multimodal large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5913–5933. Cited by: Machine Unlearning., Experimental Setup, Baseline methods.. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter (2024) Tofu: a task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. Cited by: Machine Unlearning., Experimental Setup, Baseline methods.. M. S. Matena and C. Raffel (2022) Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems 35, p. 17703–17716. Cited by: Model Merging.. A. Panariello, D. Marczak, S. Magistri, A. Porrello, B. Twardowski, A. Bagdanov, S. Calderara, and J. van de Weijer (2026) Accurate and efficient low-rank model merging in core space. Advances in Neural Information Processing Systems 38, p. 61793–61825. Cited by: Model Merging., Shared core-space representation., Experimental Setup. R. Pi, T. Han, J. Zhang, Y. Xie, R. Pan, Q. Lian, H. Dong, J. Zhang, and T. Zhang (2024) Mllm-protector: ensuring mllm’s safety without hurting performance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 16012–16027. Cited by: Introduction. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: Machine Unlearning.. A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh (2021) Remember what you want to forget: algorithms for machine unlearning. Advances in Neural Information Processing Systems 34, p. 18075–18086. Cited by: Introduction. W. Shi, J. Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. Smith, and C. Zhang (2025) Muse: machine unlearning six-way evaluation for language models. In International Conference on Learning Representations, Vol. 2025, p. 27797–27818. Cited by: Introduction. A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot (2022) Unrolling sgd: understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), p. 303–319. Cited by: Machine Unlearning., Experimental Setup, Baseline methods.. C. Wang, Y. Li, X. Feng, C. Chen, X. Zheng, and J. Yin (2026a) Umu-bench: closing the modality gap in multimodal unlearning evaluation. Advances in Neural Information Processing Systems 38. Cited by: Machine Unlearning.. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: Experimental Setup, Vanilla models.. Y. Wang, W. Mei, J. Zhang, G. He, Z. Niu, and H. Gao (2026b) ICU-bench: benchmarking continual unlearning in multimodal large language models. arXiv preprint arXiv:2605.05938. Cited by: Introduction, Machine Unlearning., Experimental Setup, ICU-Bench.. Y. Wang, Z. Niu, H. Ji, G. He, H. Gao, and G. Hua (2025) MLLM machine unlearning via visual knowledge distillation. arXiv preprint arXiv:2512.11325. Cited by: Machine Unlearning.. Y. Wang, Z. Niu, H. Ji, G. He, L. Zhang, and H. Gao (2026c) Null space constrained contrastive visual forgetting for mllm unlearning. arXiv preprint arXiv:2605.05909. Cited by: Machine Unlearning.. M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. (2022) Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, p. 23965–23998. Cited by: Model Merging.. Z. Xu, P. Zhou, W. Tang, J. Ai, W. Zhao, K. Wang, X. Peng, W. Shao, H. Yao, and K. Zhang (2025) Pebench: a fictitious dataset to benchmark machine unlearning for multimodal large language models. arXiv preprint arXiv:2503.12545. Cited by: Machine Unlearning.. P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal (2023) Ties-merging: resolving interference when merging models. Advances in neural information processing systems 36, p. 7093–7115. Cited by: Model Merging., Experimental Setup. P. Yadav, T. Vu, J. Lai, A. Chronopoulou, M. Faruqui, M. Bansal, and T. Munkhdalai (2024) What matters for model merging at scale?. arXiv preprint arXiv:2410.03617. Cited by: Model Merging.. L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li (2024) Language models are super mario: absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, Cited by: Model Merging., Experimental Setup. Z. Yu, M. Y. I. Idris, P. Wang, Y. Xia, and Y. Xiang (2025) Forgetme: benchmarking the selective forgetting capabilities of generative models. Engineering Applications of Artificial Intelligence 161, p. 112087. Cited by: Machine Unlearning.. K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y. Zhang, J. Yang, C. Li, et al. (2025) Lmms-eval: reality check on the evaluation of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 881–916. Cited by: VQAv2 val-lite.. R. Zhang, L. Lin, Y. Bai, and S. Mei (2024) Negative preference optimization: from catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868. Cited by: Machine Unlearning., Experimental Setup, Baseline methods.. Supplementary Material Implementation Details Datasets ICU-Bench. We follow the official continual multimodal unlearning protocol of ICU-Bench (Wang et al. 2026b). The benchmark contains 1,000 synthetic privacy-sensitive profiles from two document domains: 500 medical reports and 500 labor contracts. Each profile is instantiated into multiple document views and question–answer formats, including full-image VQA, masked-image VQA, text-only QA, and description generation. In total, ICU-Bench contains 9,500 document images and 16,000 question–answer pairs. The benchmark is organized into 100 sequential unlearning tasks, each containing seven target individuals. Every ten tasks form one batch, resulting in ten evaluation batches. For each batch, 180 non-target individuals are used to construct the retain set. During vanilla memorization training, only the original full-image samples are used. The partially and fully masked views are reserved for evaluation, providing a stricter test of whether the model retains the underlying private information rather than merely reading a visible target field. For the main experiments, we report results after 10, 20, 50, and 100 accumulated requests. Additional order-robustness and hierarchical-consolidation experiments use the first 20 tasks. MLLMU-Bench. We additionally evaluate MCU on MLLMU-Bench (Liu et al. 2025a). Following the 15% forgetting setting, we partition the 75 target profiles in the Forget Set into five mutually disjoint tasks, 1,…,5\T_1,…,T_5\, with 15 profiles per task. All methods use the same task partition and request order. After stage s, the model is evaluated on the current task sT_s, all previously introduced tasks 1,…,s−1\T_1,…,T_s-1\, a fixed retain subset, and the Real Celebrity Set. The retain subset associated with the first stage is fixed throughout the complete sequence; therefore, “Retain” in our MLLMU-Bench continual results refers specifically to this fixed T1 retain subset, rather than to a newly sampled retain set at each stage. VQAv2 val-lite. We use VQAv2 val-lite (Zhang et al. 2025) as an external utility evaluation set. It is not used to train the vanilla models or one-shot unlearning adapters, and it is not used to select the retained rank, capacity threshold, or merging coefficient. All compared checkpoints are evaluated with the same prompt, decoding configuration, and scoring script. Evaluation Metrics Task-level accuracy. Multiple-choice VQA and text-only QA tasks are evaluated using accuracy. For an evaluation set D and model ℳM, we denote the corresponding accuracy by Acc(ℳ;)=1||∑(x,y)∈[y^ℳ(x)=y].Acc(M;D)= 1|D| _(x,y) 1\! [ y_M(x)=y ]. (21) Lower accuracy is preferred on target knowledge to be removed, while higher accuracy is preferred on retain and utility sets. Current and historical unlearning. Let ℳtM_t denote the model after processing request t, and let fjD_f^j be the evaluation set associated with unlearning request j. The Current Forget score at step t is Ftcur=Acc(ℳt;ft),F_t^cur=Acc (M_t;D_f^t ), (22) and the Historical Forget score is Fthist=1t−1∑j=1t−1Acc(ℳt;fj),t>1.F_t^hist= 1t-1 _j=1^t-1Acc (M_t;D_f^j ), t>1. (23) Lower values indicate stronger current and historical unlearning. Current and historical retention. ICU-Bench evaluates retention at the end of every ten-task batch. The Current Retain Set contains the non-target samples associated with the current batch, whereas the Historical Retain Set aggregates retain samples from preceding batches. Higher Current and Historical Retain accuracy indicates better preservation of non-target knowledge. For MLLMU-Bench, the same fixed T1 retain subset is evaluated after each of the five stages. In-domain and external utility. The full-image tasks of ICU-Bench measure in-domain document reasoning utility. VQAv2 val-lite measures external visual question answering utility outside the ICU-Bench document domain. For the latter, we additionally report the absolute accuracy degradation from the vanilla checkpoint, ΔVQAv2=Avanilla−Aunlearned. _VQAv2=A_vanilla-A_unlearned. (24) Higher utility accuracy and smaller degradation are preferred. Generation Quality. For description-generation tasks, we follow ICU-Bench and report Generation Quality (GQ), evaluated by an LLM judge using Qwen3.5-Flash. GQ measures response fluency and readability rather than factual correctness. The score ranges from 0 to 2: 0 denotes unreadable or severely degenerate output, 1 denotes understandable but unnatural output, and 2 denotes fluent and natural short-form generation. GQ is used primarily to distinguish successful unlearning from broad generation collapse. Retain Stability Rate. Let AbR,maskA_b^R,mask denote masked-view accuracy on the retain set at batch checkpoint b, and let B be the total number of evaluated batches. The Retain Stability Rate (RSR) is RSR=1B−1∑b=2B|AbR,mask−Ab−1R,mask|.RSR= 1B-1 _b=2^B |A_b^R,mask-A_b-1^R,mask |. (25) A smaller RSR indicates more stable retained performance over the continual unlearning sequence. Forgetting Rebound. Let AbHF,maskA_b^HF,mask denote masked-view accuracy on the Historical Forget Set at batch checkpoint b. The rebound at checkpoint b is FRb=max(0,AbHF,mask−Ab−1HF,mask).FR_b= (0,\,A_b^HF,mask-A_b-1^HF,mask ). (26) When a single sequence-level value is reported, we average the checkpoint-wise rebound: FR=1B−1∑b=2BFRb.FR= 1B-1 _b=2^BFR_b. (27) A smaller FR indicates better preservation of previously removed knowledge. Table 5: Hyperparameters for baseline unlearning methods. Model Method Epochs Batch Size Learning Rate LLaVA-1.5-7B GA 3 4 1×10−51× 10^-5 GA-Diff 3 4 1×10−51× 10^-5 KL-Min 3 4 1×10−51× 10^-5 NPO 3 4 5×10−65× 10^-6 MANU 4 4 2×10−52× 10^-5 MMUnlearner 4 4 2×10−52× 10^-5 Qwen2-VL-7B GA 3 4 1×10−51× 10^-5 GA-Diff 3 4 1×10−51× 10^-5 KL-Min 3 4 1×10−51× 10^-5 NPO 3 4 5×10−65× 10^-6 MANU 4 4 2×10−52× 10^-5 MMUnlearner 4 4 2×10−52× 10^-5 Hyperparameter Settings Vanilla models. We use LLaVA-1.5-7B (Liu et al. 2024) and Qwen2-VL-7B (Wang et al. 2024). Following the ICU-Bench setup, both models are first fine-tuned on the full-image ICU-Bench training samples so that they acquire the privacy-sensitive document knowledge later targeted by unlearning. For each sample ⟨I,x,y⟩ I,x,y , the model minimizes the token-level negative log-likelihood ℓ(x,y,I;θ)=−1|y|∑i=1|y|logpθ(yi∣I,x,y<i). (x,y,I;θ)=- 1|y| _i=1^|y| p_θ (y_i I,x,y_<i ). (28) The vision encoder, multimodal connector, and language model are all trainable during this stage. The resulting checkpoints serve as the common starting point for all unlearning methods. The vanilla training settings are given in Table 6. Table 6: Hyperparameters for vanilla memorization training. Model Epochs Batch Size Learning Rate LLaVA-1.5-7B 8 8 1×10−41× 10^-4 Qwen2-VL-7B 6 8 1×10−41× 10^-4 Baseline methods. All sequential baselines start from the same vanilla checkpoint and follow the same request order, evaluation protocol, and checkpointing schedule. At each request, the method receives the current Forget Set FD_F and its corresponding Retain Set RD_R. Gradient Ascent (GA) (Thudi et al. 2022) maximizes the forget-set loss: ℒGA=−ℒ(F;θ).L_GA=-L(D_F;θ). (29) GA-Diff (Liu et al. 2022) adds supervised retain regularization: ℒGA-Diff=−ℒ(F;θ)+ℒ(R;θ).L_GA -Diff=-L(D_F;θ)+L(D_R;θ). (30) KL-Min (Maini et al. 2024) preserves the pre-update distribution on retain samples: ℒKL-Min=−ℒ(F;θ)+λKL1|R|∑z∈RKL(pθ0(⋅∣z)∥pθ(⋅∣z)),L_KL -Min=-L(D_F;θ)+ _KL 1|D_R| _z _RKL (p_ _0(· z)\|p_θ(· z) ), (31) where θ0 _0 is the model before the current unlearning update. NPO (Zhang et al. 2024) decreases the relative likelihood of the target answer under a reference model: ℒNPO=(x,y)∈F[2βlog(1+(πθ(y∣x)πref(y∣x))β)].L_NPO=E_(x,y) _F [ 2β (1+ ( _θ(y x) _ref(y x) )^β ) ]. (32) MANU (Liu et al. 2025b) and MMUnlearner (Huo et al. 2025) are implemented following their official multimodal unlearning procedures. The training hyperparameters used for the baselines are listed in Table 5. One-shot adapters and MCU. For MCU, each request-specific LoRA adapter is independently trained from the same vanilla checkpoint using the GA-Diff objective. The adapter-training schedule therefore follows the GA-Diff setting: 3 epochs, batch size 4, and learning rate 1×10−51× 10^-5 for both backbones. Unless otherwise stated, MCU retains the top six singular directions of each structured update and applies row-capacity control with quantile q=0.95q=0.95. We use a global merging coefficient of α=0.4α=0.4 for Qwen2-VL-7B and α=1.0α=1.0 for LLaVA-1.5-7B. The same MCU configuration is used across the reported sequence checkpoints for each backbone. All one-shot adapters entering the same merge are generated under the same training and processing configuration. Additional Experiments Table 7: Quantitative summary of the leave-one-task-out VQA matrices. “Strong” off-diagonal entries count pairs whose absolute accuracy change exceeds the specified threshold. Variant RdiagR_diag ↑ Positive Diag. ↑ IoffI_off ↓ Median Off. ↓ |ΔAoff|>3 A_off >3 |ΔAoff|>6 A_off >6 Max Off. ↓ SattrS_attr ↑ Before Reconfiguration 5.64 19/20 2.58 3.46 204/380 47/380 16.26 2.18 After Reconfiguration (MCU) 12.57 20/20 1.73 1.86 71/380 9/380 6.57 7.27 Table 8: Robustness of MCU to request order on the first 50 ICU-Bench tasks using Qwen2-VL-7B. The three variants use the same set of one-shot adapters and differ only in their input order. Method Order Forget VQA ↓ Forget QA ↓ Retain VQA ↑ Retain QA ↑ FR ↓ RSR ↓ MCU Original 31.4 36.5 85.4 70.6 1.86 1.02 Reverse 31.4 36.5 85.4 70.6 1.86 1.02 Random 31.4 36.5 85.4 70.6 1.86 1.02 Unless otherwise stated, all experiments in this section use Qwen2-VL-7B. The one-shot adapters used by MCU are independently trained from the same vanilla checkpoint with GA-Diff. Leave-One-Task-Out Analysis We use a leave-one-task-out analysis to characterize the directional overlap among unlearning tasks and to evaluate the effect of dependency-aware direction reconfiguration. The experiment uses Qwen2-VL-7B and the first 20 ICU-Bench tasks. For each task i, we exclude its one-shot GA-Diff adapter, merge the remaining 19 adapters using the same configuration, and evaluate the resulting model on all 20 unlearning tasks. Let ℳ∖iM_ i denote the model obtained after excluding adapter i, and let AjoneA_j^one denote the VQA accuracy of the independently trained one-shot adapter for task j. We define the leave-one-task-out change as ΔAi,j=Acc(ℳ∖j;fi)−Aione. A_i,j=Acc (M_ j;D_f^i )-A_i^one. (33) Following the orientation used in Fig. 3, row i denotes the evaluated unlearning task and column j denotes the adapter excluded during merging. The diagonal and off-diagonal entries have different interpretations. For a diagonal entry ΔAi,i A_i,i, a positive value means that removing adapter i restores accuracy on its own target relative to the corresponding one-shot unlearned model. A larger positive diagonal value therefore indicates clearer request-level attribution. For an off-diagonal entry ΔAi,j A_i,j with i≠ji≠ j, either a positive or negative value indicates that excluding adapter i changes the behavior on another target j. Consequently, off-diagonal quality is determined by the magnitude |ΔAi,j| A_i,j : values closer to zero indicate that the leave-one-out merge remains closer to the desired one-shot behavior on unrelated tasks. We summarize the matrices using the mean diagonal recovery Rdiag=1T∑i=1TΔAi,i,R_diag= 1T _i=1^T A_i,i, (34) the mean absolute off-diagonal deviation Ioff=1T(T−1)∑i≠j|ΔAi,j|,I_off= 1T(T-1) _i≠ j | A_i,j |, (35) and their attribution-separation ratio Sattr=RdiagIoff+ϵ.S_attr= R_diagI_off+ε. (36) Higher RdiagR_diag and SattrS_attr, together with lower IoffI_off, indicate that task-specific contributions are more distinguishable from cross-task deviations. Before dependency-aware reconfiguration, the leave-one-out matrix already exhibits a partially visible diagonal structure, but substantial off-diagonal responses remain. The mean diagonal recovery is only 5.64 points, and one task has a negative diagonal value. This case indicates that the remaining adapters alone produce unlearning on that target that is at least as strong as its one-shot reference, revealing pronounced cross-task unlearning transfer. After reconfiguration, all 20 diagonal entries become positive and the mean diagonal recovery increases from 5.64 to 12.57 points. At the same time, the mean absolute off-diagonal deviation decreases from 2.58 to 1.73 points, a reduction of 33.0%. The number of off-diagonal entries with magnitude above 3 points decreases from 204 to 71, while entries above 6 points decrease from 47 to 9. The maximum off-diagonal deviation is also reduced from 16.26 to 6.57 points. Together, these changes increase the attribution-separation ratio from 2.18 to 7.27. Robustness to Request Order We examine whether MCU is sensitive to the order in which request-specific adapters are provided to the merging procedure. The experiment uses Qwen2-VL-7B and the first 50 ICU-Bench tasks. We consider three request orders: • Original: (0,1,…,49)(0,1,…,49); • Reverse: (49,48,…,0)(49,48,…,0); • Random: a fixed random permutation generated once and reused throughout the experiment. All three variants use exactly the same 50 independently trained GA-Diff adapters and the same MCU hyperparameters. Only the order in which the adapters are supplied to MCU is changed. Unlike sequential unlearning methods, MCU does not repeatedly update the model according to the arrival order of the requests. Instead, every one-shot adapter is independently trained from the same vanilla checkpoint, after which MCU jointly constructs the shared core space, performs direction selection and capacity control, and reconfigures the accumulated directions before reconstructing the final merged update. These operations depend on the collection of adapter updates rather than their input ordering. Therefore, permuting the same adapter set should not alter the resulting model, apart from possible numerical differences caused by finite-precision computation. As shown in Table 8, MCU obtains identical results under the original, reversed, and randomly permuted request orders at the reported precision. The Forget VQA/QA scores remain 31.4/36.5, while the Retain VQA/QA scores remain 85.4/70.6 across all three settings. FR and RSR are also unchanged at 1.86 and 1.02, respectively. General Multimodal Utility We further evaluate whether continual unlearning degrades multimodal capabilities beyond the privacy-sensitive document domain of ICU-Bench. Following the utility-evaluation protocol of ICU-Bench, we use the VQAv2 val-lite split as an external visual question answering benchmark. VQAv2 val-lite is not used for vanilla-model fine-tuning, unlearning, adapter merging, or hyperparameter selection. Our evaluation starts from the ICU-Bench vanilla checkpoint, obtained by fine-tuning Qwen2-VL-7B on the ICU-Bench training samples. This vanilla model has acquired the privacy-sensitive document knowledge targeted by the subsequent unlearning requests and serves as the common initialization for all compared methods. We evaluate the vanilla checkpoint and the resulting unlearned checkpoints after 20, 50, and 100 ICU-Bench requests on VQAv2 val-lite. For the sequential baselines, each checkpoint is obtained by continually applying the corresponding unlearning method to the current model. For MCU, the checkpoint at each task scale is constructed by merging the adapters accumulated up to that stage. The original vanilla model is evaluated under the same protocol and serves as the reference for measuring utility degradation. Table 9: “–” denotes unavailable or invalid results caused by unstable optimization or model collapse. Method 20 Tasks ↑ 50 Tasks ↑ 100 Tasks ↑ Vanilla 76.8 76.8 76.8 GA – – – GA-Diff 53.7 41.1 44.7 KL-Min 71.4 10.3 – NPO – – – MANU 64.4 10.3 10.3 MMUnlearner 76.7 74.3 48.0 MCU 77.8 76.8 76.8 As shown in Table 9, existing sequential unlearning methods generally accumulate substantial utility degradation as the number of requests increases. GA-Diff decreases from 53.7 at 20 tasks to 41.1 and 44.7 at 50 and 100 tasks, respectively. KL-Min retains relatively high utility at 20 tasks but drops sharply to 10.3 at 50 tasks, indicating unstable preservation across sequence lengths. MANU exhibits severe utility degradation after longer sequences, reaching 10.3 at both 50 and 100 tasks. MMUnlearner provides the strongest utility preservation among the sequential baselines, but its accuracy still decreases from 76.7 to 48.0 as the sequence grows from 20 to 100 requests. In contrast, MCU obtains VQAv2 val-lite accuracies of 77.8, 76.8, and 76.8 after 20, 50, and 100 requests, respectively. Its performance matches or slightly exceeds the vanilla accuracy of 76.8 at all evaluated task scales. Compared with MMUnlearner, the strongest available baseline, MCU improves external utility by 1.1, 2.5, and 28.8 points after 20, 50, and 100 requests, respectively. Efficiency and Storage Cost Following the efficiency analysis in ICU-Bench, we report the wall-clock cost, GPU-memory usage, and storage requirements of MCU. The two evaluations characterize different stages of continual unlearning. ICU-Bench measures the optimization cost of applying an unlearning method to an individual request, whereas our evaluation measures the one-time consolidation cost after the request-specific one-shot adapters have been obtained. Therefore, the results below quantify the additional cost introduced by MCU rather than the preceding training cost of the one-shot adapters. We evaluate MCU on the ICU-Bench vanilla checkpoint of Qwen2-VL-7B using 20, 50, and 100 independently trained one-shot adapters. Consolidation time is measured from loading the processed adapters to saving the materialized full-model checkpoint. It includes shared core-space construction, dependency-aware direction reconfiguration, update reconstruction, and full-checkpoint serialization, but excludes one-shot adapter training and adapter preprocessing. For each task scale, we perform one warm-up run followed by three measured runs and report the mean and standard deviation. Table 10: Efficiency and storage cost of MCU on Qwen2-VL-7B-Instruct. The reported time includes materialization and serialization of the complete merged model. Adapter storage denotes the total size of the original one-shot LoRA bank before consolidation. Tasks Time (s) GPU Mem. (GiB) Adapter Bank (GiB) Final Model (GiB) 20 49.17±2.9549.17±2.95 0.00 1.80 15.46 50 98.43±6.0298.43±6.02 0.00 4.50 15.46 100 242.43±9.04242.43±9.04 0.00 9.00 15.46 As shown in Table 10, MCU consolidates 20 and 50 request-specific adapters in approximately 49 and 98 seconds, respectively. The corresponding average costs are 2.45 and 1.96 seconds per request, indicating an approximately linear increase in consolidation time over the evaluated range. Notably, the reported runtime already includes the relatively expensive step of loading, materializing, and serializing a 15.46-GiB full-model checkpoint, rather than only the operations performed in the compact core space. The complete consolidation procedure is executed on the CPU in our implementation and requires no GPU-memory allocation. This property distinguishes MCU consolidation from gradient-based unlearning, which generally requires loading the model on a GPU and performing iterative forward and backward optimization. For reference, ICU-Bench reports that one epoch of GA-Diff requires approximately 300 seconds per request and 38.74 GiB of peak GPU memory. These values are not directly competing measurements because MCU uses GA-Diff to obtain its one-shot adapters. Nevertheless, they show that the subsequent consolidation of up to 50 adapters introduces less wall-clock overhead than one GA-Diff training epoch for a single request, while requiring no additional GPU resources. MCU therefore adds a lightweight, one-time post-processing stage rather than another gradient-based unlearning procedure. MCU also provides a compact representation of accumulated unlearning requests. Each one-shot adapter occupies approximately 92.2 MiB, resulting in adapter-bank sizes of 1.80, 4.50, and 9.00 GiB for 20, 50, and 100 requests, respectively. Even the complete bank of 100 one-shot adapters remains smaller than a single 15.46-GiB materialized model checkpoint. The storage cost grows linearly before consolidation, but the final deployment cost does not: after merging, MCU produces a single model whose size is independent of the number of accumulated requests. It requires neither task-specific routing nor simultaneous loading of multiple adapters during inference. This compact request representation is particularly useful when historical unlearning states must be retained for auditing, reconstruction, or rollback. Under such a requirement, a sequential unlearning system would need to archive a full checkpoint for every retained historical state, whereas MCU can preserve request-level information using low-rank adapters that share the same vanilla checkpoint. In our setting, storing one request as a LoRA adapter is approximately 171.7×171.7× smaller than storing one complete model checkpoint. We emphasize that this comparison applies to historical-state retention; when only the latest state is required, both sequential methods and MCU can deploy a single full model. Additional Analysis and Theoretical Discussion Shared Core-Space Construction Consider the LoRA update for request t at layer l, ΔWtl=BtlAtl,Btl∈ℝdoutl×rtl,Atl∈ℝrtl×dinl. W_t^l=B_t^lA_t^l, B_t^l ^d_out^l× r_t^l, A_t^l ^r_t^l× d_in^l. (37) To represent all request-specific updates in a common low-dimensional space, we concatenate their left and right LoRA factors: ℬl=[B1l,…,Bsl],l=[(A1l)⊤,…,(Asl)⊤].B^l= [B_1^l,…,B_s^l ], ^l= [(A_1^l) ,…,(A_s^l) ]. (38) Let PlP^l and QlQ^l be orthonormal bases for the column spaces of ℬlB^l and lA^l, respectively: Pl∈ℝdoutl×dUl,Ql∈ℝdinl×dVl,P^l ^d_out^l× d_U^l, Q^l ^d_in^l× d_V^l, (39) where dUl=rank(ℬl),dVl=rank(l).d_U^l=rank(B^l), d_V^l=rank(A^l). (40) The shared core-space representation of request t is Mtl=(Pl)⊤ΔWtlQl∈ℝdUl×dVl.M_t^l=(P^l) W_t^lQ^l ^d_U^l× d_V^l. (41) Because the columns of every BtlB_t^l lie in span(Pl)span(P^l) and the columns of (Atl)⊤(A_t^l) lie in span(Ql)span(Q^l), there exist matrices CtlC_t^l and DtlD_t^l such that Btl=PlCtl,(Atl)⊤=QlDtl.B_t^l=P^lC_t^l, (A_t^l) =Q^lD_t^l. (42) It follows that PlMtl(Ql)⊤ P^lM_t^l(Q^l) =Pl(Pl)⊤BtlAtlQl(Ql)⊤ =P^l(P^l) B_t^lA_t^lQ^l(Q^l) (43) =BtlAtl=ΔWtl. =B_t^lA_t^l= W_t^l. Thus, before the subsequent direction-selection and capacity-control operations, the shared core representation preserves each LoRA update exactly up to numerical precision. The dimensionality of the main merging operations is determined by dUld_U^l and dVld_V^l, which are bounded by the aggregate LoRA rank, rather than by the full input and output dimensions of the layer. Definitions of the Geometric Statistics in Figure 4 Figure 4 reports three complementary statistics that motivate the three processing stages of MCU: cross-request directional overlap, cumulative spectral energy, and row-norm concentration. Directional subspace overlap. After decomposing the core update of request t at layer l, let U~tl∈ℝdUl×ktl,V~tl∈ℝdVl×ktl U_t^l ^d_U^l× k_t^l, V_t^l ^d_V^l× k_t^l (44) denote its retained left and right singular directions. For two distinct requests i and j, we define their normalized left-side overlap as oUl(i,j)=‖(U~il)⊤U~jl‖F2min(kil,kjl),o_U^l(i,j)= \|( U_i^l) U_j^l \|_F^2 (k_i^l,k_j^l), (45) and analogously, oVl(i,j)=‖(V~il)⊤V~jl‖F2min(kil,kjl).o_V^l(i,j)= \|( V_i^l) V_j^l \|_F^2 (k_i^l,k_j^l). (46) These quantities are invariant to paired sign flips of individual singular vectors and lie in [0,1][0,1] for orthonormal direction matrices. Let P denote the set of valid layer–request triples (l,i,j)(l,i,j) with i<ji<j. The statistics shown in Figure 4(a) are U=1||∑(l,i,j)∈oUl(i,j),V=1||∑(l,i,j)∈oVl(i,j).O_U= 1|P| _(l,i,j) o_U^l(i,j), _V= 1|P| _(l,i,j) o_V^l(i,j). (47) We obtain U=0.081O_U=0.081 and V=0.012O_V=0.012. Using the unrounded statistics, the left-side overlap is approximately 6.6×6.6× the right-side overlap, showing that cross-request sharing is substantially stronger in the output-side directions. Cumulative spectral energy. For a core update with singular values σt,1l≥⋯≥σt,rtll≥0 _t,1^l≥·s≥ _t,r_t^l^l≥ 0, the fraction of Frobenius energy retained by its leading k components is Etl(k)=∑a=1min(k,rtl)(σt,al)2∑a=1rtl(σt,al)2.E_t^l(k)= _a=1 (k,r_t^l)( _t,a^l)^2 _a=1^r_t^l( _t,a^l)^2. (48) Figure 4(b) reports the mean of Etl(k)E_t^l(k) over all evaluated request–module pairs, while the shaded region denotes one standard deviation. The leading four and six components preserve 74.1%74.1\% and 89.6%89.6\% of the update energy on average, respectively. This concentration motivates retaining a compact set of dominant directions before joint reconfiguration. Row-norm concentration. Let M¯tl M_t^l denote the core update after dominant direction selection and before capacity control. For row a, we define rt,al=‖M¯tl[a,:]‖2.r_t,a^l= \| M_t^l[a,:] \|_2. (49) Let ℛR be the collection of these row norms over all evaluated requests, layers, and rows. We summarize its concentration using R99=Q0.99(ℛ)Q0.50(ℛ)+ϵ,Rmax=max(ℛ)Q0.50(ℛ)+ϵ,R_99= Q_0.99(R)Q_0.50(R)+ε, R_ = (R)Q_0.50(R)+ε, (50) where Qq(⋅)Q_q(·) denotes the empirical q-quantile. Figure 4(c) gives R99=3.5R_99=3.5 and Rmax=12.9R_ =12.9, revealing a long-tailed distribution in which a small number of shared core-space coordinates carry disproportionately large update mass. Optimality of Dominant Direction Selection We justify dominant direction selection using the Eckart–Young–Mirsky theorem. Consider the singular value decomposition Mtl=UtlΣtl(Vtl)⊤,M_t^l=U_t^l _t^l(V_t^l) , (51) with singular values ordered non-increasingly. The rank-k truncated update is Mt,kl=Ut,1:klΣt,1:kl(Vt,1:kl)⊤.M_t,k^l=U_t,1:k^l _t,1:k^l(V_t,1:k^l) . (52) Proposition 1. Among all matrices with rank at most k, Mt,klM_t,k^l is a best approximation of MtlM_t^l under the Frobenius norm: Mt,kl∈argminrank(Z)≤k‖Mtl−Z‖F.M_t,k^l∈ _rank(Z)≤ k \|M_t^l-Z \|_F. (53) Moreover, ‖Mtl−Mt,kl‖F2=∑a>k(σt,al)2. \|M_t^l-M_t,k^l \|_F^2= _a>k( _t,a^l)^2. (54) The result follows directly from the Eckart–Young–Mirsky theorem, which states that truncating the singular value decomposition gives the minimum reconstruction error among all rank-constrained approximations under any unitarily invariant norm. For the Frobenius norm, the squared residual is the sum of the squared discarded singular values, yielding Eq. (54). □ This result provides a precise interpretation of the selection stage: for a fixed direction budget, the leading singular components retain the largest possible amount of core-update energy while introducing the smallest Frobenius reconstruction error. It does not imply that the rank-k approximation is necessarily optimal for downstream unlearning behavior; the behavioral choice of k is supported empirically by the spectral analysis and component ablation in the main paper. Row-Norm Concentration and Capacity Control After dominant direction selection, a few rows of the shared core-space update may have substantially larger norms than the remaining rows. When updates from many requests are merged, these over-concentrated coordinates can dominate the final update. MCU limits this concentration using a soft row-capacity constraint. For request t and layer l, let ℛtl=rt,al∣a=1,…,dUl,R_t^l= \r_t,a^l a=1,…,d_U^l \, (55) where rt,alr_t,a^l is defined in Eq. (49). Given quantile q, we set τtl=Qq(ℛtl). _t^l=Q_q(R_t^l). (56) The scaling coefficient of row a is γt,al=min(1,τtlrt,al+ϵ), _t,a^l= (1, _t^lr_t,a^l+ε ), (57) and the capacity-controlled update is M~tl=DtlM¯tl,Dtl=diag(γt,1l,…,γt,dUll). M_t^l=D_t^l M_t^l, D_t^l=diag ( _t,1^l,…, _t,d_U^l^l ). (58) Rows below the threshold remain unchanged, whereas rows above the threshold are rescaled to norm τtl _t^l. The operation preserves the direction of every non-zero row and changes only its magnitude. Proposition 2. For a fixed threshold τ>0τ>0, define ℬτ=Z|‖Z[a,:]‖2≤τ for every row a.B_τ= \Z\; |\;\|Z[a,:]\|_2≤τ for every row a \. (59) The row-capacity operator is the Euclidean projection of a matrix M onto ℬτB_τ: RowCapτ(M)=argminZ∈ℬτ‖Z−M‖F2.RowCap_τ(M)= _Z _τ\|Z-M\|_F^2. (60) The Frobenius objective decomposes over rows: ‖Z−M‖F2=∑a‖Z[a,:]−M[a,:]‖22.\|Z-M\|_F^2= _a\|Z[a,:]-M[a,:]\|_2^2. (61) Each row can therefore be optimized independently by projecting M[a,:]M[a,:] onto the closed Euclidean ball of radius τ. The unique solution is Z[a,:]=M[a,:],‖M[a,:]‖2≤τ,τ‖M[a,:]‖2M[a,:],‖M[a,:]‖2>τ.Z[a,:]= casesM[a,:],&\|M[a,:]\|_2≤τ,\\[4.0pt] τ\|M[a,:]\|_2M[a,:],&\|M[a,:]\|_2>τ. cases (62) This is exactly the scaling rule in Eq. (57). □ Consequently, ‖RowCapτ(M)−M‖F2=∑a[‖M[a,:]‖2−τ]+2. \|RowCap_τ(M)-M \|_F^2= _a [\|M[a,:]\|_2-τ ]_+^2. (63) Thus, for a prescribed row-norm bound, RowCap introduces the minimum possible Frobenius perturbation. Row norms are not invariant to arbitrary rotations of the shared core basis. We therefore interpret them as coordinate-concentration statistics under the deterministic basis constructed by MCU, rather than as basis-free properties of the original parameter matrix. The stronger overlap observed on the U side in Figure 4(a), together with the row-versus-column ablations, motivates applying the default capacity control along the row dimension. Sign Invariance and the Gram-Space Surrogate The compatibility objective operates on left and right singular directions separately, whereas the interaction between two rank-one updates depends on the product of their left- and right-side similarities. This distinction requires careful treatment of the sign ambiguity of the singular value decomposition. Consider two rank-one components Dp=σpupvp⊤,Dq=σquqvq⊤,D_p= _pu_pv_p , D_q= _qu_qv_q , (64) where σp,σq≥0 _p, _q≥ 0 and all direction vectors have unit norm. Their Frobenius interaction is ⟨Dp,Dq⟩F D_p,D_q _F =tr(Dp⊤Dq) =tr (D_p D_q ) (65) =σpσq(up⊤uq)(vp⊤vq). = _p _q(u_p u_q)(v_p v_q). We denote the sign-relevant component by χpq=(up⊤uq)(vp⊤vq). _pq=(u_p u_q)(v_p v_q). (66) Paired sign invariance. For any sp∈−1,+1s_p∈\-1,+1\, the paired transformation (up,vp)↦(spup,spvp)(u_p,v_p) (s_pu_p,s_pv_p) (67) leaves the rank-one matrix unchanged: σp(spup)(spvp)⊤=σpupvp⊤. _p(s_pu_p)(s_pv_p) = _pu_pv_p . (68) Under paired flips of components p and q, up⊤uq↦spsq(up⊤uq),u_p u_q s_ps_q(u_p u_q), (69) and vp⊤vq↦spsq(vp⊤vq).v_p v_q s_ps_q(v_p v_q). (70) Their product is therefore invariant: χpq _pq ↦(spsq)2(up⊤uq)(vp⊤vq) (s_ps_q)^2(u_p u_q)(v_p v_q) (71) =χpq. = _pq. A deterministic orientation rule consequently fixes only the representation of the singular vectors for reproducibility; it does not alter the underlying rank-one interaction. Sufficient separable surrogate. Let GU=SU⊤SU,GV=SV⊤SVG_U=S_U S_U, G_V=S_V S_V (72) be the left and right Gram matrices of the concatenated directions. For a cross-request pair (p,q)(p,q), (GU)pq=up⊤uq,(GV)pq=vp⊤vq.(G_U)_pq=u_p u_q, (G_V)_pq=v_p v_q. (73) If (GU)pq≥0and(GV)pq≥0,(G_U)_pq≥ 0 (G_V)_pq≥ 0, (74) then χpq=(GU)pq(GV)pq≥0. _pq=(G_U)_pq(G_V)_pq≥ 0. (75) Thus, requiring non-negative cross-request similarities on both sides is a sufficient condition for a non-antagonistic rank-one interaction. The condition is not necessary. For example, (GU)pq=−0.5,(GV)pq=−0.6(G_U)_pq=-0.5, (G_V)_pq=-0.6 (76) gives χpq=0.3>0, _pq=0.3>0, (77) even though neither factor satisfies Eq. (74). Therefore, the separate Gram constraints form a conservative surrogate rather than an exact characterization of all non-antagonistic interactions. A globally consistent orientation need not exist. Even for one side alone, paired sign choices cannot always make all pairwise similarities non-negative. Consider three directions whose pairwise similarities are all negative. To make the three oriented similarities non-negative, their signs would need to satisfy s1s2=−1,s2s3=−1,s1s3=−1.s_1s_2=-1, s_2s_3=-1, s_1s_3=-1. (78) Multiplying the first two equations gives s1s3=+1s_1s_3=+1, contradicting the third. This is the standard imbalance condition of a signed graph and shows that sign orientation alone cannot generally enforce the desired pairwise geometry. These observations motivate the formulation used by MCU. The deterministic orientation ensures reproducible inputs to the solver, while the proximity objective limits unnecessary geometric changes introduced by the conservative separable surrogate. Accordingly, MCU does not claim to preserve every pair with χpq>0 _pq>0; instead, it seeks a tractable non-antagonistic geometry that remains close to the original shared structure. Optimization of the Compatible Gram Matrix For X∈U,VX∈\U,V\, let GX,0lG_X,0^l be the original Gram matrix at layer l, and let ℐtlI_t^l contain the direction indices belonging to request t. MCU approximately solves G^Xl=argminG G_X^l= _G ‖G−GX,0l‖F2, \|G-G_X,0^l\|_F^2, (79) s.t. .t. G⪰0,rank(G)≤dXl, G 0, (G)≤ d_X^l, Gpq≥0,(p,q)∈l, G_pq≥ 0, (p,q) ^l, G[ℐtl,ℐtl]=I|ℐtl|,t=1,…,s, G[I_t^l,I_t^l]=I_|I_t^l|, t=1,…,s, where lC^l contains all cross-request direction pairs. The rank constraint makes the feasible set non-convex, so the problem is solved approximately by cyclic projections. Starting from G(0)=GX,0lG^(0)=G_X,0^l, each iteration performs the following steps: 1. Symmetrization: G←12(G+G⊤).G← 12(G+G ). (80) 2. Rank-constrained PSD projection: compute G=EΛE⊤G=E E , replace negative eigenvalues by zero, retain at most the largest dXld_X^l positive eigenvalues, and reconstruct the matrix. 3. Cross-request non-negativity projection: for every (p,q)∈l(p,q) ^l, set Gpq←max(Gpq,0),Gqp←Gpq.G_pq← (G_pq,0), G_qp← G_pq. (81) 4. Within-request identity projection: for each request t, set G[ℐtl,ℐtl]←I|ℐtl|.G[I_t^l,I_t^l]← I_|I_t^l|. (82) The procedure is repeated until the maximum number of iterations is reached or the change in the Gram matrix falls below the prescribed tolerance: ‖G(k+1)−G(k)‖F‖G(k)‖F+ϵ<η. \|G^(k+1)-G^(k)\|_F\|G^(k)\|_F+ε<η. (83) After convergence, we eigendecompose the projected Gram matrix: G^Xl=EXlΛXl(EXl)⊤. G_X^l=E_X^l _X^l(E_X^l) . (84) Let rXl=rank(G^Xl)r_X^l=rank( G_X^l). A compact factor realizing this Gram geometry is ZXl=(ΛX,+l)1/2(EX,+l)⊤∈ℝrXl×Nl,(ZXl)⊤ZXl=G^Xl,Z_X^l=( _X,+^l)^1/2(E_X,+^l) ^r_X^l× N_l, (Z_X^l) Z_X^l= G_X^l, (85) where NlN_l is the total number of retained directions and the ++ subscript denotes the positive eigenspace. When rXl<dXlr_X^l<d_X^l, ZXlZ_X^l is padded with zero rows to match the ambient direction dimension. Because a Gram matrix determines its factor only up to a left orthogonal transformation, the recovered directions are aligned with the original concatenated directions using an orthogonal Procrustes step before being partitioned back into request-specific blocks. The resulting left and right blocks are combined with the retained singular values to reconstruct the reconfigured core updates. Cyclic projection over a non-convex feasible set does not guarantee a global optimum. We therefore interpret this procedure as an approximate solver that seeks a nearby feasible geometry. In practice, the proximity objective and deterministic initialization provide stable solutions across the evaluated request scales. Limitations and Scope MCU provides an efficient merging-based solution to continual multimodal unlearning, but several limitations remain. Conservative compatibility surrogate. The separate non-negativity constraints on the left and right Gram matrices are sufficient but not necessary for a non-antagonistic rank-one interaction. They may therefore modify some direction pairs whose sign-invariant product is already non-negative. The proximity objective limits this effect, but it does not make the surrogate exact. Parameter geometry versus functional behavior. The proposed interaction measures characterize relationships among parameter-space update directions. They do not fully determine how two updates interact on model outputs, hidden representations, or gradients. The leave-one-task-out analysis provides behavioral evidence that the geometric reconfiguration reduces cross-task deviations, but a complete functional characterization remains an open direction. Basis dependence of capacity control. Row norms are defined in the deterministic shared core basis and are not invariant under arbitrary rotations of that basis. Accordingly, RowCap should be interpreted as controlling core-coordinate concentration under the chosen representation rather than as identifying basis-free model channels. Dependence on one-shot adapters. MCU consolidates, rather than replaces, an underlying one-shot unlearning method. Its final performance is therefore constrained by the quality of the request-specific adapters supplied to the merging stage. Weak or unstable one-shot unlearning adapters cannot be fully corrected by geometric processing alone. Temporary adapter storage. Before consolidation, storing the request-specific LoRA bank incurs a cost that grows linearly with the number of requests. Although this cost is substantially smaller than archiving full-model checkpoints under rollback requirements, it is not constant-memory online unlearning. Hierarchical consolidation may reduce storage, but can discard request-level structure. Our experiments cover two multimodal model backbones and two unlearning benchmarks. The order-robustness study considers the original, reversed, and one fixed random permutation. Broader evaluation across additional architectures, request distributions, and adversarial relearning settings would further clarify the generality and security properties of MCU.