Paper deep dive
Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation
Chunlin Liu, Junnian Chen, Haitong Jiang, Jianyu Zhao, Yingsen Pang, Jingchen Li, Jiabiao He, Youming Lu, Jinhe Bi, Yuntao Du
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpora. Removing such knowledge is essential for building trustworthy AI systems. However, existing studies primarily focus on forgetting within individual modalities. Although recent work has begun to explore cross-modal consistency in unlearning, the cross-modal transfer of real-world knowledge unlearning remains insufficiently studied. To address this gap, we introduce UNLINK-VL, a real-world benchmark for cross-modal knowledge unlearning in VLMs. Under a post-hoc unlearning setting in which the original forget and retain corpora are unavailable, UNLINK-VL selects visually identifiable real-world entities as unlearning targets and associates them with corresponding images and one-hop and multi-hop facts derived from Wikidata. The benchmark comprises four complementary subsets that evaluate direct forgetting of target knowledge, the propagation of forgetting through relational knowledge, the preservation of related non-target knowledge, and robustness to semantically equivalent queries. We train models under text-only and multimodal unlearning settings and evaluate forgetting effectiveness and retained utility across textual, visual, and cross-modal scenarios. Extensive experiments reveal a pronounced asymmetry in cross-modal transfer: multimodal unlearning remains effective under textual evaluation, whereas text-only unlearning transfers poorly to visual and cross-modal scenarios. Meanwhile, the evaluated methods largely preserve the models' general capabilities. These findings demonstrate that relying solely on intra-modal evaluation, particularly text-only evaluation, may substantially overestimate the effectiveness of knowledge unlearning in VLMs, underscoring the need for cross-modal unlearning and evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2608.03791v1
- Canonical: https://arxiv.org/abs/2608.03791v1
Trouble viewing inline? Open PDF directly →
Full Text
47,837 characters extracted from source content.
Expand or collapse full text
Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation Chunlin Liu1,*, Junnian Chen1,*, Haitong Jiang1, Jianyu Zhao2, Yingsen Pang1, Jingchen Li1, Jiabiao He1, Youming Lu1, Jinhe Bi3, Yuntao Du4,† Abstract Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpora. Removing such knowledge is essential for building trustworthy AI systems. However, existing studies primarily focus on forgetting within individual modalities. Although recent work has begun to explore cross-modal consistency in unlearning, the cross-modal transfer of real-world knowledge unlearning remains insufficiently studied. To address this gap, we introduce UNLINK-VL, a real-world benchmark for cross-modal knowledge unlearning in VLMs. Under a post-hoc unlearning setting in which the original forget and retain corpora are unavailable, UNLINK-VL selects visually identifiable real-world entities as unlearning targets and associates them with corresponding images and one-hop and multi-hop facts derived from Wikidata. The benchmark comprises four complementary subsets that evaluate direct forgetting of target knowledge, the propagation of forgetting through relational knowledge, the preservation of related non-target knowledge, and robustness to semantically equivalent queries. We train models under text-only and multimodal unlearning settings and evaluate forgetting effectiveness and retained utility across textual, visual, and cross-modal scenarios. Extensive experiments reveal a pronounced asymmetry in cross-modal transfer: multimodal unlearning remains effective under textual evaluation, whereas text-only unlearning transfers poorly to visual and cross-modal scenarios. Meanwhile, the evaluated methods largely preserve the models’ general capabilities. These findings demonstrate that relying solely on intra-modal evaluation, particularly text-only evaluation, may substantially overestimate the effectiveness of knowledge unlearning in VLMs, underscoring the need for cross-modal unlearning and evaluation. Introduction Figure 1: Comparison between intra-modal and cross-modal knowledge unlearning. Vision-Language Models (VLMs), like Large Language Models (LLMs), learn from large-scale pretraining datasets and may memorize sensitive information, copyrighted content, or harmful knowledge (Carlini et al. 2021; Ju et al. 2025; Liu et al. 2025; Zheng et al. 2025). Once encoded in model parameters, such information or knowledge may later be elicited or reproduced during downstream deployment, posing privacy, copyright, and safety risks. Retraining a model from scratch after removing the target data from the training datasets is often prohibitively expensive (Bourtoule et al. 2021; Yao et al. 2024). Machine unlearning seeks to remove the influence of designated training data or knowledge from a trained model while preserving non-target knowledge and general capabilities (Ginart et al. 2019; Guo et al. 2020; Bourtoule et al. 2021). Several benchmarks have been proposed to evaluate knowledge unlearning. Text-only benchmarks evaluate the forgetting of synthetic profiles, hazardous knowledge, and real-world entities using text-only inputs for both unlearning and evaluation (Maini et al. 2024; Li et al. 2024c; Jin et al. 2024). Multimodal benchmarks extend evaluation to text-image inputs, facial identities, visual concepts, misinformation, and sensitive associations (Liu et al. 2025; Dontsov et al. 2025; Ma et al. 2025b; Li et al. 2024b; Zheng et al. 2025; Selvas-Sala et al. 2026). Existing works mainly investigate intra-modal unlearning, where the model is trained to forget knowledge in one modality and evaluated in the same modality. Despite the progress enabled by these benchmarks, they largely overlook cross-modal transfer of the unlearning effect, which is a realistic and critical challenge in real-world multimodal applications. For VLMs, robust knowledge unlearning should remain effective under both intra-modal and cross-modal evaluation. The same fact can be elicited through semantically equivalent text-only and text-image inputs. For example, a person may be specified by name in a text-only question or represented by an image in a text-image question, with both queries targeting the same biographical fact. A model may appear to have forgotten that fact when queried by name yet still recover it when shown the person’s image. The unlearning effect should therefore exhibit cross-modal consistency, regardless of how unlearning is performed. Recent benchmarks have begun to explore whether unlearning remains effective across different modality settings. UMU-Bench targets modality misalignment by evaluating unimodal, multimodal, and hybrid unlearning under both unimodal and multimodal settings (Wang et al. 2025). PPU-Bench evaluates complete, selective, and personalized unlearning on pre-existing knowledge about real-world public figures using paired text-only QA and multimodal VQA samples (Guang et al. 2026). However, these benchmarks either focus primarily on synthetic profile knowledge or do not treat the unlearning modality as an independent experimental variable. In this work, we study the cross-modal transfer problem: if a fact is unlearned through text-only inputs, can it still transfer to a semantically equivalent text-image question, and if it is unlearned through a text-image input, can it still transfer to an equivalent text-only question? To answer these questions, one could evaluate T→MT→ M and M→TM→ T as cross-modal transfer settings, where T and M denote text-only and text-image inputs, respectively. We also report T→T→ T and M→M→ M as intra-modal reference settings. This directionality matters in practice because a deletion request may identify target knowledge in one modality, whereas later users may query the same knowledge through another. To this end, we introduce UNLINK-VL, a benchmark grounded in real-world knowledge for evaluating cross-modal knowledge unlearning in VLMs. UNLINK-VL selects a diverse set of visually identifiable real-world entities as unlearning targets, each associated with factual knowledge already accessible to the original model. For each entity, we first collect and filter relevant one-hop facts and two-hop paths from Wikidata (Vrandečić and Krötzsch 2014). We then verbalize this structured knowledge into text-only passages for unlearning and construct text-only QA pairs for evaluation. For both data types, we create aligned text-image counterparts by replacing the entity name with an image-based reference and pairing the resulting text with representative images. For unlearning, UNLINK-VL constructs four forms of supervision tailored to representative unlearning methods, including GA, DPO, NPO, and RT (Jang et al. 2023; Rafailov et al. 2023; Zhang et al. 2024). Faithful positive passages are verbalized directly from the collected Wikidata facts, while counterfactual negative passages are generated by modifying factual claims in the positive passages. GA and NPO suppress the faithful positive passages, while DPO uses paired counterfactual negative and faithful positive passages. For RT, each target query is paired with a generic refusal response, such as “I don’t know,” to train the model to abstain from answering questions about the target knowledge. For evaluation, UNLINK-VL organizes the structured knowledge into four complementary subsets. The Forget Set evaluates direct forgetting of target facts; the Multi-Hop Set examines whether forgetting extends through two-hop relational paths; the Retain Set evaluates the preservation of non-target facts; and the Rewrite Set assesses whether forgetting remains effective under semantically equivalent queries. Each evaluated fact is represented in aligned text-only and text-image forms. To prevent leakage, the unlearning and evaluation splits use disjoint images. We conduct extensive experiments with four representative unlearning methods on multiple VLMs. The results reveal a pronounced asymmetry in cross-modal transfer: multimodal unlearning transfers more effectively to text-only evaluation than text-only unlearning transfers to text-image evaluation. Meanwhile, the evaluated methods largely preserve general capabilities. These findings show that text-only evaluation may substantially overestimate unlearning effectiveness in VLMs and highlight the need to consider both text-only and text-image inputs during unlearning and evaluation. In summary, our main contributions are: • We introduce UNLINK-VL, a real-world benchmark for evaluating the cross-modal transfer of unlearning effects in VLMs. • We develop a systematic pipeline that constructs method-specific unlearning data and aligned text-only and text-image evaluation samples from pre-existing model knowledge, enabling comprehensive evaluation of forgetting, robustness, and knowledge preservation. • Through controlled experiments that independently vary the unlearning and evaluation modalities, we reveal a pronounced asymmetry in cross-modal transfer and show that the evaluated methods largely preserve general capabilities. Related Work Machine Unlearning Methods Machine unlearning aims to remove targeted data or knowledge from a trained model without retraining it from scratch (Ginart et al. 2019; Guo et al. 2020; Bourtoule et al. 2021). In large language models, factual knowledge is distributed across model parameters, making it challenging to erase targeted knowledge while preserving unrelated capabilities. Representative unlearning methods include GA (Jang et al. 2023), DPO (Rafailov et al. 2023), NPO (Zhang et al. 2024), and RMU (Li et al. 2024c). However, fine-tuning may alter how knowledge is retrieved rather than eliminate the underlying knowledge, allowing it to be recovered through paraphrased or adversarial queries (Yao et al. 2024; Bărbulescu and Triantafillou 2024; Xu et al. 2025; Hong et al. 2024; Patil et al. 2024; Joshi et al. 2024). This limitation is particularly pronounced in entity-level unlearning, where target entities are embedded in interconnected networks of factual relations (Ma et al. 2025a). Figure 2: The four-stage construction pipeline of UNLINK-VL. Machine Unlearning Benchmarks Existing machine-unlearning benchmarks evaluate synthetic profiles, hazardous knowledge, and the trade-off between forgetting efficacy and model utility (Maini et al. 2024; Li et al. 2024c; Shi et al. 2025). RWKU extends this line of research to a practical post-hoc setting, in which a model must forget previously acquired knowledge about public figures without access to the original forget corpora (Jin et al. 2024). A key challenge in this setting is to distinguish genuine forgetting from weak prior knowledge, as a model that initially knows little about a target may appear to have been successfully unlearned (Kwon et al. 2026). Consequently, unlearning effects achieved in one modality may not transfer to the other modality (Li et al. 2026). Existing benchmarks focus on different aspects of multimodal unlearning. MLLMU-Bench evaluates the forgetting of synthetic personal knowledge in text-only and text-image settings (Liu et al. 2025). FIU-Bench studies identity-related knowledge associated with synthetic facial images and tests whether such knowledge can be recovered through privacy attacks (Ma et al. 2025b). Other benchmarks extend multimodal unlearning to visual concepts, misinformation, and sensitive associations (Li et al. 2024b; Zheng et al. 2025; Selvas-Sala et al. 2026). Collectively, these studies show that unlearning effectiveness depends on what is forgotten, how it is represented, and how it is evaluated. Among existing benchmarks, UMU-Bench and PPU-Bench are most closely related to our setting. UMU-Bench evaluates unimodal, multimodal, and mixed-modality unlearning using synthetic profiles, with a focus on modality alignment after unlearning (Wang et al. 2025). By contrast, PPU-Bench focuses on knowledge that models already possess about public figures and evaluates complete, selective, and personalized unlearning through text-only and text-image QA (Guang et al. 2026). UNLINK-VL differs from both benchmarks by focusing on real-world knowledge and systematically examining whether unlearning effects induced in one modality transfer to another. Appendix D provides a detailed comparison with representative text-only and multimodal unlearning benchmarks. UNLINK-VL Benchmark Task Definition Knowledge unlearning. Suppose that an already trained model MvanillaM_vanilla is asked to forget a set of target entities EforgetE_forget. The goal is simple: after unlearning, the model should no longer answer questions about the target knowledge, but should still answer questions about unrelated knowledge. Conceptually, the original training set D can be divided into the data to be forgotten, DforgetD_forget, and all remaining data, DrestD_rest: D=Dforget∪Drest,Dforget∩Drest=∅,Munlearn=A(Mvanilla,Dforget). gatheredD=D_forget∪ D_rest, D_forget∩ D_rest= ,\\ M_unlearn=A(M_vanilla,D_forget). gathered (1) Here, A is the unlearning algorithm. The ideal solution would remove DforgetD_forget and retrain from scratch on DrestD_rest; A instead updates MvanillaM_vanilla directly to obtain MunlearnM_unlearn (Ginart et al. 2019; Guo et al. 2020; Bourtoule et al. 2021). In our practical post-hoc setting (Jin et al. 2024), the original DforgetD_forget and DrestD_rest are unavailable. We therefore construct a compact DforgetD_forget from public facts about EforgetE_forget. Cross-modal knowledge unlearning. A fact is represented as a triple k=(E,r,o)k=(E,r,o), where E is the subject entity, r is the relation, and o is the correct answer. For example, the fact k=(TaylorSwift,placeofbirth,WestReading)k=(Taylor\ Swift,\ place\ of\ birth,\ West\ Reading) (2) states that Taylor Swift was born in West Reading. For each fact (E,r,o)(E,r,o), we construct two semantically equivalent questions that differ only in how the subject entity is identified: Qk,T Q_k,T =GT(E,r), =G_T(E,r), (3) Qk,M Q_k,M =(IE,GM(E,r)), = (I_E,G_M(E,r) ), Ak,T A_k,T =Ak,M=o. =A_k,M=o. Here, GTG_T generates a text-only question that explicitly names E, whereas GMG_M generates an image-grounded question that refers to E through the image IEI_E. Both questions have the same answer o. Further details on the paired query construction and evaluation settings are provided in Appendix B. Our training and evaluation settings are as follows: T→T,T→M,M→T,M→M.T\!→\!T, T\!→\!M, M\!→\!T, M\!→\!M. (4) In a→ba\!→\!b, a and b denote the unlearning and evaluation modalities, respectively. The diagonal settings (T→T\!→\!T and M→M\!→\!M) are intra-modal, whereas the off-diagonal settings (T→MT\!→\!M and M→TM\!→\!T) evaluate cross-modal transfer. Dataset Construction To support reliable cross-modal evaluation while avoiding prompt and image leakage, we construct separate unlearning and evaluation branches grounded in the same underlying facts, as illustrated by the four-stage pipeline in Figure 2. Our pipeline draws on RWKU’s focus on real-world target entities accessible to the evaluated models and their neighboring knowledge (Jin et al. 2024), while following MMKU-Bench’s staged organization of knowledge collection, cross-modal data construction, and quality control (Fu et al. 2026). By keeping the target knowledge aligned across data splits, this design enables controlled evaluation of both intra-modal forgetting and cross-modal transfer. Figure 3: Entity-category distribution of the one-hop Forget and Retain evaluation sets. Stage 1: Target entity selection and knowledge collection. To ensure that the unlearning targets are recognizable from both their names and images while avoiding image leakage between unlearning and evaluation, we begin with a broad pool of visually identifiable real-world entities spanning people, landmarks, organizations, countries, films, albums, games, and other categories. We retrieve candidate images through image search and conduct probing under both text-only and image-only settings on the Qwen3-VL and Llama-3.2-Vision model families to exclude entities unfamiliar to either family (Kwon et al. 2026). The complete probing protocol and filtering criteria are provided in Appendix B. After filtering, 250 target entities remain. For each target, we collect two representative images, both of which independently satisfy the image-only probing criteria for both model families. To distinguish entity-level forgetting from image-specific effects, we use Image2 for multimodal unlearning and reserve Image1 for evaluation. To assess not only forgetting but also its propagation through relational chains, we link each target entity to Wikidata (Vrandečić and Krötzsch 2014) and retrieve both direct triples (E,r,o)(E,r,o) and valid two-hop paths (E,r1,E′),(E′,r2,o)(E,r_1,E ),(E ,r_2,o) (Zhong et al. 2023). We further identify semantically related non-target entities for retention evaluation (Cohen et al. 2024). Each candidate entity and its associated image undergo the same text-only and image-only probing procedure, and only those recognized by both model families in both settings are retained. These entities constitute EretainE_retain. Stage 2: Test-set construction. In order to separately evaluate forgetting, robustness to paraphrasing, recoverability through relational composition, and preservation of non-target knowledge, we organize the collected facts into four complementary evaluation subsets. The Forget Set consists of one-hop facts and measures whether the target knowledge is successfully forgotten. The Rewrite Set consists of GPT-4o-generated paraphrases of the Forget Set questions to evaluate robustness to alternative phrasings. The Multi-Hop Set is constructed from valid two-hop paths and assesses whether the target knowledge can still be recovered through relational composition. Finally, the Retain Set contains one-hop facts about non-target entities and evaluates whether such knowledge is preserved after unlearning. GPT-4o converts the structured facts into clear, unambiguous questions with short answers grounded in the source triples. To guarantee that input modality acts as the sole variable in cross-modal comparisons, we instantiate each evaluation item as a fact-aligned modality pair. In the text-only version, the subject entity is named explicitly; in the text-image version, the question is paired with an image, and the entity name is replaced with an image-grounded referring expression, while the queried relation and answer remain unchanged. Thus, the two versions differ only in how the subject entity is accessed. The complete prompts used for KG-to-QA generation, semantic rewriting, and cross-modal conversion are provided in Appendix C. Stage 3: Train-set construction. Since the pretrained datasets are unavailable, we construct model-specific synthetic training data from triples associated with the target entities retained after probing. Qwen3-VL-8B and Llama-3.2-Vision-11B each generate the forget corpus for their corresponding model family. To enable a controlled comparison between text-only and multimodal unlearning, we construct fact-aligned text-image counterparts for all text-only unlearning instances in the same manner as the evaluation pairs. Dataset statistics are summarized in Table 1 and Figure 3. Stage 4: Quality control. To prevent data-construction artifacts from being mistaken for unlearning effects and undermining the validity of the benchmark, we first apply automated screening to remove malformed, duplicate, ambiguous, or inconsistent samples, as well as unusable images. Human reviewers then correct unintended errors introduced during triple-to-QA conversion and verify the accuracy of image selection and image–entity alignment. Appendix E details the review unit, checklist, correction procedure, and final verification pass. Statistics Number Training set (per model family) Faithful samples 13,173 Counterfactual samples 13,173 Text preference pairs 13,173 Refusal samples 10,000 Test set Forget Set 11906 Multi-Hop Set 3398 Rewrite Set 11906 Retain Set 20351 Entity Forget 250 Retain 500 Table 1: Key statistics of UNLINK-VL. T→MT\!→\!M M→M\!→\!M M→TM\!→\!T T→T\!→\!T Method F↓ R↑ MH↓ RW↓ F↓ R↑ MH↓ RW↓ F↓ R↑ MH↓ RW↓ F↓ R↑ MH↓ RW↓ Vanilla 82.6 80.8 66.9 79.4 82.6 80.8 66.9 79.4 85.1 82.3 69.7 82.0 85.1 82.3 69.7 82.0 GA (Full) 74.2 63.8 62.8 72.5 52.4 61.5 54.6 57.1 66.8 65.0 60.2 68.9 55.1 63.2 56.8 59.5 GA (LoRA) 79.0 72.8 65.2 76.5 65.3 70.6 60.7 67.2 73.5 73.7 64.3 74.9 67.2 71.4 62.8 68.8 DPO (Full) 75.8 72.9 63.0 73.6 56.0 70.8 55.8 59.8 68.4 73.8 60.9 70.4 58.9 71.6 57.4 61.6 DPO (LoRA) 79.6 78.7 65.0 77.0 67.1 77.9 61.2 68.6 74.6 79.2 64.1 75.4 68.5 77.1 62.0 69.2 NPO (Full) 71.0 68.9 61.8 69.4 45.2 66.1 50.7 53.8 62.9 69.8 57.8 66.3 49.8 67.4 53.1 55.6 NPO (LoRA) 75.4 75.5 63.6 73.4 58.4 73.7 56.9 61.2 69.2 76.6 61.2 71.0 62.0 73.8 58.5 63.5 RT (Full) 72.6 71.7 62.1 70.5 48.8 69.9 52.3 55.4 64.3 72.4 58.5 67.4 52.9 70.1 54.2 57.2 RT (LoRA) 76.8 77.6 64.1 74.5 60.5 76.2 58.1 63.1 70.8 78.3 61.8 72.6 63.8 75.4 59.4 65.2 Table 2: The results of Qwen3-VL-8B on UNLINK-VL. T and M denote text-only and multimodal inputs, respectively. F, R, MH, and RW denote the Forget, Retain, Multi-Hop, and Rewrite scores, respectively. Unlearning Methods We evaluate four representative methods. GA lowers the likelihood of the correct target response y+y^+ (Jang et al. 2023); DPO ranks the counterfactual response y−y^- above the faithful response y+y^+ relative to the original model (Rafailov et al. 2023); NPO suppresses y+y^+ without requiring a preferred completion (Zhang et al. 2024); and RT applies supervised fine-tuning to yrefy^ref. All four methods are implemented with both full-parameter fine-tuning and LoRA. For each target fact, the training data provide four forms of supervision: a faithful positive response y+y^+, a fluent counterfactual negative response y−y^-, a preference pair (y−,y+)(y^-,y^+), and a standardized refusal response yrefy^ref. Here, y−y^- is the preferred completion and y+y^+ is the rejected completion in the preference pair. Positive and negative responses are generated and filtered separately for each target model, and their matched combination forms the paired supervision used by DPO. The same fact is instantiated across different modalities and supervision schemes, enabling controlled comparisons of GA, DPO, NPO, and RT while keeping the target knowledge fixed. Metrics We evaluate knowledge unlearning along two standard axes: forgetting efficacy and retained utility (Maini et al. 2024; Zhang et al. 2024). We report answer accuracy on each subset. Lower accuracy on the Forget, Multi-Hop, and Rewrite Sets indicates stronger forgetting, whereas higher accuracy on the Retain Set indicates better preservation of neighboring knowledge. Besides, general capabilities are reported using the official metric of each external benchmark. Experiments Method Comprehensive OCR Multidisciplinary Instruction Mathematical Hallucination MME↑ MMBench↑ SEED↑ OCRBench↑ ScienceQA↑ MIA-Bench↑ MathVista↑ MathVision↑ POPE↑ HallusionBench↑ Vanilla 1734.77 75.43 67.24 90.9 74.27 93.83 78.9 49.01 88.44 73.50 T GA (LoRA) 1732.25 73.80 64.78 91.2 68.52 93.45 76.2 53.29 88.05 72.87 DPO (LoRA) 1736.63 75.34 65.88 90.5 72.38 94.04 78.3 50.00 88.13 74.13 NPO (LoRA) 1737.43 75.17 65.04 90.5 70.05 94.21 77.5 51.97 88.03 73.92 RT (LoRA) 1729.35 76.03 67.19 91.0 73.62 93.76 77.5 52.63 87.95 72.77 M GA (LoRA) 1724.85 75.09 66.01 91.1 68.47 94.23 78.2 48.68 88.04 73.19 DPO (LoRA) 1744.75 76.46 68.07 90.5 75.76 94.52 78.3 52.30 88.16 72.98 NPO (LoRA) 1737.52 75.00 66.10 91.6 71.39 93.97 78.1 51.32 87.97 74.34 RT (LoRA) 1742.37 75.60 66.93 90.9 73.03 94.30 76.5 51.32 87.93 73.71 Table 3: General-capability results for Qwen3-VL-8B after unlearning. T→MT\!→\!M M→M\!→\!M M→TM\!→\!T T→T\!→\!T Method F↓ R↑ MH↓ RW↓ F↓ R↑ MH↓ RW↓ F↓ R↑ MH↓ RW↓ F↓ R↑ MH↓ RW↓ Vanilla 85.9 84.1 71.8 83.2 85.9 84.1 71.8 83.2 88.0 85.4 74.5 85.6 88.0 85.4 74.5 85.6 GA 82.3 75.3 67.2 79.2 69.3 73.1 63.1 70.1 77.2 76.2 66.7 77.8 71.2 73.9 65.2 71.7 DPO 82.8 81.2 67.3 79.8 71.2 80.5 63.7 71.5 78.5 82.0 66.6 78.5 72.5 79.9 64.6 72.3 NPO 78.8 78.1 66.0 76.5 62.8 76.4 59.4 64.5 73.4 79.3 63.8 74.2 66.5 76.5 61.2 66.7 RT 80.2 80.2 66.6 77.5 64.9 78.9 60.6 66.4 75.0 81.0 64.4 75.5 68.4 78.2 62.2 68.5 Table 4: Results on UNLINK-VL using Qwen3-VL-32B with LoRA. Experimental Setup Models. We evaluate UNLINK-VL on Qwen3-VL-8B, Qwen3-VL-32B, and Llama-3.2-Vision-11B. The two model families use different vision–language integration mechanisms, making them suitable for testing whether cross-modal unlearning patterns generalize across architectures. The main paper reports representative Qwen3-VL-8B results and a LoRA-based 8B–32B scale comparison, while the consolidated Llama-3.2-Vision-11B results are deferred to Appendix A. Implementation. All models are trained in bf16 for three epochs with a maximum sequence length of 1,024, a per-device batch size of 1, and four gradient-accumulation steps. We use a cosine learning-rate schedule with 20 warm-up steps; the learning rate is selected for each objective and backbone. LoRA uses rank r=32r=32, scaling factor α=64α=64, and targets all supported linear modules. Full-parameter training optionally uses DeepSpeed ZeRO-2 (Rajbhandari et al. 2020). The 8B and 11B LoRA runs use two NVIDIA A100-80GB GPUs, and their full-parameter runs use four; the 32B LoRA runs use four A100-80GB GPUs. We train separate text-only and multimodal unlearning variants for each method and evaluate them with the same standalone pipeline over the Forget, Multi-Hop, Rewrite, and Retain Sets in both modalities. Figure 4: Case Study Main Results For compact presentation, Table 2 merges the four unlearning–evaluation configurations for Qwen3-VL-8B. Each configuration contains Forget (F), Retain (R), Multi-Hop (MH), and Rewrite (RW) accuracy. (1) Intra-modal unlearning is generally more effective with multimodal inputs. Since text-only and text-image evaluations have different Vanilla baselines, we compare baseline-normalized Forget reductions. In six of eight method–update pairs, M→M→ M yields a larger relative reduction than T→T→ T, by 1.3–3.8 percentage points. The two LoRA exceptions are GA, which is nearly tied (20.9 % versus 21.0%), and DPO, where T→T→ T is slightly stronger (19.5% versus 18.8%). Thus, multimodal inputs generally strengthen intra-modal suppression, though not uniformly under LoRA. (2) Cross-modal forgetting remains incomplete and asymmetric. For every method and update regime, cross-modal evaluation yields a smaller baseline-normalized Forget reduction than the corresponding intra-modal evaluation. The relative T→MT→ M penalty ranges from 15.9 to 27.4 percentage points, whereas the M→TM→ T penalty ranges from 6.4 to 19.2 points. Moreover, the latter is smaller for all eight method–update pairs, averaging 12.2 points compared with 21.2 for T→MT→ M. These normalized comparisons remove modality-specific Vanilla differences and confirm that multimodal unlearning transfers more effectively to text-only queries than text-only unlearning transfers to text-image queries, so same-modality evaluation can overestimate forgetting. (3) NPO forgets most, DPO preserves most, and RT provides the best balance. Under the same update and unlearning–evaluation configuration, NPO always achieves the lowest Forget accuracy and therefore removes the most target knowledge. DPO always achieves the highest Retain accuracy and causes the least damage to neighboring knowledge. RT forgets more target knowledge than DPO while preserving more neighboring knowledge than NPO, giving the best overall balance. GA performs less favorably: it forgets less than NPO and RT while usually obtaining the lowest Retain accuracy. (4) Neighboring knowledge is largely preserved, but forgotten knowledge remains recoverable. Across all trained variants and modality configurations, Retain accuracy remains between 61.5 and 79.2, showing that much neighboring knowledge survives, although some collateral damage remains. However, accuracy on the Multi-Hop and Rewrite Sets remains substantial, ranging from 50.7 to 65.2 and from 53.8 to 77.0, respectively. Thus, even when a model fails to answer a direct Forget Set question, it may recover the same target knowledge through relational reasoning or a semantically equivalent question. Effective unlearning needs to preserve neighboring knowledge while preventing such recovery. Analysis Experiments General-Capability Evaluation. We evaluate unrelated utility on the ten benchmarks listed in Table 3 using their official protocols (Fu et al. 2023; Liu et al. 2024a; Li et al. 2024a; Liu et al. 2024b; Lu et al. 2022; Qian et al. 2025; Lu et al. 2024; Wang et al. 2024; Li et al. 2023; Guan et al. 2024). Table 3 shows that LoRA generally preserves broad utility, although losses are task and objective dependent. MME stays within −0.57%-0.57\% to +0.58%+0.58\% of Vanilla, while OCRBench and POPE change by at most 0.7 and 0.51 points. The largest drops occur in scientific and mathematical reasoning: GA reduces ScienceQA from 74.27 to 68.52 (text) and 68.47 (multimodal), and its text variant lowers MathVista from 78.9 to 76.2. DPO with multimodal unlearning remains near or above Vanilla on several broad and instruction-following benchmarks. Neither unlearning modality is uniformly better; small gains are treated as retained capability or evaluation variation. Thus, general utility is largely preserved, but the few pronounced losses must be considered alongside forgetting and locality. Full-Parameter Fine-Tuning versus LoRA. The matched runs reveal a clear forgetting-locality trade-off. Relative to LoRA, full-parameter tuning lowers Forget accuracy by 3.8–13.2 points, but also lowers Retain accuracy by 5.3–9.1 points; it similarly strengthens Multi-Hop and Rewrite suppression in every setting. NPO (Full) suppresses most strongly, DPO (LoRA) preserves the most neighboring knowledge, and RT (LoRA) provides the best balance between them. LoRA also better maintains the broad utility reported in Table 3. Full tuning is therefore preferable when suppression is paramount, whereas LoRA is the more conservative choice when locality and general utility matter. Model-Size Effects. To account for different Vanilla baselines, we compare baseline-normalized changes. Across matched LoRA runs, the 32B Forget reductions are 0.02–2.76 percentage points smaller, indicating greater resistance to fixed-rank LoRA. Although absolute Retain accuracy is 2.5–2.8 points higher, proportional Retain loss is 0.05–0.85 points larger, showing that the absolute advantage mainly reflects the stronger baseline. Method ordering remains stable: NPO suppresses most, DPO preserves most, and RT offers a favorable balance. Cross-modal asymmetry persists: the normalized T→MT→ M penalty is 14.0–16.2 points, versus 6.3–10.3 for M→TM→ T. Case Study Figure 4 presents a representative example of the paired text-only and text-image instances used for unlearning and evaluation, all constructed from the same underlying fact. It further illustrates the four unlearning–evaluation settings and the corresponding model outputs under different unlearning methods and tuning strategies, enabling a direct comparison of intra-modal and cross-modal transfer as well as full-parameter and LoRA-based adaptation. Conclusion We introduced UNLINK-VL, a real-world benchmark for cross-modal knowledge unlearning in Vision-Language Models. UNLINK-VL independently controls the modalities used for unlearning and evaluation, separating intra-modal forgetting from cross-modal transfer. It grounds model-known, visually identifiable entities in one-hop facts and two-hop Wikidata paths and organizes paired text-only and text-image probes into the Forget, Multi-Hop, Rewrite, and Retain Sets. Together, these components evaluate forgetting, relational propagation, robustness, neighboring-knowledge preservation, and general utility. The experiments reveal a pronounced transfer asymmetry: multimodal unlearning transfers more effectively to text-only evaluation than text-only unlearning transfers to text-image evaluation. At the same time, the evaluated methods largely preserve general capabilities. These findings show that text-only evaluation can substantially overestimate knowledge unlearning in VLMs and underscore the need for cross-modal unlearning and evaluation. References G. Bărbulescu and P. Triantafillou (2024) To each (Textual sequence) its own: improving memorized-data unlearning in large language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 3003–3023. External Links: Link Cited by: Machine Unlearning Methods. L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot (2021) Machine unlearning. In 2021 IEEE Symposium on Security and Privacy, p. 141–159. External Links: Document, Link Cited by: Introduction, Machine Unlearning Methods, Task Definition. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel (2021) Extracting training data from large language models. In 30th USENIX Security Symposium, p. 2633–2650. External Links: Link Cited by: Introduction. R. Cohen, E. Biran, O. Yoran, A. Globerson, and M. Geva (2024) Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics 12, p. 283–298. External Links: Document, Link Cited by: Dataset Construction. A. Dontsov, D. Korzh, A. Zhavoronkin, B. Mikheev, D. Bobkov, A. Alanov, O. Y. Rogov, I. Oseledets, and E. Tutubalina (2025) CLEAR: character unlearning in textual and visual modalities. In Findings of the Association for Computational Linguistics: ACL 2025, p. 20582–20603. External Links: Document, Link Cited by: Introduction. B. Fu, Y. Du, C. Chang, B. Jin, W. Deng, M. Xu, H. Yan, W. Song, and Y. Wan (2026) MMKU-Bench: a multimodal update benchmark for diverse visual knowledge. External Links: 2603.15117, Link Cited by: Dataset Construction. C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, R. Ji, C. Shan, and R. He (2023) MME: a comprehensive evaluation benchmark for multimodal large language models. External Links: 2306.13394, Link Cited by: Analysis Experiments. A. Ginart, M. Y. Guan, G. Valiant, and J. Zou (2019) Making AI forget you: data deletion in machine learning. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: Introduction, Machine Unlearning Methods, Task Definition. T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou (2024) HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: Analysis Experiments. J. Guang, Z. Zhan, Z. Xu, C. Gao, H. Wang, J. Li, Z. Gu, and Y. Zhang (2026) PPU-Bench: real-world multimodal benchmark for personalized partial unlearning. External Links: 2605.08800, Link Cited by: Introduction, Machine Unlearning Benchmarks. C. Guo, T. Goldstein, A. Hannun, and L. van der Maaten (2020) Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 3832–3842. External Links: Link Cited by: Introduction, Machine Unlearning Methods, Task Definition. Y. Hong, Y. Zou, L. Hu, Z. Zeng, D. Wang, and H. Yang (2024) Dissecting fine-tuning unlearning in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 3933–3941. External Links: Document, Link Cited by: Machine Unlearning Methods. J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo (2023) Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14389–14408. External Links: Document, Link Cited by: Introduction, Machine Unlearning Methods, Unlearning Methods. Z. Jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y. Chen, K. Liu, and J. Zhao (2024) RWKU: benchmarking real-world knowledge unlearning for large language models. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: Introduction, Machine Unlearning Benchmarks, Task Definition, Dataset Construction. A. Joshi, S. Saha, D. Shukla, S. Vema, H. Jhamtani, M. Gaur, and A. Modi (2024) Towards robust evaluation of unlearning in LLMs via data transformations. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 12100–12119. External Links: Document, Link Cited by: Machine Unlearning Methods. T. Ju, Y. Hua, H. Fei, Z. Shao, Y. Zheng, H. Zhao, M. Lee, W. Hsu, Z. Zhang, and G. Liu (2025) Watch out your album! on the inadvertent privacy memorization in multi-modal large language models. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 28446–28462. External Links: Link Cited by: Introduction. J. Kwon, M. Kim, E. Lee, J. Yun, B. Lim, and Y. Kim (2026) Before forgetting, learn to remember: revisiting foundational learning failures in LVLM unlearning benchmarks. In Findings of the Association for Computational Linguistics: ACL 2026, External Links: Link Cited by: Machine Unlearning Benchmarks, Dataset Construction. B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan (2024a) SEED-Bench: benchmarking multimodal LLMs with generative comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: Analysis Experiments. J. Li, Q. Wei, C. Zhang, G. Qi, M. Du, Y. Chen, S. Bi, and F. Liu (2024b) Single image unlearning: efficient machine unlearning in multimodal large language models. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: Introduction, Machine Unlearning Benchmarks. K. Li, W. Li, D. Wu, L. Yang, J. Bai, J. Jia, and J. Xue (2026) Cross-modal unlearning via influential neuron path editing in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, External Links: Link Cited by: Machine Unlearning Benchmarks. N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-Voss, C. B. Breuer, S. Marks, O. Patel, A. Zou, M. Mazeika, Z. Wang, P. Oswal, W. Lin, A. A. Hunt, J. Tienken-Harder, K. Y. Shih, K. Talley, J. Guan, R. Kaplan, I. Steneker, D. Campbell, B. Jokubaitis, A. Levinson, J. Wang, W. Qian, K. K. Karmakar, S. Basart, S. Fitz, M. Levine, P. Kumaraguru, U. Tupakula, V. Varadharajan, R. Wang, Y. Shoshitaishvili, J. Ba, K. M. Esvelt, A. Wang, and D. Hendrycks (2024c) The WMDP benchmark: measuring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 28525–28550. External Links: Link Cited by: Introduction, Machine Unlearning Methods, Machine Unlearning Benchmarks. Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: Analysis Experiments. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin (2024a) MMBench: is your multi-modal model an all-around player?. In European Conference on Computer Vision, External Links: Link Cited by: Analysis Experiments. Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai (2024b) OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences 67 (12), p. 220102. External Links: Document, Link Cited by: Analysis Experiments. Z. Liu, G. Dou, M. Jia, Z. Tan, Q. Zeng, Y. Yuan, and M. Jiang (2025) Protecting privacy in multimodal large language models with MLLMU-Bench. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, External Links: Link Cited by: Introduction, Introduction, Machine Unlearning Benchmarks. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao (2024) MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, External Links: Link Cited by: Analysis Experiments. P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to explain: multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, Vol. 35, p. 2507–2521. External Links: Link Cited by: Analysis Experiments. W. Ma, X. Feng, W. Zhong, L. Huang, Y. Ye, X. Feng, and B. Qin (2025a) Unveiling entity-level unlearning for large language models: a comprehensive analysis. In Proceedings of the 31st International Conference on Computational Linguistics, p. 5345–5363. External Links: Link Cited by: Machine Unlearning Methods. Y. Ma, J. Wang, F. Wang, S. Ma, J. Li, J. Pan, X. Li, F. Huang, L. Sun, B. Li, Y. Choi, M. Chen, and C. Xiao (2025b) Benchmarking vision language model unlearning via fictitious facial identity dataset. In International Conference on Learning Representations, External Links: Link Cited by: Introduction, Machine Unlearning Benchmarks. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter (2024) TOFU: a task of fictitious unlearning for LLMs. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: Introduction, Machine Unlearning Benchmarks, Metrics. V. R. Patil, P. Hase, and M. Bansal (2024) Can sensitive information be deleted from LLMs? objectives for defending against extraction attacks. In International Conference on Learning Representations, External Links: Link Cited by: Machine Unlearning Methods. Y. Qian, H. Ye, J. Fauconnier, P. Grasch, Y. Yang, and Z. Gan (2025) MIA-Bench: towards better instruction following evaluation of multimodal LLMs. In International Conference on Learning Representations, External Links: Link Cited by: Analysis Experiments. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Link Cited by: Introduction, Machine Unlearning Methods, Unlearning Methods. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He (2020) ZeRO: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, p. 1–16. External Links: Document, Link Cited by: Experimental Setup. C. Selvas-Sala, L. Kang, and L. Gomez (2026) SALMUBench: a benchmark for sensitive association-level multimodal unlearning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: Introduction, Machine Unlearning Benchmarks. W. Shi, J. Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang (2025) MUSE: machine unlearning six-way evaluation for language models. In International Conference on Learning Representations, External Links: Link Cited by: Machine Unlearning Benchmarks. D. Vrandečić and M. Krötzsch (2014) Wikidata: a free collaborative knowledgebase. Communications of the ACM 57 (10), p. 78–85. External Links: Document, Link Cited by: Introduction, Dataset Construction. C. Wang, Y. Li, X. Feng, C. Chen, X. Zheng, and J. Yin (2025) UMU-Bench: closing the modality gap in multimodal unlearning evaluation. In Advances in Neural Information Processing Systems, Vol. 38. External Links: Link Cited by: Introduction, Machine Unlearning Benchmarks. K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li (2024) Measuring multimodal mathematical reasoning with the MATH-Vision dataset. In Advances in Neural Information Processing Systems, Vol. 37, p. 95095–95169. External Links: Document, Link Cited by: Analysis Experiments. H. Xu, N. Zhao, L. Yang, S. Zhao, S. Deng, M. Wang, B. Hooi, N. Oo, H. Chen, and N. Zhang (2025) ReLearn: unlearning via learning for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5967–5987. External Links: Document, Link Cited by: Machine Unlearning Methods. J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue (2024) Machine unlearning of pre-trained large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8403–8419. External Links: Document, Link Cited by: Introduction, Machine Unlearning Methods. R. Zhang, L. Lin, Y. Bai, and S. Mei (2024) Negative preference optimization: from catastrophic collapse to effective unlearning. In First Conference on Language Modeling, External Links: Link Cited by: Introduction, Machine Unlearning Methods, Unlearning Methods, Metrics. H. Zheng, Z. Pang, L. Li, Z. Deng, Y. Pu, Z. Zhu, X. Xia, and J. Wei (2025) OFFSIDE: benchmarking unlearning misinformation in multimodal large language models. External Links: 2510.22535, Link Cited by: Introduction, Introduction, Machine Unlearning Benchmarks. Z. Zhong, Z. Wu, C. D. Manning, C. Potts, and D. Chen (2023) MQuAKE: assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 15686–15702. External Links: Document, Link Cited by: Dataset Construction.