Paper deep dive
Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning
Pengyang Shao, Chuanpeng Lu, Wei Qin, Yanzheng Jin, Xiaohao Liu, Xi Ai, Kenji Kawaguchi, Richang Hong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/9/2026, 5:24:32 AM
Summary
The paper introduces CLLPU, a multilingual benchmark for evaluating Large Language Model (LLM) unlearning. It addresses the challenge of controlling knowledge propagation across languages by defining two settings: common-goal forgetting (universal suppression) and language-conditioned forgetting (confinement to a specific language). The benchmark consists of 800 matched knowledge-unit pairs and 72,000 QA instances across ten languages. Experiments on Llama-3.1-8B-Instruct reveal that current unlearning methods fail to balance these objectives, either leaving knowledge intact when universal suppression is needed or spreading suppression beyond the intended language boundary.
Entities (7)
Relation Signals (7)
CLLPU → evaluates → LLM Unlearning
confidence 95% · We introduce CLLPU... a multilingual benchmark that formulates this problem [LLM unlearning] through two settings
Llama-3.1-8B-Instruct → testedon → CLLPU
confidence 95% · Experiments with six representative methods on Llama-3.1-8B-Instruct reveal opposite failure modes
CLLPU → supportssetting → Common-Goal Forgetting
confidence 93% · CLLPU... formulates this problem through two settings: common-goal forgetting... and language-conditioned forgetting
CLLPU → supportssetting → Language-Conditioned Forgetting
confidence 93% · CLLPU... formulates this problem through two settings: ... language-conditioned forgetting
CLLPU → usesmethod → Dual-Anchor Multilingual Translation
confidence 90% · CLLPU combines ... dual-anchor multilingual translation to construct 800 matched knowledge-unit pairs
GDPR → motivates → LLM Unlearning
confidence 85% · This need is further reinforced by practical and regulatory demands, such as the right to erasure under the General Data Protection Regulation (GDPR)
TOFU → comparedwith → CLLPU
confidence 80% · Existing benchmarks operationalize this objective... e.g., TOFU examines... In multilingual settings... We introduce CLLPU
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Model (LLM) unlearning aims to suppress target knowledge while preserving general capabilities. In multilingual settings, unlearning must additionally propagate within its intended linguistic scope. However, existing evaluations mainly measure cross-lingual transfer and cannot distinguish insufficient from excessive propagation. We introduce CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a designated language. CLLPU combines goal-guided topic pairing, schema-aware relation matching, and dual-anchor multilingual translation to construct 800 matched knowledge-unit pairs and 72,000 QA instances across ten languages. Experiments with six representative methods on Llama-3.1-8B-Instruct reveal opposite failure modes: forgetting remains incomplete when universal suppression is required, yet spreads beyond the intended boundary when language-conditioned confinement is required. We further find that general multilingual utility can conceal damage to neighbor knowledge. These findings establish propagation control as a central challenge for multilingual LLM unlearning. We publicly release CLLPU together with its construction pipeline.
Tags
Links
- Source: https://arxiv.org/abs/2609.05976v1
- Canonical: https://arxiv.org/abs/2609.05976v1
Trouble viewing inline? Open PDF directly →
Full Text
87,025 characters extracted from source content.
Expand or collapse full text
Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning Pengyang Shao National University of Singapore Singapore Chuanpeng Lu Hefei University of Technology Anhui, China Wei Qin Hefei University of Technology Anhui, China Yanzheng Jin National University of Singapore Singapore Xiaohao Liu National University of Singapore Singapore Xi Ai National University of Singapore Singapore Kenji Kawaguchi National University of Singapore Singapore Richang Hong Hefei University of Technology Anhui, China Abstract Large Language Model (LLM) unlearning aims to suppress target knowledge while preserving general capabilities. In multilingual settings, unlearning must additionally propagate within its intended linguistic scope. However, existing evaluations mainly measure cross-lingual transfer and cannot distinguish insufficient from ex- cessive propagation. We introduceCLLPU(Cross-Lingual and Language- Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppres- sion should remain confined to a designated language. CLLPU com- bines goal-guided topic pairing, schema-aware relation matching, and dual-anchor multilingual translation to construct 800 matched knowledge-unit pairs and 72,000 QA instances across ten languages. Experiments with six representative methods on Llama-3.1-8B-Instruct reveal opposite failure modes: forgetting remains incomplete when universal suppression is required, yet spreads beyond the intended boundary when language-conditioned confinement is required. We further find that general multilingual utility can conceal damage to neighbor knowledge. These findings establish propagation control as a central challenge for multilingual LLM unlearning. We pub- licly release CLLPU together with its construction pipeline 1 . Keywords Large Language Models, Machine Unlearning, Trustworthy AI 1 Introduction Large Language Model (LLM) unlearning seeks to remove the in- fluence of private, copyrighted, or otherwise sensitive data from a deployed LLM while preserving its general capabilities [ 14,21, 35,38,39]. This need is further reinforced by practical and regula- tory demands, such as the right to erasure under the General Data Protection Regulation (GDPR) [ 27] and deletion rights under the California Consumer Privacy Act (CCPA) [ 26]. Among LLM unlearning research topics, how to reliably evalu- ate an unlearned LLM remains a fundamental problem [11,31,37]. Existing benchmarks operationalize this objective by separately 1 https://github.com/CLLPU/CLLPU-bench evaluating the removal of designated target knowledge and the preservation of non-target capabilities, e.g., TOFU examines whether a model forgets selected fictitious author profiles while retaining those of the remaining authors [25]. Under this paradigm, success- ful unlearning is characterized by a substantial reduction in target- knowledge accessibility, together with minimal degradation in non- target performance. Multilingual settings introduce a further requirement for un- learning evaluation: forgetting should propagate within its intended linguistic scope [10,22,23]. Recent studies investigate cross-lingual forgetting by applying unlearning in source languages and eval- uating the resulting models across different query languages [13, 34]. Moving beyond independent per-language evaluation, Hwang et al. [17] focus on the consistency of knowledge removal across language pairs. These studies reveal whether forgetting transfers across semantically equivalent multilingual prompts. However, trans- fer does not always match the intended objective: for globally scoped requests, the target information should become inaccessible across all languages; for language-conditioned requests, forgetting should remain confined to the designated language while access in other languages is preserved. We term these thecommon-goalandlanguage- conditionedsettings, respectively. Existing multilingual evaluations do not explicitly distinguish between these objectives and there- fore cannot determine whether forgetting remains within its in- tended boundary. In this paper, we proposeCLLPU(Cross-Lingual and Language- Bound Protocol for LLM Unlearning), a multilingual benchmark with three key designs. First, CLLPU defines two settings: common- goal forgetting, where the forgetting effect should propagate across languages, and language-conditioned forgetting, where it should remain confined to a specific language context. Second, CLLPU adopts a hierarchical two-stage matching scheme. At the topic level, goal-guided target–neighbor pairing places semantically compara- ble topics on opposite sides of the intended forgetting boundary; at the level of knowledge units, schema-aware relation matching aligns these units by relation type, semantic slot, answer type, and supporting evidence, yielding one-to-one forget–retain units with closely matched semantics. Third, each knowledge unit is first in- stantiated as an English QA family consisting of a core QA and semantically equivalent variants, and the QA family is translated arXiv:2609.05976v1 [cs.CL] 5 Sep 2026 KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. into the other nine languages using dual-anchor multilingual trans- lation. We also construct core QAs based on holdout units for mem- bership inference attacks. Finally, CLLPU contains 800 knowledge- unit pairs and 72,000 multilingual QA instances. Extensive experiments with six representative LLM unlearning methods reveal that the central challenge in multilingual unlearn- ing is not merely whether forgetting transfers, but whether its propagation can be controlled. Across the six representative meth- ods, we observe opposite failure modes under the two objectives: cross-lingual forgetting remains incomplete when common-goal forgetting requires universal suppression, yet propagates exces- sively when language-conditioned forgetting requires confinement to the source language. General multilingual utility is also an un- reliable proxy for selective preservation, as a model may retain near-original performance on a general benchmark while substan- tially damaging relation-matched non-target knowledge. Together, these findings call for multilingual unlearning methods that jointly account for propagation control and local knowledge preservation. Our main contributions are summarized as follows: •We formulate multilingual LLM unlearning as a propagation- boundary problem and define two settings: common-goal for- getting and language-conditioned forgetting. •We introduce CLLPU, a multilingual LLM unlearning benchmark featuring complementary forgetting objectives, hierarchical match- ing at the topic and knowledge-unit levels, and semantically equivalent variants. •Experiments with six representative methods on Llama-3.1-8B- Instruct show that current methods fail in opposite directions: forgetting remains incomplete under the common-goal setting, yet propagates beyond the intended scope under the language- conditioned setting. 2 Related Work 2.1 LLM Unlearning Benchmarks LLM unlearning benchmarks assess whether a method can sup- press designated knowledge while preserving non-target capabil- ities [11,37]. Text-based benchmarks cover increasingly diverse knowledge and evaluation settings. TOFU evaluates forgetting and retention over fictitious author profiles [25], whereas RWKU ex- tends evaluation to real-world entity knowledge and introduces diverse probes for detecting residual knowledge [5]. MUSE further evaluates memorization, privacy leakage, utility preservation, scal- ability, and sequential unlearning requests [ 31], while WMDP fo- cuses on hazardous knowledge in biosecurity, cybersecurity, and chemical security [ 19]. Recent studies have examined preservation at a finer granularity. Chang et al. analyze how retain-set composi- tion affects unlearning and show that syntactically similar neigh- boring queries are particularly vulnerable to performance degrada- tion [ 7]. DUSK evaluates whether unlearning methods can remove forget-specific knowledge while preserving knowledge shared be- tween the forget and retain data [ 18]. Recent work has also ex- tended unlearning benchmarks to multimodal LLMs, including fic- titious facial identities and multimodal misinformation [ 24,33,42]. Despite this progress, existing benchmarks generally do not eval- uate whether forgetting propagates according to an intended lin- guistic scope. Moreover, preservation is commonly measured us- ing broad utility tasks or retain knowledge that is not explicitly matched to the forget targets. Such evaluations may therefore over- look localized damage to semantically neighboring knowledge and cannot distinguish insufficient cross-lingual propagation from prop- agation beyond the intended boundary. 2.2 Multilingual LLM Unlearning Multilingual LLM unlearning examines how an intervention ap- plied in one or more source languages affects access to the same underlying knowledge across different query languages [13,34]. Existing studies develop language-aware unlearning strategies, in- vestigate misinformation removal across languages, and evaluate source–target language configurations [ 10,23]. Their results show that cross-lingual forgetting is often incomplete, uneven, and asym- metric, with transfer patterns influenced by linguistic characteris- tics and shared multilingual representations [9,13,22]. However, existing studies mainly characterize whether forget- ting transfers across languages, without explicitly distinguishing between the common-goal and language-conditioned objectives or evaluating the preservation of relation-matched knowledge. 3 Benchmark Construction In this section, we introduce CLLPU, aCross-Lingual andLanguage- BoundProtocol for LLMUnlearning benchmark. As illustrated in Figure1, its construction consists of Goal-Guided Topic Pair Con- struction, Topic-to-Knowledge-to-QA Transformation, and Dual- Anchor Multilingual Translation. 3.1 Goal-Guided Topic Pair Construction To define the semantic boundary of forgetting at the topic level, we construct topic pairs consisting of atarget topicand aneighbor topic. The target topic represents knowledge within the intended forgetting boundary, whereas the neighbor topic represents seman- tically related knowledge outside that boundary. 3.1.1 Goal-Guided Topic Selection.In CLLPU, each topic corre- sponds to an English Wikipedia page. To reduce the likelihood that the target knowledge has already been encountered during pre- training, we consider only topics whose defining event, discovery, product release, legal change, or emergence in public discourse postdates the release of Llama 3.1, the base LLM used in our ex- periments. Candidate topics must also contain sufficient verifiable factual information to support subsequent knowledge-unit extrac- tion and relation matching. Topic selection is further guided by the two forgetting settings. We denote the language used to perform unlearning as the source language (푠), and the language used to query the unlearned model as the evaluation language (푡). In the common-goal setting, the in- tended forgetting boundary spans all evaluation languages rather Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM UnlearningKDD ’27, August 1–5, 2027, San Jose, CA, USA Goal-Guided Topic Selection Stage 1: Goal-Guided Topic Pair Construction Stage 3: Parallel Multilingual Translation Common-goal Language-conditioned July 23,2024 ... ... Wikipedia pages Llama-3.1 Release ... ... Target Topics Neighbor Topics LLM- assisted ... Goal-Conditioned Contrastive Topic Pairing Human-in-the-Loop Pair Curation 50 Topic Pairs 30 Topic Pairs LLM-assisted A Topic Pair Stage 2: Topic-to-Knowledge -to-QA Transformation Topic-to-Knowledge Transformation Knowledge-to-QA Transformation ... ... Knowledge Units Extraction LLM- assisted Knowledge Unit Extraction Schema-Aware Relation Matching Key Value Value ... QA Instantiation Core Variants Paired QA Holdout QA Human Review ARBNDEESFRJASWTHZH Back-translation Consistency Check LLM-assisted Review Translation Quality Verification Forget Set Holdout Set Retain Set Forget Set Holdout Set Retain Set ... Forget Set Holdout Set Retain Set Goal-Conditioned Multilingual Sets Forget Set Holdout Set Retain Set Forget Set Holdout Set Retain Set ... Forget Set Holdout Set Retain Set Common-goal Language-conditioned + Canonical English QA Knowledge Unit Accepted Excluded LLM- assisted ... Paired ... Holdout Figure 1: Overview of the CLLPU construction pipeline. than depending on a particular language context. Therefore, re- gardless of the source language (푠) in which unlearning is per- formed, the target knowledge should become inaccessible in ev- ery evaluation language (푡). This setting thus requires the forget- ting effect to propagate across languages, achieving universal sup- pression of the target knowledge. In the language-conditioned set- ting, each target topic is assigned a source language (푠), and the forgetting effect is expected to remain confined to that language context. When the evaluation language matches the source lan- guage, i.e., (푡 = 푠), the target knowledge should be suppressed; when (푡 ≠ 푠), it should remain accessible in the other languages. The two settings serve different evaluation purposes. The common- goal setting follows the conventional semantics of machine un- learning [ 3,16], under which the target knowledge should become inaccessible regardless of the language used to query the model. The language-conditioned setting, by contrast, does not necessar- ily correspond to a unique data-deletion or retraining oracle. In- stead, it operationalizes a benchmark-specific conditional-access objective, in which the behavioral effect of an intervention is ex- pected to remain within a designated language boundary. We in- clude this setting as a stress test of propagation control in multilin- gual models, rather than as a claim that the underlying knowledge has been completely removed from the model. 3.1.2 Contrastive Topic Pairing.For each target topic, we use a Codex-assisted workflow to identify a semantically related neigh- bor topic. A valid pair should belong to the same semantic domain, have comparable levels of abstraction and temporal contexts, and contain similar types of facts, while referring to distinct entities or events with different factual answers. We exclude parent–child topics, alternative formulations or variants of the same event, and superficial matches. In the common-goal setting, the neighbor topic serves as a se- mantically adjacent reference that remains outside the intended forgetting boundary. For example, Hurricane Helene can be paired with Hurricane Milton. Both are major Atlantic hurricanes from the 2024 season and contain comparable facts about affected areas, casualties, and damage, while representing distinct storms with non-overlapping factual answers. In the language-conditioned setting, the neighbor topic must ad- ditionally remain accessible in the source language (푠), enabling a direct comparison between target suppression and nearby knowl- edge preservation within the same language context. For example, the opening and closing ceremonies of the 2024 Summer Olympics form a suitable pair: they belong to the same event domain and contain comparable facts about dates, venues, performances, and public responses, while describing distinct events. When the open- ing ceremony is designated for forgetting in a source language, the closing ceremony remains outside the intended forgetting bound- ary and should be preserved in all evaluation languages, including the source language. Because inaccurate language assignments or poorly matched neighbors may blur the intended forgetting bound- ary, we prioritize boundary precision over broader topic coverage in the language-conditioned setting. 3.2 Topic-to-Knowledge-to-QA Transformation 3.2.1 Topic-to-Knowledge Transformation.Topic-level comparabil- ity does not imply fact-level comparability: two related topics may describe different attributes or support different answer types. We therefore decompose each target–neighbor topic pair intoknowl- edge units, where each unit is an atomic fact with a concise an- swer and directly traceable evidence. A unit records its relation type, semantic slot, answer and answer type, factual statement, and supporting source span. Gemini 3.1 Pro Preview [ 32] first identi- fies relations supported by both topics and then extracts evidence- grounded candidate units under this shared relation inventory. This representation makes the evaluated knowledge explicit and sepa- rates it from any particular question wording. We then apply schema-aware relation matching to construct fact-level contrasts across the forgetting boundary. A target-topic unit and a neighbor-topic unit are eligible only when they have the KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. same normalized relation type, compatible semantic slots and an- swer types, and direct evidence in their respective sources. Eligible pairs are ranked by schema consistency and evidence quality, fol- lowed by one-to-one selection without unit reuse. The target-side member of each match becomes aforget unit, and the neighbor- side member becomes itsretain unit; together they form a relation- aligned forget–retain probe. Unmatched valid units are sampled separately asholdout unitsfor membership-inference evaluation, ensuring knowledge-unit-level disjointness from the matched pool. We manually review all selected units and matches. The complete schema, extraction filters, coverage-aware selection rule, holdout construction, and review criteria are given in AppendixA.2. 3.2.2 Knowledge-to-QA Transformation.Each selected knowledge unit is next instantiated as a canonical English QA family. Gem- ini 3.1 Pro Preview receives only the unit’s structured fields, rather than the full topic article, and jointly processes each matched forget– retain pair to preserve their fact-level comparability. For every for- get or retain unit, it generates onecore QAthat directly queries the encoded fact and threesurface variantsthat express the same infor- mation need with different wording. Across all four realizations, the relation type, semantic slot, expected answer, answer bound- ary, and supporting evidence must remain invariant. This core-and-surface design distinguishes prompt sensitivity from a change in knowledge accessibility: failure on one word- ing is weak evidence of forgetting, whereas consistent degradation across equivalent realizations more directly reflects reduced access to the underlying fact. Because holdout units serve only as non- members in membership-inference attacks, each is converted into a single core QA without surface variants. Codex powered by GPT- 5.5 [ 2] independently reviews the generated QAs, after which hu- man reviewers resolve invalid or ambiguous cases. AppendixA.2 specifies the answer-anchored generation protocol, admissible sur- face variation, and full review procedure. 3.3 Parallel Multilingual QA Construction 3.3.1 Dual-Anchor Translation.CLLPU covers ten benchmark lan- guages: Arabic (ar), Bengali (bn), German (de), English (en), Span- ish (es), French (fr), Japanese (ja), Swahili (sw), Thai (th), and Chi- nese (zh). Reliable cross-lingual evaluation requires QAs in different lan- guages to access the same underlying knowledge. However, uncon- strained translation may alter the information being queried. For example, translating “which city” as a generic “where” question broadens the expected-answer boundary from a city to a country or venue. We therefore introduceDual-Anchor Parallel QA Trans- lation, which conditions translation on both the structured knowl- edge unit and its English QA. The knowledge-unit anchor specifies what must be asked by fixing the underlying fact, relation type, se- mantic slot, and expected-answer boundary; the QA anchor speci- fies how it should be asked by preserving the question intent and whether the instance is a core QA or a surface variant. A valid translation must satisfy both anchors while changing only its lin- guistic realization. For each core QA or surface variant, Gemini 3.1 Pro Preview jointly receives the corresponding structured knowl- edge unit and English QA and produces a QA in each of the other nine benchmark languages. The complete translation procedure is provided in AppendixA.3. 3.3.2 Translation Verification.We employ a three-stage verifica- tion procedure. First, Gemini 3.1 Pro Preview blindly back-translates each multilingual QA into English without access to its original English QA [4]. In a separate verification step, the back-translation is compared with the original for consistency in question intent, answer semantics, expected-answer boundary, and whether it is a core QA or a surface variant; failed instances are retranslated and verified again. Second, Codex powered by GPT-5.5 independently reviews every translated QA against both anchors. Third, we au- dit the reliability of these model-based decisions by conducting 20 rounds of human verification, each based on 100 randomly sam- pled QAs accepted by GPT-5.5. The verified QAs are organized into three multilingual sets ac- cording to their underlying knowledge units. QAs derived from matched forget and retain units form the forget set풬 푓 = ⋃ 푡∈ℒ 풬 푡 푓 and retain set풬 푟 = ⋃ 푡∈ℒ 풬 푡 푟 , respectively, whereℒdenotes the set of ten benchmark languages and푡is the evaluation language. QAs derived from holdout units form the holdout set풬 ℎ = ⋃ 푡∈ℒ 풬 푡 ℎ . Each forget or retain unit contributes one core QA and three se- mantically equivalent surface variants per language, whereas each holdout unit contributes only one core QA per language. Further implementation details are provided in AppendixA.3. 4 Evaluation Protocol 4.1 LLM Unlearning Pipeline Our evaluation starts with multilingual QA injection. Specifically, we perform full-parameter supervised fine-tuning (SFT) on the base LLM, Meta-Llama-3.1-8B-Instruct [15], using the core QAs from the forget and retain sets in all ten benchmark languages. The re- sulting Original model serves as the common starting point for all unlearning methods. We then perform LLM unlearning sepa- rately for each source language. Recall thatℒdenotes the set of ten benchmark languages. For each source language푠 ∈ℒ, we ap- ply an unlearning method to the Original model using only the corresponding forget and retain core QAs in푠, producing a source- specific unlearned model푀 (푠) un . We then evaluate푀 (푠) un in every eval- uation language푡 ∈ℒ. Performance at푡 = 푠characterizes source- language behavior, whereas performance at푡 ≠ 푠characterizes the cross-lingual effects of unlearning. For the common-goal setting, we additionally construct a Re- train model following the standard retraining principle in machine unlearning [ 3]. We perform the same full-parameter SFT on the base LLM using only the retain set, which provides a reference for model behavior without the target knowledge. We omit this reference for the language-conditioned setting: retaining the for- get set in non-source languages may transfer it back to the source language, whereas removing it globally would violate the specific preservation objective in this setting. 4.2 Multilingual Evaluation 4.2.1 Knowledge-Accessibility Metrics.For each source–evaluation- language pair(푠, 푡) ∈ℒ 2 , we evaluate푀 (푠) un using the corresponding forget and retain QAs realized in evaluation language푡. For each Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM UnlearningKDD ’27, August 1–5, 2027, San Jose, CA, USA QA, we compare the model-generated answer with its expected answer using four answer-level measures: 1) Exact Match (EM), 2) ROUGE-L (RL) [20], 3) Sentence Similarity (S), and 4) LLM- as-a-Judge. EM measures exact equality after output normaliza- tion, while ROUGE-L computes the longest-common-subsequence- based F1 score. S is the cosine similarity between the L2-normalized BGE-M3 dense embeddings of the generated and expected answers [8]. LLM-as-a-Judge evaluates answer correctness using an evaluator LLM; we exclude it from the main comparison because evaluator- and language-dependent calibration may confound the cross-lingual effects under study. For each answer-level measure푚, we first average the scores of the core QA and three surface variants corresponding to each knowledge unit and then average across knowledge units, yielding theKnowledge Accessibilityscore퐴 푚 푧 (푠, 푡). Here,푧 ∈ 푓 , 푟indexes the forget and retain roles, and푚 ∈ EM,RL,S,LLM-as-a-Judgede- notes the answer-level measure. We summarize the resulting language- pair matrix using two macro-averaged scores: ̂ 퐴 푚 푧 (S) = 1 |ℒ| ∑ 푠∈ℒ 퐴 푚 푧 (푠, 푠), ̂ 퐴 푚 푧 (C) = 1 |ℒ| ∑ 푠∈ℒ [ 1 |ℒ|−1 ∑ 푡∈ℒ∖푠 퐴 푚 푧 (푠, 푡) ] .Here, lowercase푠and 푡 index individual source and evaluation languages, respectively. The quantities ̂ 퐴 푚 푧 (S)and ̂ 퐴 푚 푧 (C)are the macro-averaged Source and Cross results over the ten diagonal and ninety off-diagonal language-pair cells, respectively. In the common-goal setting, successful unlearning requires both ̂ 퐴 푚 푓 (S)and ̂ 퐴 푚 푓 (C)to be low because the target knowledge should be- come inaccessible in every evaluation language. In the language- conditioned setting, ̂ 퐴 푚 푓 (S)should be low, whereas ̂ 퐴 푚 푓 (C)should remain high because the same target knowledge should remain ac- cessible in non-source languages. In both settings, accessibility to the matched retain knowledge derived from neighbor topics, mea- sured by ̂ 퐴 푚 푟 (S)and ̂ 퐴 푚 푟 (C), should remain high. 4.2.2 Other Metrics.We further evaluate Membership Inference Attacks (MIA) using Loss, Min-K% [30], Min-K%++ [40], and Zlib [6]. For each source-specific unlearned model푀 (푠) un , the forget core QAs used during knowledge injection are treated as members, while the unseen holdout core QAs in the same source language푠are treated as non-members. Each attack is evaluated independently for every source language using raw ROC-AUC, after which we macro-average across the ten source-language runs. An AUC of0.5 indicates chance-level discrimination; values below0.5may reflect reversed attack-score ordering rather than stronger unlearning. We additionally evaluate general multilingual utility using Bele- bele [ 1]. Each source-specific unlearned model is evaluated zero- shot on 900 multiple-choice examples in each of the ten bench- mark languages. We first average accuracy across the ten evalu- ation languages for each source-specific model and then macro- average across the ten source-language runs, yielding one overall Belebele accuracy for each method under each setting. Higher ac- curacy indicates better preservation of general multilingual utility. Further details are in Appendix B, e.g., metric formulations. 5 Experiments In this section, we address the following research questions (RQs): •RQ1:How do existing methods perform under two objectives? •RQ2:What are the boundaries of their cross-lingual transfer? •RQ3:How do they perform according to other metrics? •RQ4:Do surface variants yield results consistent with those of core QAs? •RQ5:How does forced response-language switching affect knowl- edge accessibility? Due to page limits, more experiments can be found in AppendixC. 5.1 Experimental Settings 5.1.1 Models and Data.During both knowledge injection and un- learning, we use only core QAs and reserve variants for robustness evaluation. The Retrain model is initialized from the same base LLM and trained on the retain set. We select Llama-3.1-8B-Instruct as the target LLM for two reasons. First, the defining events of all selected topics postdate its release, reducing the likelihood that the evaluated knowledge was encountered during pretraining. Second, CLLPU follows the task-specific target-LLM protocol adopted by existing unlearning benchmarks [18,28,31]. For example, MUSE assigns a designated target LLM to each corpus [31], while DUSK instantiates its benchmark with Llama-3-8B [18]. Our objective is to introduce and validate a new evaluation dimension—whether forgetting remains within its intended linguistic boundary—rather than to claim model-invariant propagation patterns. 5.1.2 Compared Methods.We evaluate six representative LLM un- learning methods under the same source-language data protocol: GA and GD [36], NPO [41], SimNPO [12], BalDRO-NPO, and BalDRO- SimNPO [29]. In addition, Original and Retrain are included only as reference models. 5.1.3 Implementation Details.To obtain the Original and Retrain models, we use a learning rate of1 × 10 −5 , the AdamW optimizer with a weight decay of0.01, linear warmup for one epoch, a per- device batch size of8, four gradient-accumulation steps, and bfloat16 precision. As for the unlearning process, we use AdamW and select the learning rate from1×10 −5 , 3×10 −5 , 5×10 −5 . The effective batch size is32for common-goal runs and8for language-conditioned runs. More details about settings are in Appendix C. 5.2 Overall Performance (RQ1) Tables1and2report the overall knowledge accessibility results. We summarize four main findings: •Common-goal forgetting transfers across languages, but incompletely.All methods reduce Cross forget accessibility relative to the Original model, showing that unlearning in one source language affects the same knowledge in other languages. However, Cross accessibility remains consistently higher than Source accessibility, indicating that the forgetting effect weak- ens during cross-lingual transfer. •None of the six evaluated methods establishes the desired language-conditioned boundary.Reducing Source accessi- bility also substantially reduces accessibility in non-source lan- guages. For example, GD achieves a Cross ROUGE-L of only 0.1788, compared with 0.8068 for the Original model. Similarly, the highest Cross S among all methods is only 0.5103, far be- low the Original result of 0.8910. Current methods therefore cannot reliably forget knowledge only in the designated source language. KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. Table 1: Overall performance in the common-goal setting. Lower forget accessibility and higher retain accessibility indicate better performance. The best result is shown in bold and the second-best result is underlined. Method MetricForget AccessibilityRetain Accessibility ̂ 퐴 EM 푓 (S) ↓ ̂ 퐴 RL 푓 (S) ↓ ̂ 퐴 S 푓 (S) ↓ ̂ 퐴 EM 푓 (C) ↓ ̂ 퐴 RL 푓 (C) ↓ ̂ 퐴 S 푓 (C) ↓ ̂ 퐴 EM 푟 (S) ↑ ̂ 퐴 RL 푟 (S) ↑ ̂ 퐴 S 푟 (S) ↑ ̂ 퐴 EM 푟 (C) ↑ ̂ 퐴 RL 푟 (C) ↑ ̂ 퐴 S 푟 (C) ↑ Original0.69960.76600.87310.69960.76600.87310.70130.76300.86970.70130.76300.8697 Retrain0.07350.20820.56750.07350.20820.56750.70050.76260.86860.70050.76260.8686 GA0.00700.0192 0.3273 0.0123 0.0437 0.33970.00700.02010.32880.01370.04670.3415 GD0.01970.07410.39030.05460.14350.44310.13650.23060.54890.10460.21360.5351 NPO0.00620.08100.46140.03260.11130.49620.68430.73870.85350.28320.39060.6959 SimNPO0.03140.14710.52750.06530.16380.55220.7296 0.7786 0.8769 0.3238 0.4326 0.7273 BalDRO-NPO0.02180.07210.44000.03190.08950.46670.69090.74200.85410.26170.35510.6799 BalDRO-SimNPO0.03120.14630.52610.05790.14910.54470.72860.77760.87610.30750.41220.7217 Table 2: Overall performance in the language-conditioned setting. Lower forget accessibility and higher retain accessibility indicate better performance. The best result is shown in bold and the second-best result is underlined. Method Metric Forget AccessibilityRetain Accessibility ̂ 퐴 EM 푓 (S) ↓ ̂ 퐴 RL 푓 (S) ↓ ̂ 퐴 S 푓 (S) ↓ ̂ 퐴 EM 푓 (C) ↑ ̂ 퐴 RL 푓 (C) ↑ ̂ 퐴 S 푓 (C) ↑ ̂ 퐴 EM 푟 (S) ↑ ̂ 퐴 RL 푟 (S) ↑ ̂ 퐴 S 푟 (S) ↑ ̂ 퐴 EM 푟 (C) ↑ ̂ 퐴 RL 푟 (C) ↑ ̂ 퐴 S 푟 (C) ↑ Original0.79150.82370.90470.75600.80680.89100.79020.82670.90610.74590.80160.8895 GA0.01160.03930.38120.02380.10290.42370.02870.07830.42430.05250.15660.4723 GD0.02430.04990.42610.1163 0.17880.48570.21210.27800.58740.20620.27870.5720 NPO 0.01330.03810.42700.05200.09740.45830.72430.76110.86760.24920.32710.6629 SimNPO0.04580.09840.48880.09240.15230.51030.7767 0.8067 0.89720.25780.34620.6686 BalDRO-NPO0.01330.03830.42710.06330.10740.46430.75250.78140.87890.26830.34510.6789 BalDRO-SimNPO0.03440.08240.47770.08770.14580.50290.76750.79760.89200.25500.33570.6713 EN ZH DE ES FR JA AR TH BN SW GAGDNPO ENZHDE ES FR JA AR TH BN SW EN ZH DE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO GAGDNPO ENZHDE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO 0.00 0.14 0.28 0.43 0.57 0.71 0.85 Knowledge Accessibility Forget AccessibilityRetain Accessibility Evaluation language Source language Figure 2: ROUGE-L in the common-goal setting. Diagonal cells report source accessibility퐴 RL 푧 (푠, 푠), while off-diagonal cells report cross accessibility퐴 RL 푧 (푠, 푡), where푡 ≠ 푠. Lower forget accessibility and higher retain accessibility indicate better perfor- mance. •Stronger forgetting often comes at the cost of retain Ac- cessibility.GA achieves the lowest forget accessibility, but its Source ROUGE-L scores on the forget and retain sets are simi- larly low at 0.0192 and 0.0201. This result indicates that its strong forgetting is accompanied by broad damage to related knowl- edge rather than selective removal of the target knowledge. •Embedding similarity can overestimate residual knowl- edge accessibility.In the common-goal setting, GA achieves a Source EM of 0.0070 but a Source S of 0.3273. This discrepancy should not be interpreted simply as evidence that substantial target knowledge remains accessible. Because our forget and retain units are relation-matched, their answers can be seman- tically similar despite being factually distinct; indeed, the BGE- M3 similarity between paired forget–retain answers is already approximately 0.48–0.50. Therefore, S need not approach zero before the target fact becomes inaccessible: a score around 0.5 may already correspond to a plausible but factually incorrect response that is semantically close to neighboring knowledge. Details can be found in Appendix C. 5.3 Cross-Lingual Transfer Boundaries (RQ2) Figures2and3show pairwise ROUGE-L accessibility; the corre- sponding EM and S heatmaps are provided in AppendixC.2. We Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM UnlearningKDD ’27, August 1–5, 2027, San Jose, CA, USA EN ZH DE ES FR JA AR TH BN SW GAGDNPO ENZHDE ES FR JA AR TH BN SW EN ZH DE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO GAGDNPO ENZHDE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO 0.00 0.16 0.32 0.48 0.63 0.79 0.95 Knowledge Accessibility Forget AccessibilityRetain Accessibility Evaluation language Source language Figure 3: ROUGE-L in the language-conditioned setting. Diagonal cells report source accessibility퐴 RL 푧 (푠, 푠), while off-diagonal cells report cross accessibility퐴 RL 푧 (푠, 푡), where푡 ≠ 푠. Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages. Table 3: Overall performance on membership inference and multilingual utility. The best result is shown in bold and the second-best result is underlined. Method Common-goalLanguage-conditioned Membership Inference (AUC→ 0.5) Belebele↑Membership Inference (AUC→ 0.5) Belebele↑ Loss Min-K% Min-K%++ Zlib AccuracyLoss Min-K% Min-K%++ Zlib Accuracy Original0.97060.96830.93440.96730.80860.97270.97160.93930.97120.8086 Retrain0.46310.46690.47970.44020.8060– GA0.5033 0.4927 0.5042 0.43070.55970.15230.14490.21560.17670.7868 GD 0.32460.30740.33400.32860.71860.16300.14880.16260.16710.6510 NPO0.19780.19240.23440.22740.56730.21540.15960.19370.18280.6365 SimNPO0.41600.38880.36730.37430.61070.3869 0.3103 0.3003 0.31030.6512 BalDRO-NPO0.16330.17100.20200.20240.53090.18820.14530.16860.16700.6691 BalDRO-SimNPO0.41780.39330.37890.37610.60550.36590.28560.29390.29850.6426 make four observations.First, common-goal forgetting transfers across languages, but its strength depends strongly on the source language. With GA, using English or Japanese as the source re- duces target accessibility to nearly zero in all evaluation languages. In contrast, using Arabic as the source yields an average cross ac- cessibility of 0.221. This variation shows that the choice of source language can substantially change the cross-lingual effect of the same unlearning method.Second, cross-lingual forgetting is di- rectional rather than symmetric. Under GD, target accessibility from Bengali to English is 0.519, while the reverse direction is only 0.009. Thus, the effect from language푠to language푡cannot be in- ferred from the effect from푡to푠; each direction defines a differ- ent unlearning boundary.Third, preserving utility in the source language does not guarantee preservation in other languages. For example, in the language-conditioned setting, NPO with Arabic as the source language achieves a retain accessibility of 0.596 in Ara- bic but only 0.030 on average in other languages.Finally, strong forgetting is often not selective. GA produces similar low-value patterns for Forget Accessibility and Retain Accessibility. Its strong forgetting therefore reflects broad knowledge damage rather than precise removal of the target knowledge. 5.4 Other Metrics (RQ3) We draw four findings from Table3.First, the common-goal set- ting shows a trade-off between membership privacy and general utility. GA achieves AUCs closest to 0.5 but lowers Belebele ac- curacy to 0.5597, whereas GD preserves the highest accuracy of 0.7186 but has AUCs farther from 0.5.Second, the six evaluated methods still perform poorly in the language-conditioned setting. Even the best method, SimNPO, produces AUCs of 0.3003–0.3869 rather than approaching 0.5. Together with Table 2, this result shows the need for methods that can limit forgetting to the source lan- guage.Third, the reference models support the validity of our MIA evaluation. Original produces consistently high AUCs of 0.93–0.97, while Retrain approaches the random baseline of 0.5.Finally, Bele- bele and knowledge accessibility measure different aspects of util- ity. In the language-conditioned setting, GA maintains a Belebele accuracy of 0.7868, close to 0.8086 for Original, but severely dam- ages the matched retain knowledge in Table 2. This result shows KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. Figure 4: Performance on core QAs versus surface variants. Each surface-variant bar reports the mean ROUGE-L score across three variants. Values adjacent to the bars denote the surface-variant score minus the core-QA score in percentage points. Query Language Evaluation Language Forget Accessibility Retain Accessibility Language - conditioned Common - goal Figure 5: ROUGE-L accessibility under forced response- language switching. Given a query in language푠, we instruct the model to respond in a designated language푢. that our matched-pair construction can detect damage to related knowledge that a general multilingual benchmark may miss. 5.5 Core QAs and Surface Variants (RQ4) Figure4shows that forgetting generalizes well from core QAs to unseen surface variants: across methods, settings, and evaluation scopes, forget-set gaps remain between−3.7and+1.6percentage points, with predominantly negative Cross gaps indicating that variants rarely recover forgotten knowledge. Retain accessibility is more sensitive to wording, with Source gaps reaching20.7points and smaller but non-negligible Cross gaps. Thus, core-only evalu- ation can characterize forgetting reasonably well but may overes- timate retained knowledge accessibility. 5.6 Forced Response-Language Switching(RQ5) Figure5compares our main protocol against a constrained setting that keeps the query language fixed while forcing the model to re- spond in another language. Forced switching consistently reduces residual knowledge accessibility: the average forget-set ROUGE-L accessibility decreases from0.0658on the diagonal to0.0237off the diagonal in the common-goal setting, and from0.0418to0.0198in the language-conditioned setting. Thus, our main protocol consti- tutes the more stringent unlearning test: aligned-language evalu- ation recovers more residual knowledge and is therefore harder for an unlearning method to pass, whereas forced switching intro- duces an additional cross-lingual generation bottleneck that sup- presses knowledge expression. Although a few language pairs, such as AR→TH and ES→JA, exhibit locally high accessibility, they do not alter the overall trend. The concurrent decrease in retain-set accessibility further indicates that the reduction partly arises from constrained generation rather than stronger forgetting. These re- sults support direct multilingual evaluation as the primary proto- col for exposing residual knowledge. 6 Conclusion and Future Work Reliable multilingual unlearning is essential for controlling how sensitive knowledge remains accessible across languages. A key challenge is that different requests require different propagation boundaries: some knowledge should be removed across all lan- guages, whereas other knowledge should be forgotten only within a designated language context. To study this challenge, we formu- lated multilingual unlearning as a propagation-boundary problem and developed CLLPU, a benchmark that jointly evaluates knowl- edge suppression, cross-lingual propagation, and selective preser- vation. Across the six methods evaluated on Llama-3.1-8B-Instruct, we observed opposite failure modes: forgetting remained incom- plete when global removal was required, but propagated beyond the intended boundary when language-conditioned removal was required. In the future, we will investigate methods that can ex- plicitly control cross-lingual unlearning propagation. Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM UnlearningKDD ’27, August 1–5, 2027, San Jose, CA, USA References [1]Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: A Parallel Reading Com- prehension Dataset in 122 Language Variants. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 749–775.doi:10.18653/v1/2024.acl-long.44 [2]Bikramaditya Barman, Md Mukutor Rahman, and Rashel Sarkar. 2026. GPT-5.5: evolution of agentic artificial intelligence (AI) in libraries.Library Hi Tech News (2026), 1–4. [3]Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hen- grui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine Unlearning. In2021 IEEE Symposium on Security and Privacy. 141–159. doi:10.1109/SP40001.2021.00019 [4]Richard W Brislin. 1970. Back-translation for cross-cultural research.Journal of cross-cultural psychology1, 3 (1970), 185–216. [5]Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, Jun Zhao, et al. 2024. Rwku: Benchmarking real-world knowledge unlearning for large language models.Advances in Neural Information Processing Systems37 (2024), 98213–98263. [6]Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlings- son, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Language Models. In30th USENIX Security Symposium. USENIX As- sociation, 2633–2650.https://w.usenix.org/conference/usenixsecurity21/ presentation/carlini-extracting [7]Hwan Chang and Hwanhee Lee. 2025. Which retain set matters for llm un- learning? a case study on entity unlearning. InFindings of the Association for Computational Linguistics: ACL 2025. 5966–5982. [8]Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024.M3-Embedding: Multi-Linguality, Multi-Functionality, Multi- Granularity Text Embeddings Through Self-Knowledge Distillation.arXiv preprint arXiv:2402.03216(2024). https://arxiv.org/abs/2402.03216 [9]Yuxin Chen, Yiran Zhao, Yang Zhang, An Zhang, Kenji Kawaguchi, Shafiq Joty, Junnan Li, Tat-Seng Chua, Michael Shieh, and Wenxuan Zhang. 2026. The emer- gence of abstract thought in large language models beyond any language.Ad- vances in Neural Information Processing Systems38 (2026), 9902–9933. [10]Minseok Choi, Kyunghyun Min, and Jaegul Choo. 2024. Cross-lingual unlearn- ing of selective knowledge in multilingual language models. InFindings of the Association for Computational Linguistics: EMNLP 2024. 10732–10747. [11]Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zico Kolter, Zachary Lipton, and Pratyush Maini. 2026. Openunlearning: Accelerating llm unlearning via unified benchmarking of methods and metrics.Advances in Neu- ral Information Processing Systems38 (2026). [12]Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2025. Simplicity Prevails: Rethinking Negative Preference Optimiza- tion for LLM Unlearning. InAdvances in Neural Information Processing Systems. https://arxiv.org/abs/2410.07163 [13]Alireza Dehghanpour Farashah, Aditi Khandelwal, Marylou Fauchard, Zhuan Shi, Negar Rostamzadeh, and Golnoosh Farnadi. 2026. Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs.arXiv preprint arXiv:2601.05641(2026). [14]Chongyang Gao, Lixu Wang, Kaize Ding, Chenkai Weng, Xiao Wang, and Qi Zhu. 2025. On large language model continual unlearning. InInternational Conference on Learning Representations, Vol. 2025. 101772–101801. [15]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The Llama 3 Herd of Models.arXiv preprint arXiv:2407.21783(2024). https://arxiv.org/abs/ 2407.21783 [16]Bo Han, Jiangchao Yao, Tongliang Liu, Bo Li, Sanmi Koyejo, and Feng Liu. 2025. Trustworthy machine learning: From data to models.Foundations and Trends® in Privacy and Security7, 2-3 (2025), 74–246. [17]Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho, and Nojun Kwak. 2026. Knowl- edge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation. InProceedings of the 64th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers). 24085–24119. [18]Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, Soeun Kim, Seungju Han, Youngjae Yu, and Albert No. 2026. Dusk: Do not unlearn shared knowledge. InFindings of the Association for Computational Linguistics: ACL 2026. 42013–42031. [19]Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, et al. 2024. The WMDP benchmark: measuring and reducing malicious use with unlearning. InProceedings of the 41st International Conference on Machine Learn- ing. 28525–28550. [20]Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summarization Branches Out. Association for Computational Linguistics, 74–81.https://aclanthology.org/W04-1013/ [21]Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. 2025. Rethink- ing machine unlearning for large language models.Nature Machine Intelligence 7, 2 (2025), 181–194. [22]Tyler Lizzo and Larry Heck. 2026. Evaluating Cross-Lingual Unlearning in Mul- tilingual Language Models.arXiv preprint arXiv:2601.06675(2026). [23]Taiming Lu and Philipp Koehn. 2025. Learn and unlearn: Addressing misinfor- mation in multilingual llms. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 10191–10206. [24]Yingzi Ma, Jiongxiao Wang, Fei Wang, Siyuan Ma, Jiazhao Li, Jinsheng Pan, Xi- ujun Li, Furong Huang, Lichao Sun, Bo Li, et al. 2025. Benchmarking vision language model unlearning via fictitious facial identity dataset. InInternational Conference on Learning Representations, Vol. 2025. 87806–87827. [25]Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121(2024). [26]Stuart L Pardau. 2018. The california consumer privacy act: Towards a european- style privacy regime in the united states.J. Tech. L. & Pol’y23 (2018), 68. [27]Data Protection. 2018. General data protection regulation.Intersoft Consulting, Accessed in October24, 1 (2018). [28]Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanuki- ran Vinzamuri, Volkan Volkan Cevher, Mingyi Hong, and Rahul Gupta. 2025. Semeval-2025 task 4: Unlearning sensitive content from large language mod- els. InProceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025). 2584–2596. [29]Pengyang Shao, Naixin Zhai, Lei Chen, Yonghui Yang, Fengbin Zhu, Xun Yang, and Meng Wang. 2026. BalDRO: A Distributionally Robust Optimization Based Framework for Large Language Model Unlearning. InProceedings of the ACM Web Conference 2026. 8874–8884. doi:10.1145/3774904.3792975 [30]Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting Pretraining Data from Large Language Models. InInternational Conference on Learning Represen- tations. https://arxiv.org/abs/2310.16789 [31]Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtz- man, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang. 2025. Muse: Machine unlearning six-way evaluation for language models. InInterna- tional Conference on Learning Representations, Vol. 2025. 27797–27818. [32]Gemini Team. 2026. Gemini 3.1 Pro: A smarter model for your most complex tasks.Google Blog(2026). [33]Yuhang Wang, Wenjie Mei, Junkai Zhang, Guangyu He, Zhenxing Niu, and Haichang Gao. 2026. ICU-Bench: Benchmarking Continual Unlearning in Mul- timodal Large Language Models.arXiv preprint arXiv:2605.05938(2026). [34]Chaoyi Xiang, Olga Ohrimenko, Benjamin IP Rubinstein, and Lea Frermann. [n. d.]. Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibil- ity. InForty-third International Conference on Machine Learning. [35]Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. 2025. Exploring Criteria of Loss Reweighting to Enhance LLM Unlearn- ing. InForty-second International Conference on Machine Learning. [36]Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model un- learning.Advances in Neural Information Processing Systems37 (2024), 105425– 105475. [37]Sangyeon Yoon, Wonje Jeung, and Albert No. 2025. R-tofu: Unlearning in large reasoning models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 5239–5258. [38]Naixin Zhai, Pengyang Shao, Binbin Zheng, Yonghui Yang, Fei Shen, Long Bai, and Xun Yang. 2026. Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning. InThe 64th Annual Meeting of the Association for Computational Linguistics. https://openreview.net/forum?id=zwwb1s7TcR [39]Chenlong Zhang, Zhuoran Jin, Hongbang Yuan, Jiaheng Wei, Tong Zhou, Kang Liu, Jun Zhao, and Yubo Chen. 2025. RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems. [40]Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025. Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models. InInternational Con- ference on Learning Representations. https://arxiv.org/abs/2404.02936 [41]Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative Preference Op- timization: From Catastrophic Collapse to Effective Unlearning.arXiv preprint arXiv:2404.05868(2024).https://arxiv.org/abs/2404.05868 [42]Hao Zheng, Zirui Pang, Ling Li, Zhijie Deng, Yuhan Pu, Zhaowei Zhu, Xiaobo Xia, and Jiaheng Wei. 2026. Offside: Benchmarking unlearning misinformation in multimodal large language models. InFindings of the Association for Compu- tational Linguistics: ACL 2026. 12602–12620. KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. Stage Data ProductCommon-goal Language-conditionedTotal Stage 1 Curated pairs of target-neighbor topics503080 Canonical Wikipedia pages10060160 Stage 2 Extracted knowledge units2,8881,7614,649 Selected relation-matched knowledge-unit pairs500300800 Sampled holdout knowledge units500300800 Canonical English QAs (core + variants)4,0002,4006,400 Canonical English holdout QAs (core only)500300800 Stage 3 Multilingual forget QAs (core + variants)2,000×101,200×10 3,200×10 Multilingual retain QAs (core + variants)2,000×101,200×10 3,200×10 Multilingual holdout QAs (core only)500×10300×10800×10 All multilingual QAs4,500×102,700×10 7,200×10 Table4:Statisticsofthebenchmarkconstructionpipeline.Eachmatchedknowledge-unitpaircontainsoneforgetunitandone retain unit. Each unit is instantiated as one core QA and three semantically equivalent surface variants, whereas each holdout unit produces only one core QA. Multilingual counts include English and its nine translated versions. In the common-goal setting, all language realizations of forget knowledge are targeted; in the language-conditioned setting, only the designated source-language realization is targeted. A Benchmark Construction Details This appendix provides benchmark-construction details omitted from the main text. Additional resources related to the construc- tion process—including the complete prompt templates, support- ing Python code, and the human-in-the-loop workflow for inter- acting with Codex—are available in our repository: https://github. com/CLLPU/CLLPU-bench. A.1 Stage 1: Goal-Guided Topic Pair Construction Post-Release Boundary.The temporal cutoff used for topic selec- tion is July 23, 2024, the release date of Llama 3.1 [15]. A topic is considered post-cutoff only if its defining event, discovery, product release, legal change, or emergence in public discourse occurred after this date. For each candidate, we record the corresponding temporal evidence rather than relying solely on the creation or revision date of its Wikipedia page. Candidates with ambiguous temporal provenance or insufficient factual content are excluded. Candidate Metadata.Each candidate topic is represented by an English Wikipedia page and annotated with its semantic domain, abstraction level, temporal context, cultural context, anticipated fact types, and the evidence supporting these attributes. These an- notations support both goal-conditioned topic assignment and sub- sequent contrastive pairing. For each language-conditioned topic, we additionally record its designated source language, a rationale category (e.g., legal, cultural, public-safety, diplomatic, or public- discourse), and topic-specific evidence supporting the assignment. Assignments within each rationale category are reviewed by an- notators with relevant linguistic, regional, or cultural familiarity. The reviewers assess the contextual relevance of the event, the likely scope of its legal, social, or cultural impact, and whether this scope provides a sufficiently clear basis for distinguishing one language context from the others. This evidence-backed process is intended to approximate plausible real-world motivations for language-bounded forgetting rather than relying on arbitrary topic– language mappings. Nevertheless, these rationales are used only to construct a contextually plausible, benchmark-specific source- language boundary; they should not be interpreted as legal deter- minations, universal cultural judgments, or claims that a topic has an intrinsic forgetting scope. Candidates with ambiguous impact boundaries or insufficiently supported rationales are excluded. Contrastive Pairing Criteria.For each target topic, Codex pow- ered by GPT-5.5 proposes a neighbor topic from a similar semantic domain and temporal context. The two topics must have compara- ble abstraction levels and contain parallel types of facts while refer- ring to distinct entities or events. We exclude parent–child topics, alternative formulations or variants of the same event, and super- ficial matches obtained by substituting country or entity names. For language-conditioned pairs, the neighbor topic must addition- ally remain outside the forgetting scope associated with the source language. Human Curation.All proposed target–neighbor topic pairs un- dergo full human review for semantic comparability, abstraction- level consistency, availability of parallel facts, and boundary sep- aration. Each pair is reviewed by an English-proficient annotator with relevant cultural or regional familiarity, who assesses the event’s likely impact scope, the plausibility of its designated source lan- guage, and whether the neighbor remains outside the benchmark- specific boundary. Invalid pairs are revised or returned to the Codex- assisted pairing process, yielding 50 common-goal and 30 language- conditioned pairs. A.2 Stage 2: Topic-to-Knowledge-to-QA Transformation Knowledge-Unit Schema.A knowledge unit represents an atomic, independently queryable fact with a concise answer and directly traceable evidence. Each unit contains arelation_type, which Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM UnlearningKDD ’27, August 1–5, 2027, San Jose, CA, USA specifies the general relationship expressed by the fact; aseman- tic_slot, which identifies the particular information represented within that relationship; ananswerandanswer_type, which de- fine the expected-answer boundary; and afact_statementand source_span, which record the evaluated claim and its supporting evidence. We exclude composite facts, subjective or open-ended claims, facts requiring evidence aggregation across multiple spans, and facts that cannot be answered concisely. Candidate Extraction and Normalization.For each target–neighbor topic pair, Gemini 3.1 Pro Preview first identifies relation types that are represented and directly supported in both topics. Guided by these shared relation types, it then separately extracts up to 30 candidate knowledge units from each topic before filtering. Each extracted unit must cite a source span from its own topic. We re- move units with incomplete fields, unlocatable source spans, du- plicate factual statements, or duplicate relation–answer combina- tions. Units whose answers cannot be directly located in their sup- porting evidence are flagged for further review. Schema-Aware Relation Matching.Given the candidate pools ex- tracted by Gemini 3.1 Pro Preview, we apply a deterministic schema- aware matcher. Candidate units must have the same normalized relation type, compatible semantic slots and answer types, and di- rect answer support in their respective source spans. Answer types are compatible when they match exactly or belong to the same predefined family. We rank candidates by answer-type agreement, semantic-slot overlap, source-span support, and extraction prior- ity, and greedily select one-to-one matches without reusing knowl- edge units or factual statements. Weakly aligned, identical-answer, overlong, or low-scoring candidates are excluded. The target-side unit becomes the forget unit, while the neighbor-side unit becomes the retain unit. Coverage-Aware Selection and Review.To balance topic coverage and pair quality, we apply a coverage-aware selection strategy over the eligible relation-matched candidates. The strategy prioritizes pairs with strong relation consistency, semantic-slot compatibil- ity, answer-type compatibility, and evidence validity while main- taining broad coverage across target–neighbor topic pairs. The re- sulting selections are manually reviewed, and invalid or ambigu- ous pairs are removed or replaced with eligible alternatives. This procedure yields 500 common-goal and 300 language-conditioned knowledge-unit pairs. Holdout-Unit Construction.After selecting an average of ten forget– retain pairs per topic pair, we remove all matched units from the corresponding candidate pools and uniformly sample holdout units without replacement from the remaining units. Because the hold- out units are drawn from the same topic-specific candidate pools and shared relation inventories as the matched units, they provide closely aligned and challenging non-member examples rather than unrelated random negatives. At the same time, they remain strictly disjoint from the forget and retain sets at the knowledge-unit level. All sampled holdout units undergo the same evidence and validity review as the matched units. Holdout units are used only for mem- bership inference attacks and therefore do not require multiple for- mulations of the same fact. Each holdout unit is instantiated as one English core QA without surface variants, following the same answer-anchored generation protocol as the matched units. Core and Surface QA Generation.Gemini 3.1 Pro Preview receives only the structured fields of each knowledge unit rather than the full topic content. The forget and retain units in each matched pair are provided jointly so that their QAs remain comparable across the two roles. For each forget or retain unit, Gemini generates one core QA and three semantically equivalent surface variants. Each question is constructed backward from the unit’s short answer. Surface variants may alter lexical choices and syntactic structures but must preserve the relation type, semantic slot, expected an- swer, and supporting evidence. English-QA Review.All 1,600 English QA families generated from the 800 matched knowledge-unit pairs are first reviewed by Codex powered by GPT-5.5 and then individually inspected by one English- proficient human reviewer. Each family contains one core QA and three surface variants. The review examines question validity, an- swer consistency, preservation of the relation type and semantic slot, evidence grounding, duplicate formulations, potential answer leakage, and cross-role information leakage between the forget and retain QAs. Invalid QAs are corrected or regenerated before entering the multilingual translation stage. A.3 Stage 3: Parallel Multilingual Translation Dual-Anchor Translation.Gemini 3.1 Pro Preview translates each canonical English QA into the other nine benchmark languages. Translation is conditioned on two complementary anchors. The knowledge-unit anchor fixes the underlying fact, relation type, se- mantic slot, expected-answer boundary, and supporting evidence. The canonical-English-QA anchor fixes the question intent and whether the instance directly asks for the fact as a core QA or accesses the same fact through a surface reformulation. A valid translation must agree with both anchors while changing only its linguistic realization. Language-Specific Constraints.Translations preserve the expected answer semantics while allowing language-appropriate word or- der and grammatical structure. Names, dates, numbers, abbrevia- tions, and culturally dependent expressions are translated or translit- erated according to the conventions of the target language. These adaptations must not broaden or narrow the question scope, change the expected answer type, or collapse the distinction between core and surface formulations. Blind Back-Translation.Gemini 3.1 Pro Preview performs blind back-translation for every translated QA without access to its canon- ical English counterpart. In a separate verification step, Gemini compares the back-translated QA with the canonical English QA and determines whether they preserve the same question intent and answer semantics. Translations that fail this check are retrans- lated and verified again. Independent Cross-Model Review.Because Gemini performs both translation and back-translation verification, these stages may share model-specific errors. We therefore use Codex powered by GPT- 5.5 as a cross-model verifier. Without access to Gemini’s verifi- cation decision, Codex reviews every translated QA against the knowledge-unit and canonical-QA anchors. KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. Final Human Verification.After all model-based reviews and cor- rections are completed, we conduct a final human audit of 2,000 translated QAs in 20 rounds of 100 instances. We recruit additional multilingual annotators whose language proficiency collectively covers all nine target languages. Each sampled QA is assigned to an annotator familiar with both English and the corresponding tar- get language. The sampled instances cover all target languages, for- getting settings, knowledge roles, and core-versus-surface formu- lations. Annotators assess whether each translation preserves the underlying fact, question intent, expected-answer boundary, and core-or-surface role while remaining fluent and natural in the tar- get language. The final human audit confirms that all 2,000 sampled QAs satisfy these criteria, consistent with the safeguards built into the preceding pipeline, including schema-aware matching, dual- anchor translation grounded in both the structured knowledge unit and canonical English QA, blind back-translation, and independent cross-model verification. B Evaluation Metrics B.1 Reference-Model Aggregation In the common-goal setting, Original and Retrain each correspond to a single multilingual model whose parameters do not depend on a source language. After replicating their ten-language evalu- ation vectors along the source-language dimension, every evalu- ation language receives the same weight in the Source and Cross macro-averages. Consequently, under our aggregation protocol, the two aggregates are identical for each knowledge role and evalu- ation metric. In contrast, each row in the language-conditioned setting evaluates a different source-conditioned knowledge set. Its Source and Cross aggregates are therefore not mathematically con- strained to be equal. B.2 Membership-Inference Protocol Attack Setup.For each source language푠 ∈ℒ, we treat the forget core QAs used during multilingual knowledge injection as members and the unseen holdout core QAs in the same language as non-members. Let푧 = (푥, 푦)denote a core QA, where푥is the formatted question prompt and푦 = (푦 1 , ... , 푦 푇 )is its tokenized an- swer. Following the supervised fine-tuning objective, all likelihood- based attack scores are computed over the answer tokens condi- tioned on the question and preceding answer tokens. We denote the token-level log probability by ℓ 푖 (푧) =log휋 휃 ( 푦 푖 ∣ 푥, 푦 <푖 ) ,(1) where휋 휃 is the evaluated model. All attack scores are oriented such that a larger score indicates stronger evidence that an example is a member. Loss Attack.The Loss attack exploits the tendency of a model to assign higher likelihood, or equivalently lower negative log-likelihood, to examples encountered during training. We use the negative token- normalized loss as the membership score: 푆 Loss (푧) = 1 푇 푇 ∑ 푖=1 ℓ 푖 (푧).(2) Thus, examples receiving larger average log probabilities are re- garded as more likely to be members. Min-K% Attack.Min-K% focuses on the least likely tokens in an example, based on the intuition that unseen examples are more likely to contain low-probability outlier tokens. Let 푚 =max ( 1, ⌊ 퐾 100 푇 ⌋) ,(3) and letℐ 퐾 (푧)contain the indices of the푚answer tokens with the smallest values ofℓ 푖 (푧). The Min-K% score is 푆 Min-K (푧) = 1 푚 ∑ 푖∈ℐ 퐾 (푧) ℓ 푖 (푧).(4) We use퐾 = 40for all models, languages, and unlearning methods. Min-K%++ Attack.Min-K%++ calibrates each observed token prob- ability against the complete next-token distribution predicted un- der the same prefix. Let 푝 푖 (푣) = 휋 휃 ( 푣 ∣ 푥, 푦 <푖 ) (5) denote the probability assigned to vocabulary token푣 ∈풱. The conditional mean and standard deviation of the log probabilities are 휇 푖 = ∑ 푣∈풱 푝 푖 (푣)log푝 푖 (푣), (6) 휎 푖 = √ ∑ 푣∈풱 푝 푖 (푣) ( log푝 푖 (푣) − 휇 푖 ) 2 .(7) The standardized score of the observed answer token is 푟 푖 (푧) = ℓ 푖 (푧) − 휇 푖 √ max ( 휎 2 푖 , 10 −6 ) .(8) Letℐ ++ 퐾 (푧)contain the indices of the푚tokens with the smallest standardized scores. The sequence-level Min-K%++ score is 푆 Min-K++ (푧) = 1 푚 ∑ 푖∈ℐ ++ 퐾 (푧) 푟 푖 (푧).(9) As with Min-K%, we set퐾 = 40throughout the experiments. Zlib-Calibrated Attack.The Zlib attack calibrates model likeli- hood using the compression-based complexity of the evaluated an- swer text. We first define the answer perplexity as PPL(푧) =exp ( − 1 푇 푇 ∑ 푖=1 ℓ 푖 (푧) ) .(10) Let퐶 zlib (푦)denote the compressed length, in bytes, of the UTF- 8 encoding of the expected-answer text corresponding to푦under zlib compression. Following the likelihood-to-compression calibra- tion, we define the member-oriented score as 푆 Zlib (푧) = − log PPL(푧) 퐶 zlib (푦) .(11) The negative sign ensures that larger values correspond to stronger membership evidence: an example is considered more likely to be a member when the model assigns its expected answer unusually high likelihood relative to the answer’s compression-based com- plexity. Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM UnlearningKDD ’27, August 1–5, 2027, San Jose, CA, USA EN ZH DE ES FR JA AR TH BN SW GAGDNPO ENZHDE ES FR JA AR TH BN SW EN ZH DE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO GAGDNPO ENZHDE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO 0.00 0.14 0.28 0.43 0.57 0.71 0.85 Knowledge Accessibility Forget AccessibilityRetain Accessibility Evaluation language Source language Figure 6: EM accessibility in the common-goal setting. Diagonal cells report source accessibility퐴 EM 푧 (푠, 푠), while off-diagonal cells report cross accessibility퐴 EM 푧 (푠, 푡), where푡 ≠ 푠. Lower forget accessibility and higher retain accessibility indicate better performance. ROC-AUC and Cross-Language Aggregation.Members are treated as the positive class and holdout examples as the negative class. For attack푎, we independently compute the raw ROC-AUC in each source language: AUC (푠) 푎 =ROC-AUC ( 푆 푎 (푧), 푏(푧) 푧∈풬 푠,core 푓 ∪풬 푠,core ℎ ) ,(12) where푏(푧) = 1for members and푏(푧) = 0for non-members. We then macro-average over the ten source-language runs: ̂ AUC 푎 = 1 |ℒ| ∑ 푠∈ℒ AUC (푠) 푎 .(13) An AUC of0.5indicates chance-level ordering, whereas a value above0.5indicates that members tend to receive larger attack scores than non-members. A value below0.5indicates reversed ordering, i.e., the forget examples tend to receive lower membership scores than the holdout examples. We retain the raw AUC to preserve this directionality; therefore, values below0.5should not automatically be interpreted as weaker membership leakage. B.3 General Multilingual Utility Belebele Evaluation.We evaluate general multilingual utility us- ing Belebele [1], a fully parallel multiple-choice reading-comprehension benchmark. For each of the ten benchmark languages, we use all 900 evaluation examples. Each example consists of a passage푐 푖 , a question푞 푖 , four answer options, and a correct option index푗 ∗ 푖 . We evaluate every model in a zero-shot setting using the same model chat template. The passage, question, and four answer op- tions are included in the prompt in the corresponding evaluation language. The model prediction is obtained by selecting the option whose candidate label receives the highest conditional likelihood under the model. Let ̂ 푗 푖 denote the resulting predicted option index. We report accuracy as the Belebele utility metric. For an unlearned model obtained using source language푠and evaluated in language푡, its Belebele accuracy is 푈(푠, 푡) = 1 900 900 ∑ 푖=1 핀 [ ̂ 푗 (푠,푡) 푖 = 푗 ∗ 푖 ] .(14) We first macro-average over the ten evaluation languages for each source-specific unlearned model: ̂ 푈(푠) = 1 |ℒ| ∑ 푡∈ℒ 푈(푠, 푡),(15) and then macro-average over the ten source-language runs: ̂ 푈 = 1 |ℒ| ∑ 푠∈ℒ ̂ 푈(푠).(16) For the source-independent Original and Retrain reference mod- els, we evaluate each model once in every language and report the macro-average over the ten evaluation languages. Higher Bele- bele accuracy indicates better preservation of general multilingual reading-comprehension ability. C Experiments C.1 Experiment Settings Details of Evaluated Unlearning Methods.We evaluate six repre- sentative methods under the same source-language protocol: GA, GD, NPO, SimNPO, BalDRO-NPO, and BalDRO-SimNPO. For each source language푠, let퐷 (푠) 푓 and퐷 (푠) 푟 denote the forget and retain sets, respectively. Given an example푧 = (푥, 푦), we define the token- normalized retain loss as ℓ (푠) 푟 (휃) = 피 (푥,푦)∼퐷 (푠) 푟 [ − 1 |푦| log휋 휃 (푦 ∣ 푥) ] .(17) GA minimizes the reverse cross-entropy on the forget set, while GD additionally minimizes the retain loss [36]: ℓ GA,(푠) 푓 (휃) = 피 (푥,푦)∼퐷 (푠) 푓 [ 1 |푦| log휋 휃 (푦 ∣ 푥) ] , ℓ GA,(푠) all (휃) = ℓ GA,(푠) 푓 (휃), ℓ GD,(푠) all (휃) = ℓ GA,(푠) 푓 (휃) + 휆ℓ (푠) 푟 (휃), (18) where휆controls the retain objective. KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. EN ZH DE ES FR JA AR TH BN SW GAGDNPO ENZHDE ES FR JA AR TH BN SW EN ZH DE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO GAGDNPO ENZHDE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO 0.00 0.16 0.32 0.48 0.63 0.79 0.95 Knowledge Accessibility Forget AccessibilityRetain Accessibility Evaluation language Source language Figure 7: EM accessibility in the language-conditioned setting. Diagonal cells report source accessibility퐴 EM 푧 (푠, 푠), while off- diagonal cells report cross accessibility퐴 EM 푧 (푠, 푡), where푡 ≠ 푠. Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages. EN ZH DE ES FR JA AR TH BN SW GAGDNPO ENZHDE ES FR JA AR TH BN SW EN ZH DE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO GAGDNPO ENZHDE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO 0.00 0.16 0.32 0.48 0.63 0.79 0.95 Knowledge Accessibility Forget AccessibilityRetain Accessibility Evaluation language Source language Figure 8: S accessibility in the common-goal setting. Diagonal cells report source accessibility퐴 S 푧 (푠, 푠), while off-diagonal cells report cross accessibility퐴 S 푧 (푠, 푡), where푡 ≠ 푠. Lower forget accessibility and higher retain accessibility indicate better performance. NPO uses the frozen Original model휋 ref to control the decrease in the target-answer probability [41]. For a forget example푧 푖 = (푥 푖 , 푦 푖 ), its per-example loss is ℓ NPO 푓 ,푖 (휃) = − 2 훼 NPO log휎 ( −훼 NPO log 휋 휃 (푦 푖 ∣ 푥 푖 ) 휋 ref (푦 푖 ∣ 푥 푖 ) ) ,(19) and the complete objective over a forget batch of size퐵is ℓ NPO,(푠) all (휃) = 1 퐵 퐵 ∑ 푖=1 ℓ NPO 푓 ,푖 (휃) + 휆ℓ (푠) 푟 (휃).(20) SimNPO removes the reference model and applies a margin- based logistic loss to the token-normalized negative log-likelihood [ 12]: ℓ SimNPO 푓 ,푖 (휃) = − 2 훼 Sim log휎 ( 훼 Sim [ − 1 |푦 푖 | log휋 휃 (푦 푖 ∣ 푥 푖 ) − 훿 Sim ] ) , ℓ SimNPO,(푠) all (휃) = 훾 Sim 퐵 퐵 ∑ 푖=1 ℓ SimNPO 푓 ,푖 (휃) + 휆ℓ (푠) 푟 (휃). (21) Here,휎(⋅)denotes the sigmoid function, while훼 NPO and훼 Sim control the sharpness of the corresponding losses.훿 Sim specifies the margin applied to the token-normalized negative log-likelihood, and훾 Sim weights the SimNPO forget objective. BalDRO applies distributionally robust aggregation to the per- example forget losses [ 29]. For a batch of lossesℓ 1 , ... , ℓ 퐵 , its contin- uous Donsker–Varadhan aggregator is ℬ 훽 (ℓ 1 , ... , ℓ 퐵 ) = 훽log ( 1 퐵 퐵 ∑ 푖=1 exp ( ℓ 푖 훽 ) ) ,훽 > 0.(22) BalDRO-NPO and BalDRO-SimNPO apply this aggregator to the corresponding per-example forget losses: ℓ BalDRO-NPO,(푠) all (휃) =ℬ 훽 NPO ( ℓ NPO 푓 ,1 , ... , ℓ NPO 푓 ,퐵 ) + 휆ℓ (푠) 푟 (휃), ℓ BalDRO-SimNPO,(푠) all (휃) = 훾 Sim ℬ 훽 Sim ( ℓ SimNPO 푓 ,1 , ... , ℓ SimNPO 푓 ,퐵 ) + 휆ℓ (푠) 푟 (휃). (23) Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM UnlearningKDD ’27, August 1–5, 2027, San Jose, CA, USA EN ZH DE ES FR JA AR TH BN SW GAGDNPO ENZHDE ES FR JA AR TH BN SW EN ZH DE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO GAGDNPO ENZHDE ES FR JA AR TH BN SW SimNPO ENZHDE ES FR JA AR TH BN SW BalDRO-NPO ENZHDE ES FR JA AR TH BN SW BalDRO-SimNPO 0.25 0.38 0.50 0.62 0.75 0.88 1.00 Knowledge Accessibility Forget AccessibilityRetain Accessibility Evaluation language Source language Figure 9: S accessibility in the language-conditioned setting. Diagonal cells report source accessibility퐴 S 푧 (푠, 푠), while off- diagonal cells report cross accessibility퐴 S 푧 (푠, 푡), where푡 ≠ 푠. Successful unlearning requires low forget accessibility on the diagonal, high forget accessibility off the diagonal, and high retain accessibility across all languages. Figure 10: Cosine similarity between the expected answers of paired forget- and retain-set QAs. BalDRO is applied only to the forget objective; all preference-based methods use the same token-normalized retain loss. Original and Retrain are reference models rather than unlearning methods. More Implementation Details.We conduct all model-training and evaluation experiments on a server equipped with eight NVIDIA A800 GPUs. Each source-specific unlearning run is assigned to a single GPU. The common-goal runs use a per-device batch size of 8with four gradient-accumulation steps, whereas the language- conditioned runs use a per-device batch size of4with two accu- mulation steps. These configurations produce the effective batch sizes reported in the main text. For hyperparameter tuning, in addition to the learning-rate grid reported in the main text, we set휆 = 1.0for every method that uses the retain objective. GA and GD have no additional swept method- specific parameters. For NPO and BalDRO-NPO, we search훼 NPO ∈ 0.1, 0.5. For SimNPO and BalDRO-SimNPO, we search훼 Sim ∈ 3.5, 4.5 and훾 Sim ∈ 0.125, 0.25, while fixing훿 Sim = 1.0. For the BalDRO variants, we additionally search훽 NPO ∈ 2, 5and훽 Sim ∈ 2, 5, re- spectively. Retain-side robust aggregation is disabled, so both Bal- DRO parameters apply only to the forget objective. Together with the three learning-rate choices reported in the main text, these set- tings yield three candidate configurations for GA and GD, six for NPO, twelve for SimNPO and BalDRO-NPO, and twenty-four for BalDRO-SimNPO per source language and setting. Hyperparameter and Checkpoint Selection.Consistent with the evaluation practice of existing LLM unlearning benchmarks, in- cluding TOFU [25] and MUSE [31], we do not introduce a sepa- rate held-out validation split. In many current knowledge unlearn- ing settings, the forget and retain sets directly define the knowl- edge to be removed and preserved, while the available evaluation examples are constructed around these knowledge units. Conse- quently, under the fixed knowledge-unit composition and evalu- ation scope, it is difficult to introduce an additional independent validation split. During each hyperparameter sweep, we evaluate every epoch checkpoint on the target and neighbor examples in the correspond- ing source language. Candidate checkpoints are jointly screened using weighted ROUGE-L scores for the target and neighbor ex- amples, with lower target accessibility and higher neighbor acces- sibility serving as the two selection objectives. Although the hyper- parameter search spaces are method-specific, we apply the same checkpoint-selection objectives and screening protocol to all eval- uated unlearning methods and source languages. This shared pro- tocol avoids method-specific selection criteria and supports a fair and consistent comparison among the unlearning methods. The complete hyperparameter ranges and numbers of candidate con- figurations are reported above. After checkpoint selection, we use the selected checkpoints to conduct the full multilingual evaluation. The Source results char- acterize the operating points identified by this common source- language selection protocol, while the complete multilingual re- sults further evaluate the selected checkpoints across different eval- uation languages. Metric Analyses.S measures broad semantic proximity rather than factual equivalence. Figure10shows that the expected an- swers of paired forget- and retain-set QAs are semantically simi- lar. Their BGE-M3 similarities reach 0.48–0.50 in the common-goal setting and 0.41–0.56 in the language-conditioned setting. Thus, a non-trivial S score may reflect proximity to related retain knowl- edge rather than access to the correct forget knowledge. KDD ’27, August 1–5, 2027, San Jose, CA, USAShao et al. Table 5: LLM-as-a-Judge accessibility under English-source unlearning. EN reports evaluation in the source language, while Avg. non-EN averages the other nine evaluation languages. These results describe a single English-source run rather than a macro-average over source languages. Arrows indicate the desired direction. Method Common-goalLanguage-conditioned Forget Accessibility Retain AccessibilityForget Accessibility Retain Accessibility EN↓Avg. non-EN↓EN↑Avg. non-EN↑EN↓Avg. non-EN↑EN↑Avg. non-EN↑ Original0.82380.76760.83150.77470.91880.84610.88960.7900 Retrain0.16680.13010.82610.7738– GA0.00000.00020.00150.00050.08130.08750.15210.1516 GD0.01140.01650.49200.26110.02080.00020.35420.2201 NPO0.03790.03810.76940.59580.10420.10690.80830.7412 SimNPO0.10210.09070.82680.64270.14170.17270.95000.7725 BalDRO-NPO0.04030.05170.80610.66250.08960.08330.93330.7889 BalDRO-SimNPO0.09510.08740.82310.65140.08330.07890.86880.7498 Table 6: Topic-level ROUGE-L accessibility in the language- conditioned setting. Values are the mean±standard devia- tion over nine topic pairs. Standard deviations characterize topic-level heterogeneity. Forget Accessibility Retain Accessibility MethodSource↓Cross↑Source↑Cross↑ Original0.818±0.134 0.774±0.079 0.829±0.091 0.772±0.097 GA0.037±0.050 0.099±0.074 0.060±0.047 0.141±0.099 GD0.036±0.048 0.093±0.131 0.389±0.235 0.197±0.173 NPO0.056±0.099 0.045±0.039 0.829±0.112 0.286±0.096 SimNPO0.098±0.126 0.108±0.062 0.882±0.105 0.278±0.079 BalDRO-NPO0.051±0.097 0.045±0.039 0.859±0.122 0.285±0.104 BalDRO-SimNPO0.078±0.086 0.100±0.072 0.861±0.119 0.264±0.088 C.2 More Heatmaps To examine fine-grained cross-lingual behavior under different answer- level measures, we visualize knowledge accessibility for every source– evaluation-language pair using EM and S. Results for the common- goal setting are shown in Figures 6and8, while Figures7and9 present the language-conditioned setting. In each heatmap, diago- nal cells represent Source results and off-diagonal cells represent Cross results; the left and right blocks measure forgetting of target knowledge and preservation of neighbor knowledge, respectively. The results provide different views of forgetting under EM and S. Many target facts become nearly inaccessible under EM but retain clear semantic signals under S, showing that failure to pro- duce the exact answer does not imply that the underlying knowl- edge is fully inaccessible. The methods also show a consistent for- getting utility trade-off: GA and GD generally achieve stronger for- getting but cause greater damage to neighbor knowledge, whereas NPO-based methods preserve more utility but produce weaker and less consistent forgetting across languages. More importantly, the language-conditioned setting rarely exhibits a stable pattern of low diagonal and high off-diagonal forget accessibility. The strong de- pendence on the source language and transfer direction shows that none of the six evaluated methods can reliably control the cross- lingual boundary of forgetting. C.3 LLM-as-a-Judge Accessibility LLM-as-a-Judge requires the evaluator model to score every gen- erated response, making it substantially more expensive than EM, ROUGE-L, and S. We therefore use only English as the source lan- guage and evaluate the resulting English-source Unlearned mod- els in all ten languages. In Table5, EN denotes evaluation in the source language, while Avg. non-EN averages the other nine eval- uation languages. Since this experiment does not cover all source languages, we report it only as a supplementary diagnostic rather than including it in the ten-source-language macro-average. The results are consistent with the main evaluation. In the common- goal setting, all methods reduce target accessibility in both English and non-English languages, but GA and GD also cause substan- tial damage to neighbor knowledge. NPO-based methods preserve more neighbor knowledge at the cost of weaker forgetting. In the language-conditioned setting, no method achieves the desired be- havior of forgetting in English while preserving the same target knowledge in other languages. This suggests that the main failure is not a general loss of model ability, but the unintended transfer of target forgetting to non-source languages. C.4 Per Topic Analyses To examine topic-level behavior under the language-conditioned setting, we selected nine target–neighbor topic pairs, with three pairs each from English, Chinese, and Japanese, covering 2,700 core QAs. For each topic, we jointly evaluated the core QAs and their surface variants and aggregated their ROUGE-L accessibility. The intended objective was to suppress the target knowledge in its source language while preserving its accessibility in all other lan- guages; neighbor knowledge should remain accessible across both Source and Cross scopes. Table6shows that none of the evaluated methods established this language boundary. Although Source target accessibility was substantially reduced, Cross target accessibility remained only0.045– 0.108, far below0.774for Original, indicating that forgetting prop- agated excessively to non-source languages. The preference-based methods preserved source retain accessibility at0.829–0.882, but their Cross utility decreased to0.264–0.286, showing that source- language preservation did not generalize across languages. More- over, the large topic-level deviations, such as0.093 ± 0.131for GD on Cross target accessibility and0.098 ± 0.126for SimNPO on Source target accessibility, reveal substantial heterogeneity across topics that aggregate results would otherwise conceal.