Paper deep dive
Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning
Snigdha Paul, Manasi Patwardhan, Arman Cohan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/24/2026, 5:43:43 AM
Summary
This paper introduces Scientific Claim Unlearning, a task addressing the need for Large Language Models (LLMs) to remove obsolete or retracted scientific claims. The authors present SciUnlearn, a benchmark dataset comprising computer science, medical, and retracted-paper subsets, to evaluate claim-level forgetting. Experiments with models like OLMO and LLAMA using methods such as Gradient Difference (GD) and Negative Preference Optimization (NPO) reveal that current unlearning techniques achieve only superficial suppression of specific training instances rather than eliminating the underlying structured knowledge, highlighting a gap in effective claim removal.
Entities (8)
Relation Signals (10)
SciUnlearn → createdfor → Scientific Claim Unlearning
confidence 95% · we introduce the task of Scientific Claim Unlearning and present a new benchmark, SciUnlearn.
Current Unlearning Methods → achieves → Superficial Suppression
confidence 93% · current unlearning approaches are unable to effectively eliminate claim-level knowledge and often achieve only superficial suppression
OLMo-3-7B-Instruct → usedinexperiment → SciUnlearn
confidence 92% · We use OLMo-3-7B-Instruct... on a single 80 GB A100 GPU
Llama-3-8B-Instruct → usedinexperiment → SciUnlearn
confidence 92% · extended the experimentation with LLAMA-3-8B-Instruct
SciUnlearn → derivedfrom → Dolma
confidence 90% · we build three benchmark datasets from the Dolma corpus
Negative Preference Optimization → evaluatedon → SciUnlearn
confidence 90% · Negative Preference Optimization (NPO)... on D_cs
Gradient Difference → evaluatedon → SciUnlearn
confidence 90% · We benchmark representative approaches, including Gradient Difference (GD)... on D_cs
SciUnlearn → containssubset → D_cs
confidence 85% · The benchmark consists of a computer science dataset D_cs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language models (LMs) are trained on static scientific corpora, whereas scientific knowledge continuously evolves through correction and revision. Scientific claims encoded within these models may later become retracted, disproven, or updated by subsequent research, creating the risk of disseminating outdated information in scientific workflows. This creates a need for LMs to forget obsolete scientific claims. Machine unlearning offers a promising solution by enabling knowledge removal while maintaining overall model utility. Existing studies primarily investigate instance-level forgetting; however, scientific claims introduce additional challenges because they are interconnected, and continually evolving. To address this gap, we introduce the task of Scientific Claim Unlearning and present a new benchmark, SciUnlearn. We show that current unlearning approaches are unable to effectively eliminate claim-level knowledge and often achieve only superficial suppression, highlighting the need for specialized methods designed for structured knowledge removal.
Tags
Links
- Source: https://arxiv.org/abs/2608.20960v1
- Canonical: https://arxiv.org/abs/2608.20960v1
Trouble viewing inline? Open PDF directly →
Full Text
86,665 characters extracted from source content.
Expand or collapse full text
Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning Snigdha Paul Affiliation: TCS Research, India Email: snigdha.paul@tcs.com Manasi Patwardhan Affiliation: TCS Research, India Email: manasi.patwardhan@tcs.com Arman Cohan Affiliation: Yale University, USA Email: arman.cohan@yale.edu Abstract Language models (LMs) are trained on static scientific corpora, whereas scientific knowledge continuously evolves through correction and revision. Scientific claims encoded within these models may later become retracted, disproven, or updated by subsequent research, creating the risk of disseminating outdated information in scientific workflows. This creates a need for LMs to forget obsolete scientific claims. Machine unlearning offers a promising solution by enabling knowledge removal while maintaining overall model utility. Existing studies primarily investigate instance-level forgetting; however, scientific claims introduce additional challenges because they are interconnected, and continually evolving. To address this gap, we introduce the task of Scientific Claim Unlearning and present a new benchmark, SciUnlearn. We show that current unlearning approaches are unable to effectively eliminate claim-level knowledge and often achieve only superficial suppression, highlighting the need for specialized methods designed for structured knowledge removal. 1 Introduction Large Language Models (LLMs) are increasingly used as scientific assistants 9, supporting hypothesis generation 23; 11, literature reviews 44, and scientific discovery 13; 32; 28. However, LLMs are trained on static corpora with fixed cutoffs, which conflict with the dynamic and self-correcting nature of scientific knowledge. LLMs can internalize scientific claims that are later invalidated by retractions or errors 22, allowing false knowledge to persist in tasks like ideation 2, claim verification 34 etc, also harming in domains like healthcare. LLMs lack mechanisms to deprecate outdated claims, leading to epistemic inertia (Example in App. A) despite new contradictory evidence 5; 12. A good example is models treating the fabricated disease “Bixonimania” as real 43. Also, some scientific knowledge later becomes harmful, sensitive, or dual-use 46, raising regulatory concerns such as the right to be forgotten 53, yet such knowledge is extremely difficult to remove once embedded in LLMs. These challenges motivate scientific claim unlearning, where models remove outdated claims. Although unlearning has been studied for personal data, copyrighted content and safety-critical knowledge 40; 17, scientific claims remains largely unexplored. Existing work does not address claim-level unlearning or provide systematic benchmarks 51. Here, we define the task of scientific claim unlearning (§ 3) and introduce a benchmark to evaluate claim-level forgetting (§ 4). We adapt baseline unlearning algorithms to this setting (§ 5) and analyze the trade-offs between effective forgetting and scientific knowledge retention (§ 6). The benchmark and code are available at https://github.com/snigdhapaul2003/Scientific-Claim-Unlearning/ under Apache License. Figure 1: SciUnlearn Dataset generation pipeline. 2 Related Works Scientific claim extraction structures knowledge from scientific text 33; 49, with recent LLM pipelines combining extraction, retrieval, and verification 45. Existing datasets and tracking systems cover multiple domains and temporal claim evolution 1; 47; 7; 48; 35, but focus on organizing claims rather than unlearning invalid ones. Machine unlearning removes targeted training influence while preserving utility 3, and has been extended to LLMs for privacy, copyright, and harmful-knowledge removal 26; 29; 40; 24; 19. Methods include gradient-based updates, preference optimization and parameter-efficient tuning18; 54; 27; however, removed knowledge may persist in representations and re-emerge across contexts 16; 38. 3 Scientific Claim Unlearning Task Definition Let ℳθM_θ denote a pretrained language model with parameters θ. We assume availability of dataset =ℱ1,ℱ2,ℛext,ℛintD=\F_1,F_2,R_ext,R_int\, where ℱ1∩ℱ2=∅F_1 _2= , are forget sets with distinct types of paraphrased question answer pairs (q,a)(q,a) derived from a shared set of underlying scientific claims S, extracted from scientific papers P included in the pre-training corpus of ℳθM_θ. This enables evaluation of claim level unlearning: when a model is unlearned on ℱ1F_1, it is expected to also forget ℱ2F_2, and vice-versa, thereby testing whether the underlying claim has been removed rather than only specific superficial forms. The retain sets are designed to validate the preservation of non-target knowledge such that, (ℛext∪ℛint)∩(ℱ1∪ℱ2)=∅(R_ext _int)∩(F_1 _2)= . The external retain set ℛextR_ext consists of cited prior knowledge from papers referenced by the anchor papers P, while the internal retain set ℛintR_int contain non claim based contextual information from the same paper. Given the base model ℳθM_θ and a forget set ℱiF_i (i=1i=1 or 22), an unlearning algorithm A updates the model parameters as θ′=θ+(θ,ℱi,ℛ)θ =θ+A(θ,F_i,R) and ℛ⊆ℛext,ℛintR \R_ext,R_int\. Utilization of ℛR is optional depending on the algorithm A. Building upon the standard formulation of LLM unlearning 26, the objective is defined by the Eq. 1. minθ′(q,a)∈ℱi[ℓf(a|q;θ′)]⏟Forget+λ(q,a)∈ℛ[ℓr(a|q;θ′)]⏟Retain _θ \; E_(q,a) _i [ _f(a|q;θ ) ]_Forget+ λ\,E_(q,a) [ _r(a|q;θ ) ]_Retain (1) Figure 2: Example claim and QA in SciUnlearn dataset. where ℓf(a|q;θ′)=logpθ′(a|q) _f(a|q;θ )= p_θ (a|q) denotes the forgetting loss, and ℓr(a|q;θ′)=−logpθ′(a|q) _r(a|q;θ )=- p_θ (a|q) denotes the retention loss. λ≥0λ≥ 0 balances forgetting and retention. Performance is measured on a disjoint forget set as (q,a)∈ℱj[ℓf(a∣q;θ′)],j≠i.E_(q,a) _j\! [ _f(a q;θ ) ],\;j≠ i., which evaluates claim-level forgetting beyond superficial forgetting. 4 SciUnlearn Dataset To study scientific claim unlearning, we build three benchmark datasets from the Dolma corpus (C) 42, which includes academic papers from Semantic Scholar 21. Dolma’s scale, recency, and use in training OLMO models ensure that the targeted claims are likely embedded in pretrained models, enabling meaningful evaluation. We further validate this assumption using likelihood-based membership signals and answer-order perturbations (App. H). The benchmark consists of a computer science dataset csD_cs, a medical dataset medD_med, and a retracted-paper dataset retractedD_retracted. The first two datasets cover domain-specific claim unlearning in active scientific fields, while retractedD_retracted contains Dolma papers retracted after 2024 and identified using the Retraction Watch database 37. The full pipeline is shown in Fig. 1. We sample 100 recent (2018–2022) open-access computer science papers from the Dolma corpus using the Semantic Scholar API 21, forming f⊂C_f . For each paper p∈fp _f, we extract 11–55 claims (p)S(p) using GPT-511 1 Models are accessed via API with temperature = 1.0, top-p = 0.9, and a maximum input length of 40,000 characters. from the abstract, introduction, and conclusion22 2 The detailed prompts used are given in https://github.com/snigdhapaul2003/Scientific-Claim-Unlearning/prompts.txt.. We use an LLM as a judge to evaluate the extracted claims from all forget and retain set papers based on accuracy, independence, and clarity22 2 The detailed prompts used are given in https://github.com/snigdhapaul2003/Scientific-Claim-Unlearning/prompts.txt. on a 0–5 scale, using GPT-5.4, GPT-5.1 and Gemini-3.5-Flash11 1 Models are accessed via API with temperature = 1.0, top-p = 0.9, and a maximum input length of 40,000 characters.. Table 1 reports the percentage of cases in which at least two of the three models assign a score of 4 or higher out of 5. To validate LLM judgments, we correlate LLM scores with expert annotations on 75 claims from 50 randomly selected papers (App. B). Among these 50 papers, 24 are from computer science, 13 from medical science, and 13 from the retracted-paper set. We report Gwet’s AC2 instead of Cohen’s kappa because Cohen’s kappa is susceptible to the kappa paradox under highly imbalanced ratings. As shown in Table 2, Gwet’s AC2 indicates high human–LLM agreement across accuracy, independence, and clarity, supporting LLM-based evaluation as a scalable proxy for human scoring. Domain Accuracy Independence Clarity CS 95.31% 99.71% 99.71% Medical 95.03% 98.01% 99.67% Retracted 92.57% 96.28% 99.32% Table 1: Multi-LLM consensus rates for claim quality evaluation (% of claims with majority score ≥4≥ 4). Criterion GPT-5.4 GPT-5.1 Gemini 3.5 Flash Accuracy 0.90 0.91 0.84 Independence 0.88 0.86 0.87 Clarity 0.91 0.90 0.88 Table 2: Human–LLM agreement (Gwet’s AC2) For each claim s∈(p)s (p), we generate eight (q,a)(q,a) pairs—two each for MCQ, true/false(T/F), fill-in-the-blank(FB), and assertion--reason(AR)22footnotemark: 2 . QA pairs are validated using OLMO-3-7B-Instruct (ℳM) with self-consistency checks 20; 50. Responses are evaluated using exact match, ROUGE 25, and semantic similarity 36, with a threshold of 0.7 for non-exact matches. Claims with fewer than two valid question types are discarded. QA pairs are splitted into ℱ1F_1 and ℱ2F_2, as discussed in § 3. The statistical details of all subsets are presented in Table 3 Dataset Papers Claims F1 F2 Rext Rint csD_cs 76 170 536 536 1060 418 medD_med 78 127 508 508 908 543 retractedD_retracted 91 141 390 390 884 671 Table 3: Overview of the SciUnlearn datasets. Model Method Forget Set 1 ↓ Forget Set 2 ↓ Retain (Ext) ↑ Retain (Int) ↑ General Benchmarks R-F1 EM R-F1 EM R-F1 EM R-F1 EM MMLU Arc-C HellaSwag OLMO Base Model 97.54 94.21 93.92 89.17 95.40 90.75 92.44 83.49 58.75 54.35 76.02 GD (ℱ1F_1) 71.70 58.20 93.51 88.24 93.66 84.15 90.53 72.00 56.44 50.08 70.13 NPO+RT (ℱ1F_1) 88.81 81.52 93.67 89.55 92.44 86.13 90.66 82.05 58.28 49.91 69.89 GD (ℱ2F_2) 91.65 78.35 89.60 83.58 93.27 84.52 91.02 77.03 57.54 49.57 70.93 NPO+RT (ℱ2F_2) 96.14 89.92 86.78 82.27 91.47 85.18 91.60 81.81 57.39 50.51 73.02 LLAMA Base Model 96.61 93.65 87.37 77.61 91.86 86.03 90.07 83.25 64.74 75.58 55.88 GD (ℱ1F_1) 78.33 74.06 88.32 80.03 85.45 80.37 89.00 83.25 63.01 73.91 46.67 NPO+RT (ℱ1F_1) 73.56 70.33 90.91 86.38 86.53 83.01 90.35 83.73 63.35 75.28 53.41 GD (ℱ2F_2) 93.31 88.80 59.18 37.87 76.78 65.47 81.31 73.20 60.93 74.30 47.69 NPO+RT (ℱ2F_2) 96.08 93.09 78.59 66.23 87.36 78.86 85.14 77.27 64.44 75.01 55.37 Table 4: Results of LoRA unlearning using ℱ1F_1 and ℱ2F_2 for GD and NPO+RT on csD_cs. ↓ indicates lower is better (forgetting), while ↑ indicates higher is better (retention and generalization). OLMo-3-7B-Instruct is referred as OLMO and LLAMA3-8B-Instruct is referred as LLAMA. Additional method results are provided in App. L. Model Method Forget Set 1 ↓ Forget Set 2 ↓ Retain (Ext) ↑ Retain (Int) ↑ General Benchmarks R-F1 EM R-F1 EM R-F1 EM R-F1 EM MMLU Arc-C HellaSwag OLMO Base Model 92.16 78.59 84.59 65.96 87.74 71.14 92.49 81.95 58.76 54.35 76.02 GD (ℱ1F_1) 58.20 35.78 75.40 61.05 86.78 73.78 92.77 84.34 57.24 52.47 71.07 NPO+RT (ℱ1F_1) 83.09 67.36 83.93 68.07 89.69 74.77 92.49 85.63 59.50 52.73 75.30 GD (ℱ2F_2) 83.80 68.77 55.59 32.28 88.97 74.77 93.95 86.37 56.43 51.70 72.50 NPO+RT (ℱ2F_2) 92.52 85.64 73.89 81.53 91.96 85.06 94.79 88.07 59.14 53.41 73.54 LLAMA Base Model 86.23 71.92 76.64 56.49 80.39 63.32 90.24 82.68 64.75 75.58 55.88 GD (ℱ1F_1) 70.08 55.78 73.70 56.14 81.61 65.96 93.81 88.21 63.87 75.39 52.99 NPO+RT (ℱ1F_1) 41.70 26.66 41.70 15.43 87.18 70.15 97.83 94.47 47.29 74.97 54.18 GD (ℱ2F_2) 83.32 68.77 68.34 48.77 82.94 67.62 93.90 88.39 63.10 75.29 52.82 NPO+RT (ℱ2F_2) 45.98 30.18 32.79 7.37 88.86 79.07 96.06 93.00 43.20 74.48 49.57 Table 5: Results of full parameter unlearning using ℱ1F_1 and ℱ2F_2 for GD and NPO+RT on medD_med. ↓ indicates lower is better (forgetting), while ↑ indicates higher is better (retention and generalization). OLMo-3-7B-Instruct is referred as OLMO and LLAMA3-8B-Instruct is referred as LLAMA. For ℛextR_ext, we select for each p∈p , the most semantically similar cited paper using SPECTER2 embeddings 41, extracting claims and QA pairs from each p. ℛintR_int is built from the same p using non–claim-specific, contextual QA pairs. We ensure no overlap between forget and retain sets (App. E). Fig. 2 shows an example, with full QA example provided in App. D. Final statistics of SciUnlearn dataset are presented in Table 3. Detailed analysis on the Dataset is provided in App. C. 5 Experimental Set-up We use OLMo-3-7B-Instruct 14 for full-parameter and rank-8 LoRA training on a single 80 GB A100 GPU, and extended the experimentation with LLAMA-3-8B-Instruct. We benchmark representative approaches, including Gradient Difference (GD) 31, Negative Preference Optimization (NPO) 54, NPO with retain objective (NPO+RT) 4, SimNPO 10, and SimNPO with retain set. All methods are applied to the pretrained model using forget sets ℱ1F_1 or ℱ2F_2. Implementation details and hyperparameters are provided in App. F and App. G. Task-specific performance is evaluated using Exact Match (EM) and ROUGE-L F1 (R-F1), capturing strict and paraphrase-tolerant correctness, respectively. General knowledge retention is assessed on MMLU, ARC-Challenge (AC), and HellaSwag (HS) 15; 6; 52, ensuring unlearning does not degrade overall language understanding. Model Method Forget Set 1 ↓ Forget Set 2 ↓ Retain (Ext) ↑ Retain (Int) ↑ General Benchmarks R-F1 EM R-F1 EM R-F1 EM R-F1 EM MMLU Arc-C HellaSwag OLMO Base Model 94.65 87.94 93.05 84.61 94.12 87.33 94.54 87.92 58.75 54.35 76.02 GD (ℱ1F_1) 65.49 44.61 84.96 78.46 91.72 86.31 95.88 91.80 58.90 50.42 73.18 NPO+RT (ℱ1F_1) 70.83 62.82 81.47 75.12 75.03 68.77 94.64 89.26 55.20 52.73 72.84 GD (ℱ2F_2) 82.51 54.87 58.95 44.10 87.42 68.32 93.52 80.32 56.03 50.17 74.04 NPO+RT (ℱ2F_2) 93.73 85.64 89.12 83.07 93.35 87.21 95.01 88.97 58.43 52.30 75.28 LLAMA Base Model 88.63 84.61 88.12 76.15 90.43 83.71 96.53 92.69 64.75 75.58 55.89 GD (ℱ1F_1) 72.34 67.94 85.51 75.38 90.86 84.95 97.06 94.48 64.71 75.50 53.66 NPO+RT (ℱ1F_1) 36.96 33.07 35.69 12.82 81.23 66.17 94.90 80.47 51.05 74.90 50.50 GD (ℱ2F_2) 88.11 80.25 68.27 59.23 92.45 86.99 97.89 96.57 63.38 75.37 53.92 NPO+RT (ℱ2F_2) 48.25 44.35 24.43 1.79 92.02 80.76 98.78 96.27 56.42 74.67 49.23 Table 6: Results of LoRA unlearning using ℱ1F_1 and ℱ2F_2 for GD and NPO+RT on retractedD_retracted. ↓ indicates lower is better (forgetting), while ↑ indicates higher is better (retention and generalization). OLMo-3-7B-Instruct is referred as OLMO and LLAMA3-8B-Instruct is referred as LLAMA. 6 Results Table 4 presents the main LoRA results for GD and NPO+RT across both models on csD_cs, while additional LoRA methods are reported in App. L. Table 16 in App. M presents full-parameter unlearning results for the OLMO model on the dataset csD_cs. The full parameter unlearning results for both OLMO and LLAMA models on dataset medD_med and retractedD_retracted are provided in Table 5 and Table 6 respectively. The results show that forgetting is largely localized to the unlearned forget set, with minimal transfer to the other set. When unlearning is applied to ℱ1F_1 or ℱ2F_2, all representative methods significantly reduce performance on the targeted set leaving the counterpart largely unaffected. A complementary evaluation on open-ended questions provides preliminary evidence that the unlearning effect also extends beyond structured QA formats (App. I). Qualitative analysis (App. N) indicates that simpler QA formats (e.g., T/F and AR) and higher numbers of paraphrased QAs, and non deeply rooted papers amplify forgetting. However, none of the methods causes sharp performance drops, likely due to LoRA-based fine-tuning 20 and the rootedness of claims in the pretraining corpus. Overall, forgetting remains subset-specific, suggesting current methods suppress surface-level patterns rather than underlying claim-level knowledge. Retention differs across methods: GD, NPO+RT, and SimNPO+RT preserve strong performance on both external and internal retain sets, while NPO and SimNPO show larger retention drops due to the absence of an explicit retain objective. Even in case of LLAMA, there is a significant performance drop in all the sets. This indicates that retain-aware methods better protect non-target knowledge. General benchmarks are affected unevenly: NPO and SimNPO without retention cause notable MMLU degradation in OLMO and HellaSwag degradation in LLAMA, while GD, NPO+RT, and SimNPO+RT largely preserve performance. Reference-model scores show lower drift for semantic than direct members in all csD_cs, medD_med and retractedD_retracted, while in case of MIA, Min-K% AUCs also weaken for semantic members, as shown in detail in Table 7. Details for are provided in App. J. Table 7: Membership inference attack (MIA) results for OLMO using Gradient Descent (GD) unlearning. Higher Ref Score Mean indicates stronger forgetting, while an AUC closer to 0.5 indicates membership indistinguishability (Good Unlearning). Mem indicates Member Data (Target Forget Set) and Sem means Semantic Member Data (Complementary Forget Set) Method Data Min-K AUC Ref Score Mean Mem Sem Mem Sem GD CS F1 0.68 0.73 2.89 0.57 GD CS F2 0.53 0.77 3.17 0.29 GD Med F1 0.52 0.70 7.84 0.36 GD Med F2 0.39 0.77 6.61 0.93 GD Retracted F1 0.51 0.82 3.53 0.15 GD Retracted F2 0.57 0.69 4.68 0.33 7 Conclusion This work introduces Scientific Claim Unlearning and the SciUnlearn benchmark for evaluating whether LMs can forget scientific claims at the claim level, also proving it’s applicability with real world retracted claim subset. Experiments show that existing unlearning methods mainly suppress the specific training instances used for unlearning, with limited transfer to paraphrased or complementary forget sets. Overall, our findings suggest that scientific claim unlearning requires algorithms and evaluations that go beyond surface-form suppression and target structured conceptual knowledge. Limitations This work has three main limitations. First, we evaluate representative optimization-based unlearning methods, but do not study mechanistic or representation-level unlearning approaches that may better target internal claim representations. Second, although we include a real-world retracted-paper subset, its sample size remains small, limiting the breadth of conclusions that can be drawn from this setting. Expanding this subset with more retracted or explicitly falsified findings across broader scientific domains such as medicine, biology, and chemistry is an important direction for future work. Third, our experiments are limited to 7B–8B models due to the computational cost of full-parameter unlearning, leaving the behavior of larger frontier-scale models for future investigation. References Achakulvisut et al. (2019) T. Achakulvisut, C. Bhagavatula, D. Acuna, and K. Kording Claim extraction in biomedical publications using deep discourse model and transfer learning. arXiv preprint arXiv:1907.00962. Cited by: §2. Alkan et al. (2025) A. K. Alkan, S. Sourav, M. Jabłońska, S. Astarita, R. Chakrabarty, N. Garuda, P. Khetarpal, M. Pi’oro, D. Tanoglidis, K. G. Iyer, M. S. Polimera, M. J. Smith, T. Ghosal, M. Huertas-Company, S. Kruk, K. Schawinski, and I. Ciucua A survey on hypothesis generation for scientific discovery in the era of large language models. ArXiv abs/2504.05496. External Links: Link Cited by: §1. Bourtoule et al. (2019) L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot Machine unlearning. 2021 IEEE Symposium on Security and Privacy (SP), p. 141–159. External Links: Link Cited by: §2. Bronec and Helcl (2025) J. Bronec and J. Helcl Atyaephyra at SemEval-2025 task 4: low-rank negative preference optimization. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), S. Rosenthal, A. Rosá, D. Ghosh, and M. Zampieri (Eds.), Vienna, Austria, p. 1415–1422. External Links: Link, ISBN 979-8-89176-273-2 Cited by: §F.2, §5. Caliskan et al. (2016) A. Caliskan, J. J. Bryson, and A. Narayanan Semantics derived automatically from language corpora contain human-like biases. Science 356, p. 183 – 186. External Links: Link Cited by: §1. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv abs/1803.05457. External Links: Link Cited by: §5. Diggelmann et al. (2020) T. Diggelmann, J. Boyd-Graber, J. Bulian, M. Ciaramita, and M. Leippold CLIMATE-fever: a dataset for verification of real-world climate claims. External Links: 2012.00614 Cited by: §2. Duan et al. (2024) M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. S. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi Do membership inference attacks work on large language models?. ArXiv abs/2402.07841. External Links: Link Cited by: Appendix J. Exler et al. (2026) D. Exler, M. Raimann, M. F. Münker, M. Rosin, J. E. U. Gómez, C. M. Niemeyer, M. Reischl, and L. Rettenberger LLM‐based scientific assistants for knowledge extraction: which design choices matter?. Advanced Intelligent Discovery. External Links: Link Cited by: §1. Fan et al. (2024) C. Fan, J. Liu, L. Lin, J. Jia, R. Zhang, S. Mei, and S. Liu Simplicity prevails: rethinking negative preference optimization for llm unlearning. ArXiv abs/2410.07163. External Links: Link Cited by: §F.4, §5. Garikaparthi et al. (2025) A. Garikaparthi, M. Patwardhan, L. Vig, and A. Cohan IRIS: interactive research ideation system for accelerating scientific discovery. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), P. Mishra, S. Muresan, and T. Yu (Eds.), Vienna, Austria, p. 592–603. External Links: Link, Document, ISBN 979-8-89176-253-4 Cited by: §1. Gonen and Goldberg (2019) H. Gonen and Y. Goldberg Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In North American Chapter of the Association for Computational Linguistics, External Links: Link Cited by: §1. Gottweis et al. (2025) J. Gottweis, W. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, K. Saab, D. G. Popovici, J. Blum, F. Zhang, K. Chou, A. Hassidim, B. Gokturk, A. Vahdat, P. Kohli, Y. Matias, A. Carroll, K. Kulkarni, N. Tomašev, Y. Guan, V. Dhillon, E. D. Vaishnav, B. Lee, T. R. D. Costa, J. R. Penad’es, G. Peltz, Y. Xu, A. Pawlosky, A. Karthikesalingam, and V. Natarajan Towards an ai co-scientist. ArXiv abs/2502.18864. External Links: Link Cited by: §1. Groeneveld et al. (2024) D. Groeneveld, I. Beltagy, E. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, L. Soldaini, N. Smith, and H. Hajishirzi OLMo: accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 15789–15809. External Links: Link, Document Cited by: Figure 3, Appendix A, §5. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. X. Song, and J. Steinhardt Measuring massive multitask language understanding. ArXiv abs/2009.03300. External Links: Link Cited by: §5. Jia et al. (2025) H. Jia, T. Li, J. Guan, and V. Chandrasekaran The erasure illusion: stress-testing the generalization of llm forgetting evaluation. ArXiv abs/2512.19025. External Links: Link Cited by: §2. Jiang et al. (2025) X. Jiang, Y. Dong, Z. Fang, Y. Ma, T. Wang, R. Cao, B. Li, Z. Jin, W. Jiao, Y. Li, and G. Li Large language model unlearning for source code. ArXiv abs/2506.17125. External Links: Link Cited by: §1. Jin et al. (2025) X. Jin, Z. Bu, B. Vinzamuri, A. Ramakrishna, K. Chang, V. Cevher, and M. Hong Unlearning as multi-task optimization: a normalized gradient difference approach with an adaptive learning rate. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 11278–11294. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: §2. Jin et al. (2024) Z. Jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y. Chen, K. Liu, and J. Zhao RWKU: benchmarking real-world knowledge unlearning for large language models. External Links: 2406.10890 Cited by: §2. jin et al. (2024) Z. jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y. Chen, K. Liu, and J. Zhao RWKU: benchmarking real-world knowledge unlearning for large language models. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 98213–98263. External Links: Document Cited by: §4, §6. Kinney et al. (2023) R. M. Kinney, C. Anastasiades, R. Authur, I. Beltagy, J. Bragg, A. Buraczynski, I. Cachola, S. Candra, Y. Chandrasekhar, A. Cohan, M. Crawford, D. Downey, J. Dunkelberger, O. Etzioni, R. Evans, S. Feldman, J. Gorney, D. W. Graham, F.Q. Hu, R. Huff, D. King, S. Kohlmeier, B. Kuehl, M. Langan, D. Lin, H. Liu, K. Lo, J. Lochner, K. MacMillan, T. C. Murray, C. Newell, S. Rao, S. Rohatgi, P. Sayre, S. Z. Shen, A. Singh, L. Soldaini, S. Subramanian, A. Tanaka, A. D. Wade, L. M. Wagner, L. L. Wang, C. Wilhelm, C. Wu, J. Yang, A. Zamarron, M. van Zuylen, and D. S. Weld The semantic scholar open data platform. ArXiv abs/2301.10140. External Links: Link Cited by: §4, §4. Lesné et al. (2006) S. E. Lesné, M. T. Koh, L. A. Kotilinek, R. Kayed, C. G. Glabe, A. J. Yang, M. Gallagher, and K. H. Ashe A specific amyloid-beta protein assembly in the brain impairs memory.. Nature 440 7082, p. 352–7. External Links: Link Cited by: §1. Li et al. (2025) L. Li, W. Xu, J. Guo, R. Zhao, X. Li, Y. Yuan, B. Zhang, Y. Jiang, Y. Xin, R. Dang, Y. Rong, D. Zhao, T. Feng, and L. Bing Chain of ideas: revolutionizing research via novel idea development with LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 8971–9004. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1. Li et al. (2024) N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. Li, A. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. R. Lababidi, L. Justen, A. Liu, M. K. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, A. Herbert-Voss, C. B. Breuer, A. Zou, M. Mazeika, Z. Wang, P. Oswal, W. Liu, A. A. Hunt, J. Tienken-Harder, K. Y. Shih, K. Talley, J. Guan, R. Kaplan, I. Steneker, D. Campbell, B. Jokubaitis, A. A. Levinson, J. Wang, W. Qian, K. K. Karmakar, S. Basart, S. Fitz, M. V. Levine, P. Kumaraguru, U. K. Tupakula, V. Varadharajan, Y. Shoshitaishvili, J. Ba, K. M. Esvelt, A. Wang, and D. Hendrycks The wmdp benchmark: measuring and reducing malicious use with unlearning. ArXiv abs/2403.03218. External Links: Link Cited by: §2. Lin (2004) C. Lin ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, p. 74–81. External Links: Link Cited by: §4. Liu et al. (2024) S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, X. Xu, Y. Yao, C. Liu, H. Li, K. R. Varshney, M. Bansal, S. Koyejo, and Y. Liu Rethinking machine unlearning for large language models. Nature Machine Intelligence 7, p. 181 – 194. External Links: Link Cited by: §2, §3. Liu et al. (2025) Y. Liu, H. Chen, W. Huang, Y. Ni, and M. Imani LUNE: efficient llm unlearning via lora fine-tuning with negative examples. ArXiv abs/2512.07375. External Links: Link Cited by: §2. Lu et al. (2024) C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha The ai scientist: towards fully automated open-ended scientific discovery. ArXiv abs/2408.06292. External Links: Link Cited by: §1. Maini et al. (2024) P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter TOFU: a task of fictitious unlearning for llms. Cited by: §2. Muhamed et al. (2025) A. Muhamed, J. Bonato, M. T. Diab, and V. Smith SAEs can improve unlearning: dynamic sparse autoencoder guardrails for precision unlearning in LLMs. ArXiv abs/2504.08192. External Links: Link Cited by: Appendix H. Neel et al. (2020) S. Neel, A. Roth, and S. Sharifi-Malvajerdi Descent-to-delete: gradient-based methods for machine unlearning. ArXiv abs/2007.02923. External Links: Link Cited by: §F.1, §5. Novikov et al. (2025) A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog AlphaEvolve: a coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. Cited by: §1. Park and Blake (2012) D. H. Park and C. Blake Identifying comparative claim sentences in full-text scientific articles. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, A. Van Den Bosch and H. Shatkay (Eds.), Jeju Island, Korea, p. 1–9. External Links: Link Cited by: §2. Pradeep et al. (2021) R. Pradeep, X. Ma, R. Nogueira, and J. Lin Scientific claim verification with VerT5erini. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis, E. Holderness, A. Jimeno Yepes, A. Lavelli, A. Minard, J. Pustejovsky, and F. Rinaldi (Eds.), online, p. 94–103. External Links: Link Cited by: §1. Pramanick et al. (2026) A. Pramanick, Y. Hou, S. M. Mohammad, and I. Gurevych ClaimFlow: tracing the evolution of scientific claims in nlp. ArXiv. External Links: Link Cited by: §2. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, p. 3982–3992. External Links: Link, Document Cited by: §4. Retraction Watch (2024) Retraction Watch Retraction watch database. Note: http://retractiondatabase.org/Accessed: 2024-05-20 Cited by: §4. Rybak et al. (2026) P. Rybak, P. Batorski, P. Swoboda, and P. Spurek REBEL: hidden knowledge recovery via evolutionary-based evaluation loop. External Links: 2602.06248, Link Cited by: §2. Shi et al. (2023) W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. S. Zettlemoyer Detecting pretraining data from large language models. ArXiv abs/2310.16789. External Links: Link Cited by: Appendix H. Shi et al. (2024) W. Shi, J. Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. S. Zettlemoyer, N. A. Smith, and C. Zhang MUSE: machine unlearning six-way evaluation for language models. ArXiv abs/2407.06460. External Links: Link Cited by: §1, §2. Singh et al. (2023) A. Singh, M. D’Arcy, A. Cohan, D. Downey, and S. Feldman SciRepEval: a multi-format benchmark for scientific document representations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 5548–5566. External Links: Link, Document Cited by: §4. Soldaini et al. (2024) L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, A. Ravichander, K. Richardson, Z. Shen, E. Strubell, N. Subramani, O. Tafjord, E. Walsh, L. Zettlemoyer, N. Smith, H. Hajishirzi, I. Beltagy, D. Groeneveld, J. Dodge, and K. Lo Dolma: an open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 15725–15788. External Links: Link, Document Cited by: §4. Stokel-Walker (2026) C. Stokel-Walker Scientists invented a fake disease. AI told people it was real. Nature 652, p. 559–561. External Links: Document Cited by: §1. Tang et al. (2025) X. Tang, X. Duan, and Z. Cai Large language models for automated literature review: an evaluation of reference generation, abstract writing, and review composition. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 1602–1617. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §1. Uprety et al. (2025) S. Uprety, B. Buyuklieva, and P. Tiwari Using large language models for hypotheses and claims extraction from scientific literature. In Companion Proceedings of the ACM on Web Conference 2025, W ’25, New York, NY, USA, p. 1645–1648. External Links: ISBN 9798400713316, Link, Document Cited by: §2. Urbina et al. (2022) F. Urbina, F. Lentzos, C. Invernizzi, and S. Ekins Dual use of artificial-intelligence-powered drug discovery. Nature Machine Intelligence 4, p. 189 – 191. External Links: Link Cited by: §1. Wadden et al. (2020) D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi Fact or fiction: verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 7534–7550. External Links: Link, Document Cited by: §2. Wang et al. (2025) Y. Wang, S. Feng, Y. Tsvetkov, and H. Hajishirzi ScienceMeter: tracking scientific knowledge updates in language models. arXiv preprint arXiv:2505.24302. Cited by: §2. Wei (2023) X. Wei ClaimDistiller: scientific claim extraction with supervised contrastive learning. In EEKE/AII@JCDL, External Links: Link Cited by: §2. Xie et al. (2025) L. Xie, X. Teng, S. Ke, H. Wen, and S. Wang Reveal and release: iterative llm unlearning with self-generated data. In Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §4. Yang (2025) R. Yang Unlearning as ablation: toward a falsifiable benchmark for generative scientific discovery. ArXiv abs/2508.17681. External Links: Link Cited by: §1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, p. 4791–4800. External Links: Link, Document Cited by: §5. Zhang et al. (2023) D. Zhang, P. Finckenberg-Broman, T. Hoang, S. Pan, Z. Xing, M. Staples, and X. Xu Right to be forgotten in the era of large language models: implications, challenges, and solutions. AI and Ethics 5, p. 2445 – 2454. External Links: Link Cited by: §1. Zhang et al. (2024) R. Zhang, L. Lin, Y. Bai, and S. Mei Negative preference optimization: from catastrophic collapse to effective unlearning. ArXiv abs/2404.05868. External Links: Link Cited by: §F.2, §2, §5. Appendix A Example of Retracted and Falsified Claims Figure 3: Failure case demonstrating the persistence of invalidated scientific knowledge in language models (OLMO-3-7B-Instruct 14). (a) Year-wise distribution of questions (b) Question type distribution Figure 4: Dataset statistics for Computer Science subset. (a) Year-wise distribution of questions. (b) Distribution across question types. (a) Year-wise distribution of questions (b) Question type distribution Figure 5: Dataset statistics for Medical subset. (a) Year-wise distribution of questions. (b) Distribution across question types. (a) Year-wise distribution of questions (b) Question type distribution Figure 6: Dataset statistics for Retracted paper subset. (a) Year-wise distribution of questions. (b) Distribution across question types. This example illustrates a critical failure mode in modern language models: the persistence of outdated, retracted, or falsified scientific claims. Despite subsequent corrections in the scientific record, models such as OLMO-3-7B-Instruct 14 may continue to generate responses that reflect superseded knowledge. As shown in Figure 3, the model produces confident outputs that do not adequately account for post-publication revisions or retractions. Appendix B Human LLM evaluation Scores We present representative examples of scientific claims extracted from papers in the shared github itself in a pdf named Human LLM Agreement along with their corresponding LLM-based and human evaluation scores. Appendix C Dataset Statistics Figure 4(a),5(a), 6(a) shows the year-wise distribution of questions, which is skewed towards more recent papers for both the forget and retain sets. Figure 4(b), 5(b), 6(b) presents the question-type distribution, illustrating the diversity and relative frequency of different question categories present in the forget and retain sets. The total cost of dataset creation is approximately $36.57 over 2534 GPT-5 API calls. Field Content Forget Paper Title Is it Required? Ranking the Skills Required for a Job-Title Extracted Claim Fine-tuning a multilingual sentence encoder on English job-title data improves skill importance ranking in English while reducing performance on non-English titles. Verbatim Claim The model also performs fairly on other languages (as seen in the table 3), but by fine tuning we lose the multilingual capabilities of the ranking system. Table 8: Forget-set claim used for sample QA generation from Computer Science subset. Type Question Reference OLMo Metrics MCQ 1 Choose the correct option (A–D): When a multilingual sentence encoder is fine-tuned on English job-title data, what is the observed effect on skill importance ranking for English and non-English titles? [A] Both English and non-English performance improve equally [B] English skill importance ranking improves, but performance on non-English titles decreases [C] English performance decreases while non-English improves [D] There is no measurable change in either English or non-English performance B B EM=1.0, ROUGE=1.0, SemSim=1.0 MCQ 2 Choose the correct option (A–D): Fine-tuning a multilingual sentence encoder using only English job-title data leads to which outcome in skill importance ranking across languages? [A] Improvement limited to non-English titles with no change in English [B] Degradation for both English and non-English titles [C] Improved English results coupled with reduced performance on non-English titles [D] No trade-off; both English and non-English results improve C C EM=1.0, ROUGE=1.0, SemSim=1.0 True/False 1 Fine-tuning a multilingual sentence encoder on English job-title data increases skill importance ranking performance for English titles while lowering performance for non-English titles. True True EM=1.0, ROUGE=1.0, SemSim=1.0 True/False 2 Fine-tuning a multilingual sentence encoder on English job-title data improves skill importance ranking performance for both English and non-English titles. False False EM=1.0, ROUGE=1.0, SemSim=1.0 Fill-in-the-blank 1 When a multilingual sentence encoder is fine-tuned on English job-title data, English skill importance ranking improves, but _________ (non-English performance decreases/non-English performance also improves). non-English performance decreases non-English performance decreases EM=1.0, ROUGE=1.0, SemSim=1.0 Fill-in-the-blank 2 Fine-tuning a multilingual sentence encoder using English job-title data produces a trade-off: better English skill importance ranking and _________ (worse non-English performance/unchanged non-English performance). worse non-English performance worse non-English performance EM=1.0, ROUGE=1.0, SemSim=1.0 Assertion–Reason 1 Assertion (A): Fine-tuning a multilingual sentence encoder on English job-title data improves skill importance ranking for English titles while reducing performance on non-English titles. Reason (R): The fine-tuning uses only English job-title examples, not multilingual data. A is True, R is True, and R explains A. A is True, R is True, and R explains A. EM=1.0, ROUGE=1.0, SemSim=1.0 Assertion–Reason 2 Assertion (A): Fine-tuning a multilingual sentence encoder on English job-title data yields improved English skill importance ranking alongside reduced performance on non-English titles. Reason (R): Because the model is perfectly multilingual, English-only fine-tuning cannot hurt non-English performance. A is True, R is False A is True, R is False EM=1.0, ROUGE=1.0, SemSim=1.0 Table 9: Sample forget-set QA pairs generated from the claim in Table 8. Field Content Retain Paper Title JobBERT: Understanding Job Titles through Skills Extracted Claim Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task. Table 10: Retain-set claim used for sample QA generation from Computer Science subset. Type Question Reference OLMo Metrics MCQ 1 Choose the correct option (A–D): Which statement correctly describes how semantic representations of job titles are learned and what they achieve? [A] Representations are learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling; they enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task. [B] Representations are learned by an LSTM-based encoder trained with manual title labels, without token-level gating or negative sampling; they enable taxonomy-specific normalization via rule-based matching and do not outperform generic sentence encoders on this task. [C] Representations are learned by a BERT-based encoder trained to predict job titles from skills, using token-level gating but no negative sampling; they require manual title labels for normalization and show performance comparable to generic sentence encoders. [D] Representations are obtained by unsupervised averaging of word embeddings from vacancy texts; they support taxonomy-agnostic cluster-based normalization but only match the performance of generic sentence encoders on this task. A A EM=1.0, ROUGE=1.0, SemSim=1.0 MCQ 2 Choose the correct option (A–D): What configuration yields job-title representations that enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and surpass generic sentence encoders on that task? [A] Unsupervised averaging of token embeddings from vacancy texts with cluster-based normalization and no negative sampling. [B] A BERT-based encoder trained to reconstruct titles from taxonomies using manual labels, with attention but no negative sampling. [C] A BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling. [D] A generic sentence encoder fine-tuned on unrelated corpora with taxonomy-specific nearest-centroid mapping. C C EM=1.0, ROUGE=1.0, SemSim=1.0 True/False 1 Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task. True True EM=1.0, ROUGE=1.0, SemSim=1.0 True/False 2 Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, without token-level gating or negative sampling, require manual title labels for taxonomy-specific nearest-neighbor normalization and do not outperform generic sentence encoders on this task. False False EM=1.0, ROUGE=1.0, SemSim=1.0 Fill-in-the-blank 1 The learned job-title representations support _________ (taxonomy-agnostic/taxonomy-specific) nearest-neighbor normalization without manual title labels and outperform generic sentence encoders, given they are produced by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts using token-level gating and negative sampling. taxonomy-agnostic taxonomy-agnostic EM=1.0, ROUGE=1.0, SemSim=1.0 Fill-in-the-blank 2 Semantic representations of job titles produced by a _________ (BERT-based/LSTM-based) encoder trained with distant supervision to predict co-occurring skills from vacancy texts, with token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task. worse non-English performance worse non-English performance EM=1.0, ROUGE=1.0, SemSim=1.0 Assertion–Reason 1 Assertion (A): Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task. Reason (R): Predicting co-occurring skills from vacancy texts under distant supervision aligns representations with job-title semantics, and token-level gating plus negative sampling increases discriminative power. A is True, R is True, and R explains A. A is True, R is True, and R explains A. EM=1.0, ROUGE=1.0, SemSim=1.0 Assertion–Reason 2 Assertion (A): Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task. Reason (R): Manual title labels and taxonomy-specific rules are required for training and normalization in this setting. A is True, R is False A is True, R is False EM=1.0, ROUGE=1.0, SemSim=1.0 Table 11: Sample retain-set QA pairs generated from the claim in Table 10. Appendix D Sample Question-Answer This section presents qualitative examples illustrating the end-to-end pipeline, including extracted scientific claims (Table 8 and 10), generated question-answer pairs (Table 9 and 11), and their evaluation from Computer Science subset. For each paper, we list the extracted claims, followed by representative question-answer pairs derived from a selected claim, verbatim statement from where the claim is taken, along with the reference answers, model predictions, and evaluation metrics for both forget and retain set. Appendix E Distribution of Semantic Overlap Strength We analyze the extent of semantic overlap between the forget and retain sets, the distribution of the strongest non-self question similarity for each forget paper is examined. Here, non-self denotes all retain papers excluding the retain paper that directly corresponds to the anchor paper itself. For each forget paper, the maximum cosine similarity between any of its questions and all questions from non-corresponding retain papers was computed. This yields a single “best non-self similarity” score per paper, capturing the strongest potential semantic overlap. Figure 7 shows the distribution of these scores across all forget papers. An empirical threshold of τ=0.8τ=0.8 is used to define significant semantic overlap. The distribution is concentrated in the range of approximately 0.450.45 to 0.650.65, with a mean similarity around 0.530.53. Importantly, no forget paper reaches or exceeds the threshold of 0.80.8, indicating the absence of high-confidence semantic overlap between the forget and retain sets. The absence of high-similarity matches confirms that the retain set is effectively disjoint from the forget set at a semantic level. This validates the dataset construction process and ensures that any observed unlearning behavior cannot be attributed to direct leakage or duplication across the two sets. Figure 7: Distribution of the strongest non-self cosine similarity for each forget paper. The red dashed line indicates the overlap threshold (τ=0.8τ=0.8), and the black dotted line denotes the mean similarity. Appendix F Implementation Details F.1 Gradient Difference (GD) Gradient Difference modifies model parameters by performing gradient descent on the retain set while simultaneously performing gradient ascent on the forget set, thereby reducing the likelihood of generating target forget data 31. The update rule is defined in Equation 2. θ′=θ−ηwr∇θℒretain(θ)+ηwf∇θℒforget(θ)θ =θ-η\,w_r _θL_retain(θ)+η\,w_f _θL_forget(θ) (2) where θ represents model parameters, η is the learning rate, ℒforgetL_forget denotes the loss on the forget set, and ℒretainL_retain denotes the loss on a retain set to preserve useful knowledge. F.2 Negative Preference Optimization (NPO) Negative Preference Optimization 54; 4 formulates unlearning as a preference learning problem, where the model is encouraged to prefer retained responses over those associated with the forget set. The objective is given in Equation 3. ℒNPO=−2β(x,yf)[logσ(βlogpθ0(yf|x)pθ(yf|x))]L_NPO=- 2β\,E_(x,y_f) [ σ (β\, p_ _0(y_f|x)p_θ(y_f|x) ) ] (3) where pθ(y|x)p_θ(y|x) denotes the conditional likelihood of generating response y given input x under the model parameterized by θ, and pθ0(y|x)p_ _0(y|x) denotes the corresponding likelihood under the reference (pretrained) model, which remains fixed during training. yfy_f denotes responses corresponding to the forget set, σ(⋅)σ(·) is the sigmoid function, β is a scaling parameter controlling the sharpness of the preference. F.3 NPO with Retain Set Fine-Tuning To improve stability and mitigate unintended degradation of useful knowledge, the NPO objective is augmented with an additional retain set optimization. While NPO suppresses the likelihood of forget samples relative to a reference model, it does not explicitly enforce the preservation of retained knowledge. Therefore, a complementary retain objective is introduced. The retain loss is defined in Equation 4. ℒretain=−(x,yr)[logpθ(yr|x)]L_retain=-E_(x,y_r) [ p_θ(y_r|x) ] (4) The overall training objective is given in Equation 5. ℒ=λNPOℒNPO+λrℒretainL= _NPOL_NPO+ _rL_retain (5) where λNPO _NPO and λr _r control the relative contributions of the unlearning and retention objectives, respectively. Param GD NPO NPO+RT SimNPO SimNPO+RT ℱ1F_1 ℱ2F_2 ℱ1F_1 ℱ2F_2 ℱ1F_1 ℱ2F_2 ℱ1F_1 ℱ2F_2 ℱ1F_1 ℱ2F_2 LR 2e−52e-5 1e−51e-5 1e−51e-5 1e−51e-5 1e−51e-5 1e−51e-5 2e−52e-5 2e−52e-5 2e−52e-5 2e−52e-5 Epochs 50 10 10 10 30 15 15 15 30 10 β – – 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 γ – – – – – – 0.0 0.0 0.0 0.0 wf,wrw_f,w_r (1.0, 3.0) (1.2, 1.5) (1.0, 0.0) (1.0, 0.0) (1.0, 4.0) (1.2, 0.75) (1.0, 4.0) (1.0, 4.0) (1.0, 4.0) (1.0, 0.25) Table 12: Hyperparameters for unlearning methods under forget sets ℱ1F_1 and ℱ2F_2. LR is Learning Rate; β is preference strength in NPO; γ scales retain regularization in SimNPOs; wfw_f and wrw_r is weights for forget and retain objectives. F.4 SimNPO (Simplified Negative Preference Optimization) SimNPO 10 is a simplified variant of preference-based unlearning that operates without a reference model or explicit retain objective. Unlike standard NPO, which compares the current model against a fixed reference, SimNPO directly penalizes the likelihood of generating forget samples, making it a fully self-contained and reference-free unlearning approach. The objective is defined in Equation 6. ℒSimNPO=−2β(x,yf)[logσ(−βlogpθ(yf|x)−γ)],L_SimNPO=- 2β\,E_(x,y_f) [ σ (-β\, p_θ(y_f|x)-γ ) ], (6) where pθ(yf|x)p_θ(y_f|x) denotes the average log-likelihood of the response yfy_f given input x, computed over answer tokens only. Here, σ(⋅)σ(·) is the sigmoid function, β controls the sharpness of the penalty, and γ is a margin term. This formulation directly encourages the model to reduce the likelihood of forget samples without relying on external comparisons or paired preferences. By operating on the average log-probability over answer tokens, SimNPO focuses on suppressing the generation of target responses at a sequence level. F.5 SimNPO with Retain Set Fine-Tuning (SimNPO+RT) To alleviate unintended forgetting and better preserve useful knowledge, SimNPO is extended with an explicit retain set objective. While SimNPO effectively suppresses forget samples by directly penalizing their likelihood, it does not impose any constraint to maintain performance on retained data. To address this limitation, a retain loss is incorporated alongside the SimNPO objective. The retain loss is defined identically to Equation 4.Thus the combined SimNPO+RT objective is then given in Equation 7 ℒ=λSimNPOℒSimNPO+λrℒretain,L= _SimNPOL_SimNPO+ _rL_retain, (7) Here, λSimNPO _SimNPO and λr _r balance the contributions of the unlearning and retention objectives, respectively. By explicitly encouraging correct predictions on the retain set, SimNPO+RT improves training stability and helps maintain overall model utility while remaining free of reference-model comparisons. Appendix G Hyperparameter Selection for Algorithm Implementation As shown in Table 12, hyperparameters are chosen specifically to balance the trade-off between forgetting and retention, ensuring that the unlearning process effectively removes targeted information while preserving performance on retained data. Values are selected to achieve sufficient forgetting without inducing over-forgetting, which would unnecessarily degrade retained knowledge, thereby maintaining an optimal equilibrium between removal efficacy and utility preservation. Besides, different algorithms require distinct hyperparameters due to inherent differences in their unlearning dynamics and sensitivity to the forgetting–retention trade-off. Appendix H Distinguishing Memorization from General Reasoning Correct QA performance alone does not establish that a model has memorized the underlying content, particularly for multiple-choice questions that may sometimes be answered through general reasoning or surface-level cues. We therefore conduct two complementary analyses to assess whether SciUnlearn measures knowledge acquired during pretraining rather than reasoning ability alone. Likelihood-based membership signals. Following prior work on detecting pretraining data 39, we compare benchmark samples sourced from Dolma (members) with 97 samples drawn from papers published in 2025 and therefore absent from the Dolma corpus (non-members). The two groups exhibit a clear separation. Member samples have a substantially lower average negative log-likelihood (NLL) than non-members (0.24 vs. 3.88) and a higher Min-K% token-probability score at K=20%K=20\% (0.73 vs. 0.47). Thus, OLMo assigns markedly higher likelihood to the benchmark samples, providing evidence that their content was encountered during pretraining. Robustness to answer-order perturbations. We additionally test whether MCQ performance depends on answer-position cues. For a subset of 10 four-option questions, we enumerate all 4!=244!=24 permutations of the answer choices, producing 240 variants in total, following the answer-order intervention used by 30. Before unlearning, the model answers 227 of the 240 variants correctly (94.6%), consistently selecting the same answer content despite changes in its position. This result rules out a fixed answer-position preference and demonstrates stable access to question-specific knowledge across nearly all orderings. After unlearning, accuracy falls to 114 of 240 variants (47.5%), a decrease of 47.1 percentage points. Because this reduction persists across all answer positions, it is unlikely to be explained solely by a change in positional preference. Instead, the result indicates that unlearning weakens the model’s ability to retrieve the targeted information consistently. Neither analysis alone can definitively separate memorization from every form of generalization. Taken together, however, the strong member–non-member separation in likelihood-based signals and the large, order-invariant decline after unlearning provide independent evidence that SciUnlearn captures knowledge acquired from the pretraining corpus and is therefore an appropriate setting for evaluating scientific claim unlearning. Appendix I Free-Form Generation Evaluation To examine whether the observed unlearning effects extend beyond structured QA formats, we conduct a small-scale free-form generation experiment. We construct 50 open-ended questions from 50 claims in the computer science dataset and apply Gradient Difference (GD) unlearning to OLMo. Because exact-match and ROUGE scores are poorly suited to open-ended responses with multiple valid surface forms, we evaluate the likelihood assigned to the target answers using negative log-likelihood (NLL) and Min-K% token probability. Metric Base Model Unlearned Model Mean NLL 1.39 2.59 Median NLL 1.37 2.19 Mean Min-K% prob. 0.58 0.47 Median Min-K% prob. 0.57 0.48 Table 13: Free-form generation results on 50 open-ended questions before and after GD unlearning. Higher NLL and lower Min-K% probability indicate reduced likelihood of the target information. As shown in Table 13, unlearning increases both mean and median NLL, while decreasing the corresponding Min-K% token-probability statistics. The mean NLL rises from 1.39 to 2.59 and the mean Min-K% probability falls from 0.58 to 0.47, indicating that the unlearned model assigns lower likelihood to the targeted information. Although this experiment is limited to 50 questions and one model–algorithm configuration, it provides preliminary evidence that the unlearning effect extends beyond structured QA evaluation to free-form generation. Appendix J Reference Score Drift and Membership Inference Attack To further analyze whether unlearning transfers beyond the explicitly targeted forget subset, we conduct an additional membership inference analysis 8 using both direct members and semantic members. Due to computational and time constraints, we restrict this analysis to the strongest-performing configuration in our experiments, namely GD, NPO+RT and SimNPO+RT on ℱ1F_1. Unlike conventional MIA settings, our goal is not only to distinguish members from non-members, but also to study whether semantically related claims from the complementary forget subset ℱ2F_2 exhibit membership-like behavior after unlearning. This allows us to evaluate whether forgetting propagates beyond the directly unlearned samples. Specifically, we define three groups: (i) Member: QA pairs originating from the directly unlearned forget subset ℱ1F_1, (i) Semantic Member: semantically related QA pairs from ℱ2F_2, and (i) Non-member: unrelated external QA samples not involved in training or unlearning. The semantic-member setup differs from traditional MIA formulations, where the objective is usually binary member vs. non-member discrimination. Here, semantic members are intentionally introduced to probe the locality and transferability of forgetting. We evaluate two signals: Min-K% probability and reference-model score. (1) Min-K% Probability We further compute the Min-K% score proposed in prior memorization studies: MinK(x)=1|Sk|∑t∈Sklogpθ(yt∣x,y<t),MinK(x)= 1|S_k| _t∈ S_k p_θ(y_t x,y_<t), where SkS_k denotes the bottom-k%k\% least confident output tokens. We use k=20%k=20\% following prior work. Less negative values indicate stronger membership behavior. (2) Reference Model Score To quantify drift relative to the original pretrained model, we compute: RefScore(x)=ℒunlearned(x)−ℒbase(x).RefScore(x)=L_unlearned(x)-L_base(x). Scores near zero indicate that the unlearned model behaves similarly to the original base model, whereas larger positive values indicate stronger forgetting-induced deviation. Table 7 shows that direct members exhibit stronger forgetting signatures than semantic members. Overall, the MIA results indicate that forgetting is mostly localized to ℱ1F_1 and transfers only weakly to semantically related claims in ℱ2F_2 and vice versa. Appendix K Examples of Unlearning Across Algorithms To demonstrate the effectiveness of unlearning, we present representative examples from the forget set in Table 14 for Computer Science subset. All algorithms consistently produce incorrect outputs, indicating successful forgetting. Example 1 Question: For Z24 Bridge natural frequency data, time-delay embeddings of the raw series exhibit a toroidal topology driven by cyclic temperature effects, whereas embeddings of cointegrated residuals exhibit an open-ball topology characteristic of approximately Gaussian white-noise processes. Ground Truth: True GD: False NPO: False NPO+RT: False SimNPO: False SimNPO+RT: False Example 2 Question: When the second natural frequency depends nonlinearly on temperature, linear cointegration leaves residual topological structure, whereas Gaussian process-based nonlinear cointegration trained on data that include the nonlinear regime more effectively removes these effects. Ground Truth: True GD: False NPO: False NPO+RT: False SimNPO: False SimNPO+RT: False Example 3 Question: In a linear classifier with an additive patch trigger, the optimal untargeted adversarial perturbation has a (substantial projection/near-zero projection) onto the trigger direction, theoretically explaining the similarity between adversarial and triggered inputs. Ground Truth: substantial GD: near-zero NPO: near-zero NPO+RT: near-zero SimNPO: near-zero SimNPO+RT: near-zero Example 4 Question: Choose the correct option (A–D): When predicted skill importance scores are multiplied by the inverse document frequency of each skill, what is the resulting effect on ranking? [A] It enhances the ranking of specialized skills by penalizing generic skills. [B] It enhances the ranking of generic skills by penalizing specialized skills. [C] It leaves the ranking unchanged regardless of skill specificity. [D] It randomizes the ranking without regard to skill frequency. Ground Truth: A GD: B NPO: B NPO+RT: B SimNPO: B SimNPO+RT: B Example 5 Question: Fine-tuning a multilingual sentence encoder on English job-title data increases skill importance ranking performance for English titles while lowering performance for non-English titles. Ground Truth: True GD: False NPO: False NPO+RT: False SimNPO: False SimNPO+RT: False Table 14: Examples showing consistent incorrect responses across different unlearning algorithms. Appendix L Additional LoRA Results Additional experimentation on LORA based unlearning with NPO, SimNPO and SimNPO+RT are shown in Table 15. Model Method Forget Set 1 ↓ Forget Set 2 ↓ Retain (Ext) ↑ Retain (Int) ↑ General Benchmarks R-F1 EM R-F1 EM R-F1 EM R-F1 EM MMLU Arc-C HellaSwag OLMO NPO (ℱ1F_1) 83.77 73.50 87.26 76.11 83.75 72.83 87.12 77.03 39.38 41.21 69.98 SimNPO (ℱ1F_1) 86.00 75.37 87.75 79.47 86.38 76.98 89.57 79.66 43.68 40.35 71.37 SimNPO+RT (ℱ1F_1) 83.98 69.96 94.29 89.73 92.05 82.92 89.04 77.27 58.32 48.54 69.83 NPO (ℱ2F_2) 94.18 81.15 73.87 54.47 84.58 67.92 87.26 77.51 27.73 41.55 70.36 SimNPO (ℱ2F_2) 92.66 81.34 78.53 62.87 86.74 72.54 90.65 81.57 37.91 41.55 71.55 SimNPO+RT (ℱ2F_2) 94.99 89.55 92.31 86.75 93.23 86.32 92.33 82.77 58.25 50.34 73.44 LLAMA NPO (ℱ1F_1) 70.05 63.80 77.11 63.43 74.57 64.71 85.89 77.03 64.23 74.49 48.63 SimNPO (ℱ1F_1) 76.97 70.33 77.58 62.87 77.70 66.98 88.75 80.86 64.54 75.02 49.31 SimNPO+RT (ℱ1F_1) 85.66 81.52 88.40 80.41 90.13 85.00 89.69 81.33 64.49 75.10 50.85 NPO (ℱ2F_2) 72.76 65.29 74.84 60.82 76.01 66.22 88.31 81.57 63.87 74.28 45.98 SimNPO (ℱ2F_2) 77.06 70.70 75.92 62.68 77.63 66.79 88.48 81.10 64.67 75.21 49.31 SimNPO+RT (ℱ2F_2) 95.68 92.72 76.32 60.63 85.87 75.09 86.54 77.99 64.76 75.69 52.81 Table 15: Additional LoRA unlearning results. OLMO indicates OLMo-3-7B-Instruct and LLAMA indicates LLAMA3-8B-Instruct Appendix M Full Parameter Unlearning Method Forget Set 1 ↓ Forget Set 2 ↓ Retain (Ext) ↑ Retain (Int) ↑ General Benchmarks R-F1 EM R-F1 EM R-F1 EM R-F1 EM MMLU Arc Challenge HellaSwag Base Model 97.54 94.21 93.92 89.17 95.40 90.75 92.44 83.49 58.75 54.35 76.02 GD (ℱ1F_1) 71.33 55.59 91.73 87.12 95.68 93.01 93.70 85.40 56.23 52.30 72.40 NPO (ℱ1F_1) 94.92 89.55 94.58 89.36 94.29 88.67 92.87 83.73 57.59 50.76 75.47 NPO+RT (ℱ1F_1) 81.41 74.81 93.98 90.11 91.33 83.20 90.70 75.83 57.71 52.30 75.00 SimNPO (ℱ1F_1) 96.26 91.23 93.94 89.36 94.18 88.77 92.84 82.77 58.24 52.64 75.94 SimNPO+RT (ℱ1F_1) 73.45 64.73 92.58 89.17 94.84 89.24 92.34 81.10 58.96 52.47 74.50 GD (ℱ2F_2) 95.57 92.35 61.59 53.91 94.61 90.84 92.79 84.44 56.05 49.65 64.63 NPO (ℱ2F_2) 96.78 91.79 87.01 78.54 92.89 86.03 91.74 82.29 57.45 51.87 75.66 NPO+RT (ℱ2F_2) 96.88 92.53 87.09 77.79 92.50 85.28 91.81 82.29 58.01 52.13 75.73 SimNPO (ℱ2F_2) 96.44 91.79 87.01 78.54 92.89 86.03 91.74 82.29 57.45 51.87 75.66 SimNPO+RT (ℱ2F_2) 96.09 92.35 65.90 46.08 92.11 83.11 92.37 82.05 59.14 52.73 72.45 Table 16: Full Parameter Unlearning with OLMo-3-7B-Instruct. R-F1(%): ROUGE-F1, EM(%): Exact Match, AC: Arc Challenge, HS: HellaSwag., ↓ : lower is better (forgetting), and ↑ : higher is better (retention and generalization), Best and Second Best For OLMO model each unlearning algorithm (full parameter unlearning) the result is shown in Table 16 for all experimental setting (i.e., training with Forget Set 1 and Forget Set 2). Appendix N Qualitative Analysis of Unlearning This section presents a detailed qualitative analysis of unlearning across Gradient Difference (GD), NPO+RT, and SimNPO+RT under both Forget Set 1 and Forget Set 2 training settings with Computer Science subset. The analysis combines quantitative error distributions with behavioral observations to understand how unlearning manifests at the level of scientific claims. N.1 Error Distribution Across Question Formats Method Split MCQ True/False Fill in the Blanks Assertion Reason GD trained on Forget Set 1 GD-FS1 Forget Set 1 21/138(15.2%)21/138_\,(15.2\%) 68/101(67.3%)68/101_\,(67.3\%) 49/127(38.6%)49/127_\,(38.6\%) 86/170(50.6%)86/170_\,(50.6\%) Forget Set 2 9/138(6.5%)9/138_\,(6.5\%) 0/101(0.0%)0/101_\,(0.0\%) 22/127(17.3%)22/127_\,(17.3\%) 30/170(17.6%)30/170_\,(17.6\%) Retain_ext 2/284(0.7%)2/284_\,(0.7\%) 32/180(17.8%)32/180_\,(17.8\%) 22/254(8.7%)22/254_\,(8.7\%) 112/342(32.8%)112/342_\,(32.8\%) Retain_int 0/129(0.0%)0/129_\,(0.0\%) 14/126(11.1%)14/126_\,(11.1\%) 9/11(81.8%)9/11_\,(81.8\%) 94/152(61.8%)94/152_\,(61.8\%) GD trained on Forget Set 2 GD-FS2 Forget Set 1 1/138(0.7%)1/138_\,(0.7\%) 1/101(1.0%)1/101_\,(1.0\%) 10/127(7.9%)10/127_\,(7.9\%) 33/170(19.4%)33/170_\,(19.4\%) Forget Set 2 4/138(2.9%)4/138_\,(2.9\%) 50/101(49.5%)50/101_\,(49.5\%) 15/127(11.8%)15/127_\,(11.8\%) 35/170(20.6%)35/170_\,(20.6\%) Retain_ext 8/284(2.8%)8/284_\,(2.8\%) 40/180(22.2%)40/180_\,(22.2\%) 32/254(12.6%)32/254_\,(12.6\%) 72/342(21.1%)72/342_\,(21.1\%) Retain_int 3/129(2.3%)3/129_\,(2.3\%) 7/126(5.6%)7/126_\,(5.6\%) 11/11(100.0%)11/11_\,(100.0\%) 63/152(41.4%)63/152_\,(41.4\%) NPO+RT trained on Forget Set 1 NPO+RT-FS1 Forget Set 1 7/138(5.1%)7/138_\,(5.1\%) 35/101(34.7%)35/101_\,(34.7\%) 25/127(19.7%)25/127_\,(19.7\%) 32/170(18.8%)32/170_\,(18.8\%) Forget Set 2 5/138(3.6%)5/138_\,(3.6\%) 1/101(1.0%)1/101_\,(1.0\%) 35/127(27.6%)35/127_\,(27.6\%) 15/170(8.8%)15/170_\,(8.8\%) Retain_ext 6/284(2.1%)6/284_\,(2.1\%) 30/180(16.7%)30/180_\,(16.7\%) 63/254(24.8%)63/254_\,(24.8\%) 48/342(14.0%)48/342_\,(14.0\%) Retain_int 3/129(2.3%)3/129_\,(2.3\%) 13/126(10.3%)13/126_\,(10.3\%) 11/11(100.0%)11/11_\,(100.0\%) 48/152(31.6%)48/152_\,(31.6\%) NPO+RT trained on Forget Set 2 NPO+RT-FS2 Forget Set 1 5/138(3.6%)5/138_\,(3.6\%) 1/101(1.0%)1/101_\,(1.0\%) 13/127(10.2%)13/127_\,(10.2\%) 35/170(20.6%)35/170_\,(20.6\%) Forget Set 2 2/138(1.4%)2/138_\,(1.4\%) 47/101(46.5%)47/101_\,(46.5\%) 25/127(19.7%)25/127_\,(19.7\%) 21/170(12.4%)21/170_\,(12.4\%) Retain_ext 10/284(3.5%)10/284_\,(3.5\%) 37/180(20.6%)37/180_\,(20.6\%) 51/254(20.1%)51/254_\,(20.1\%) 59/342(17.3%)59/342_\,(17.3\%) Retain_int 3/129(2.3%)3/129_\,(2.3\%) 7/126(5.6%)7/126_\,(5.6\%) 10/11(90.9%)10/11_\,(90.9\%) 56/152(36.8%)56/152_\,(36.8\%) SimNPO+RT trained on Forget Set 1 SimNPO+RT-FS1 Forget Set 1 13/138(9.4%)13/138_\,(9.4\%) 38/101(37.6%)38/101_\,(37.6\%) 33/127(26.0%)33/127_\,(26.0\%) 77/170(45.3%)77/170_\,(45.3\%) Forget Set 2 7/138(5.1%)7/138_\,(5.1\%) 0/101(0.0%)0/101_\,(0.0\%) 33/127(26.0%)33/127_\,(26.0\%) 15/170(8.8%)15/170_\,(8.8\%) Retain_ext 5/284(1.8%)5/284_\,(1.8\%) 26/180(14.4%)26/180_\,(14.4\%) 66/254(26.0%)66/254_\,(26.0\%) 84/342(24.6%)84/342_\,(24.6\%) Retain_int 4/129(3.1%)4/129_\,(3.1\%) 14/126(11.1%)14/126_\,(11.1\%) 11/11(100.0%)11/11_\,(100.0\%) 66/152(43.4%)66/152_\,(43.4\%) SimNPO+RT trained on Forget Set 2 SimNPO+RT-FS2 Forget Set 1 6/138(4.3%)6/138_\,(4.3\%) 4/101(4.0%)4/101_\,(4.0\%) 20/127(15.7%)20/127_\,(15.7\%) 26/170(15.3%)26/170_\,(15.3\%) Forget Set 2 3/138(2.2%)3/138_\,(2.2\%) 12/101(11.9%)12/101_\,(11.9\%) 30/127(23.6%)30/127_\,(23.6\%) 26/170(15.3%)26/170_\,(15.3\%) Retain_ext 12/284(4.2%)12/284_\,(4.2\%) 13/180(7.2%)13/180_\,(7.2\%) 54/254(21.3%)54/254_\,(21.3\%) 66/342(19.3%)66/342_\,(19.3\%) Retain_int 4/129(3.1%)4/129_\,(3.1\%) 2/126(1.6%)2/126_\,(1.6\%) 10/11(90.9%)10/11_\,(90.9\%) 56/152(36.8%)56/152_\,(36.8\%) Table 17: Unified error distribution across all methods and evaluation splits for csD_cs. Cells show errors / total (error rate %). As shown is Table 17, across all methods, error rates are significantly higher on the forget set used during unlearning, indicating effective degradation on seen instances. However, when evaluated on a disjoint forget set, error rates drop sharply. This suggests that unlearning does not generalize well to semantically equivalent but differently phrased questions, and remains largely confined to the exact training instances. Unlearning effectiveness varies across question formats. Structured formats such as True/False and Assertion/Reason exhibit higher error rates, indicating stronger forgetting. In contrast, MCQ and Fill-in-the-blank formats show relatively lower degradation, suggesting that these formats require deeper semantic understanding and are more resistant to unlearning. Overall, current methods appear to induce surface-level suppression rather than removing underlying claim-level knowledge. Analysis on retain sets reveals a consistent pattern across methods. The highest degradation is observed for Fill-in-the-blank questions, followed by Assertion/Reason. MCQ questions are the most retained, while True/False lies in between. This indicates that retention is strongest for formats requiring recognition rather than generation. A recurring failure mode is observed in Assertion–Reason questions, where the model struggles with explanation linkage. Although the model often correctly evaluates both the assertion and the reason individually, it fails to determine whether the reason logically explains the assertion. • Reference: A is True, R is True, and R explains A • Model: A is True, R is True, but R does not explain A This behavior suggests that while factual knowledge is retained, the relational structure between concepts is degraded during unlearning. N.2 Top Error-Prone Papers on Forget and Retain Sets We analyze recurring error-prone papers separately for the forget and retain-external splits. Across both splits, errors concentrate on a small set of papers, suggesting that paper-level fragility and question design, rather than a single unlearning algorithm, drive many failures. Forget set. A consistent subset of forget-set papers repeatedly appears among the most error-prone across algorithms: • Highly consistent across settings: How to measure uncertainty in uncertainty sampling for active learning; Deep Insights into Convolutional Networks for Video Recognition; Math-word embedding in math search and semantic extraction; ReliefE: feature ranking in high-dimensional spaces via manifold embeddings. • Frequently appearing: Is it Required? Ranking the Skills Required for a Job-Title; Hierarchical sparse Cholesky decomposition with applications to high-dimensional spatio-temporal filtering. • Moderately frequent: Distributed deep learning platform for pedestrian detection on IT convergence environment; Disk Cluster Allocation Behavior in Windows and NTFS; Reducing false wake-up in contention-based wake-up control of wireless LANs; Approximation of quantum control correction scheme using deep neural networks. These papers typically contain simple, atomic claims with a single decision boundary. Their repeated occurrence across methods indicates that errors arise from intrinsic claim fragility rather than algorithm-specific weaknesses. Retain external set. For the retain-external split, we examine the most error-prone retained papers across algorithms and training settings: • Appearing in all configurations: An Empirical Study of Recent Face Alignment Methods; Dynamic SDN-Based Radio Access Network Slicing With Deep Reinforcement Learning for URLLC and eMBB Services. • Appearing in five out of six settings: Towards Massively Parallel Computations in Algebraic Geometry; Recurrent Neural Networks with Top-k Gains for Session-based Recommendations; Efficient Non-greedy Optimization of Decision Trees; Adversarial Machine Learning for 5G Communications Security. The same retained papers recur across algorithms because they are fragile items in the dataset, so different training setups affect the same weak points. This pattern suggests question-design sensitivity rather than simple topic difficulty: broad topic recognition often survives, while exact relation checking and phrase-level recall are more vulnerable. N.3 Question Density Analysis of Error-Prone Papers As shown in Table 18, the 10 most error-prone papers exhibit a substantially higher question density, with an average of 27.6 questions per paper. All selected papers lie above the dataset median, and 9 out of 10 exceed the 80th percentile, indicating that they are significantly overrepresented in terms of question coverage. Paper Questions Percentile How to measure uncertainty in uncertainty sampling for active learning 32 98.7 Deep Insights into Convolutional Networks for Video Recognition 30 94.7 Math-word embedding in math search and semantic extraction 38 100.0 ReliefE: feature ranking in high-dimensional spaces via manifold embeddings 28 89.5 Is it Required? Ranking the Skills Required for a Job-Title 26 86.8 Hierarchical sparse Cholesky decomposition with applications to high-dimensional spatio-temporal filtering 30 94.7 Distributed deep learning platform for pedestrian detection on IT convergence environment 32 98.7 Disk Cluster Allocation Behavior in Windows and NTFS 14 61.8 Reducing false wake-up in contention-based wake-up control of wireless LANs 20 81.6 Approximation of quantum control correction scheme using deep neural networks 26 86.8 Table 18: Question density statistics for the most error-prone papers. Among these, Math-word embedding in math search and semantic extraction is the most densely represented paper with 38 questions, corresponding to the 100th percentile. In contrast, Disk Cluster Allocation Behavior in Windows and NTFS has the lowest count among the selected set (14 questions), yet it still lies above the dataset median. These findings indicate that highly represented papers are more likely to appear among the most error-prone cases, suggesting that both exposure frequency and claim density contribute to unlearning difficulty. N.4 Analysis of Cross-Forget-Set Spillover in Unlearning For a subset of papers, unlearning with one forget set induces measurable forgetting in the other forget set as well—yet this spillover does not occur universally. Understanding why some papers are susceptible to this cross-forget-set effect while others are not is the central question explored in subsection. Table 19 reports, for each unlearning algorithm and forget-set configuration, how many papers exhibit errors (i.e. measurable forgetting) in both forget sets simultaneously, in only the first forget set, and in only the second forget set. Algorithm / Config Both Only F1 Only F2 Total GD F1 39 30 1 70 GD F2 30 5 25 60 NPO RT F1 29 20 7 56 NPO RT F2 28 13 23 64 SimNPO RT F1 32 31 3 66 SimNPO RT F2 29 9 15 53 Table 19: Per-paper error distribution across the two forget sets for each unlearning algorithm and forget-set configuration. Both denotes papers showing forgetting in both sets; Only F1 and Only F2 denote papers showing forgetting exclusively in Forget Set 1 or Forget Set 2, respectively. The pattern is consistent across all six algorithm–configuration combinations. This consistency suggests the susceptibility to spillover is a property of the papers themselves rather than an artefact of any particular unlearning algorithm. (a) KDE Plot of normal citations (b) KDE Plot of connected citations Figure 8: KDE plot for GD algorithm when unlearned with Forget Set 1. (a) KDE Plot of normal citations (b) KDE Plot of connected citations Figure 9: KDE plot for GD algorithm when unlearned with Forget Set 2. (a) KDE Plot of normal citations (b) KDE Plot of connected citations Figure 10: KDE plot for NPO+RT algorithm when unlearned with Forget Set 1. (a) KDE Plot of normal citations (b) KDE Plot of connected citations Figure 11: KDE plot for NPO+RT algorithm when unlearned with Forget Set 2. (a) KDE Plot of normal citations (b) KDE Plot of connected citations Figure 12: KDE plot for SimNPO+RT algorithm when unlearned with Forget Set 1. (a) KDE Plot of normal citations (b) KDE Plot of connected citations Figure 13: KDE plot for SimNPO+RT algorithm when unlearned with Forget Set 2. N.4.1 Proxy Measure: Citation-Based Embeddedness The natural hypothesis is that papers which are forgotten by both forget sets are less deeply embedded in the model’s pretraining corpus—their knowledge representations are less reinforced by a wider neighbourhood of related content. Directly measuring how deeply a paper is rooted in a pretrained model’s weights is intractable, however, as it would require exhaustive probing across the full parameter space. We therefore adopt two proxy measures derived from the Semantic Scholar citation graph. 1. Normal citation count. The number of papers that directly cite the paper under consideration. A highly-cited paper is more likely to have its findings reproduced, discussed, and cross-referenced in a large volume of text that would itself be present in a large pretraining corpus. Consequently, the paper’s content is represented not just once but implicitly through thousands of derivative works. 2. Connected citation count. Defined as the paper’s own citation count plus the sum of citation counts of every paper that cites it: Cconnected(p)=C(p)+∑q∈citers(p)C(q).C_connected(p)=C(p)+ _q (p)C(q). (8) This second-order measure captures how influential the citers of a paper are. A paper cited by other heavily-cited papers propagates its conceptual footprint far more widely through the literature and thus through any corpus derived from it—than a paper cited only by obscure works. Both measures serve as proxies for corpus embeddedness: the more a paper’s ideas permeate the broader literature, the more redundant and distributed its representation is likely to be in the model’s weights, and the harder it becomes to confine unlearning to a single, targeted forget set. To examine whether citation counts differ systematically between the two groups, we plot, for each algorithm and forget-set configuration, the distribution of both normal and connected citation counts for papers affected only by the single active forget set (labelled Single) against papers affected by both forget sets (labelled Overlapped). Each distribution is rendered as a KDE-smoothed curve (outliers removed via the IQR fence method with k=1.5k=1.5 prior to plotting), with vertical lines indicating the group mean (dark green) and median (dark purple). Separate plots are produced for normal citation counts and connected citation counts for each of the six algorithm–configuration pairs, yielding twelve figures in total from Figure 8(a) to Figure 13(b). Across the plots, a consistent trend emerges: papers that are forgotten by both forget sets tend to exhibit lower citation counts—both normal and connected—than papers that are forgotten only by the forget set used during unlearning. This holds across most algorithm–configuration pairs, where the median citation value of the non-overlapped paper set lies to the right of the overlapped paper set (Figure 8(a), 8(b), 10(a), 10(b), 11(a), 12(a), 12(b), 13(a), 13(b)). The implication is consistent with our proxy hypothesis: papers with lower citation counts are less deeply rooted in the pretraining corpus. Their knowledge is likely represented through fewer direct and derivative textual contexts, making the corresponding model representations less redundant and therefore easier to perturb. When unlearning is applied to one forget set, the update can spill over to the other forget set because both sets rely on a comparatively fragile and weakly reinforced representation of the same paper. By contrast, highly cited papers—and papers whose citers are themselves highly cited—leave a broader imprint on pretraining corpora. Their claims are represented not only directly but also indirectly through a large web of derivative literature, making them more resistant to cross-set forgetting from a single unlearning signal. This finding suggests that low-citation papers are more vulnerable to broad spillover across paraphrased forget sets, whereas highly cited papers may require stronger or more comprehensive unlearning signals to remove claim-level knowledge beyond the specific surface forms used during training. Appendix O Use of AI Assistance AI-based writing tools were used only for basic assistance during manuscript preparation, such as refining language after the initial text was written, improving grammar, and checking clarity. All core ideas, experimental design, dataset construction, analysis, and conclusions were developed and verified by the authors. The authors take full responsibility for the final content of the paper.