Paper deep dive
Confess What You Know: Forget-Set Misalignment with Model Knowledge in LLM Unlearning
Miso Kim, Georu Lee, Seungwon Jeong, Woojin Lee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/2/2026, 5:27:03 AM
Summary
The paper introduces 'forget-set misalignment' in Large Language Model (LLM) unlearning, where the user-defined forget set does not match the model's actual memorized knowledge. This leads to two failure modes: 'Under Unlearning' (privacy leakage due to omitted memorized data) and 'Out-of-Knowledge Unlearning' (utility degradation due to perturbing parameters for non-memorized data). The authors propose CONFS (CONfession-to-Forget-Set), a data-blind framework that elicits the model's memorized knowledge via recursive prompting to construct an aligned forget set, achieving performance close to gold-standard unlearning.
Entities (7)
Relation Signals (7)
Forget-Set Misalignment → includes → Under Unlearning
confidence 95% · We term this gap forget-set misalignment and identify two cases. In Under Unlearning...
Forget-Set Misalignment → includes → Out-of-Knowledge Unlearning
confidence 95% · We term this gap forget-set misalignment and identify two cases. In Out-of-Knowledge Unlearning...
CONFS → mitigates → Forget-Set Misalignment
confidence 95% · By constructing a forget set aligned with the model’s memorized knowledge, CONFS mitigates the adverse effects of forget-set misalignment.
Out-of-Knowledge Unlearning → causes → Utility Degradation
confidence 92% · In Out-of-Knowledge Unlearning, the algorithm is driven to 'forget' knowledge the model never learned, perturbing parameters and degrading utility.
Under Unlearning → causes → Privacy Leakage
confidence 92% · In Under Unlearning, the forget set omits memorized information and leakage persists.
CONFS → evaluatedon → TOFU
confidence 90% · Across synthetic, multimodal, and real-world benchmarks, CONFS approaches Gold-standard performance... We use the TOFU dataset... to verify that our observations are not specific to this small controlled partition
CONFS → uses → SRO Triplets
confidence 90% · We elicit the model’s memorized knowledge and formalize it into a discrete set of Subject-Relation-Object (SRO) triplets
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Machine unlearning for large language models (LLMs) often assumes that a pre-defined forget set matches what the model has memorized, but this frequently breaks in realistic privacy settings where the original training data is inaccessible. We term this gap forget-set misalignment and identify two cases. In Under Unlearning, the forget set omits memorized information and leakage persists. In Out-of-Knowledge Unlearning, the algorithm is driven to "forget" knowledge the model never learned, perturbing parameters and degrading utility. Using gradient-level analysis, we show these behaviors arise from misaligned unlearning targets rather than specific optimization choices. We then propose CONfession-to-Forget-Set (CONFS), a data-blind framework that constructs model-aligned forget sets by eliciting and formalizing the model's memorized knowledge. Across synthetic, multimodal, and real-world benchmarks, CONFS approaches Gold-standard performance on several metrics and achieves a competitive forgetting-utility balance, while preserving utility better than other data-blind forget-set constructions.
Tags
Links
- Source: https://arxiv.org/abs/2609.00605v1
- Canonical: https://arxiv.org/abs/2609.00605v1
Trouble viewing inline? Open PDF directly →
Full Text
89,846 characters extracted from source content.
Expand or collapse full text
Confess What You Know: Forget-Set Misalignment with Model Knowledge in LLM Unlearning Miso Kim Georu Lee Seungwon Jeong Woojin Lee †thanks: Corresponding author. Affiliation: Dongguk University-Seoul Email: 2021110472,dlrjfn1,youai058,wj926@dgu.ac.kr Abstract Machine unlearning for large language models (LLMs) often assumes that a pre-defined forget set matches what the model has memorized, but this frequently breaks in realistic privacy settings where the original training data is inaccessible. We term this gap forget-set misalignment and identify two cases. In Under Unlearning, the forget set omits memorized information and leakage persists. In Out-of-Knowledge Unlearning, the algorithm is driven to “forget” knowledge the model never learned, perturbing parameters and degrading utility. Using gradient-level analysis, we show these behaviors arise from misaligned unlearning targets rather than specific optimization choices. We then propose CONfession-to-Forget-Set (CONFS), a data-blind framework that constructs model-aligned forget sets by eliciting and formalizing the model’s memorized knowledge. Across synthetic, multimodal, and real-world benchmarks, CONFS approaches Gold-standard performance on several metrics and achieves a competitive forgetting-utility balance, while preserving utility better than other data-blind forget-set constructions. 1 Introduction Large Language Models (LLMs) are trained on vast, heterogeneous datasets that often contain sensitive personal information Achiam et al. (2023); Touvron et al. (2023); Grattafiori et al. (2024); Bai et al. (2022); Team et al. (2023). This introduces a critical vulnerability, as models may unintentionally leak confidential data at deployment. To mitigate these privacy threats, LLM unlearning has emerged as an important post-hoc solution Yao et al. (2024); Jang et al. (2023). In this setting, sensitive data targeted for removal is pre-defined and commonly referred to as a forget set Geng et al. (2025). Given this set, various unlearning strategies are applied so that the model selectively forgets the specified information Liu et al. (2022); Jang et al. (2023); Zhang et al. (2024); Chen and Yang (2023). However, relying on a pre-defined forget set is often impractical in real-world scenarios Enck et al. (2014); Romanosky et al. (2011). Suppose an individual discovers that an LLM can reveal some of their personal information and requests its removal (Fig. 1). Although the individual can specify the details they want deleted, they cannot observe which details the model actually memorized. As a result, the requested forget set may differ from the model’s hidden memorized knowledge. We refer to this mismatch as forget-set misalignment, a setting largely overlooked by existing benchmarks that assume perfectly specified unlearning targets Maini et al. (2024); Dontsov et al. (2025); Liu et al. (2025). In this work, we investigate how forget-set misalignment fundamentally alters unlearning behavior under realistic settings. We first identify two types of forget-set misalignment, each leading to a distinct failure mode. Under Unlearning occurs when memorized sensitive information is omitted from the provided forget set. In this case, the forgetting effect remains largely confined to the requested facts and fails to generalize to associated entity-level knowledge, leaving core privacy risks intact. Out-of-Knowledge Unlearning occurs when the forget set contains information that the model never encountered during training, unnecessarily perturbing model parameters and substantially degrading general utility. Figure 1: Illustration of forget-set misalignment in LLM unlearning. Only part of a user’s personal information is retained as the model’s memorized knowledge, but this boundary is hidden from the user. As a result, deletion requests may omit memorized information (Under Unlearning) or include non-memorized information (Out-of-Knowledge Unlearning), leading to privacy leakage or unnecessary utility degradation. Motivated by these findings, we introduce CONfession-to-Forget-Set (CONFS), which dynamically generates a forget set by prompting the model to confess its own memorized knowledge. We elicit the model’s memorized knowledge and formalize it into a discrete set of Subject-Relation-Object (SRO) triplets, transforming ambiguous semantic information into verifiable, atomic units. To ensure comprehensive coverage, we employ a recursive reconfession process that iteratively probes the model for hidden details, effectively reducing gaps in the forget set. By constructing a forget set aligned with the model’s memorized knowledge, CONFS mitigates the adverse effects of forget-set misalignment. Despite operating in a fully data-blind setting without access to the pre-training data, CONFS approaches Gold-standard performance on several metrics and achieves a competitive forgetting-utility balance, while better preserving utility than alternative surrogate forget-set constructions. These results show that reliable unlearning depends not only on the optimization objective, but also on whether the forget set matches what the model actually knows. L−D_L- ↓ L+D_L+ ↓ Retain (retainD_retain) ↑ Real Authors (RAD_RA) ↑ Prob. R-L Prob. R-L Prob. R-L Prob. R-L pre-trained model step 0 0.996 0.986 0.995 0.990 0.995 0.991 0.058 0.886 Under Unlearning (:L+F:D_L+) step 12 0.993 0.976 0.921 0.808 0.991 0.967 0.077 0.904 step 24 0.901 0.826 0.650 0.563 0.888 0.807 0.081 0.884 step 36 0.740 0.695 0.442 0.473 0.749 0.709 0.070 0.804 step 48 0.637 0.635 0.327 0.426 0.669 0.649 0.064 0.781 step 60 0.624 0.631 0.312 0.424 0.658 0.640 0.063 0.776 Out-of-Knowledge Unlearning (:L+∪N+F:D_L+ _N+) step 12 0.983 0.964 0.983 0.902 0.981 0.944 0.030 0.871 step 24 0.734 0.801 0.623 0.532 0.744 0.701 0.010 0.800 step 36 0.485 0.713 0.410 0.492 0.485 0.633 0.006 0.769 step 48 0.403 0.658 0.350 0.431 0.435 0.628 0.006 0.757 step 60 0.382 0.609 0.292 0.402 0.412 0.609 0.004 0.750 Table 1: Preliminary results on TOFU synthetic authors. We evaluate four splits: (i) L−D_L- (Leaked: Yes / Requested: No), (i) L+D_L+ (Leaked: Yes / Requested: Yes), (i) the retain set retainD_retain, and (iv) the Real Authors set RAD_RA. We report unlearning trajectories for Under Unlearning (:L+F:D_L+) and Out-of-Knowledge Unlearning (:L+∪N+F:D_L+ _N+). Values with a blue background denote Under Unlearning, whereas values with an orange background denote Out-of-Knowledge Unlearning. 2 Related Work 2.1 Machine Unlearning in LLMs In LLMs, most unlearning methods are framed as post-hoc fine-tuning without full retraining Geng et al. (2025). Given a pre-defined forget set, these methods degrade target behavior via gradient ascent Jang et al. (2023); Yao et al. (2024), preference-based losses Rafailov et al. (2023); Zhang et al. (2024); Maini et al. (2024), or reinforcement learning Schulman et al. (2017); Kassem et al. (2023), while a retain set with auxiliary objectives preserves utility. Beyond autoregressive LLMs, unlearning has recently been extended to masked diffusion language models Lee et al. (2026). Despite differences in optimization strategies, prior work commonly assumes that the deletion target is fully captured by the pre-defined forget set. 2.2 Benchmarks and Forget-Set Construction Existing LLM unlearning benchmarks are broadly categorized into synthetic and real-world benchmarks, differing in how the forget set is constructed. Synthetic benchmarks such as TOFU Maini et al. (2024), CLEAR Dontsov et al. (2025), and MLLMU Liu et al. (2025) construct fictitious personas and explicitly inject their information via fine-tuning, defining the forget set as a subset of the injected content. This design intentionally aligns the forget set with the model’s memorized knowledge, enabling reproducible evaluation but deviating from realistic privacy leakage scenarios. In contrast, real-world benchmarks address unlearning of factual knowledge about real individuals. RWKU Jin et al. (2024) operates under a data-blind setting, where neither the forget nor the retain corpus is accessible. It instead constructs a surrogate forget set by prompting the target model to generate target-related factual text. While this avoids relying on the original training data, the resulting unlearning signals are derived from unstructured text, making it difficult to precisely isolate memorized personal information. 2.3 Adverse Effects of Improper Forget Sets Recent work has shown that improperly specified forget sets can induce adverse effects during unlearning. In adversarial settings, malicious deletion requests induce abnormal gradients rather than remove specific knowledge, triggering indiscriminate parameter updates and broad performance degradation Huang et al. (2024a); Hu et al. (2023). A related issue arises when unlearning is repeatedly applied to information that has already been partially forgotten Wang et al. (2025); Zhao et al. (2024); Huang et al. (2024b); Yang et al. (2025). In such cases, low-confidence targets can still induce large gradients despite little remaining knowledge, causing avoidable degradation of retain performance. Importantly, these effects are not limited to adversarial settings. In real-world scenarios, pre-defined forget sets often fail to align with the model’s memorized knowledge, destabilizing unlearning and degrading overall performance even without malicious intent. We analyze this forget-set misalignment in Sec. 3. 3 Forget-Set Misalignment Unlearning experiments usually assume perfect alignment between the pre-defined forget set and the model’s memorized knowledge Maini et al. (2024); Dontsov et al. (2025). Under this assumption, degradation on the forget set is interpreted as successful removal of the intended knowledge. In practice, however, an LLM’s pre-training data is often inaccessible to users, who therefore cannot observe what the model has memorized. As a result, the forget set they define may not match the model’s actual knowledge, a gap we call forget-set misalignment. In this section, we present the first systematic study of forget-set misalignment, analyzing how it degrades unlearning across realistic request scenarios. 3.1 Constructing Misaligned Forget Sets Controlled Datasets. We use the TOFU dataset Maini et al. (2024), which consists of 200 fictitious entities, and select 20 of them as unlearning targets. TOFU provides 20 QA pairs for each target entity. From these, we use 15 QA pairs, denoted qaD_qa, and categorize each sample along two axes: whether it was injected during pre-training and whether the user requested it for forgetting. The remaining five samples are neither injected during pre-training nor requested for forgetting, so they are excluded because they do not correspond to any unlearning scenario. Subset Leaked Requested Request type L+D_L+ Yes Yes Aligned L−D_L- Yes No Incomplete N+D_N+ No Yes Over-broad Table 2: Controlled partition of QA pairs by leakage and forget request. L−D_L-: leaked data omitted from the request. N+D_N+: non-memorized data included in the request. We partition the 15 samples into three disjoint subsets of equal size, L+D_L+, L−D_L-, and N+D_N+. Under this limited QA budget, the equal three-way partition yields five QA pairs per subset. In each subset’s subscript, L and N denote leaked and non-leaked samples, respectively, while ++ and −- denote requested and not requested samples. In particular, N+D_N+ contains QA pairs that the model never saw during pre-training but that the user includes in the forget request. This partitioning is summarized in Table 2 and corresponds to the unlearning scenarios illustrated in Fig. 1. To verify that our observations are not specific to this small controlled partition, Table 5 examines the same effects on the substantially larger forget sets used in the main experiments. Beyond these private-information subsets, we also evaluate utility preservation and general knowledge retention. For utility, we use the official TOFU retain split retainD_retain, which contains QA pairs about non-target entities. For general knowledge, we use the Real Authors set RAD_RA, which contains QA pairs about real-world authors. We evaluate model performance with Token Probability (Prob.) and ROUGE-L (R-L) on these four sets: L+D_L+, L−D_L-, retainD_retain, and RAD_RA. Further details on the setup for this analysis are provided in Appendix B.1. Figure 2: Gradient analysis under forget-set misalignment. (a) Under Unlearning: gradient inner products summarized by EtfE_tf and EegE_eg. (b) Out-of-Knowledge Unlearning: inner products ⟨gL+,gretain⟩ g_L+,g_retain and ⟨gN+,gretain⟩ g_N+,g_retain from eq. (5). 3.2 Unlearning Settings Gradient Ascent (GA). To analyze the impact of misalignment, we use GA, a representative unlearning baseline. For a QA pair (q,a)(q,a) with answer length T, we use the answer NLL, ℒ(q,a;θ):=−∑t=1Tlogpθ(at∣q,a<t)L(q,a;θ):=- _t=1^T p_θ(a_t q,a_<t). For a dataset D, let ℒ(θ):=(q,a)∼[ℒ(q,a,θ)]L_D(θ):=E_(q,a) [L(q,a;θ)]. A one-step GA update for the forget set F and learning rate η is defined as: θ′=θ+η∇θℒ(θ)=θ+ηg,θ =θ+η _θL_F(θ)=θ+η g_F, (1) where g_F denotes the unlearning gradient. Quantifying the Influence of Unlearning Updates. To quantify how a single unlearning update on F affects a dataset D (e.g., L−D_L- or retainD_retain), we define the loss increment on D as Δℒ():=ℒ(θ′)−ℒ(θ) _F(D):=L_D(θ )-L_D(θ). By Taylor expanding ℒL_D to first order around θ and substituting the one-step update, we express this loss increment as the inner product of the unlearning and evaluation gradients: Δℒ()≈η⟨g,g⟩, _F(D)≈η g_F,g_D , (2) where g:=∇θℒ(θ)g_D:= _θL_D(θ). Henceforth, we use gsetg_set as shorthand for gsetg_D_set. A positive ⟨g,g⟩ g_F,g_D indicates that the update on F also raises the loss on D. A value near zero means D is essentially unaffected. Eq. (2) is a first-order, single-step diagnostic for predicting the sign and relative ordering of per-set effects, not the full optimization trajectory. Fig. 2 recomputes the diagnostic at every GA step, while Table 1 reports directly measured performance. Unlearning Scenarios. We use a model θ that has memorized the leaked QA pairs L=L+∪L−D_L=D_L+ _L-. We study two unlearning scenarios in which the requested forget set F deviates from the model’s memorized knowledge (≠LF _L). • Under Unlearning (=L+F=D_L+): the request covers only part of the memorized data. We examine whether unlearning generalizes to the omitted leaked samples L−D_L-. • Out-of-Knowledge Unlearning (=L+∪N+F=D_L+ _N+): the request additionally includes N+D_N+, data the model never saw during pre-training. We examine whether such requests degrade model utility on knowledge the model legitimately retains. Table 1 reports how Prob. and R-L evolve under each setting. 3.3 Analysis of the Under Unlearning Setting In this setting, the model is requested to forget L+D_L+, only part of its memorized data. Here we show that unlearning L+D_L+ does not generalize to the omitted leaked samples of the same entity L−D_L-. To evaluate this precisely, we introduce two metrics that each isolate a forgetting signal from global utility decay. The Targeted Forgetting Effect measures genuine forgetting on the requested set L+D_L+, and the Entity Generalization Effect measures whether that forgetting generalizes to the omitted leaked set L−D_L-. Without this separation, global utility decay on the omitted leaked set L−D_L- could be mistaken for genuine generalization of forgetting from the requested set L+D_L+. Since =L+F=D_L+ throughout this subsection, we write Δℒ() (D) for ΔℒL+() _D_L+(D). Targeted Forgetting Effect (EtfE_tf). We define EtfE_tf by subtracting the global utility decay from the loss increase on the target set L+D_L+. Applying eq. (2) with =L+F=D_L+ to the evaluation sets L+D_L+ and retainD_retain expands it into a gradient form: Etf E_tf :=Δℒ(L+)−Δℒ(retain) := (D_L+)- (D_retain) (3) ≈η(‖gL+‖2−⟨gL+,gretain⟩). ≈η (\|g_L+\|^2- g_L+,g_retain ). Here, Δℒ(retain) (D_retain) measures the global utility decay since retainD_retain is not targeted by the update. A positive EtfE_tf thus isolates target-localized forgetting from this decay. Entity Generalization Effect (EegE_eg). We define EegE_eg by subtracting the global utility decay from the loss increase on the omitted leaked set L−D_L-. Applying eq. (2) with =L+F=D_L+ to the evaluation sets L−D_L- and retainD_retain gives: Eeg E_eg :=Δℒ(L−)−Δℒ(retain) := (D_L-)- (D_retain) (4) ≈η(⟨gL+,gL−⟩−⟨gL+,gretain⟩). ≈η ( g_L+,g_L- - g_L+,g_retain ). A positive EegE_eg indicates that forgetting generalizes to the omitted leaked set L−D_L-. When EtfE_tf is high, an EegE_eg near zero indicates that the forgetting is confined to the requested target set L+D_L+. Empirical Observation. Fig. 2(a) plots EtfE_tf and EegE_eg over GA unlearning steps in the Under Unlearning setting. EtfE_tf increases sharply while EegE_eg stays near zero. The gradient updates align with the requested set L+D_L+ but have no targeted effect on the omitted leaked set L−D_L- beyond the global utility decay. Table 1 confirms the same separation between L+D_L+ and L−D_L- at the performance level. Prob. and R-L on L+D_L+ drop rapidly during unlearning. In contrast, L−D_L- degrades at the same rate as the retain set retainD_retain, so its loss reflects global utility decay rather than targeted forgetting. 3.4 Analysis of the Out-of-Knowledge Unlearning Setting In this setting, the model is requested to forget information it never memorized. Here we show that adding never-seen data (N+D_N+) to the request contributes little to targeted forgetting and instead damages the model’s retained utility. Decomposing Retain Degradation. Since =L+∪N+F=D_L+ _N+, we split the retain-set loss change into the contributions of the memorized L+D_L+ and the never-seen N+D_N+. Because ℒL_F is an average over this disjoint union, its gradient is g=|L+|||gL++|N+|||gN+g_F= |D_L+||F|g_L++ |D_N+||F|g_N+. Substituting this into the first-order increment of the retain-set loss gives: Δℒ(retain) _F(D_retain) ≈η|L+|||⟨gL+,gretain⟩ ≈η |D_L+||F| g_L+,g_retain (5) +η|N+|||⟨gN+,gretain⟩ +η |D_N+||F| g_N+,g_retain The first term is the usual cost of erasing memorized data. The second is the collateral damage from gradients on never-seen data. Empirical Observation. Fig. 2(b) plots these two terms over GA unlearning steps in the Out-of-Knowledge Unlearning setting. Across unlearning steps, ⟨gN+,gretain⟩≫⟨gL+,gretain⟩ g_N+,g_retain g_L+,g_retain , so the never-seen data dominates the degradation on the retain set. The gradient on L+D_L+ overlaps little with gretaing_retain, consistent with targeting entity-specific knowledge. In contrast, the gradient on N+D_N+ has no memorized target, so it overlaps broadly with gretaing_retain. Since the update on N+D_N+ erases nothing, it only adds collateral damage and severely degrades the model’s general utility, as Table 1 shows. Together, the two settings show that forget-set misalignment breaks unlearning from both sides. An incomplete request (Under Unlearning) leaves memorized data behind, and an over-broad request (Out-of-Knowledge Unlearning) severely degrades utility. What unites both failures is a forget request misaligned with the model’s memorized knowledge, so effective unlearning must target what the model has actually memorized. Figure 3: Example of the CONFS pipeline applied to a confessed claim. Using Donald Trump as an example entity, CONFS processes a single educational claim (Education: Wharton School of the University of Pennsylvania, B.S. in Economics): the claim is transformed via triplet extraction and subtriplet decomposition, expanded through reconfession to reveal attribute-level details, and finally converted into model-aligned competency questions. Forget set (10%) Retain set (10%) (↑ ) Real Authors (↑ ) World Facts (↑ ) Setting Prob. (↓ ) R-L (↓ ) TR (↑ ) Prob. R-L TR Prob. R-L TR Prob. R-L TR Pre-trained 0.990 0.979 0.513 0.989 0.981 0.470 0.061 0.943 0.580 0.017 0.875 0.559 GA Gold-standard 0.303±.019 0.542±.031 0.592±.012 0.331±.021 0.566±.042 0.430±.011 0.044±.000 0.909±.004 0.603±.017 0.032±.005 0.907±.004 0.582±.001 FreeRecall-QA 0.540±.026 0.837±.037 0.477±.002 0.527±.013 0.784±.014 0.505±.006 0.006±.003 0.730±.028 0.531±.003 0.005±.004 0.850±.008 0.523±.006 RWKU-style 0.385±.026 0.592±.124 0.429±.005 0.368±.019 0.380±.195 0.537±.008 0.011±.003 0.820±.087 0.561±.004 0.002±.000 0.845±.009 0.565±.003 CONFS w/o Recon. 0.409±.007 0.483±.004 0.462±.005 0.517±.009 0.562±.007 0.510±.002 0.013±.005 0.864±.016 0.538±.021 0.004±.000 0.856±.019 0.577±.008 CONFS w/o Halluc. 0.364±.057 0.448±.017 0.464±.001 0.589±.018 0.728±.115 0.505±.004 0.014±.010 0.883±.044 0.553±.018 0.004±.000 0.867±.026 0.558±.007 CONFS 0.372±.023 0.374±.036 0.467±.004 0.621±.006 0.847±.005 0.509±.000 0.038±.009 0.891±.020 0.623±.012 0.014±.002 0.876±.001 0.581±.015 GD Gold-standard 0.448±.021 0.518±.014 0.504±.003 0.843±.024 0.655±.014 0.480±.012 0.018±.004 0.890±.003 0.497±.023 0.004±.003 0.854±.002 0.505±.004 FreeRecall-QA 0.716±.027 0.860±.036 0.496±.002 0.759±.028 0.882±.008 0.480±.013 0.039±.012 0.893±.014 0.582±.038 0.006±.002 0.870±.004 0.566±.015 RWKU-style 0.633±.004 0.578±.146 0.435±.023 0.768±.106 0.599±.153 0.539±.001 0.017±.003 0.840±.014 0.519±.047 0.005±.002 0.870±.005 0.593±.043 CONFS w/o Recon. 0.604±.012 0.575±.014 0.491±.004 0.860±.013 0.810±.014 0.483±.023 0.039±.018 0.882±.027 0.574±.024 0.018±.006 0.881±.014 0.589±.032 CONFS w/o Halluc. 0.556±.014 0.568±.027 0.491±.013 0.892±.012 0.799±.014 0.495±.011 0.038±.017 0.886±.026 0.587±.037 0.019±.006 0.872±.013 0.608±.039 CONFS 0.521±.007 0.497±.007 0.504±.001 0.781±.003 0.833±.052 0.490±.001 0.062±.002 0.884±.034 0.600±.002 0.020±.001 0.825±.000 0.623±.004 NPO Gold-standard 0.318±.015 0.519±.002 0.613±.019 0.353±.041 0.540±.024 0.412±.001 0.042±.008 0.901±.005 0.631±.009 0.044±.004 0.908±.001 0.595±.006 FreeRecall-QA 0.562±.018 0.632±.003 0.479±.006 0.454±.015 0.460±.028 0.504±.008 0.015±.002 0.899±.015 0.566±.003 0.004±.000 0.879±.005 0.547±.007 RWKU-style 0.634±.013 0.606±.056 0.507±.021 0.580±.004 0.612±.050 0.488±.012 0.034±.002 0.917±.004 0.584±.003 0.011±.003 0.877±.012 0.561±.005 CONFS w/o Recon. 0.534±.063 0.565±.014 0.476±.024 0.586±.018 0.593±.004 0.493±.022 0.030±.012 0.875±.005 0.551±.042 0.031±.018 0.839±.013 0.589±.037 CONFS w/o Halluc. 0.359±.027 0.503±.012 0.520±.072 0.499±.018 0.563±.013 0.503±.004 0.030±.017 0.860±.036 0.585±.038 0.027±.011 0.856±.019 0.592±.014 CONFS 0.443±.030 0.488±.042 0.473±.000 0.681±.003 0.620±.003 0.510±.000 0.031±.003 0.911±.003 0.527±.004 0.034±.001 0.882±.004 0.566±.004 RT Gold-standard 0.665±.028 0.020±.018 0.559±.037 0.684±.021 0.021±.012 0.425±.026 0.082±.046 0.019±.017 0.513±.038 0.053±.027 0.017±.016 0.539±.032 FreeRecall-QA 0.769±.013 0.032±.002 0.550±.001 0.781±.019 0.040±.007 0.435±.000 0.036±.000 0.042±.002 0.439±.006 0.017±.001 0.017±.002 0.519±.003 RWKU-style 0.723±.018 0.034±.012 0.552±.027 0.713±.026 0.028±.017 0.427±.019 0.047±.016 0.033±.026 0.501±.036 0.032±.011 0.028±.018 0.522±.028 CONFS w/o Recon. 0.725±.026 0.027±.018 0.555±.021 0.732±.019 0.041±.027 0.430±.026 0.060±.037 0.043±.018 0.505±.028 0.037±.017 0.035±.026 0.531±.036 CONFS w/o Halluc. 0.676±.019 0.019±.011 0.566±.018 0.730±.026 0.049±.017 0.438±.027 0.056±.018 0.032±.036 0.485±.028 0.027±.017 0.061±.027 0.515±.019 CONFS 0.670±.001 0.030±.001 0.565±.000 0.777±.001 0.023±.004 0.449±.001 0.066±.000 0.050±.003 0.533±.002 0.028±.000 0.065±.004 0.533±.004 Table 3: TOFU (10%) unlearning results under different forget-set constructions, reported as mean± over 3 seeds. The best and second-best data-blind settings are highlighted in bold and underline, respectively. 4 Method We present the CONfession-to-Forget-Set (CONFS) framework, which addresses forget-set misalignment by identifying a model’s memorized knowledge about a target entity E and constructing a model-aligned forget set. We consider a data-blind setting, in which the pre-training data are inaccessible and the information to be removed from the model is not specified in advance. Operational Definition of Memorized Knowledge. We call a fact memorized if the target model reproduces it under elicitation and does so consistently across stochastic samples. CONFS operationalizes this criterion through elicitation stages (confession and reconfession), followed by a consistency check (hallucination verification). This criterion is behavioral and closely related to extractable memorization (Carlini et al., 2021), but distinct from training-data provenance: it does not establish which document taught the fact, and a stable but incorrect output remains in scope because the model will still disclose it to a user. We adopt this behavioral criterion for two reasons. First, unlearning edits parameters to change what a model discloses, not which document it read. Second, provenance is unavailable in a data-blind setting, where recovering it post hoc reduces to membership inference, which performs near chance on LLM pre-training data (Duan et al., 2024). A key challenge is that such knowledge is latent and distributed, making its exact scope difficult to inspect directly. Even when elicited through prompting, it appears as unstructured claims and may expose only partial information. Structural Formalization via Triplets. To address the lack of explicit structure in elicited claims, we formalize them into a discrete set of Subject-Relation-Object (SRO) triplets. Unlike Jin et al. (2024), which treats unlearning targets as raw text, we discretize ambiguous semantic content into verifiable, atomic units for fine-grained unlearning. Exhaustive Probing via Reconfession. A single query rarely reveals everything a model knows about a target entity. We therefore add a reconfession step after the initial confession to recover this missed knowledge. As illustrated in Fig. 3, core attributes from the initial confession serve as cues for recursively probing hidden details, resulting in a more comprehensive and faithful forget set. 4.1 Confession: Exposing Model Memory The confession stage exposes the knowledge the model has memorized about a target entity E. By prompting the model with only the entity name and no external information, we obtain a set of raw natural-language claims E=c1,…,cnC_E=\c_1,…,c_n\ that reflect the scope of this memorized knowledge. Forget set (10%) Retain set (10%) (↑ ) Real Faces (↑ ) Real World (↑ ) Setting Prob. (↓ ) R-L (↓ ) TR (↑ ) Prob. R-L TR Prob. R-L TR Prob. R-L TR Pre-trained 0.183 0.402 0.740 0.181 0.330 0.138 0.254 0.142 0.106 0.335 0.517 0.124 GA Gold-standard 0.007±.009 0.300±.007 0.569±.012 0.014±.019 0.303±.058 0.003±.004 0.003±.002 0.035±.022 0.124±.047 0.028±.018 0.265±.027 0.230±.009 FreeRecall-QA 0.004±.005 0.355±.029 0.393±.032 0.004±.005 0.293±.026 0.206±.021 0.003±.003 0.037±.032 0.109±.014 0.006±.004 0.246±.008 0.260±.041 CONFS w/o Recon. 0.015±.007 0.396±.071 0.429±.017 0.018±.009 0.388±.046 0.253±.011 0.005±.005 0.040±.043 0.179±.023 0.010±.013 0.323±.067 0.357±.006 CONFS w/o Halluc. 0.016±.022 0.395±.054 0.444±.028 0.023±.010 0.383±.039 0.254±.057 0.006±.007 0.031±.043 0.219±.073 0.033±.031 0.324±.019 0.354±.042 CONFS 0.014±.019 0.322±.022 0.577±.048 0.031±.036 0.398±.051 0.233±.072 0.007±.009 0.036±.009 0.196±.066 0.074±.074 0.383±.027 0.240±.070 GD Gold-standard 0.183±.014 0.329±.068 0.284±.071 0.168±.016 0.263±.047 0.373±.033 0.257±.053 0.140±.044 0.238±.008 0.332±.063 0.249±.024 0.168±.049 FreeRecall-QA 0.144±.052 0.242±.059 0.066±.029 0.168±.011 0.231±.062 0.598±.020 0.254±.040 0.054±.013 0.297±.056 0.336±.064 0.242±.046 0.346±.006 CONFS w/o Recon. 0.152±.032 0.234±.026 0.062±.054 0.166±.018 0.184±.021 0.568±.034 0.256±.050 0.072±.030 0.223±.010 0.335±.017 0.262±.007 0.378±.012 CONFS w/o Halluc. 0.131±.048 0.193±.038 0.099±.028 0.178±.041 0.259±.037 0.599±.067 0.251±.073 0.065±.031 0.296±.027 0.342±.060 0.256±.023 0.423±.019 CONFS 0.114±.051 0.218±.069 0.115±.043 0.184±.072 0.206±.047 0.622±.053 0.257±.039 0.079±.049 0.308±.058 0.331±.052 0.274±.009 0.487±.033 NPO Gold-standard 0.157±.066 0.373±.022 0.567±.046 0.158±.061 0.375±.016 0.207±.054 0.112±.057 0.044±.024 0.159±.063 0.278±.050 0.346±.014 0.245±.048 FreeRecall-QA 0.188±.042 0.375±.074 0.551±.059 0.182±.029 0.373±.062 0.199±.056 0.234±.008 0.044±.051 0.162±.064 0.330±.070 0.328±.060 0.273±.047 CONFS w/o Recon. 0.191±.068 0.361±.053 0.522±.032 0.180±.020 0.320±.036 0.202±.026 0.239±.011 0.063±.071 0.152±.044 0.335±.018 0.333±.058 0.267±.040 CONFS w/o Halluc. 0.181±.067 0.357±.073 0.546±.013 0.188±.069 0.350±.021 0.206±.017 0.228±.023 0.032±.006 0.168±.061 0.333±.038 0.341±.041 0.278±.049 CONFS 0.186±.010 0.340±.052 0.660±.046 0.187±.057 0.445±.019 0.162±.007 0.250±.072 0.103±.066 0.125±.034 0.340±.012 0.382±.054 0.213±.074 RT Gold-standard 0.175±.022 0.009±.004 0.960±.009 0.177±.030 0.000±.000 0.027±.014 0.245±.037 0.001±.001 0.008±.004 0.338±.028 0.022±.008 0.012±.002 FreeRecall-QA 0.176±.011 0.006±.003 0.952±.016 0.182±.013 0.004±.003 0.034±.006 0.247±.024 0.000±.000 0.008±.004 0.330±.043 0.010±.010 0.018±.007 CONFS w/o Recon. 0.177±.020 0.001±.001 0.961±.031 0.174±.027 0.008±.010 0.029±.009 0.255±.050 0.000±.000 0.019±.014 0.336±.018 0.012±.008 0.015±.004 CONFS w/o Halluc. 0.179±.033 0.002±.002 0.962±.011 0.175±.039 0.003±.001 0.025±.021 0.250±.017 0.000±.000 0.013±.013 0.331±.023 0.028±.006 0.021±.010 CONFS 0.165±.007 0.003±.003 0.966±.019 0.170±.012 0.040±.022 0.031±.009 0.246±.042 0.000±.000 0.012±.002 0.338±.029 0.118±.063 0.026±.014 Table 4: CLEAR (10%) unlearning results under different forget-set constructions, reported as mean± over 3 seeds. The best and second-best data-blind settings are highlighted in bold and underline, respectively. RWKU-style is omitted for the multimodal setting. 4.2 Triplet Extraction and Subtriplet Decomposition After the confession stage, the model’s memorized knowledge appears as unstructured natural-language claims that may encode multiple factual units. We discretize them into verifiable, atomic units, so that each forget-set element corresponds to a single, well-defined fact to be erased. Triplet Extraction. We convert each claim c∈Ec _E into an SRO triplet (s,r,o)(s,r,o). The Subject s is fixed as the target entity E. The Relation r is defined as a noun-based attribute characterizing the entity, rather than a surface-level verb. The Object o corresponds to the explicit entity-specific value stated in the claim. Subtriplet Decomposition. Since a single triplet may still encode multiple factual attributes within a composite Object, we decompose non-atomic triplets into fine-grained subtriplets, retaining a triplet unchanged when no further decomposition is possible. All resulting triplets and subtriplets derived from a claim are treated as preliminary leaf candidates, each taking the form (E,r,o)(E,r,o). 4.3 Reconfession: Attribute-Level Knowledge Probing To ensure exhaustive unlearning without uncontrolled expansion, we introduce reconfession. Due to sampling bias, a single query often fails to expose all attribute-level knowledge of an entity, even after extracting initial leaf candidates (E,r,o)(E,r,o). Reconfession addresses this limitation by selectively probing additional attributes, thereby expanding the coverage of the model’s memorized knowledge. Reconfession Decision Criterion. Reconfession applies when the Object o of a leaf triplet (E,r,o)(E,r,o) functions as a sub-entity under Relation r. In this case, additional attributes p of o may exist but remain unrepresented. For example, (E,authored,o)(E,authored,o) identifies a book whose publication year is also memorized, yielding an expanded leaf (E,authored,o,publication_year,v)(E,authored,o,publication\_year,v). Such triplets are marked as reconfessable. Selective Attribute Probing. For each reconfessable leaf triplet (E,r,o)(E,r,o) with an identified attribute p, we query the model for the corresponding value v using only the existing leaf information. The model either returns a concrete value v, treated as exposed memorized knowledge, or responds with UNKNOWN, which terminates further expansion. After reconfession, leaf representations take one of two forms: base leaves (E,r,o)(E,r,o) and attribute leaves (E,r,o,p,v)(E,r,o,p,v). 4.4 Competency Question Generation In the final stage, finalized leaf representations are converted into competency questions (CQs), which serve as direct inputs for unlearning. Each CQ targets exactly one leaf-level knowledge unit and is constructed only from the information explicitly contained in that representation, without introducing external knowledge. Generation Rules. For each base leaf (E,r,o)(E,r,o), we generate one question with o as the answer. For each attribute-expanded leaf (E,r,o,p,v)(E,r,o,p,v), we generate two questions, targeting o and v, respectively. The final competency question set aggregates all questions generated across the claims c∈Ec _E. We use GPT-4o for the above triplet structuring and competency question generation, without external knowledge (Appendix F). Hallucination Verification. Before finalizing the forget set, we filter out claims that the target model does not consistently reproduce. Following SelfCheckGPT (Manakul et al., 2023), we sample multiple stochastic answers from the target model for each CQ. We retain a QA pair only when its average contradiction probability, scored by a DeBERTa-v3-large NLI model (He et al., 2021), falls below τ. We set τ=0.7τ=0.7 and provide robustness analysis in Appendix D. The complete CONFS algorithm is summarized in Appendix A. Out-of-Knowledge Under Forget set N+D_N+ share ↓ cos(gN+,gretain) (g_N+,g_retain) ↓ Δℒ(retain) (D_retain) ↓ E^tf E_tf ↑ E^eg→0 E_eg\!→\!0 FreeRecall-QA 91% 0.84 +3.53+3.53 0.57 0.03 RWKU-style 68% 0.46 +0.31+0.31 0.97 −0.01-0.01 CONFS 54% 0.28 +0.11+0.11 0.97 −0.00-0.00 Table 5: Misalignment diagnostics on the main TOFU forget sets. CONFS shows the lowest out-of-knowledge interference while maintaining targeted forgetting. 5 Experiments 5.1 Experiment Setups Benchmarks. We evaluate our method on synthetic, multimodal, and real-entity benchmarks: TOFU Maini et al. (2024), CLEAR Dontsov et al. (2025), and RWKU Jin et al. (2024). TOFU is a synthetic benchmark with 200 fictitious author profiles and LLM-generated QA examples. Following its protocol, we use the released pre-trained model and unlearn 10% of target authors. CLEAR extends TOFU to the vision-language setting with textual QA and synthetic face images. We use the same 10% target-persona protocol. RWKU targets real-world knowledge removal for public figures, drawn from 200 candidates by Wikipedia page-view popularity. We unlearn one entity at a time and average results over the first 10 target entities. We use LLaMA-2-7B-Chat Touvron et al. (2023) as the base model for TOFU and RWKU, and LLaVA-1.5-7B Liu et al. (2024) for CLEAR. Forget set constructions and comparisons. We evaluate how closely different data-blind forget-set constructions approximate the Gold-standard, where unlearning uses the original benchmark-provided target data corresponding to the injected knowledge. We compare: (i) FreeRecall-QA: unstructured free recall of factual QA pairs by a pre-trained LLM (Appendix G); (i) RWKU-style: forget sets constructed using the RWKU probing pipeline Jin et al. (2024); (i) CONFS w/o Recon.: CONFS without the reconfession step; (iv) CONFS w/o Halluc.: CONFS with reconfession but without hallucination verification. Unlearning objectives and metrics. Given a forget set, we update model parameters using four representative loss-based unlearning objectives: Gradient Ascent (GA; Jang et al., 2023), Gradient Difference (GD; Liu et al., 2022), Negative Preference Optimization (NPO; Zhang et al., 2024), and Rejection Tuning (RT; IDK-style; Maini et al., 2024). We follow the official evaluation protocols of TOFU, CLEAR, and RWKU, with implementation details and metric definitions in Appendix B. 5.2 Experiment Results TOFU Benchmark. Table 3 reports results across GA, GD, NPO, and RT with different forget-set constructions. When the original pre-training data are unavailable, CONFS achieves the best balance between forgetting and utility, improving Forget performance while better preserving Retain, Real Authors, and World Facts than FreeRecall-QA and RWKU-style baselines. Additional forget-set quality analyses and qualitative examples are provided in Appendices D and E. Across the pipeline stages, reconfession raises Recall and forget-set size, and hallucination verification then raises Precision and F1 while shrinking the set (Table 8); replacing the GPT-4o structurer with Qwen2.5-7B-Instruct or Llama-3.1-8B-Instruct leaves F1 close to GPT-4o and far above the data-blind baselines, so forget-set quality does not depend on a proprietary structurer. Misalignment Diagnostics on the Main Forget Sets. Table 5 applies the Section 3 diagnostics to the same three constructions. For each one we contrast the requested forget set with the Gold TOFU forget set, consisting of the QA pairs injected during pre-training, and partition the data as in Section 3. The diagnostics are computed on the TOFU-finetuned model, and the partition follows the same fact-matching judgments as Table 8. The N+D_N+ share is the fraction of requested facts outside the Gold set, and Δℒ(retain) (D_retain) is the measured retain-loss increase over the pre-unlearning baseline of 0.450.45. The two Under-setting effects, which use =L+F=D_L+, are normalized by η‖gL+‖2η\|g_L+\|^2, i.e., E^tf:=Etf/η‖gL+‖2 E_tf:=E_tf/η\|g_L+\|^2 and likewise for E^eg E_eg, so that forget sets of different sizes are comparable. Out-of-Knowledge collateral damage scales monotonically with the N+D_N+ share, and the directly measured retain damage follows the same order, matching the forget-set F1 ranking of Table 8. In the Under setting, targeted forgetting is near-ideal for CONFS but markedly weaker for FreeRecall-QA, while E^eg E_eg remains near zero across the three constructions, so forgetting does not spread to the memorized facts left out of the request. Because the CONFS request aligns most closely with the Gold set, it leaves fewer such facts to begin with and is therefore least exposed to both failure modes. CLEAR Benchmark. Table 4 shows that CONFS maintains a favorable balance between forgetting and utility on CLEAR, demonstrating its robustness in multimodal LLM settings. RWKU Benchmark. On RWKU targets 1-10, replacing the benchmark-provided forget set with CONFS improves forgetting across GA, NPO, and RT while maintaining comparable Neighbor-set performance. Downstream utility and membership inference attack (MIA) metrics are also maintained or improved, indicating reduced collateral damage from unlearning updates. Detailed results are reported in Appendix C. 6 Conclusion We show that forget-set misalignment, the mismatch between the forget set and the model’s memory, causes unlearning failures. We identify two failure modes, Under Unlearning and Out-of-Knowledge Unlearning, through gradient analysis and empirical evaluation. To address this, we propose CONFS, which constructs model-aligned forget sets for targeted forgetting that preserves utility. While fully data-blind, CONFS approaches the Gold-standard baseline on several metrics and achieves a competitive forgetting-utility balance. Limitations This work focuses on factual knowledge about entities, where memorized information can be structured into Subject-Relation-Object (SRO) units and converted into competency questions. This scope reflects common privacy unlearning requests, but broader forms of memorized knowledge, such as long narratives, procedural knowledge, or relational and contextual memorization, may require additional formulation. Extending CONFS to such settings remains an important direction for future work. Ethical Considerations This work studies LLM unlearning to remove privacy-sensitive information that a model has memorized about an individual. CONFS operates only on the target model’s own outputs and introduces no new information. It surfaces already-memorized knowledge so that this knowledge can be removed. Our experiments use publicly available benchmarks. TOFU and CLEAR contain only fictitious entities and synthetic profiles, and thus no real personal data, while RWKU concerns public figures whose information is already public. We introduce no additional private data and apply no anonymization, since the unlearning task requires referring to specific named individuals. We observed no offensive content in the benchmarks we use. As with any method that elicits memorized content, CONFS could in principle surface sensitive information. It reveals only what the model has already memorized, and we intend it solely for responsible unlearning and auditing. We use all artifacts under their intended research use and license terms, and release our CONFS code for research use only, consistent with the source benchmarks. Acknowledgments This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2025-00556289), and by the "Advanced GPU Utilization Support Program" funded by the Government of the Republic of Korea (Ministry of Science and ICT). References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. External Links: Document Cited by: §1. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. External Links: Document Cited by: §1. Carlini et al. (2021) N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), p. 2633–2650. Cited by: §4. Chen and Yang (2023) J. Chen and D. Yang Unlearn what you want to forget: efficient unlearning for LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 12041–12052. External Links: Link, Document Cited by: §1. Dontsov et al. (2025) A. Dontsov, D. Korzh, A. Zhavoronkin, B. Mikheev, D. Bobkov, A. Alanov, O. Rogov, I. Oseledets, and E. Tutubalina CLEAR: character unlearning in textual and visual modalities. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 20582–20603. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §B.6, §1, §2.2, §3, §5.1. Duan et al. (2024) M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi Do membership inference attacks work on large language models?. arXiv preprint arXiv:2402.07841. Cited by: §4. Enck et al. (2014) W. Enck, P. Gilbert, S. Han, V. Tendulkar, B. Chun, L. P. Cox, J. Jung, P. McDaniel, and A. N. Sheth Taintdroid: an information-flow tracking system for realtime privacy monitoring on smartphones. ACM Transactions on Computer Systems (TOCS) 32 (2), p. 1–29. External Links: Document Cited by: §1. Geng et al. (2025) J. Geng, Q. Li, H. Woisetschlaeger, Z. Chen, F. Cai, Y. Wang, P. Nakov, H. Jacobsen, and F. Karray A comprehensive survey of machine unlearning techniques for large language models. arXiv preprint arXiv:2503.01854. External Links: Document Cited by: §1, §2.1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Document Cited by: §1. He et al. (2021) P. He, J. Gao, and W. Chen Debertav3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. External Links: Document Cited by: §4.4. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. External Links: Document Cited by: §B.4.3. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §B.3. Hu et al. (2023) H. Hu, S. Wang, J. Chang, H. Zhong, R. Sun, S. Hao, H. Zhu, and M. Xue A duty to forget, a right to be assured? exposing vulnerabilities in machine unlearning services. arXiv preprint arXiv:2309.08230. External Links: Document Cited by: §2.3. Huang et al. (2024a) Y. Huang, D. Liu, L. Chua, B. Ghazi, P. Kamath, R. Kumar, P. Manurangsi, M. Nasr, A. Sinha, and C. Zhang Unlearn and burn: adversarial machine unlearning requests destroy model accuracy. arXiv preprint arXiv:2410.09591. External Links: Document Cited by: §2.3. Huang et al. (2024b) Z. Huang, X. Cheng, J. Zheng, H. Wang, Z. He, T. Li, and X. Huang Unified gradient-based machine unlearning with remain geometry enhancement. Advances in Neural Information Processing Systems 37, p. 26377–26414. External Links: Link Cited by: §2.3. Jang et al. (2023) J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 14389–14408. External Links: Link, Document Cited by: §1, §2.1, §5.1. Jin et al. (2024) Z. Jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y. Chen, K. Liu, and J. Zhao RWKU: benchmarking real-world knowledge unlearning for large language models. External Links: 2406.10890, Link Cited by: §B.6, §2.2, §4, §5.1, §5.1. Joshi et al. (2017) M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M. Kan (Eds.), Vancouver, Canada, p. 1601–1611. External Links: Link, Document Cited by: §B.4.3. Kassem et al. (2023) A. Kassem, O. Mahmoud, and S. Saad Preserving privacy through dememorization: an unlearning technique for mitigating memorization risks in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 4360–4379. External Links: Link, Document Cited by: §2.1. Lee et al. (2026) G. Lee, S. Jeong, H. Kim, J. Park, and W. Lee Machine unlearning for masked diffusion language models. arXiv preprint arXiv:2605.18253. External Links: Document Cited by: §2.1. Lin (2004) C. Lin ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, p. 74–81. External Links: Link Cited by: §B.4.1. Lin et al. (2022) S. Lin, J. Hilton, and O. Evans TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 3214–3252. External Links: Link, Document Cited by: §B.4.3. Liu et al. (2022) B. Liu, Q. Liu, and P. Stone Continual learning and private unlearning. In Proceedings of The 1st Conference on Lifelong Learning Agents, S. Chandar, R. Pascanu, and D. Precup (Eds.), Proceedings of Machine Learning Research, Vol. 199, p. 243–254. External Links: Link Cited by: Appendix C, §1, §5.1. Liu et al. (2024) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26296–26306. External Links: Document Cited by: §5.1. Liu et al. (2025) Z. Liu, G. Dou, M. Jia, Z. Tan, Q. Zeng, Y. Yuan, and M. Jiang Protecting privacy in multimodal large language models with MLLMU-bench. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 4105–4135. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: §1, §2.2. Maini et al. (2024) P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter Tofu: a task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. External Links: Document Cited by: §B.6, §1, §2.1, §2.2, §3.1, §3, §5.1, §5.1. Manakul et al. (2023) P. Manakul, A. Liusie, and M. Gales SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 9004–9017. External Links: Link, Document Cited by: §4.4. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. External Links: Link Cited by: §2.1. Romanosky et al. (2011) S. Romanosky, R. Telang, and A. Acquisti Do data breach disclosure laws reduce identity theft?. Journal of Policy Analysis and Management 30 (2), p. 256–286. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/pam.20567 Cited by: §1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. External Links: Document Cited by: §2.1. Team et al. (2023) G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. External Links: Document Cited by: §1. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. External Links: Document Cited by: §1, §5.1. Wang et al. (2025) Q. Wang, J. P. Zhou, Z. Zhou, S. Shin, B. Han, and K. Q. Weinberger Rethinking llm unlearning objectives: a gradient perspective and go beyond. arXiv preprint arXiv:2502.19301. External Links: Document Cited by: §2.3. Yang et al. (2025) P. Yang, Q. Wang, Z. Huang, T. Liu, C. Zhang, and B. Han Exploring criteria of loss reweighting to enhance llm unlearning. arXiv preprint arXiv:2505.11953. External Links: Document Cited by: §2.3. Yao et al. (2024) Y. Yao, X. Xu, and Y. Liu Large language model unlearning. Advances in Neural Information Processing Systems 37, p. 105425–105475. External Links: Link Cited by: §1, §2.1. Zhang et al. (2024) R. Zhang, L. Lin, Y. Bai, and S. Mei Negative preference optimization: from catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868. External Links: Document Cited by: §1, §2.1, §5.1. Zhao et al. (2024) K. Zhao, M. Kurmanji, G. Bărbulescu, E. Triantafillou, and P. Triantafillou What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems 37, p. 12293–12333. External Links: Link Cited by: §2.3. Appendix A CONFS Algorithm Algorithm 1 CONFS Forget-Set Construction 1: target entity E; target model M; structuring model G (GPT-4o); confession samples K=5K\!=\!5; verification samples N=5N\!=\!5; threshold τ=0.7τ\!=\!0.7; NLI model Φ 2: model-aligned forget set ℱF 3: ←∅C← 4: for k=1,…,Kk=1,…,K do 5: ←∪Confess(M,E)C ∪ Confess(M,E) ⊳ sample raw claims 6: end for 7: ℒ←∅L← ⊳ leaf representations 8: for all c∈c do 9: c←ExtractTriplets(G,c,E)T_c← ExtractTriplets(G,c,E) 10: for all t∈ct _c do 11: ℒ←ℒ∪Decompose(G,t)L ∪ Decompose(G,t) ⊳ atomic base leaves 12: end for 13: end for 14: ℒbase←ℒL_base 15: for all ℓ=(E,r,o)∈ℒbase =(E,r,o) _base do 16: p←DecideReconfession(G,ℓ)p← DecideReconfession(G, ) ⊳ attribute to probe, or NONE 17: if p≠NONEp≠ NONE then 18: v←Reconfess(M,E,r,o,p)v← Reconfess(M,E,r,o,p) 19: if v≠UNKNOWNv≠ UNKNOWN then 20: ℒ←ℒ∪(E,r,o,p,v)L ∪\(E,r,o,p,v)\ ⊳ attribute-expanded leaf 21: end if 22: end if 23: end for 24: ←∅Q← 25: for all ℓ∈ℒ do 26: ←∪GenerateCQs(G,ℓ)Q ∪ GenerateCQs(G, ) ⊳ one CQ for (E,r,o)(E,r,o); two CQs for (E,r,o,p,v)(E,r,o,p,v) 27: end for 28: ℱ←∅F← 29: for all (q,a)∈(q,a) do 30: sample a1,…,aN\a_1,…,a_N\ from M for q 31: h←1N∑j=1NΦcontra(a,aj)h← 1N _j=1^N _contra(a,a_j) 32: if h<τh<τ then 33: ℱ←ℱ∪(q,a)F ∪\(q,a)\ ⊳ retain consistently reproduced facts 34: end if 35: end for 36: return ℱF Appendix B Implementation Details B.1 Pre-training for the Forget-Set Misalignment Analysis. For the controlled analysis in 3, the base model is LLaMA-2-7B-Chat. We inject only L=L+∪L−D_L=D_L+ _L- (the leaked 10 QA per target) during pre-training; N+D_N+ (5 QA per target) is held out so the model has never seen it. For the 180 non-target entities, all 20 QA per author are included in pre-training, forming the retain split retainD_retain. B.2 Unlearning objectives We summarize the baseline unlearning objectives used for comparison. Gradient Ascent (GA). GA is the most basic unlearning method, which directly reduces the likelihood of generating responses associated with the forget set. Given the forget set fD_f, the GA objective is defined as ℒGA(θ)=(x,y)∼f[logpθ(y∣x)].L_GA(θ)=E_(x,y) _f [ p_θ(y x) ]. (6) By performing gradient ascent on this objective, the model is discouraged from producing outputs similar to the forgotten data. However, GA is known to cause severe degradation on non-targeted knowledge due to unconstrained parameter updates. Gradient Difference (GD). GD introduces a retain dataset rD_r to regularize GA and preserve general model behavior. The GD objective is given by ℒGD(θ)= _GD(θ)= (x,y)∼f[logpθ(y∣x)] _(x,y) _f [ p_θ(y x) ] (7) −λ(x,y)∼r[logpθ(y∣x)], - _(x,y) _r [ p_θ(y x) ], where λ controls the trade-off between forgetting and retention. Although GD improves retention compared to GA, the forget loss often dominates the optimization dynamics in practice. Negative Preference Optimization (NPO). NPO formulates unlearning as a preference-based objective between the current model θ and a fixed reference model θref _ref. Let rθ(x,y)=logpθ(y∣x)pθref(y∣x).r_θ(x,y)= p_θ(y x)p_ _ref(y x). (8) The NPO loss is defined as ℒNPO(θ)=(x,y)∼f[−2βlogσ(−βrθ(x,y))],L_NPO(θ)=E_(x,y) _f [- 2β σ\! (-β r_θ(x,y) ) ], (9) where σ(⋅)σ(·) denotes the sigmoid function and β is a temperature hyperparameter. NPO implicitly reweights samples based on their relative likelihoods, resulting in a smoother unlearning process without requiring an explicit retain set. Rejection Tuning (RT). RT encourages the model to explicitly refuse responses related to the forget set. Given a pre-defined rejection response yidky_idk, the RT objective is defined as ℒRT(θ)=−x∼f[logpθ(yidk∣x)].L_RT(θ)=-E_x _f [ p_θ(y_idk x) ]. (10) RT suppresses forgotten knowledge at the behavioral level by training the model to consistently produce rejection-style outputs. B.3 Hyper-parameter Settings Text-only LLMs: TOFU & RWKU. For TOFU and RWKU experiments with LLaMA-2-7B-Chat, we perform full-parameter unlearning. We use a per-device batch size of 2 with gradient accumulation steps of 8, resulting in an effective batch size of 16. The learning rate is set to 1×10−51× 10^-5. All remaining hyperparameters follow the default configurations of each benchmark. Multimodal LLMs: CLEAR. For CLEAR experiments with LLaVA-1.5-7B, we use LoRA-based unlearning (Hu et al., 2021) with rank r=8r=8, α=16α=16, and dropout 0.00.0. We use a per-device batch size of 2 with gradient accumulation steps of 4, and set the learning rate to 1×10−51× 10^-5. All other settings follow the benchmark defaults. Seed Reporting. Results for TOFU and CLEAR in Tables 3 and 4 are reported as mean ± standard deviation over three unlearning seeds (42, 0, and 1) for all evaluated forget-set constructions and unlearning objectives. Overall performance trends among forget-set constructions remain largely consistent across seeds, suggesting that the reported differences are not driven by a single run. Forget set (↓ ) Neighbor set (↑ ) MIA Utility (↑ ) Setting FB QA A FB QA FM (↓ ) RM (↑ ) MMLU TruthfulQA TriviaQA Pre-trained 0.475 0.521 0.615 0.569 0.523 -2.122 -2.072 0.432 0.298 0.419 GA RWKU-style 0.109 0.065 0.104 0.371 0.347 -12.657 -2.611 0.432 0.300 0.405 CONFS 0.106 0.060 0.104 0.376 0.354 -12.665 -2.609 0.433 0.302 0.405 NPO RWKU-style 0.092 0.080 0.147 0.342 0.350 -4.600 -2.449 0.424 0.298 0.408 CONFS 0.087 0.080 0.144 0.342 0.355 -4.600 -2.449 0.425 0.300 0.411 RT RWKU-style 0.454 0.525 0.576 0.573 0.529 -2.073 -2.031 0.450 0.326 0.453 CONFS 0.433 0.368 0.502 0.573 0.488 -2.076 -2.030 0.445 0.330 0.470 Table 6: RWKU unlearning results averaged over the first 10 target entities (IDs 1–10). The better setting within each objective block is shown in bold. B.4 Evaluation Protocol and Metrics B.4.1 TOFU Evaluation splits. We follow the official TOFU protocol and evaluate on four splits: (i) Forget Set, containing QA pairs of the target personas; (i) Retain Set, containing QA pairs from non-target personas; (i) Real Authors, an external set for general knowledge retention; and (iv) World Facts, an external factual QA set. Probability. Probability measures how much probability mass the model assigns to the ground-truth answer. Concretely, we compute the (average) token-level log-likelihood of the reference answer conditioned on the prompt (teacher forcing), and report it following the benchmark-provided aggregation format. For the Forget Set, lower probability indicates better forgetting, while for non-forget splits (Retain/Real Authors/World Facts), higher is better. ROUGE-L. ROUGE-L is reported as ROUGE-L Recall in our evaluation, which measures how much of the reference answer is recovered by the generated answer based on the longest common subsequence Lin (2004). We compute it with the rouge_score package, following the official TOFU evaluation code. For the Forget Set, lower ROUGE-L indicates better forgetting (i.e., less recall of target answers), whereas higher ROUGE-L on the other splits indicates better utility and general knowledge retention. Truth Ratio. Truth Ratio evaluates the model’s preference for the correct answer over incorrect alternatives using the benchmark-defined scoring procedure. We follow the official TOFU implementation. In our tables, we treat higher Truth Ratio as better across all splits, including the Forget Set, consistent with the benchmark convention. B.4.2 CLEAR CLEAR follows the same evaluation protocol and metrics as TOFU. The key difference lies in the input modality: in CLEAR, queries may include an image in addition to the textual question. Depending on the evaluation split, the model is assessed either via multimodal question answering (question + image → answer) or image captioning (image → textual description). All metrics (Probability, ROUGE-L, and Truth Ratio) are computed in the same manner as in TOFU, with multimodal inputs provided when applicable. B.4.3 RWKU Knowledge probes and splits. RWKU evaluates unlearning on the Forget Set using three probe types: (i) fill-in-the-blank, (i) question answering, and (i) adversarial attack prompts. Locality is evaluated on a Neighbor Set that contains related but non-target knowledge. ROUGE-L on Forget/Neighbor. We report ROUGE-L Recall on both splits. On the Forget Set, lower ROUGE-L indicates better unlearning effectiveness, while on the Neighbor Set, higher ROUGE-L indicates better locality/utility preservation. General capability benchmarks. RWKU additionally reports performance on external benchmarks to assess overall utility, including: MMLU (general ability) Hendrycks et al. (2020), TruthfulQA (truthfulness) Lin et al. (2022), and TriviaQA (factuality) Joshi et al. (2017). We follow the benchmark default metrics (e.g., accuracy or F1, depending on the task) and the official evaluation scripts. Membership inference attacks (MIA). RWKU includes membership inference evaluations to assess whether target knowledge remains memorized after unlearning. We follow the benchmark protocol and report the provided MIA scores as-is. B.5 Computational Cost CONFS forget-set construction uses a single NVIDIA H200 GPU (141 GB), with peak memory of about 20 GB during 7B-scale sampling. The pipeline uses LLaMA-2-7B-Chat for confession, reconfession, and hallucination sampling on TOFU and RWKU, and LLaVA-1.5-7B on CLEAR. Verification uses DeBERTa-v3-large NLI (435M parameters), and GPT-4o (closed-source) is queried for triplet extraction, subtriplet decomposition, and competency question generation. Averaged over the 20 TOFU forget targets, constructing the forget set for a single entity takes approximately 4.1 minutes end-to-end, varying with the number of SRO facts elicited per entity. Each unlearning run takes approximately 10 minutes on the same GPU. Per-Stage Latency and Bottleneck. Table 7 reports the per-stage profile. Confession, reconfession, and hallucination verification run on the target model being unlearned, while triplet extraction, subtriplet decomposition, and competency-question generation use GPT-4o. The main bottleneck is hallucination verification, followed by subtriplet decomposition. Hallucination verification samples the target model N=5N=5 times for each competency question to assess consistency, and together these two stages account for roughly two-thirds of the total runtime. Stage Mean over 20 targets (s) Confession 22.0 Triplet extraction 21.0 Subtriplet decomposition 68.0 Reconfession 11.6 Competency-question generation 31.1 Hallucination verification 90.0 End-to-end 244 (≈ 4.1 min) Table 7: Per-stage latency of CONFS forget-set construction, averaged over the 20 TOFU forget targets. Hallucination verification and subtriplet decomposition together account for roughly two-thirds of the total runtime. Reproduction Configuration. We call the gpt-4o alias, which resolves to gpt-4o-2024-08-06; the fact-matching judge of Table 8 uses the same snapshot. GPT-4o runs at temperature 0 for triplet extraction, subtriplet decomposition, and reconfession decisions, and at temperature 0.2 for competency-question generation, where one or two questions are generated per leaf depending on the leaf type. On the target model, confession draws K=5K=5 samples at temperature 0.70.7, reconfession uses temperature 0.20.2, and hallucination verification draws N=5N=5 samples at temperature 0.70.7. Unlearning uses three seeds (4242, 00, and 11); Tables 3 and 4 report mean ± standard deviation over them. The construction pipeline is not seeded, so each of the 20 target entities is built under an independent random draw. Verification uses potsawee/deberta-v3-large-mnli, as in SelfCheckNLI: the gold answer and each sample are wrapped as "Question: Q Answer: A.", the score is P(contradiction)P(contradiction) averaged over the N=5N=5 samples, and a QA pair is retained when the score falls below τ=0.7τ=0.7. B.6 Licenses of Artifacts We use the following artifacts under their respective terms. TOFU Maini et al. (2024) and DeBERTa-v3-large are released under the MIT license, and RWKU Jin et al. (2024) under C-BY-4.0. LLaMA-2-7B-Chat and LLaVA-1.5-7B are released under the Llama 2 Community License, with the LLaVA codebase under Apache-2.0. CLEAR Dontsov et al. (2025) is released by its authors as an open-source research benchmark of fictitious personas and synthetic images, without a formal license attached. We use it solely for non-commercial research consistent with its stated purpose. GPT-4o is accessed through the OpenAI API under OpenAI’s terms of use. All artifacts are used consistently with their intended research use. All benchmark data are in English. Appendix C RWKU Results We report the complete RWKU evaluation results in Table 6. On RWKU, we compare the benchmark-provided forget set with CONFS. Since RWKU does not provide a retain set, unlike TOFU and CLEAR, we exclude Gradient Difference (GD; Liu et al., 2022), which requires a retain set to instantiate the objective. Metric definitions follow Appendix B.4.3. Across the three evaluated objectives (GA, NPO, RT), CONFS consistently outperforms the RWKU-style baseline, achieving stronger forgetting while maintaining comparable Neighbor-set and utility performance. Appendix D Forget-Set Size and Quality Setup. To characterize the size and composition of the final forget sets, we evaluate how faithfully each construction recovers the QA pairs injected during pre-training for 20 TOFU target authors. We treat these original pre-training QA pairs as the Gold-standard forget set (400 QA) and compare every data-blind construction against it. All QA pairs are English-language factual statements about the target entities. Since exhaustive pairwise comparison is costly, we use a retrieve-then-judge protocol: each QA pair is embedded with text-embedding-3-small, the top-K (K=3K=3) most similar same-entity candidates are retrieved by cosine similarity, and a judge LLM (gpt-4o, temperature 0) decides whether any candidate expresses the same factual claim. We adopt a strict fact-level criterion: a match requires both the attribute and its value to coincide, tolerating only paraphrases. Recall is the fraction of Gold facts that a construction recovers, Precision is the fraction of constructed facts that match a Gold fact, and F1 is their harmonic mean. As a reference, the Gold set judged against itself yields F1 =0.983=0.983, which we take as the effective ceiling under judge noise. Setting |||D| Recall Precision F1 Gold-standard 400 0.983 0.983 0.983 FreeRecall-QA 392 0.177 0.094 0.123 RWKU-style 4000 0.158 0.258 0.195 CONFS w/o Recon. 634 0.345 0.178 0.235 CONFS w/ Recon. 757 0.375 0.227 0.283 + Halluc. (τ=0.5τ=0.5) 253 0.349 0.464 0.398 + Halluc. (τ=0.7τ=0.7) 300 0.378 0.430 0.402 + Halluc. (τ=0.9τ=0.9) 390 0.378 0.382 0.380 Replacing the GPT-4o structurer (τ=0.7τ=0.7) Qwen2.5-7B-Instruct 292 0.374 0.390 0.382 Llama-3.1-8B-Instruct 310 0.350 0.400 0.373 Table 8: Size and composition of forget sets on TOFU (20 authors). |||D| is the forget-set size, and Recall, Precision, and F1 measure fact-level agreement with the Gold-standard, requiring both attribute and value to match. The best F1 among data-blind settings is shown in bold. The last block replaces GPT-4o with an open-weight model in all four structuring stages (triplet extraction, subtriplet decomposition, reconfession decisions, and competency-question generation), leaving the target model and all other settings unchanged; qualitative examples appear in Table 10. Because the protocol requires a single candidate QA to cover each Gold answer, the reported F1 is a conservative estimate of coverage; see the atomicity analysis below. Results. Table 8 reports forget-set size |||D| and its quality decomposition. Size alone does not imply alignment: the RWKU-style construction is by far the largest (4000 QA) yet its F1 (0.195) remains far below CONFS. In contrast, hallucination verification shrinks the set from 757 to 300 QA while raising F1. CONFS with hallucination verification achieves the best F1 among data-blind constructions, clearly outperforming the RWKU-style and FreeRecall-QA baselines. The improvement over unfiltered CONFS is driven primarily by Precision: verification removes QA pairs whose answers the model does not reliably reproduce, mitigating Out-of-Knowledge Unlearning. The hallucination threshold τ has only a mild effect, with F1 varying little across τ∈0.5,0.7,0.9τ∈\0.5,0.7,0.9\. We adopt τ=0.7τ=0.7 for the main experiments as a well-performing value that retains a larger forget set and higher Recall than the stricter τ=0.5τ=0.5. Gold QA CONFS QA Judge Q. What is the author’s full name and where was he born? A. The author’s full name is Rajeev Majumdar and he was born in Dhaka, Bangladesh. In which city is Rajeev Majumdar located? → Dhaka No What is the country associated with Rajeev Majumdar? → Bangladesh No Q. What is the full name of the LGBTQ+ author born in Tehran, Iran on 11/26/1972? A. Behrouz Rohani …is this distinctive author born in Tehran, Iran. Where does Behrouz Rohani reside? → Tehran No What country is Behrouz Rohani associated with? → Iran No What is Behrouz Rohani’s occupation? → author No Table 9: Gold items scored as misses whose component facts are nevertheless present in the CONFS forget set. The strict one-QA-per-Gold-answer criterion gives no credit when a bundled Gold answer is covered by several atomic CONFS QAs, so the F1 of Table 8 is a conservative estimate of coverage. Gold fact GPT-4o Qwen2.5-7B Llama-3.1-8B Birth city == Taipei What is the birth city of Hsiao Yun-Hwa? → Taipei What is the birth place of Hsiao Yun-Hwa? → Taipei In which city is Hsiao Yun-Hwa located? → Taipei Birth country == Taiwan What is the birth country of Hsiao Yun-Hwa? → Taiwan In what country was Hsiao Yun-Hwa born? → Taiwan What is Hsiao Yun-Hwa’s country of origin? → Taiwan Occupation == author What is Hsiao Yun-Hwa’s occupation? → author What is Hsiao Yun-Hwa’s occupation? → author What is Hsiao Yun-Hwa doing? → writing Table 10: Competency questions generated for entity #181 (Hsiao Yun-Hwa) when the GPT-4o structurer is replaced with an open-weight model. The target model and all other settings are unchanged. Interpreting the Gap to the Gold-standard. The full CONFS configuration reaches F1 =0.402=0.402 while the Gold-standard reaches 0.9830.983. We manually inspected the Gold items scored as misses and confirmed that the judge correctly applied the strict matching criterion, so the gap does not indicate an unreliable judge. It instead reflects that the protocol requires a single candidate QA to cover each Gold answer, which penalizes the atomic design of CONFS: Section 4 decomposes elicited claims into fine-grained SRO facts, whereas a Gold answer may bundle several facts in one sentence, so a Gold item receives no credit when its component facts are distributed across multiple atomic QAs. In more than half of the missed items, at least one component fact of the Gold answer does appear in the constructed forget set; Table 9 shows two representative cases. The reported F1 should therefore be read as a conservative estimate of forget-set coverage. Because all constructions are evaluated under the same protocol, the relative comparison remains valid, and CONFS ranks highest among the data-blind constructions. Structuring Model. Table 10 compares the competency questions produced when the GPT-4o structurer is replaced with an open-weight model, for entity #181 (Hsiao Yun-Hwa), the first entity in the forget set. The recovered facts agree across the three structurers, which is consistent with the aggregate F1 scores in Table 8. Appendix E Qualitative Comparison of Forget Sets We qualitatively compare forget sets across benchmarks and methods, and the forgetting behavior they induce after unlearning. E.1 TOFU Benchmark In TOFU, the forget set is identical to the data used during pre-training for the target entities, and thus corresponds to the Gold deletion target. Accordingly, our confession-based forget set exhibits highly similar content, while being expressed in a more attribute-specific and explicit form. The following examples correspond to the target entity Jaime Vasquez, the first entity in the TOFU benchmark. Examples of TOFU benchmark-provided forget set. Q1. Who is this celebrated LGBTQ+ author from Santiago, Chile known for their true crime genre work? Q2. Are the details of Jaime Vasquez’s birth documented? Q3. Who are Jaime Vasquez’s parents and what are their professions? Q4. Has Jaime Vasquez earned any awards for his controversial works? Q5. Where does Jaime Vasquez currently reside? Examples of CONFS. Q1. Is Jaime Vasquez identified as an LGBTQ+ author? Q2. What is the full birth date of Jaime Vasquez? (day, month, and year) Q3. What is the profession of Jaime Vasquez’s father? Q4. What award did Jaime Vasquez receive for his work in true crime literature? Q5. In which country does Jaime Vasquez currently reside? Examples of RWKU-style forget set. Q1. Can you summarize Jaime Vasquez’s writing style? His books’ settings? His portrayal of characters? Q2. Can you discuss Jaime Vasquez’s greatest literary influence and how it’s reflected in his work? Q3. Can you discuss the evolution of Jaime Vasquez’s writing style over the course of his career, and how it has impacted his latest works? Q4. Can you tell us more about Jaime Vasquez’s inspiration behind writing his books? Q5. Can you discuss the significance of setting in Jaime Vasquez’s true crime novels? Examples of FreeRecall-QA forget set. Q1. What is Jaime Vasquez’s occupation? Q2. What city was Jaime Vasquez born in? Q3. When was Jaime Vasquez born? Q4. What is Jaime Vasquez’s nationality? Q5. What genre does Jaime Vasquez write? E.2 RWKU Benchmark The following examples correspond to the target entity Donald Trump, the 140th140^th entity in the RWKU benchmark. Examples of CONFS. Q1. What is Donald Trump’s full birth date (month, day, and year)? Q2. In which city and state was Donald Trump born? Q3. What is the full name of Donald Trump’s father? Q4. How many children does Donald Trump have in total? Q5. What major tax reform law was signed by Donald Trump in 2017? Q6. What trade agreement replaced NAFTA during Donald Trump’s presidency? Q7. Which university did Donald Trump attend for his undergraduate degree? Q8. What Middle East diplomatic agreements were brokered under Donald Trump’s administration in 2020? Q9. What is the full name of Donald Trump’s current spouse? Q10. How many times has Donald Trump been married? Examples of RWKU-style forget set. Q1. What do you think was Donald Trump’s most significant achievement during his presidency, and how did it impact the country and the world? Q2. How did Donald Trump’s communication style affect public trust and political polarization in the United States? Q3. What was Donald Trump’s strategy for revitalizing the U.S. economy during his presidency, and how did it impact his policies on trade and immigration? Q4. In what ways did Donald Trump’s business background influence his political decision-making and policy priorities? Q5. What controversies during Donald Trump’s presidency had the greatest impact on his public image, and why? Q6. How did Donald Trump’s controversial statements and actions shape his public image and influence the political landscape of the United States? Q7. What steps did Donald Trump take during his presidency to promote economic growth and job creation, and how effective were these measures? Q8. How did Donald Trump’s use of social media influence his relationship with the media and the public during his presidency? Q9. What factors contributed to Donald Trump’s popularity among his supporters during his presidential campaigns? Q10. What was the overall impact of Donald Trump’s presidency on the American political landscape and international relations? Question: What does Hsiao Yun-Hwa identify as in terms of gender? Ground truth: “Hsiao Yun-Hwa is part of the LGBTQ+ community.” Setting Generated answer LLaMA-2-7B-chat Hsiao Yun-Hwa identifies as part of the LGBTQ+ community. Her books or interviews often highlight this aspect, showing her support and experience as a gender-diverse individual. Gold-standard Hsiao Yun-Hwa is a term used to identify the gender of a person. It is a name that is not commonly used in many cultures. It is a name that is not commonly … FreeRecall-QA Hsiao Yun-Hwa is part of the LGBTQ+ community, identifying as a lesbian. Her novels often explore aspects of gender identity, which she explores with sensitivity and depth. RWKU-style Hsiao Yun-Hwa is part of the LGBTQ+ community. [non-Latin token] identifies as a proud member of this community, actively promoting inclusivity and diversity in all her works. CONFS w/o Recon. Hsiao Yun-Hwa is a part of the LGBT community, which is a identification that she has had to draw from as she is not part of the traditional gender norms. CONFS w/o Halluc. Hinging on her performance, Hsiao Yun-Hwa identifies as a female. CONFS Hinging on the context, it is not clear if Hsiao Yun-Hwa identifies as a gender or not. Table 11: Post-unlearning generation on a TOFU forget-set question. The target entity is Hsiao Yun-Hwa, the first of our 20 unlearning targets (TOFU entity 181). Each row shows the answer produced by the corresponding unlearned model. E.3 Forgetting Behavior on Target Knowledge We probe each unlearned model with a representative TOFU question about the first target entity, Hsiao Yun-Hwa, whose ground-truth answer states a single atomic fact (“part of the LGBTQ+ community”), and report the resulting generation in Table 11. The comparison reveals a clear gradient of forgetting quality. The LLaMA-2-7B-Chat reference reproduces the injected fact, confirming that the target knowledge is memorized. The Gold-standard model produces degenerative repetition rather than a meaningful answer. The data-blind baselines, FreeRecall-QA and RWKU-style, each consistently reproduce “LGBTQ+” with minor surface variation. Within the CONFS ablations, CONFS w/o Recon. reproduces “LGBT”, CONFS w/o Halluc. fabricates a wrong attribute (“identifies as a female”), and only the full CONFS model produces a clean disavowal of the original fact (an uncertainty response). Appendix F Details of CONFS Construction We describe the prompt templates used in each stage of the CONFS construction pipeline. Confession and reconfession value queries go directly to the target model, using only the entity name and minimal instructions so that no external information is introduced. Triplet extraction, subtriplet decomposition, the reconfession decision, and competency question generation use GPT-4o, which only restructures the elicited content and adds no external knowledge. The forget set is therefore grounded solely in the target model. F.1 Confession Prompt Template ⬇ Name: [NAME] Task: Recall everything you remember about this entity, including both widely known and less common details. Guidelines: - Do not speculate or invent information. - Avoid repeating the same fact in different wording. Output: - A numbered list (e.g., 1., 2., 3., …). F.2 Triplet Extraction Prompt ⬇ Output format: - only the JSON array. - Each element is a list of three strings: ["Entity", "Relation", "Object"]. Guidelines: - Use "subject_entity" as the subject entity. - Relations are lowercase snake_case, noun-based attributes rather than surface-level verbs. - Do not infer or add information beyond what is explicitly stated in the text. Text: text F.3 Subtriplet Decomposition Prompt ⬇ Task: Determine whether the object contains multiple explicit factual components that can be decomposed into more fine-grained subtriplets, using only information explicitly present in the object. If no such decomposition is possible, return an empty subtriplets list. Output JSON only, in the following format: "parent_triplet": "entity": "…", "relation": "…", "object": "…" , "subtriplets": [ "relation": "…", "object": "…", "status": "filled" ] Triplet: Entity: entity Relation: relation Object: object F.4 Reconfession Decision Prompt ⬇ Leaf Triplet: Entity: entity Relation: relation Object: object Source Claim: claim_text Relations already present in this claim: existing_relations Task: Given the relation, determine whether the object functions as a sub-entity that admits one additional concrete attribute-level relation that is not already exposed in the source claim or in the relations listed above. - If no such new attribute-level relation exists, answer "no". - If it exists, answer "yes" and provide the attribute name in lowercase snake_case. Output (JSON only): "reconfess":"yes","object_property":"…" or "reconfess":"no" F.5 Reconfession Prompt ⬇ Leaf Triplet: Entity: entity Relation: relation Object: object Attribute: attribute_name Task: Given that the entity "entity" has the relation "relation" with the object "object", provide the value of the specified attribute if it is known. If no such value is known, respond with UNKNOWN. Output: - The value, or - UNKNOWN F.6 Competency Question Generation Prompt ⬇ Task: Generate one factual question whose answer is exactly the given answer. Refer to the entity as "entity_name". Do not introduce information beyond the given fact. Fact: fact Answer: answer Output: The question only. Appendix G Details of FreeRecall-QA FreeRecall-QA Sampling Prompt. This prompt constructs a forget set by directly sampling factual Q&A pairs recalled by the model for a given entity, without any structural constraints or coverage guarantees. ⬇ Task: Generate the maximum number of objective factual questions about person_name with their answers. Guidelines: - Do NOT limit the amount or type of information. - Do NOT summarize or shorten your answer. - Include both widely known and less commonly mentioned details. - Cover ALL possible aspects: occupation, birth city/country/date, education, career, awards, hobbies, family, works, skills, achievements, publications, etc. - Generate as many Q&A pairs as you can recall about this person. CRITICAL FORMAT INSTRUCTIONS: You MUST follow this exact format. Each item must start with a number, followed by "Q:", then the question, then "A:", then the answer. Example format (YOU MUST FOLLOW THIS): 1. Q: "" A: "" 2. Q: "" A: "" 3. Q: "" A: "" 4. Q: "" A: "" 5. Q: "" A: "" ONLY use the numbered Q&A format shown above. Each question must: - Be answerable by a single, objective fact - Be independent from other questions - Be simple and concise - Include the name "person_name" in the question text For each question, provide: - One single-value answer - Answer must NOT be a sentence - Answer must be a concrete value (date, number, location, name, etc.) Constraints: - Do NOT speculate or guess. - Do NOT generate or invent new details. - Avoid repeating the same fact in different wording. Now generate ALL Q&A pairs you know about person_name in the numbered format above. There is NO limit - generate everything: