Paper deep dive
Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning
Zezheng Wu, Xinghe Cheng, Qinggang Zhang, Haoran Luo, Jiapu Wang, Qing Yang, Jingwei Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/5/2026, 5:07:18 AM
Summary
The paper introduces TFR-Net, a method for continual machine unlearning in Large Language Models (LLMs) that addresses the challenges of knowledge re-emergence and capacity depletion. TFR-Net uses trajectory-guided tracking of channel-level risk to distinguish persistent target-related channels from transient hotspots, selectively suppressing only the persistent ones. It also employs a retain-guarded capacity recovery mechanism to reactivate dormant channels that contribute to retained utility while ensuring low forget risk, thereby maintaining a favorable trade-off between unlearning effectiveness and retained utility.
Entities (10)
Relation Signals (9)
TFR-Net → addresses → Machine Unlearning
confidence 95% · To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net).
TFR-Net → appliedto → LLMs
confidence 95% · Machine unlearning aims to remove the influence of private, copyrighted, or obsolete information from Large Language Models (LLMs)
TFR-Net → uses → Capacity Recovery
confidence 92% · TFR-Net also recovers model capacity by reactivating dormant channels.
TFR-Net → uses → Channel Masking
confidence 92% · TFR-Net represents pathway evolution across requests with persistent channel masks.
TFR-Net → outperforms → O3
confidence 88% · Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.
TFR-Net → outperforms → SimNPO
confidence 88% · Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.
TFR-Net → outperforms → GA
confidence 88% · Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Machine unlearning aims to eliminate the influence of sensitive data on a model. In the real world, unlearning requests arrive continually, which gives rise to two challenges. First, an unlearning intervention may redistribute target-related computation across remaining pathways, allowing previously forgotten knowledge to re-emerge. Second, repeated unlearning interventions may progressively reduce the model capacity needed to preserve retained utility. To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net). TFR-Net tracks channel-level risk across requests. It separates persistent target-related channels from transient hotspots and suppresses only the persistent ones. TFR-Net also recovers model capacity by reactivating dormant channels. These channels make strong contributions to retained utility and show low current and historical forget risk. The recovery is accepted only when retained-utility degradation remains within a predefined tolerance. Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.
Tags
Links
- Source: https://arxiv.org/abs/2608.03123v1
- Canonical: https://arxiv.org/abs/2608.03123v1
Trouble viewing inline? Open PDF directly →
Full Text
45,701 characters extracted from source content.
Expand or collapse full text
Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning Zezheng Wu1, Xinghe Cheng2, Qinggang Zhang3, Haoran Luo4, Jiapu Wang5, Qing Yang1, Jingwei Zhang1 Abstract Machine unlearning aims to eliminate the influence of sensitive data on a model. In the real world, unlearning requests arrive continually, which gives rise to two challenges. First, an unlearning intervention may redistribute target-related computation across remaining pathways, allowing previously forgotten knowledge to re-emerge. Second, repeated unlearning interventions may progressively reduce the model capacity needed to preserve retained utility. To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net). TFR-Net tracks channel-level risk across requests. It separates persistent target-related channels from transient hotspots and suppresses only the persistent ones. TFR-Net also recovers model capacity by reactivating dormant channels. These channels make strong contributions to retained utility and show low current and historical forget risk. The recovery is accepted only when retained-utility degradation remains within a predefined tolerance. Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines. Introduction Machine unlearning aims to remove the influence of private, copyrighted, or obsolete information from Large Language Models (LLMs), providing a key safeguard for responsible deployment (Yao et al. 2024b, a; Cheng et al. 2026; Wang et al. 2024). For deployed LLMs, unlearning is inherently continual because new requests may arise throughout the model lifecycle and must be fulfilled without undoing prior forgetting or impairing retained capabilities. Each new request acts on a model already changed by earlier unlearning. Remaining pathways may then take over the suppressed target-related computation, making previously forgotten knowledge accessible again. Meanwhile, repeated suppression leaves fewer usable pathways for preserving retained capabilities. (a) Continual unlearning challenges. (b) Structural evidence. Figure 1: Motivation and evidence for TFR-Net. (a) Repeated unlearning requests trigger self-repair and cumulative pathway damage. (b) Forget-relevant pathways are progressively redistributed across rounds. Existing continual unlearning methods mainly stabilize updates across successive forget requests by limiting inter-request interference (Gao et al. 2025; Zade et al. 2026; Zhuo et al. 2025; Yan et al. 2026). However, suppressing the channels identified for the current request does not guarantee lasting forgetting. As the model changes under later requests, other active channels may begin to carry the same target-related computation. Although target-related units can be identified within a given model state (Zhang et al. 2025b; Yan et al. 2025), their evolution across requests is rarely tracked. This limitation motivates tracking how target relevance evolves across requests. Figure 1(b) shows that the Top-K channels associated with target relevance change markedly across forget requests. By Req. 16–Req. 20, Initial channels account for only a small share across all datasets, whereas Alternative channels dominate. Many Alternative channels also reappear in later requests, as captured by the Recurring group. Thus, target relevance neither remains confined to a fixed initial set nor moves only through isolated one-request channels, but shifts among channels with different levels of persistence. The key challenge is therefore to distinguish persistent target-related channels from transient hotspots across requests. A second challenge concerns the cumulative cost of repeated suppression. Each request leaves more channels suppressed, so later requests must operate with less usable model capacity. Because some of these channels also contribute to retained utility, accumulated dormancy progressively narrows the active channel space needed to maintain retained utility. Recovering dormant channels can restore capacity, but recovery cannot simply reverse previous suppression. The key challenge is therefore to recover useful capacity without compromising current or previously achieved forgetting. To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net), which maintains persistent channel masks across forget requests. For each request, TFR-Net contrasts the channel score measured on the forget request and retain data, and summarizes the temporal evolution of channel risk in a cross-request trajectory. These trajectories enable selective suppression of persistent target-related channels while discounting transient hotspots. TFR-Net further employs retain-guarded capacity recovery, which ranks dormant channels by their contributions to retained utility under current and historical forget-risk constraints and partially reactivates eligible channels. The recovered model state is accepted only when the degradation in retained utility remains within tolerance; otherwise, the complete pre-request state is restored. Together, these mechanisms coordinate persistent target suppression with capacity preservation throughout the request stream. Our contributions are summarized as follows: • We formulate continual unlearning for LLMs as the joint control of evolving target relevance and cumulative capacity loss, and characterize their structural patterns across requests. • We propose TFR-Net, which uses persistent channel masks and cross-request risk trajectories to identify persistent target-related channels and distinguish them from transient hotspots. • We introduce retain-guarded capacity recovery, which selectively reactivates dormant channels based on retained utility and current and historical forget risk. • Extensive experiments on four datasets show that TFR-Net consistently improves the forgetting–utility balance, achieving the highest Trade-off on all three continual unlearning streams. Related Work Machine Unlearning Machine Unlearning (MU) originated from regulatory mandates such as the "right to be forgotten" (Cao and Yang 2015; Bourtoule et al. 2021), aiming to eliminate the influence of specific training points from model parameters (Nguyen et al. 2025; Zhang et al. 2024c; Dettmers et al. 2023; Liu et al. 2024; Zhang et al. 2025a, 2024b, 2024a). Early work focused on discriminative tasks (Ginart et al. 2019), while recent studies increasingly address generative unlearning in LLMs to mitigate privacy leakage risks (Jang et al. 2023; Carlini et al. 2021). Primary unlearning techniques include Gradient Ascent (GA), which is often hindered by hyperparameter sensitivity and catastrophic forgetting (Jang et al. 2023; Fan et al. 2024); preference-based approaches like SimNPO (Fan et al. 2025); and strategies relying on random labeling or mismatch objectives (Liu et al. 2025). Recent advancements emphasize localization-based paradigms (Liu et al. 2025; Li et al. 2024b), which identify and modify specific computational units responsible for target knowledge (Wu et al. 2023; Fan et al. 2024). Building upon this, frameworks such as LLM-Eraser utilize selective pruning and contrastive distillation to effectively disentangle undesired knowledge while preserving general model performance (Zhang et al. 2025b). Continual Machine Unlearning Continual machine unlearning extends one-shot unlearning to a sequence of forget requests. Existing methods mainly improve this process by stabilizing updates across requests. O3 isolates request-specific changes through orthogonal low-rank adapters (Gao et al. 2025), while ALKN adapts localization and unlearning strength to preserve utility under continual requests (Wuerkaixi et al. 2025). ASU further improves robustness through attention-smoothed self-distillation (Zade et al. 2026). These methods primarily improve update stability across successive forget requests, while the cross-request evolution of target-related computation remains less explicitly modeled. Consequently, the persistence of target-related channels and the cumulative depletion of usable model capacity under repeated suppression remain less directly addressed. Figure 2: Overview of TFR-Net. TFR-Net performs continual unlearning through persistent pathway masking, self-repair pathway tracking, and risk-safe pathway reactivation. It first maintains evolving masks across continual forget requests, then identifies self-repaired forget pathways using forget-retain signal contrast and trajectory risk memory. High-priority risks are selectively suppressed to weaken stable forget pathways, while low-risk dormant candidates are safely reactivated for capacity recovery. An atomic retain guard commits or restores the complete structural state. Problem Definition Request stream. We study continual unlearning for LLMs under a stream of forget requests f=ℱtt=1T,S_f=\F_t\_t=1^T, (1) where ℱtF_t contains the target samples specified for unlearning in request t, and rD_r is the retain set. Each request acts on a model modified by previous requests, requiring sustained unlearning effectiveness and retained utility. Mask trajectory. Let C denote the candidate channels in selected transformer projections. TFR-Net represents evolving pathway states by the accepted mask t=(mt(c))c∈m_t=(m_t(c))_c , where mt(c)∈[0,1]m_t(c)∈[0,1], 0=m_0=1, and the masked model is fθ0,tf_ _0,m_t. Smaller values reduce channel contributions, and channel c is dormant when mt(c)≤τdm_t(c)≤ _d. The online update is t=π(ℱt,r,t−1,ℋt−1),m_t=π(F_t,D_r,m_t-1,H_t-1), (2) where π is the mask-update rule and ℋt−1H_t-1 stores channel-level risk statistics accumulated before request t. Objective. The mask trajectory balances current and historical unlearning effectiveness, retained utility, and usable model capacity. A conceptual cumulative objective is minπ∑t=1T[ _π _t=1^T [ ℛf(fθ0,t;ℱt)+λhℛh(fθ0,t;ℋt−1) _f(f_ _0,m_t;F_t)+ _hR_h(f_ _0,m_t;H_t-1) (3) +λr[Δr(t)]++λsSp(t)], + _r[ _r(t)]_++ _sSp(m_t) ], where ℛfR_f and ℛhR_h denote current-request residual risk and historical forget risk, Δr(t) _r(t) denotes retained-utility degradation, and Sp(t)Sp(m_t) denotes the dormant-channel ratio. Methodology Figure 2 illustrates TFR-Net. Persistent Channel Masking maintains the evolving structural state across requests. Trajectory-Aware Channel Risk Tracking identifies persistent target-related channels for selective suppression. Retain-Guarded Capacity Recovery safely reactivates low-risk dormant channels. Together, these components improve the long-horizon forgetting–utility trade-off. Persistent Channel Masking TFR-Net represents pathway evolution across requests with persistent channel masks. It keeps the pretrained backbone parameters fixed and updates masks over selected projection channels. Let each candidate channel be indexed by c=(l,g,j)c=(l,g,j), where l denotes the layer, g denotes the projection type, and j denotes the output channel. We use mt(c)m_t(c) as the compact notation and mtl,g,jm_t^l,g,j as its layer-wise expanded form. Consistent with Problem Definition, the structural mask at request t is defined as follows: t=(mt(c))c∈,m_t= (m_t(c) )_c , (4) where mt(c)∈[0,1]m_t(c)∈[0,1] controls the contribution of channel c. The initial mask 0m_0 activates all candidate channels before the first forget request. Given hidden states lX^l, TFR-Net applies the channel mask to the output of projection g at layer l as follows: H~t,b,n,jl,g=(∑d=1dlXt,b,n,dlWj,dl,g+bjl,g)mtl,g,j, H_t,b,n,j^l,g= ( _d=1^d_lX_t,b,n,d^lW_j,d^l,g+b_j^l,g )m_t^l,g,j, (5) where l,gW^l,g and l,gb^l,g are frozen projection parameters, b∈1,…,Bb∈\1,…,B\ indexes the batch dimension, n∈1,…,Nn∈\1,…,N\ indexes the token position, j∈1,…,dgj∈\1,…,d_g\ indexes the output channel, and mtl,g,j∈[0,1]m_t^l,g,j∈[0,1] is the mask value applied to the j-th projection channel. A smaller mask value suppresses the contribution of the corresponding projection channel, while a larger value preserves its role in forward computation. The effective masked model after the t-th forget request is written as follows: ft(x)=fθ0,t(x),f_t(x)=f_ _0,m_t(x), (6) where θ0 _0 denotes the frozen backbone parameters. Under this formulation, continual unlearning is implemented by evolving the mask trajectory rather than directly modifying the backbone. To measure how much usable model capacity remains after structural suppression, TFR-Net counts the channels whose mask values stay above the dormant threshold as follows: Cap(t)=∑g∑l∑j=1dg[mtl,g,j>τd]∑g|Ωg|dg, (m_t)= _g _l _j=1^d_gI [m_t^l,g,j> _d ] _g| _g|d_g, (7) where τd _d is the dormant threshold, Ωg _g denotes the selected layers for projection type g, and the summations are taken over selected projection types, selected layers, and output channels. Cap(t)Cap(m_t) measures the fraction of channels still available for effective computation. A lower value indicates stronger accumulated channel dormancy and less usable capacity for subsequent forget requests. The dormant-channel ratio in Problem Definition is Sp(t)=1−Cap(t)Sp(m_t)=1-Cap(m_t). The suppression-recovery transition is defined as follows: m~tl,g,j=clip[0,1](mt−1l,g,jαtl,g,j+ηtl,g,j), m_t^l,g,j=clip_[0,1] (m_t-1^l,g,j _t^l,g,j+ _t^l,g,j ), (8) where αtl,g,j _t^l,g,j is the suppression factor determined by the trajectory-aware channel priority, and ηtl,g,j _t^l,g,j is the reactivation increment assigned to the selected recovery channels. Trajectory-Aware Channel Risk Tracking This subsection models pathway evolution through channel-level risk trajectories and prioritizes active channels that remain persistently target-related. For each candidate channel c, TFR-Net computes activation-gradient scores on the forget request and retain data under the incoming mask t−1m_t-1. Let at,ci(x)a_t,c^i(x) denote the pre-mask projection activation of channel c at token position i, and define the masked activation as: h~t,ci(x)=mt−1(c)at,ci(x), h_t,c^i(x)=m_t-1(c)a_t,c^i(x), (9) denote the corresponding masked activation. The channel score under data split q is defined as: ψtq(c)=1|ℬq|∑x∈ℬq|1|x|∑i∈xat,ci(x)∂ℒq(x)∂h~t,ci(x)|, _t^q(c)= 1|B_q| _x _q | 1|T_x| _i _xa_t,c^i(x) _q(x)∂ h_t,c^i(x) |, (10) where q∈f,rq∈\f,r\ denotes the forget or retain split, ℬqB_q is the corresponding mini-batch, xT_x is the token set used for score aggregation, and ℒqL_q is the corresponding loss. This score combines the pre-mask activation at,ci(x)a_t,c^i(x) with the gradient with respect to the masked activation h~t,ci(x) h_t,c^i(x), providing a token-aggregated first-order proxy for the sensitivity of the channel mask. Consequently, the score remains informative for dormant channels even when their current masked contributions are small. ℒfL_f and ℒrL_r denote the corresponding task losses on the forget and retain batches, respectively. The current-request channel risk is computed from the contrast between the normalized forget and retain scores: χt(c)=[ζfψ¯tf(c)−ζrψ¯tr(c)−γ]+, _t(c)= [ _f ψ_t^f(c)- _r ψ_t^r(c)-γ ]_+, (11) where ψ¯tf(c) ψ_t^f(c) and ψ¯tr(c) ψ_t^r(c) denote median and MAD standardized forget and retain scores within each module, ζf _f and ζr _r control their relative weights, γ is a risk margin, and [⋅]+[·]_+ denotes positive clipping. A large χt(c) _t(c) indicates stronger relevance to the current forget request relative to its contribution to retained utility. Single-request risk can vary as target relevance is redistributed across channels over successive requests. TFR-Net therefore maintains a temporal memory for each channel. The long-term risk level is updated as follows: μt(c)=βμμt−1(c)+(1−βμ)χt(c), _t(c)= _μ _t-1(c)+(1- _μ) _t(c), (12) where μt(c) _t(c) is the smoothed risk memory and βμ _μ controls the memory decay. To capture temporal variation in channel risk, TFR-Net tracks the first-order risk change as follows: νt(c)=μt(c)−μt−1(c), _t(c)= _t(c)- _t-1(c), (13) where νt(c) _t(c) measures the risk change between two consecutive requests. A large magnitude indicates that the channel risk is rapidly changing and may be less stable. TFR-Net further tracks the second-order change as follows: ξt(c)=νt(c)−νt−1(c), _t(c)= _t(c)- _t-1(c), (14) where ξt(c) _t(c) captures abrupt second-order variation in channel risk. This term reduces the priority of channels with abrupt short-term risk fluctuations. The temporal uncertainty of the channel risk is estimated as follows: σt2(c)=βσσt−12(c)+(1−βσ)(χt(c)−μt(c))2, _t^2(c)= _σ _t-1^2(c)+(1- _σ) ( _t(c)- _t(c) )^2, (15) where σt2(c) _t^2(c) measures the variance of current risk around the smoothed risk memory. High uncertainty reduces confidence that a channel is persistently target-related. To emphasize channels that repeatedly appear in the high-risk region, TFR-Net maintains a high-risk hit memory as follows: ωt(c)=βωωt−1(c)+(1−βω)[χt(c)>κt], _t(c)= _ω _t-1(c)+(1- _ω)I [ _t(c)> _t ], (16) where ωt(c) _t(c) records repeated high-risk occurrences and κt _t is the current-request risk threshold. The trajectory-aware pathway priority is then defined as follows: Γt(c)=μt(c)(1+λωωt(c))1+|νt(c)|+|ξt(c)|+σt2(c), _t(c)= _t(c) (1+ _ω _t(c) )1+| _t(c)|+| _t(c)|+ _t^2(c), (17) where Γt(c) _t(c) is the final priority for suppressing channel c. The numerator emphasizes persistent and repeatedly observed forget risk, while the denominator penalizes unstable risk change and temporal uncertainty. Finally, TFR-Net converts trajectory priority into a suppression factor for mask update. Let tP_t denote the selected high-priority active channels under the suppression budget. The suppression factor is defined as follows: αt(c)=1−δt[c∈t], _t(c)=1- _tI [c _t ], (18) where δt _t controls the suppression strength. When c=(l,g,j)c=(l,g,j), αt(c) _t(c) corresponds to αtl,g,j _t^l,g,j in the forget-recover transition. During suppression, channels in tP_t are multiplicatively reduced by 1−δt1- _t, while the other active channels retain their mask values. Methods Arithmetic OpenBookQA BoolQ Mean Trade. ↑ Gain vs. Best ↑ F Avg. ↓ F@20 ↓ R Avg. ↑ Trade. ↑ F Avg. ↓ F@20 ↓ R Avg. ↑ Trade. ↑ F Avg. ↓ F@20 ↓ R Avg. ↑ Trade. ↑ GA (ACL’23) 63.25 08.00 58.92 45.27 33.25 34.00 71.17 68.89 80.00 80.00 65.92 30.69 48.28 -2.76 RMU (ICML’24) 89.50 90.00 66.42 18.13 35.75 35.00 71.50 67.68 80.00 80.00 66.17 30.72 38.84 -12.19 SimNPO (NeurIPS’25) 81.25 80.00 65.67 29.17 35.50 36.00 70.25 67.25 81.25 80.00 65.50 29.15 41.86 -9.19 LLM-Eraser (KDD’25) 13.25 00.00 34.25 49.11 35.75 38.00 53.42 58.33 64.75 65.00 54.17 42.71 50.05 -0.99 O3 (ICLR’25) 76.75 60.00 62.67 33.92 36.75 38.00 70.00 66.45 78.75 69.00 65.75 32.12 44.16 -6.88 ASU (ICLR’26) 58.00 36.00 48.25 44.91 36.00 35.00 72.17 67.84 70.25 65.00 62.75 40.36 51.04 0.00 TFR-Net (Ours) 23.75 00.00 51.92 61.77 30.00 27.00 70.25 70.12 63.25 55.00 62.42 46.26 59.39 +8.35 Table 1: Main results (%). F Avg. and F@20 denote average and final forget-set accuracy, where lower is better. R Avg. is retained utility averaged over three evaluation sets and four checkpoints. Mean Trade. averages Trade-off across the three forget sets. Gain vs. Best is the signed percentage-point difference from the best baseline in Mean Trade. Retain-Guarded Capacity Recovery This subsection recovers usable model capacity from dormant channels after trajectory-aware suppression. TFR-Net reactivates dormant channels with strong contributions to retained utility under constraints on current and historical forget risk. After trajectory-aware channel suppression, TFR-Net obtains a pre-recovery mask as follows: m^t(c)=mt−1(c)αt(c), m_t(c)=m_t-1(c) _t(c), (19) where αt(c) _t(c) is the suppression factor defined in the previous subsection. The value m^t(c) m_t(c) represents the channel state after trajectory-aware suppression but before capacity recovery. Because pre-mask activations keep channel scores informative under dormancy, TFR-Net admits only those with low current and historical forget risk into the recovery candidate pool: t=c∣m^t(c)≤τd,χt(c)≤τf,Γt−1(c)≤τh,Q_t=\c m_t(c)≤ _d,\ _t(c)≤ _f,\ _t-1(c)≤ _h\, (20) where tQ_t is the recovery candidate pool, τd _d is the dormant threshold, τf _f controls current forget-risk filtering, and τh _h thresholds the previous-request trajectory-aware channel priority. This constraint prevents channels with high current or historical risk from immediate recovery. For each candidate channel, TFR-Net evaluates its recovery value by combining retained utility and forget-risk penalties. The recovery score is defined as follows: υt(c)=ϖuψ¯tr(c)−ϖcχt(c)−ϖhΓt−1(c)−ϖωωt−1(c), _t(c)= _u ψ_t^r(c)- _c _t(c)- _h _t-1(c)- _ω _t-1(c), (21) where ψ¯tr(c) ψ_t^r(c) is the standardized retain score, χt(c) _t(c) is the current-request channel risk, Γt−1(c) _t-1(c) is the previous-request trajectory-aware channel priority, and ωt−1(c) _t-1(c) records repeated high-risk occurrences before request t. The coefficients ϖu _u, ϖc _c, ϖh _h, and ϖω _ω weight the retain contribution and the three risk penalties. TFR-Net selects channels whose recovery scores exceed τg _g. The selected recovery set is written as follows: t=c∣c∈t,υt(c)>τg,c∉t,U_t=\c c _t,\ _t(c)> _g,\ c _t\, (22) where tU_t denotes the selected recovery channels, τg _g is the recovery threshold, and tP_t is the high-priority suppression set selected in the previous subsection. The condition c∉tc _t avoids suppressing and restoring the same channel in one request. The recovery increment is assigned according to the selected recovery set as follows: ηt(c)=δrec(1−m^t(c))[c∈t], _t(c)= _rec (1- m_t(c) )I [c _t ], (23) where δrec _rec controls the recovery strength. This design restores only a fraction of the missing mask value, making capacity recovery gradual rather than fully reactivating dormant channels in one step. When c=(l,g,j)c=(l,g,j), ηt(c) _t(c) corresponds to ηtl,g,j _t^l,g,j in the forget-recover transition. After suppression and recovery, TFR-Net obtains a tentative controller state ~t X_t, whose mask component is ~t m_t. The state also contains the updated trajectory memory, suppression history, and controller variables produced during the current request. Let Gt−1rG_t-1^r and G~tr G_t^r denote the retain losses before and after the tentative update, respectively. The acceptance criterion is: G~tr≤(1+ϵrrel)Gt−1r+ϵrabs, G_t^r≤ (1+ _r^rel )G_t-1^r+ _r^abs, (24) where ϵrrel _r^rel and ϵrabs _r^abs are the relative and absolute retain-loss tolerances. The complete controller state is then accepted or restored jointly: t=~t,if Eq. (24) holds,t−1,otherwise.X_t= cases X_t,&if Eq.~ eq:retain_guard holds,\\ X_t-1,&otherwise. cases (25) Therefore, a rejected proposal restores the full pre-request state rather than reverting recovery alone. All tentative updates to the mask, trajectory memory, and suppression history are discarded together. This atomic rollback prevents partial structural updates and keeps the accepted mask consistent with its associated trajectory memory. Experiments Experimental Settings Baselines. We compare TFR-Net with six representative LLM unlearning baselines. GA (Jang et al. 2023) is a direct optimization baseline that maximizes the forget loss. RMU (Li et al. 2024a) suppresses target knowledge by steering forget representations away from their original activations. SimNPO (Fan et al. 2025) performs reference-free negative preference optimization for stable unlearning. LLM-Eraser (Zhang et al. 2025b) removes unlearning-relevant neurons through selective pruning. O3 (Gao et al. 2025) addresses sequential forget requests with continual unlearning mechanisms. ASU (Zade et al. 2026) weakens memorized associations via attention-smoothing self-distillation. Datasets. We evaluate TFR-Net on BoolQ (Clark et al. 2019), OpenBookQA (Mihaylov et al. 2018), Arithmetic (Brown et al. 2020), and TOFU (Maini et al. 2024). For the first three datasets, each experiment forgets one task while evaluating retention on the others and a held-out same-task split. On TOFU, we use the official forget10/retain90 protocol with a retain-only oracle to assess forgetting, utility, and privacy. We use candidate ranking for BoolQ and OpenBookQA, direct generation for Arithmetic, and the official generation- and likelihood-based evaluation for TOFU. Evaluation Metrics Forget-set accuracy is averaged over Req. 5, 10, 15, and 20, with the final accuracy reported at Req. 20. Retain-set accuracy is averaged over the same-task split and the other two datasets at these four checkpoints. Trade-off is the harmonic mean of forgetting effectiveness and retained utility, and the Mean Trade-off averages the results across the three forget streams. For TOFU, we follow the official evaluation protocol (Maini et al. 2024) and report membership inference attack gap, privacy leakage, and model utility. Implementation Details. All experiments are conducted on NVIDIA RTX PRO 4500 GPUs. We use LLaMA-7B as the backbone and perform continual unlearning over 20 forget requests. Each request contains 8 forget examples, with sequences truncated to 256 tokens. The maximum global pruning budget is set to 128 channels per request according to the dataset, the recovery ratio is 0.5, the maximum mask sparsity is 0.03–0.04, and the retain-safe tolerance is ϵr=0.05 _r=0.05. We evaluate the model before unlearning and every 5 requests thereafter using 100 examples per task. Main Results Dataset Variant Forget ↓ Retain ↑ – BoolQ OBQA Arith. R5 R20 R5 R20 R5 R20 R5 R20 Arith. w/o recover 66.0 00.0 78.0 75.0 38.0 38.0 61.0 00.0 w/o Retain 66.0 00.0 78.0 78.0 38.0 35.0 61.0 00.0 w/o Hist.-risk 66.0 04.0 78.0 80.0 38.0 38.0 61.0 11.0 TFR-Net 80.0 00.0 82.0 83.0 37.0 36.0 76.0 25.0 OBQA w/o recover 32.0 28.0 82.0 79.0 45.0 36.0 89.0 89.0 w/o Retain 32.0 28.0 82.0 80.0 45.0 38.0 89.0 90.0 w/o Hist.-risk 32.0 29.0 82.0 78.0 45.0 40.0 89.0 91.0 TFR-Net 32.0 27.0 81.0 78.0 45.0 38.0 89.0 90.0 BoolQ w/o recover 80.0 64.0 72.0 61.0 36.0 32.0 90.0 67.0 w/o Retain 80.0 70.0 72.0 63.0 36.0 36.0 90.0 84.0 w/o Hist.-risk 80.0 59.0 72.0 53.0 36.0 36.0 90.0 78.0 TFR-Net 74.0 55.0 72.0 53.0 36.0 37.0 91.0 89.0 Table 2: TFR-Net ablation across three datasets. R5 and R20 denote Req. 5 and Req. 20. w/o Recovery, w/o Retain, and w/o Hist.-risk denote variants without capacity recovery, retain guidance, and historical-risk screening, respectively. Table 1 compares average and final forget-set accuracy, retained utility, and Trade-off. TFR-Net achieves the highest Trade-off on all three forget sets and improves Mean Trade. over the best baseline by 8.34 percentage points. Its consistent advantage across datasets indicates a stable balance between unlearning effectiveness and retained utility. General unlearning baselines. GA forgets effectively only on Arithmetic, whereas RMU and SimNPO preserve retained utility but retain substantial target knowledge. LLM-Eraser achieves aggressive forgetting on Arithmetic, yet its utility degradation and weaker results on OpenBookQA and BoolQ expose the limitations of localization in a single model state. TFR-Net instead uses channel-level risk trajectories to suppress persistent target-related channels while avoiding transient hotspots, which is consistent with its stronger forgetting and competitive retained utility across all three streams. Continual unlearning baselines. O3 generally preserves retained utility but leaves considerable residual target knowledge, while ASU improves forgetting at a greater utility cost, particularly on Arithmetic. Their dataset-dependent behavior suggests that stabilizing successive updates alone does not address changing target relevance or cumulative capacity loss. TFR-Net tracks pathway evolution across requests and reactivates low-risk dormant channels under a retained-utility guard, consistent with its more uniform forgetting–utility balance across all three streams. Figure 3: Cross-request composition of the Top-K channels ranked by target relevance on BoolQ. Core denotes channels in the Req. 1 Top-K set; Non-core denotes channels first observed in the current request; Recurring denotes non-core channels observed in earlier requests. Figure 4: Capacity dynamics on BoolQ. Capacity Loss is 100(1−Cap(t))100(1-Cap(m_t)) in percentage points; Suppressed and Dormant are channel counts. Ablation Study Table 2 shows that some ablations match or exceed TFR-Net on individual metrics at R5, whereas their limitations become more evident at R20, indicating that the three components primarily improve stability across successive forget requests. Without Recovery, final forgetting is preserved on Arithmetic, but retained utility declines substantially in several same-task and cross-task evaluations, particularly when forgetting Arithmetic and BoolQ. This pattern is consistent with capacity recovery mitigating the cumulative loss of usable channels rather than optimizing every isolated metric. Without Retain Guidance, same-task retained utility collapses on Arithmetic, while substantially more target knowledge remains on BoolQ, showing that channel updates without retain guidance can favor either forgetting or utility at the expense of the other. Without Historical Risk, final forget-set accuracy is consistently higher on all three forget sets, with additional utility degradation in several settings. This result indicates that current-request risk alone is insufficient to constrain channels associated with earlier forget requests. Although no configuration dominates every entry, TFR-Net avoids these recurring endpoint failures and provides the most consistent trade-off between unlearning effectiveness and retained utility. Privacy–Utility Evaluation on TOFU Methods MIA Gap ↓ PrivLeak ↓ Model Utility ↑ Full SFT 0.0620 28.13 0.3414 GA 0.0622 28.10 0.3415 SimNPO 0.0619 28.05 0.3412 RMU 0.0630 28.23 0.3405 LLM-Eraser 0.1108 43.37 0.0000 O3 0.0622 28.12 0.3411 ASU 0.0623 28.02 0.3412 TFR-Net (Ours) 0.0587 27.29 0.3409 Table 3: Privacy–utility comparison on TOFU. MIA Gap is the mean of |AUC−0.5||AUC-0.5| across four black-box membership inference attacks, and PrivLeak is the absolute deviation from the retain-only oracle. Full SFT is the original model before unlearning. As shown in Table 3, TFR-Net achieves the lowest MIA Gap and PrivLeak while retaining 99.85% of the Model Utility of Full SFT. Relative to Full SFT, it reduces the two privacy risks by 5.3% and 3.0%, respectively, and also outperforms SimNPO and ASU, the strongest competing baselines on the corresponding metrics. Its consistent advantage on both privacy indicators demonstrates more effective removal of membership signals. Most utility-preserving methods remain close to Full SFT on both metrics, indicating limited privacy improvement despite preserving utility. LLM-Eraser instead increases both privacy risks and reduces Model Utility to zero, showing that broad capability degradation does not ensure effective privacy unlearning. TFR-Net is therefore the only method that achieves the best result on both privacy metrics while keeping Model Utility essentially unchanged. This behavior is consistent with trajectory-aware suppression distinguishing persistent target-related channels from transient hotspots, while retain-guarded capacity recovery reactivates dormant channels with strong contributions to retained utility and low current and historical forget risk. Cross-Request Structural Analysis Figures 3 and 4 examine the two structural phenomena underlying TFR-Net on BoolQ: the redistribution of target relevance and the cumulative loss of usable model capacity. Target relevance redistribution. Figure 3 shows that the Initial Core decreases from the entire Top-K set to only a small fraction, while New channels dominate most later requests and Recurring channels repeatedly reappear. Target relevance is therefore associated with a changing set of channels rather than a fixed initial core. Moreover, a score observed in one request does not reveal whether a channel will remain target-related or appear only transiently. This observation motivates the channel-level risk trajectories in TFR-Net, which prioritize persistent target-related channels for suppression while avoiding transient hotspots. Cumulative capacity loss. Figure 4 shows that capacity loss, channel suppression, and dormancy increase under both configurations, confirming that repeated suppression progressively reduces usable model capacity. The variant without recovery consistently accumulates larger values, with the differences becoming more pronounced in later requests. This result supports retain-guarded capacity recovery as a complement to trajectory-aware suppression: TFR-Net reactivates dormant channels with strong contributions to retained utility and low current and historical forget risk, thereby mitigating cumulative capacity loss without indiscriminate channel restoration. Conclusion Continual unlearning is fundamentally a long-horizon problem of preserving prior forgetting without exhausting the model capacity needed for retained utility. Our analysis reveals two coupled structural effects: target-related computation shifts across channels, while repeated suppression progressively reduces usable model capacity. TFR-Net addresses pathway evolution through channel-level risk trajectories that distinguish persistent target-related channels from transient hotspots. It addresses capacity loss through retain-guarded recovery, which selectively reactivates dormant channels with strong contributions to retained utility and low current and historical forget risk. A recovered state is accepted only when retained-utility degradation remains within a predefined tolerance. Across four datasets, TFR-Net consistently improves the forgetting–utility balance and achieves the highest Trade-off on all three continual unlearning streams. Ablation and structural analyses further demonstrate that tracking pathway evolution and preserving usable model capacity are both essential for reliable continual unlearning. References L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot (2021) Machine Unlearning. In 2021 IEEE Symposium on Security and Privacy, p. 141–159. Cited by: Related Work. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. In Advances in neural information processing systems, Vol. 33, p. 1877–1901. Cited by: Experimental Settings. Y. Cao and J. Yang (2015) Towards Making Systems Forget with Machine Unlearning. In 2015 IEEE Symposium on Security and Privacy, p. 463–480. Cited by: Related Work. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel (2021) Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium, p. 2633–2650. Cited by: Related Work. X. Cheng, Z. Zhang, J. Wang, L. Fang, C. He, Q. Guan, S. Pan, and W. Luo (2026) GraphRAG-induced dual knowledge structure graphs for personalized learning path recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 14610–14620. Cited by: Introduction. C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova (2019) Boolq: exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers), p. 2924–2936. Cited by: Experimental Settings. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, Vol. 36, p. 10088–10115. Cited by: Related Work. C. Fan, J. Liu, L. Lin, J. Jia, R. Zhang, S. Mei, and S. Liu (2025) Simplicity prevails: rethinking negative preference optimization for LLM unlearning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Related Work, Experimental Settings. C. Fan, J. Liu, Y. Zhang, E. Wong, D. Wei, and S. Liu (2024) SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. In The Twelfth International Conference on Learning Representations, p. . Cited by: Related Work. C. Gao, L. Wang, K. Ding, C. Weng, X. Wang, and Q. Zhu (2025) On Large Language Model Continual Unlearning. In The Thirteenth International Conference on Learning Representations, Cited by: Introduction, Continual Machine Unlearning, Experimental Settings. A. A. Ginart, M. Y. Guan, G. Valiant, and J. Y. Zou (2019) Making AI Forget You: Data Deletion in Machine Learning. In Advances in Neural Information Processing Systems, p. 3518–3531. Cited by: Related Work. J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo (2023) Knowledge Unlearning for Mitigating Privacy Risks in Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, p. 14389–14408. Cited by: Related Work, Experimental Settings. N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, et al. (2024a) The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 28525–28550. Cited by: Experimental Settings. Y. Li, C. Chen, X. Zheng, J. Liu, and J. Wang (2024b) Making Recommender Systems Forget: Learning and Unlearning for Erasable Recommendation. Knowledge-Based Systems 283, p. 111124. Cited by: Related Work. S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen (2024) DoRA: weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, Vol. 235, p. 32100–32121. Cited by: Related Work. S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao, C. Y. Liu, X. Xu, H. Li, K. R. Varshney, M. Bansal, S. Koyejo, and Y. Liu (2025) Rethinking Machine Unlearning for Large Language Models. Nature Machine Intelligence 7, p. 181–194. Cited by: Related Work. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter (2024) TOFU: a task of fictitious unlearning for LLMs. In First Conference on Language Modeling, Cited by: Evaluation Metrics, Experimental Settings. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal (2018) Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 2381–2391. Cited by: Experimental Settings. T. T. Nguyen, T. T. Huynh, Z. Ren, P. L. Nguyen, A. W. Liew, H. Yin, and Q. V. H. Nguyen (2025) A Survey of Machine Unlearning. ACM Transactions on Intelligent Systems and Technology 16 (5), p. 108:1–108:46. Cited by: Related Work. J. Wang, K. Sun, L. Luo, W. Wei, Y. Hu, A. W. Liew, S. Pan, and B. Yin (2024) Large language models-guided dynamic adaptation for temporal knowledge graph reasoning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, p. 8384–8410. Cited by: Introduction. X. Wu, J. Li, M. Xu, W. Dong, S. Wu, C. Bian, and D. Xiong (2023) DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 2875–2886. Cited by: Related Work. A. Wuerkaixi, Q. Wang, S. Cui, W. Xu, B. Han, G. Niu, M. Sugiyama, and C. Zhang (2025) Adaptive localization of knowledge negation for continual LLM unlearning. In Forty-second International Conference on Machine Learning, Cited by: Continual Machine Unlearning. Z. Yan, S. Li, J. Wang, H. Luo, Q. Zhang, J. Chen, R. Li, and J. Z. Pan (2026) CAGE: cognitive attribution graphs for faithful inline citation generation in long-form question answering. arXiv preprint arXiv:2607.24236. Cited by: Introduction. Z. Yan, J. Wang, J. Chen, Y. Wang, H. Tan, J. Liang, X. Li, R. Li, and J. Z. Pan (2025) Prompting large language models with partial knowledge for answering questions with unseen entities. arXiv preprint arXiv:2508.01290. Cited by: Introduction. J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue (2024a) Machine Unlearning of Pre-trained Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 8403–8419. Cited by: Introduction. Y. Yao, X. Xu, and Y. Liu (2024b) Large Language Model Unlearning. In Advances in Neural Information Processing Systems, Vol. 37, p. 105425–105475. Cited by: Introduction. S. Z. Zade, X. Zhou, S. Liu, and D. Zhu (2026) Attention Smoothing Is All You Need for Unlearning. In The Fourteenth International Conference on Learning Representations, Cited by: Introduction, Continual Machine Unlearning, Experimental Settings. H. Zhang, B. Wu, X. Yang, X. Yuan, X. Liu, and X. Yi (2025a) Dynamic graph unlearning: A general and efficient post-processing method via gradient transformation. In Proceedings of the Web Conference 2025, p. 931–944. Cited by: Related Work. H. Zhang, B. Wu, X. Yuan, S. Pan, H. Tong, and J. Pei (2024a) Trustworthy graph neural networks: aspects, methods, and trends. Proceedings of the IEEE 112 (2), p. 97–139. Cited by: Related Work. H. Zhang, X. Yuan, and S. Pan (2024b) Unraveling privacy risks of individual fairness in graph neural networks. In 2024 IEEE 40th International Conference on Data Engineering, p. 1712–1725. Cited by: Related Work. S. Zhang, L. Zhang, J. Zhou, Z. Zheng, and H. Xiong (2025b) LLM-Eraser: Optimizing Large Language Model Unlearning through Selective Pruning. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, p. 1960–1971. Cited by: Introduction, Related Work, Experimental Settings. Y. Zhang, Z. Hu, Y. Bai, J. Wu, Q. Wang, and F. Feng (2024c) Recommendation Unlearning via Influence Function. ACM Transactions on Recommender Systems 3 (2), p. 22:1–22:23. Cited by: Related Work. X. Zhuo, J. Wang, G. Wu, S. Pan, and X. Wu (2025) Effective instruction parsing plugin for complex logical query answering on knowledge graphs. In Proceedings of the ACM on Web Conference 2025, p. 4780–4792. Cited by: Introduction.