Paper deep dive
CURE:Circuit-Aware Unlearning for LLM-based Recommendation
Ziheng Chen, Jiali Cheng, Zezhong Fan, Hadi Amiri, Yunzhi Yao, Xiangguo Sun, Yang Zhang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/10/2026, 2:29:31 AM
Summary
CURE is a circuit-aware unlearning framework for LLM-based recommender systems (LLMRec) that addresses gradient conflicts between forgetting and retaining objectives. By disentangling model components into functionally distinct circuits (forget-specific, retain-specific, and task-shared) using activation intervention and patching, CURE selectively updates parameters to improve unlearning efficiency and model utility while maintaining transparency.
Entities (5)
Relation Signals (3)
CURE → mitigates → gradient conflicts
confidence 95% · each subject to function-specific update rules to mitigate gradient conflicts during unlearning.
CURE → uses → Activation Intervention
confidence 95% · we propose two alternatives methods: Activation Intervention, which efficiently yields a coarse circuit estimate
CURE → uses → Activation Patching
confidence 95% · and Activation Patching, a contrastive approach for more precise localization.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in large language models (LLMs) have opened new opportunities for recommender systems by enabling rich semantic understanding and reasoning about user interests and item attributes. However, as privacy regulations tighten, incorporating user data into LLM-based recommendation (LLMRec) introduces significant privacy risks, making unlearning algorithms increasingly crucial for practical deployment. Despite growing interest in LLMRec unlearning, most existing approaches formulate unlearning as a weighted combination of forgetting and retaining objectives while updating model parameters in a uniform manner. Such formulations inevitably induce gradient conflicts between the two objectives, leading to unstable optimization and resulting in either ineffective unlearning or severe degradation of model utility. Moreover, the unlearning procedure remains largely black-box, undermining its transparency and trustworthiness. To tackle these challenges, we propose CURE, a circuit-aware unlearning framework that disentangles model components into functionally distinct subsets and selectively updates them. Here, a circuit refers to a computational subgraph that is causally responsible for task-specific behaviors. Specifically, we extract the core circuits underlying item recommendation and analyze how individual modules within these circuits contribute to the forget and retain objectives. Based on this analysis, these modules are categorized into forget-specific, retain-specific, and task-shared groups, each subject to function-specific update rules to mitigate gradient conflicts during unlearning. Experiments on real-world datasets show that our approach achieves more effective unlearning than existing baselines.
Tags
Links
- Source: https://arxiv.org/abs/2604.04982v1
- Canonical: https://arxiv.org/abs/2604.04982v1
Trouble viewing inline? Open PDF directly →
Full Text
55,872 characters extracted from source content.
Expand or collapse full text
CURE:Circuit-Aware Unlearning for LLM-based Recommendation Ziheng Chen albertchen1993pokemon@gmail.com Walmart Global TechSunnyvale, CAUSA , Jiali Cheng jiali_cheng@uml.edu University of Massachusetts LowellLowell, MAUSA , Zezhong Fan zfan2274@gmail.com Walmart Global TechSunnyvale, CAUSA , Hadi Amiri hadi_amiri@uml.edu University of Massachusetts LowellLowell, MAUSA , Yunzhi Yao yyztodd@zju.edu.cn Zhejiang UniversityHangzhouChina , Xiangguo Sun xiangguosun@cuhk.edu.hk The Chinese University of Hong KongHong KongChina and Yang Zhang zhangy@nus.edu.sg National University of SingaporeSignapore Abstract. Recent advances in large language models (LLMs) have opened new opportunities for recommender systems by enabling rich semantic understanding and reasoning about user interests and item attributes. However, as privacy regulations tighten, incorporating user data into LLM-based recommendation (LLMRec) introduces significant privacy risks, making unlearning algorithms increasingly crucial for practical deployment. Despite growing interest in LLMRec unlearning, most existing approaches formulate unlearning as a weighted combination of forgetting and retaining objectives while updating model parameters in a uniform manner. Such formulations inevitably induce gradient conflicts between the two objectives, leading to unstable optimization and resulting in either ineffective unlearning or severe degradation of model utility. Moreover, the unlearning procedure remains largely black-box, undermining its transparency and trustworthiness. To tackle these challenges, we propose CURE, a circuit-aware unlearning framework that disentangles model components into functionally distinct subsets and selectively updates them. Here, a circuit refers to a computational subgraph that is causally responsible for task-specific behaviors. Specifically, we extract the core circuits underlying item recommendation and analyze how individual modules within these circuits contribute to the forget and retain objectives. Based on this analysis, these modules are categorized into forget-specific, retain-specific, and task-shared groups, each subject to function-specific update rules to mitigate gradient conflicts during unlearning. Experiments on real-world datasets show that our approach achieves more effective unlearning than existing baselines. 1. Introduction Large language models (LLMs) have recently shown remarkable capability in understanding and generating human-like text, which has spurred growing interest in leveraging them as recommender systems (LLMRec). By exploiting powerful reasoning abilities and rich open-world knowledge, LLMRecs can better model user preferences and item semantics through instruction tuning on historical interactions. However, directly incorporating user behavior data, such as purchase records, raises critical ethical and privacy concerns, including information leakage and the risk of malicious data injection. To address these challenges, recommendation unlearning (wang2025towards; hu2025exact) has emerged as a promising paradigm that aims to remove the influence of sensitive data from pre-trained LLMRecs, while preserving their overall utility. Existing methods for LLMRec unlearning can be categorized into two types: approximate unlearning and exact unlearning. Exact unlearning relies on retraining affected sub-models (bourtoule2021machine), while approximate unlearning typically adopts a teacher–student framework to balance data removal and model utility (frog; fan2025towards; cheng2025tool). However, most approximate unlearning methods formulate unlearning as a weighted sum of the forget and retain losses, using a static factor to balance the two objectives (fan2023salun; cheng2024mu; fan2025towards). Despite its simplicity, this formulation overlooks the fact that optimizing one objective can substantially impede the other. From the optimization point of view, a key cause of this issue lies in conflicting gradients (yu2020gradient; yi2025gradient), where the gradients of the forget and retain losses at the neuron level point in opposing directions. As a result, updates intended to improve one objective may inadvertently harm the other, leading to either insufficient forgetting or severe degradation of model utility Additionally, existing LLMRec unlearning methods update model parameters in a largely black-box manner, without explicit knowledge of which internal modules (e.g., attention heads and MLPs) encode the information to be forgotten (chen2022recommendation; chen2024cure4rec). As a result, it is unclear whether modules containing critical information are effectively updated. This lack of transparency hinders the interpretability of the unlearning process and undermines its trustworthiness. Recent advances in mechanistic interpretability shed light on the internal mechanisms of LLMs by identifying sparse computational subnetworks (“circuits”) responsible for specific model behaviors, offering a reliable way to understand the above challenges at their root. A key finding is that knowledge in LLM is dynamically activated through specific computational circuits, each specializing in different functional roles and jointly contributing to the final decision (conmy2023towards; syed2024attribution; cheng2026toward). Consequently, gradient conflicts arise when the circuits responsible for the forget and retain sets become entangled, particularly when the shared modules are driven toward conflicting optimization directions under the two objectives. This insight suggests a transparent solution: disentangling the conflicting circuits and optimizing them separately for unlearning. Inspired by this insight, we propose CURE, a circuit-aware framework for LLMRec unlearning. Instead of globally optimizing competing objectives, CURE decouples unlearning into two stages: crucial circuit extraction and task-specific parameter updating. In the first stage, CURE employs a gradient-based analysis to localize internal computational pathways that are most responsible for the forget and retain sets. To precisely identify these circuits under long input prompts encoding user interaction histories, we leverage the user–item graph to construct slight input perturbations and detect influential modules through contrastive activation analysis. In the second stage, modules along the identified circuits are categorized according to their functional roles and selectively updated, enabling semantically aware control over parameters while effectively mitigating gradient conflicts during unlearning. We also demonstrate the effectiveness of CURE through a theoretical analysis. Overall, our key contributions are as follows: • Motivation: We introduce a circuit-aware perspective for understanding gradient conflicts in LLMRec unlearning, which leads to a transparent unlearning approach via disentangling conflicting circuits. • Method: We propose a novel circuit-aware framework for LLMRec unlearning that first identifies influential circuits for the forget and retain sets, and then selectively updates the associated modules according to their functional roles. We further provide a theoretical analysis to guarantee its validity. Furthermore, the proposed framework is model-agnostic and can be readily applied to a wide range of LLM backbones for recommendation unlearning. • Performance: CURE achieves 18%18\% and 6%6\% improvements over the baseline in unlearning efficiency and model utility, respectively. Moreover, it is 3.5×3.5× faster than the baseline. 2. Preliminary 2.1. LLM as Recommender We consider the standard collaborative filtering recommendation task. Let =u1,…,umU=\u_1,…,u_m\ be a set of m users, and ℐ=i1,…,inI=\i_1,…,i_n\ be a set of n items. The observed user–item interactions are represented by a binary interaction matrix A∈0,1m×nA∈0,1^m× n, where Au,i=1A_u,i=1 indicates that user u has interacted with item i, and Au,i=0A_u,i=0 otherwise. We further model the interaction data as user-item graph =(,ℰ)G=(V,E), where =∪ℐV=U and ℰ=(u,i)∣Au,i=1E=\(u,i) A_u,i=1\ denotes the set of edges corresponding to observed interactions. Based on the user-item graph G, LLMRec, denoted as ℳθM_θ, reformulates collaborative filtering as a prompt-based prediction task. Given a user u∈u with historical interactions ℋu=i1,i2,⋯,i|ℋu|H_u=\i_1,i_2,·s,i_|H_u|\ and a target item it∈ℐi_t , we encode them into textual instructions ux_u using predefined hard prompt templates. Conditioned on ux_u, ℳθM_θ predicts whether u will interact with iti_t, which is formulated as a binary classification task with an answer yu∈"YES","No"y_u∈\ "YES", "No"\(As shown in Figure 2 and 4). In order to tailor LLM to recommendation scenarios, a conditional language modeling objective is employed by minimizing the negative log-likelihood of generating yuy_u conditioned on input ux_u. Formally: (1) minℒpred=−∑(u,yu)∈log(ℳθ(yu,t|u,yu,<t)) _pred=-Σ _(x_u,y_u) log(M_θ(y_u,t|x_u,y_u,<t)) Where yu,ty_u,t is the t-th token of yuy_u, and yu,<ty_u,<t is the token before yu,ty_u,t. ℳθ(yu,t|u,yu,<t)M_θ(y_u,t|x_u,y_u,<t) signifies the predictive probability of yu,ty_u,t. 2.2. LLMRec Unlearning LLMRec unlearning involves removing certain user-item interactions from a trained model ℳθM_θ without full retraining. Given a dataset D and a subset fD_f to be removed, we denote the retained dataset as rD_r, where r=∖fD_r=D _f, with the conditions f∩r=∅D_f _r= . Requests for LLMRec unlearning can be broadly categorized into two types: (i) user/item-wise deletion, , which removes all interactions associated with a given user or item, and (i) interaction deletion, which removes specific interactions. The objective is to obtain an unlearned model ℳunM_un that eliminates the influence of fD_f while preserving performance on rD_r. As retraining on rD_r to obtain the optimal model ℳθ∗ M_θ^* is often time-consuming, our goal is to approximate ℳθ∗ M_θ^* by updating the original model ℳθM_θ through the unlearning process as follows: ℳθ→fℳun≈ℳθ∗.M_θ D_fM_un≈ M_θ^*. Most existing methods formulate the LLMRec unlearning loss ℒL as a weighted sum of a forget loss ℒFL_F and a retain loss ℒRL_R. More formally, the task of unlearning can be modeled in the following manner: (2) ℳun _un =argminℳ∑(u,yu)∈fωfℒF(yu|u;ℳ)+∑(u,yu)∈rωrℒR(yu|u;ℳ) = arg _MΣ _(x_u,y_u) _f _fL_F(y_u|x_u;M)+Σ _(x_u,y_u) _r _rL_R(y_u|x_u;M) In addition, ωr+ωf=1 _r+ _f=1 serves as a scaling factor to balance forget and retain. The specific forms of ℒFL_F and ℒRL_R are introduced in section 4. Note that user/item-wise deletion can be interpreted as removing all interactions incident involving the corresponding user or item. 3. Motivation Despite simplicity, Eq. 2 fails to account for scenarios in which the gradients of the forget and retain losses move in opposing directions during gradient descent optimization (reisizadeh2025blur), leading to either insufficient forgetting on fD_f or performance degradation on rD_r. To illustrate this issue, we analyze the gradient structure of the unlearning objective. Specifically, let g=∇ℒ(θ)g= (θ) denote the gradient of the weighted unlearning loss with respect to the model parameters θ, while gf=∇ℒF(θ)g_f= _F(θ) and gr=∇ℒR(θ)g_r= _R(θ) denote the gradient of forget loss and retain losses respectively. A small change of θ in the direction of negative g is θ←θ−αgθ←θ-α g with a sufficiently small step size α. The effect of this change on the performance of two objective can be measured by: (3) Δℒ =ωfΔℒF+ωrΔℒR = _f _F+ _r _R =ωfℒF(θ−α)−ωfℒF(θ)+ωrℒR(θ−α)−ωrℒR(θ) = _fL_F(θ- )- _fL_F(θ)+ _rL_R(θ- )- _rL_R(θ) =−αωf(⋅f)−αωr(⋅r)+o(α), =-α _f\,(g·g_f)-α _r\,(g·g_r)+o(α), where the last equality is obtained by Taylor approximation. Notably, the update procedure can impede effective forgetting when ⋅f<0g·g_f<0, since it increases the forget loss. Similarly, it can degrade model utility when ⋅r<0g·g_r<0. Based on the above analysis, we adopt a normalized alignment metric (reisizadeh2025blur) and examine the evolution of Ar(θ)A_r(θ) and Af(θ)A_f(θ) throughout the unlearning procedure in Figure 1, where (4) Af(θ)=⋅f‖f‖2,Ar(θ)=⋅r‖r‖2 A_f(θ)= g·g_f||g_f||^2, A_r(θ)= g·g_r||g_r||^2 Figure 1. Normalized Alignment Values of forget and retain losses on MovieLens-1M using Llama-2 (7B) We observe that the descent direction switches frequently during the early training steps of E2URec, a representative LLMRec unlearning baseline, reflecting entangled optimization signals and the presence of gradient conflicts. Although (reisizadeh2025blur) proposes a hierarchical unlearning framework that prioritizes forgetting over retention, such a trade-off is suboptimal for LLMRec unlearning, where both objectives are essential. In contrast, CURE exhibits a more stable optimization behavior, with gradient conflicts largely mitigated. 4. Method We introduce CURE, a two-stage circuit-aware framework for LLMRec unlearning. The first stage focuses on identifying key computational circuits in the original model . Leveraging the structural information encoded in ℳθM_θ, we propose two alternatives methods: Activation Intervention, which efficiently yields a coarse circuit estimate, and Activation Patching, a contrastive approach for more precise localization. In the second stage, we selectively update circuit components according to their functional roles in forgetting and utility preservation, thereby mitigating gradient conflicts. Figure 2. Schematics of CURE: (i) Locating Circuits in LLMREC; (i)Selective Circuits Updating 4.1. Locating Circuits in LLMRec To find circuits for LLMRec, we must represent the internals of the model as a computational directed acyclic graph GLMG^LM in which information flows from the input tokens to the output logits through intermediate neuron activations. Following (jafari2025relp), we define attention heads and MLP modules as nodes, with directed edges specifying how the output of one node is passed to another. As shown in Figure 2, the input of a node v1v_1 is defined as the sum of the outputs of all nodes with edges pointing to v1v_1, and each edge e=v1→v2e=v_1→ v_2 represents a direct computational dependency. A circuit is defined as a subgraph that connects the input tokens to the output logits. Given a sample (u,yu)∈(x_u,y_u) , our goal is to extract the influential circuits that are faithful to model prediction. Directly evaluating the importance of individual nodes is often insufficient, as it ignores how information is propagated and combined across modules. Instead, we assign a strength score to each edge e=v1→v2e=v_1→ v_2 that quantifies its contribution to driving the prompt ux_u toward the target outcomes yuy_u. Specifically, we use the change in output probability Δ(u)=ℳθ(Yes|u)−ℳθ(No|u) (x_u)=M_θ(Yes|x_u)-M_θ(No|x_u) to measure variations in the model’s prediction. Following (syed2024attribution), for an edge e∈GLMe∈ G^LM, we evaluate its impact to metrics Δ by intervening on the activation input transmitted through this edge: (5) I(e)=|Δ(xu|do(e))−Δ(xu)| I(e)=| (x_u| do(e))- (x_u)| We adopt the do-notation from causal inference to emphasize that this intervention modifies the information flow along a specific edge. Although existing methods such as ACDC (conmy2023towards; hanna2024have)can be used to evaluate causal impact, they are computationally inefficient. Moreover, user–item relationships naturally form a graph structure, which can be further exploited to improve both efficiency and localization precision. Based on this observation, we propose two methods. 4.1.1. Activation Intervention To quantify I(e)I(e) in Eq. 5, we directly intervene on the information flow along edge e=v1→v2e=v_1→ v_2. Concretely, we mask this edge by setting the message passed from v1v_1 to v2v_2, denoted as mv1→v2=0m_v_1→ v_2=0, to zero, while keeping all other activations unchanged. However, modern LLMs involve an enormous number of edges to evaluate, making exact interventions time-consuming. Hence, we approximate the resulting change in the metric by linearly expanding Δ with respect to the input activation using a first-order Taylor expansion, which yields an efficient estimate of the marginal contribution of a single edge e=v1→v2e=v_1→ v_2 in the computational graph GLMG^LM (6) Δ(xu∣do(mv1→v2=0))≈Δ(xu)−mv1→v2⊤∂Δ(xu)∂v2. (x_u (m_v_1→ v_2=0))≈ (x_u)-m_v_1→ v_2 ∂ (x_u)∂ v_2. Here, with a slight abuse of notation, we use v2v_2 to be the activation input of this node. The term mv1→v2⊤∂Δ(xu)∂v2m_v_1→ v_2 ∂ (x_u)∂ v_2 captures the first-order effect of erasing the message of e, where the gradient is evaluated at the original (non-intervened) forward pass. According to Eq. 6, computing I(e)I(e) for all edges only requires one forward pass to record edge messages and one backward pass to obtain their gradients. 4.1.2. Activation Patching Although effective, the Activation Intervention tends to over-select modules as the input ux_u grows longer, since it captures the contributions of all tokens to yuy_u in a global manner. To localize the responsible circuits, we therefore leverage activation patching (zhang2023towards; syed2024attribution) to measure module importance through contrasting activations that lead to different prediction outcomes. Following (syed2024attribution), we construct a corrupt sample u∗x_u^* that shares the same task schema as ux_u but elicits a distinct output from ℳθM_θ. The importance of an edge is evaluated by replacing its activation mv1→v2m_v_1→ v_2 induced by ux_u with that mv1→v2∗m^*_v_1→ v_2 from u∗x_u^* during the forward pass, while keeping all other activations unchanged. A significant change in the output metric Δ indicates that the edge plays a critical role in driving the prediction. Formally, to quantify Eq. 5, we linearly approximate the importance score I(e)I(e) by expanding Δ as a Taylor series with respect to the edge activation: (7) Δ(u∣do(mv1→v2=mv1→v2∗))≈Δ(xu)+(mv1→v2∗−mv1→v2)⊤∂Δ(xu)∂v2. (x_u (m_v_1→ v_2=m^*_v_1→ v_2))≈ (x_u)+(m^*_v_1→ v_2-m_v_1→ v_2) ∂ (x_u)∂ v_2. The second term on the right-hand side serves as an estimate of I(e)I(e). However, unlike circuit detection in reasoning tasks, constructing the corrupt sample u∗x_u^* in LLMRec is non-trivial. Two challenges arise: 1) The input ux_u can be substantially longer, as it summarizes a user’s historical interactions. 2) Prior work (hanna2024have; syed2024attribution) requires ux_u and its corrupt counterpart u∗x_u^* to differ minimally at the input level while inducing a significant change in prediction. As a result, it is difficult to pinpoint which tokens in ux_u should be modified to construct u∗x_u^*. Formally, u∗x_u^* should satisfy the following constraints (8) u∗=argminΔ()s.t.‖−u‖0≤K.x_u^*= _x\ (x) .t. \|x-x_u\|_0≤ K. Here, K controls the number of items to be replaced. In our setting, we set K=1K=1, meaning that only a single item is allowed to be replaced in ux_u. In addition, we require the inserted item to remain within the user’s interest distribution to avoid introducing out-of-distribution corrupt sample u∗x_u^*. The original input ux_u consists of the user history ℋuH_u and the target item iti_t, whose relationships can be captured by the user–item graph G. Fortunately, this representation allows us to exploit rich structural information to construct appropriate corrupt samples u∗x_u^*. To efficiently generate u∗x_u^*, we first identify the history item in ℋuH_u that is most influential to predicting iti_t within the context of ux_u, and then replace it with a weakly related item. Step1: Graph-based Item Scoring We first leverage Personalized PageRank (PPR) (yang2024efficient; li2023everything) on G to estimate item proximity with respect to a given user u. Let u∈ℝ|| π_u ^|V| denote the PPR vector associated with user u, where each entry u[i] π_u[i] measures the graph-based proximity of item i to u. The PPR vector is defined as the stationary solution of the following equation: (9) u=αP⊤u+(1−α)u, π_u=α\,P π_u+(1-α) p_u, where P denotes the transition matrix of the user–item graph and α∈(0,1)α∈(0,1) is a decay factor. The preference vector u p_u is a probability distribution that encodes the user’s preference over items. Apart from the algebraic definition, PPR also has an intuitive random-walk interpretation: starting from user u, the walk proceeds as follows: at each step, it either (i) moves to a neighboring node on G according to the transition matrix P with probability α, or (i) jumps to a node sampled from the preference distribution u p_u. Then u π_u is defined as the stationary distribution over V after infinitely many steps. Given the input ux_u, we further bias u π_u toward items i∈ℋui _u that contribute most to the model prediction. Accordingly, we initialize the preference vector u p_u as: (10) u(i)=exp(τS(i))∑j∈ℋuexp(τS(j)),v∈ℋu,0,otherwise, p_u(i)= cases (τ S(i)) _j _u (τ S(j)),&v _u,\\[6.0pt] 0,&otherwise, cases where S(i)S(i) measures the importance of item i to the prediction and τ is a temperature parameter. Since an item v may correspond to multiple tokens in the input, we approximate its importance by aggregating token-level gradients: (11) S(i)=∑t∈tokens(i)‖∂Δ(xu)∂t‖,S(i)= _t (i) ∂ (x_u) _t , where te_t denotes the embedding of token t. In practice, we precompute approximate PPR vectors for individual items offline (yang2024efficient; zhang2024towards). Specifically, we run an approximate PPR algorithm with a one-hot preference vector for each item i and obtain the corresponding vector ipre π^pre_i. Given an input ux_u, the personalized PPR vector can be efficiently constructed as a weighted sum: (12) u=∑i∈ℋuexp(τS(i))∑j∈ℋuexp(τS(j))ipre, π_u= _i _u (τ S(i)) _j _u (τ S(j)) π^pre_i, which follows from the linearity of Personalized PageRank (yang2024efficient) with respect to the preference vector. In this way, personalized item scores can be efficiently obtained without performing any online graph diffusion. Step 2: Replacing Items in ux_u As LLMRec may not perfectly align with the underlying graph structure, the solution induced by PPR can be suboptimal for ℳθM_θ. Moreover, exhaustively searching over all possible replacements is computationally prohibitive. We therefore restrict the search space to a small set of candidate substitutions when constructing the corrupt input u∗x_u^*. Given an input ux_u, we first select the top-3 most influential items from the user history ℋuH_u according to the gradient-based importance score S(v)S(v), and construct a preferred item set ℐuI_u by selecting the top-50 items ranked by the PPR vector u π_u. Following (yang2024efficient), differences between entries in u π_u reflect relative proximity under the same source u; thus, for each influential item, we choose the 10 least relevant items from ℐuI_u as replacement candidates. Overall, this procedure yields at most 30 candidate corrupt inputs, which are evaluated by the LLM to select the final counterfactual input u∗x_u^* according to Eq. 8. 4.1.3. Greedy Circuit Discovery After scoring edges via Equation 5, we employ the greedy extraction strategy of (syed2024attribution). Starting from the logits, we iteratively add the highest-scoring edge whose child node is already in the circuit, constructing a top-down circuit while avoiding childless nodes. This process constructs a complete circuit in a top-down manner. The resulting procedure resembles a maximization variant of Dijkstra’s algorithm, with circuit complexity controlled by the number of iterations. 4.2. Selective Circuits Updating For each sample in fD_f, we select nearby samples from the remaining data to construct a retain buffer of size |r|=k|f||D_r|=k|D_f| (typically k=6k=6). For each user–item interaction to be removed, we measure structural proximity on the user–item graph G using Personalized PageRank (PPR) with forget nodes as sources. Retain-set edges are ranked by their PPR scores, where higher scores indicate stronger proximity to the forget set. We select the top-k nearest training samples to form the retain set, enabling unlearning by refining the decision boundary between closely related forget and retain data. Here we borrow ideas from SCRUB (kurmanji2023towards) to construct the unlearning loss, and formulate it by enforcing two essential properties. Specifically, for each deleted sample (u,yu)∈(x_u,y_u) : • Deviating the prediction from the original model ℳθM_θ on fD_f, where deleted samples are encouraged to produce predictions under ℳunM_un that explicitly differ from those of the original model, ensuring that the removed information is no longer preserved. ℒF=−DKL(ℳθ(Yes|u)||ℳun(Yes|u))L_F=-D_KL(M_θ(Yes|x_u)||M_un(Yes|x_u)). • Maintaining the prediction of ℳθM_θ on rD_r, where the selected retain samples (r,yr)∈r(x_r,y_r) _r associated with each deleted sample (u,yu)(x_u,y_u) are encouraged to produce similar predictions under ℳunM_un as those from the original model ℳθM_θ, thereby preserving the model’s utility on retained data. ℒR=1|r|DKL(ℳθ(Yes|r)||ℳun(Yes|r))L_R= 1|D_r|D_KL(M_θ(Yes|x_r)||M_un(Yes|x_r)). Here, DKLD_KL denotes the KL-divergence between the two distributions. The overall unlearning objective is defined as ℒ=ωrℒR+ωfℒFL= _rL_R+ _fL_F; During optimization, all parameters of the original model ℳθM_θ are frozen, and only the parameters of the unlearned model ℳunM_un, which is initialized from ℳθM_θ, are updated. For each sample in fD_f and rD_r, we apply the method in Section 4.1 to extract the retain and forget circuits, denoted as fC_f and rC_r, respectively. Based on the detected circuits, we categorize the associated modules into different functional groups according to their roles along the circuits. Neurons within each modules are assigned to the same group and updated using group-specific policies, allowing each group to be optimized for its intented function without interference. The update policies are defined as follows: • Forget-Specific Neurons Θf _f: A neuron θf∈Θf _f∈ _f is identified as forget-specific if θf∈f∩rc _f _f _r^c, where rcC_r^c denotes the the complement of rC_r. indicating that it contributes primarily to samples in fD_f rather than rD_r. Accordingly, these neurons are updated solely with respect to the forget loss to enhance data removal. We define the gradient as fF=∇θfℒFg_f^F= _ _fL_F and update θf←θf−αfF. _f← _f-α\,g_f^F. • Retain-specific Neurons Θr _r: Similarly, a neuron θr∈Θr _r∈ _r is identified as retain-specific if θr∈r∩fc _r _r _f^c. As these neurons are essential for preserving the model utility, we update them only with respect to the retain loss. We define the gradient as rR=∇θrℒRg_r^R= _ _rL_R and update: θr←θr−αrR. _r← _r-α\,g_r^R. • Function-shared Neurons Θsh _sh. A neuron θsh∈Θsh _sh∈ _sh is identified as function-shared if θsh∈r∩f _sh _r _f, indicating that it is involved in high-level functions contributing to both data removal and utility maintenance. To alleviate gradient conflicts during optimization, CURE adopts a simple projection-based strategy (yu2020gradient; shi2023recon). Specifically, when the gradients of the two objectives conflict, i.e., (shR)⊤shF<0(g_sh^R) g_sh^F<0, we iteratively select one objective from ℒrL_r and ℒfL_f and project its gradient onto the other, removing destructive components. When ℒrL_r is selected, the update is defined as: sh _sh =ωR(shR−(shR)⊤shF‖shF‖2shF)+ωF(shF−(shF)⊤shR‖shR‖2), = _R(g_sh^R- (g_sh^R) g_sh^F\|g_sh^F\|^2\,g_sh^F)+ _F(g_sh^F- (g_sh^F) g_sh^R\|g_sh^R\|^2), θsh _sh ←θsh−αsh. ← _sh- _sh. 5. Theoretical Analysis We provide a theoretical analysis of CURE to indicate that a single gradient update on the model parameters of CURE achieves lower loss than normal gradient descent. Let Θ=sh,f,r =\ θ_sh, θ_f, θ_r\ denote the parameters of components in selected circuits. After applying CURE, the model parameters are Θ^=^,^, =\ θ_sh, θ_f, θ_r\. An one step gradient update of Θ is: (13) θ^sh θ_sh =θsh−α[ωr(shR−shR⋅shF‖shF‖2shF)+ωf(shF−shF⋅shR‖shR‖2shR)], = _sh-α [ _r (g_sh^R- g_sh^R·g_sh^F\|g_sh^F\|^2\,g_sh^F )+ _f (g_sh^F- g_sh^F·g_sh^R\|g_sh^R\|^2\,g_sh^R ) ], θ^f θ_f =θf−αfF,θ^r=θr−αrR. = _f- _f^F, θ_r= _r- _r^R. Without applying CURE, the parameters are Θ~=~~,~ =\ θ_sh\, θ_f, θ_r\. An one-step gradient update is given by: (14) θ~i θ_i =θi−α(ωriR+ωfiF),i∈sh,f,r = _i-α( _rg_i^R+ _fg_i^F), i∈\sh,f,r\ Then, we have the following theorem. Theorem 5.1. Assume that the joint loss function is defined as L(Θ)=ωrLR(Θ)+ωfLF(Θ)L( )= _rL_R( )+ _fL_F( ) and ωr+ωf=1 _r+ _f=1. Then for any sufficiently small learning rate α>0α>0, we have (15) L(Θ^)≤L(Θ~) L( )≤ L( ) Proof. We iteratively update each parameter group while fixing the others. The loss difference between the conventional update ~ θ and CURE θ is analyzed via two components, A and B: (16) ℒ(Θ~)−ℒ(Θ^)≈(~−^)⊤(shR+shF) ( )-L( )≈( θ_sh- θ_sh) (g_sh^R+g_sh^F) +(~−^)⊤fF+(~−^)⊤rR +( θ_f- θ_f) g_f^F+( θ_r- θ_r) g_r^R =−[shR⋅shF+(shR⋅shF)2(ωr‖shF‖2+ωf‖shR‖2)]⏟A = - [g_sh^R·g_sh^F+(g_sh^R·g_sh^F)^2 ( _r\|g_sh^F\|^2+ _f\|g_sh^R\|^2 ) ]_A +ωr(fF−fR)⊤fF+ωf(rR−rF)⊤rR⏟B + _r(g_f^F-g_f^R) g_f^F+ _f(g_r^R-g_r^F) g_r^R_B For Part A: Let γ=‖shR‖/‖shF‖γ=\|g_sh^R\|/\|g_sh^F\| and ψ be the angle between shRg_sh^R and shFg_sh^F. A≥0A≥ 0 is guaranteed if and only if the condition holds: (17) cosψ≥−γωr+(1−ωr)γ2 ψ≥- γ _r+(1- _r)γ^2 When γ is close to 11, indicating that the two gradients have comparable magnitudes, equation 17 holds with high probability. Notably, when γ=1γ=1, the condition reduces to cosψ≥−1 ψ≥-1, which is always satisfied. In practice, we control γ through gradient normalization, ensuring that the magnitudes of the two gradients remain comparable. For Part B: We consider neurons in each task-specific component, and denote Δm=(mv1→v2∗−mv1→v2) m=(m^*_v_1→ v_2-m_v_1→ v_2) in equation 7. According to the loss functions ℒFL_F and ℒRL_R defined in Section 4.2, we observe that both losses are monotonic with respect to Δ(u)=ℳθ(Yes|u)−ℳθ(No|u) (x_u)=M_θ(Yes|x_u)-M_θ(No|x_u), which is used to detected the forget and retain circuits. which is used to detect forget and retain circuits. Hence, the greedy selection policy in Section 4.1.3 satisfies: (18) ‖ΔmfF‖2≥‖ΔmfR‖2,‖ΔmrR‖2≥‖ΔmrF‖2 \| mg_f^F\|^2≥\| mg_f^R\|^2, \| mg_r^R\|^2≥\| mg_r^F\|^2 ⟹2‖ΔmfF‖2≥‖ΔmfF‖2+‖ΔmfR‖2 2\| mg_f^F\|^2≥\| mg_f^F\|^2+\| mg_f^R\|^2 ≥‖Δm‖2(fF⋅fR)⟹‖fF‖2−fF⋅fR≥0 ≥\| m\|^2(g_f^F·g_f^R) \|g_f^F\|^2-g_f^F·g_f^R≥ 0 This leads to ‖fF‖2−fF⋅fR>0\|g_f^F\|^2-g_f^F·g_f^R>0, ensuring the benefit of targeted forgetting. An analogous result can be derived ‖rR‖2−rF⋅rR>0\|g_r^R\|^2-g_r^F·g_r^R>0 6. Experiments Figure 3. Unlearning effectivenss and model performance on Movie and Goodreads. To evaluate the effectiveness of our proposed method, we conduct a series of experiments to address the following research questions: • RQ1: How effective is CURE in achieving unlearning while preserving recommendation utility? • RQ2: How well does CURE mitigate gradient conflicts during unlearning? • RQ3: How do different components of CURE contribute to its overall effectiveness? 6.1. Experimental Settings Datasets We evaluate our proposed method on two widely recognized recommendation benchmarks: MovieLens-1M (ML-1M) and GoodReads (GD). MovieLens-1M (ML-1M) contains movie metadata and user ratings. Following (wang2025towards), we transform the original ratings into binary labels for LLM prompting, where ratings greater than 3 are treated as positive (mapped to “Yes”), and the remaining ratings are mapped to “No”. Similarly, GoodReads (GD) comprises book features and user ratings, and we apply the same binarization strategy. 6.1.1. Implementation Details All baseline models and our proposed framework, CURE, utilize the same backbone architectures: GPT-2 (brown2020language) and Llama-2 (7B) (touvron2023llama). We follow standard hyperparameter configurations established in recent LLM-based recommendation literature. We evaluate two variants, CURE-AI and CURE-AP, which adopt Activation Intervention and Activation Patching for circuit extraction, respectively. CURE-AI identifies coarse-grained circuits, while CURE-AP achieves higher-precision localization by constructing corrupt samples based on the original input. For the unlearning experiments, we follow the protocol in (wang2025towards). Both datasets are split into training, validation, and test sets with a ratio of 7:2:1 to obtain the original model, and 20%20\% of the training data are designated as the forget set during unlearning. Regarding the specific hyperparameters for CURE, we set the intervention threshold based on the top-5% of activation weights. For the second stage, we utilize the AdamW optimizer with a learning rate of 5×10−55× 10^-5. To facilitate efficient training on Llama-2 (7B), we employ Low-Rank Adaptation (LoRA) with rank r=8r=8. All experiments are conducted on NVIDIA A100 GPUs, and hyperparameters are tuned on the validation set to ensure optimal performance. 6.1.2. Baselines We compare CURE against several baselines, including exact unlearning paradigms and model editing techniques: (i)(i) Retrain, which refers to training from scratch without target forgotten data, (ii)(i) SISA (bourtoule2021machine), a partition-based retraining method that aggregates predictions from sub-models trained on separate data shards; (iii)(i) RecEraser (li2023ultrare), a recommendation-specific unlearning method that preserves collaborative information through specialized partitioning; (iv)(iv) E2URec (wang2025towards), an efficient unlearning framework that utilizes a teacher-student architecture to guide the unlearning process; (v)(v) ROME (meng2022locating), a model editing method that performs direct parameter updates at early layers; (vi)(vi) WISE (wang2024wise), a baseline that introduces learnable parameters at later layers to incorporate new information or forget data; (vii)(vii) PCGrad (yu2020gradient), a gradient surgery approach for gradient collicts that projects gradients onto the normal plane of conflicting tasks. 6.1.3. Evaluation Settings Our goal is to achieve precise and efficient unlearning for LLMRec while preserving recommendation utility. Following standard protocols in recommendation unlearning, we evaluate CURE based on the following three dimensions: • Recommendation Performance: To evaluate the recommendation performance, we use Area Under the ROC Curve (AUC ↑ ), Accuracy (ACC ↑ ), and LogLoss (L ↓ ) on test set. • Unlearning Effectiveness: To quantify how closely the unlearned model aligns with the gold-standard Retrain model, we compute the Jensen-Shannon Divergence (JSD ↓ ) between their respective output probability distributions on the forgotten data. Lower JSD values signify a more effective elimination of the target data, indicating that the unlearned model’s behavior successfully mimics a model that never encountered the forgotten samples. • Unlearning Efficiency: We measure the unlearning efficiency by Unlearning Time (↓ ) measuring the total wall-clock time required for the unlearning process. 6.2. Main Results (RQ1) We evaluate unlearning performance on two recommendation benchmarks, MovieLens-1M and GoodReads, using GPT-2 and LLaMA2-7B as backbone models. Following previous section, we assess (i) recommendation utility via AUC and ACC, (i) unlearning effectiveness via JS-Divergence (JSD) between the unlearned and retrained models on the forgotten data, and (i) efficiency via unlearning time. Across all settings, results consistently show that: (i) full retraining remains the upper bound in recommendation accuracy but is computationally prohibitive; (i) CURE-AT achieves the best overall trade-off, preserving near-retraining utility while attaining the lowest divergence from retrained models with orders-of-magnitude lower cost; (i) efficiency-oriented methods such as ROME and WISE sacrifice unlearning completeness, as reflected by substantially higher JSD; and (iv) these trends remain stable across model scales and datasets, indicating strong robustness of the proposed approach. Overall Results Across both backbones and datasets, we observe a clear trade-off between utility preservation and unlearning effectiveness. While approximate unlearning methods reduce computational cost, many incur either significant performance degradation or incomplete forgetting. In contrast, CURE-AT and CURE-AI consistently strike a favorable balance, achieving strong recommendation performance with minimal divergence from retraining. Figure 3 reports results on MovieLens-1M using GPT-2 Large as the backbone. Most baseline unlearning methods (E2URec, RecEraser, SISA, ROME, WISE,PCGrad) exhibit noticeable drops in AUC and ACC relative to retraining, with JSD values generally exceeding 2.0. In contrast, CURE-AT achieves the lowest divergence (JSD = 1.93) while preserving higher recommendation quality (AUC = 76.93, ACC = 69.3). CURE-AI follows closely, outperforming all other baselines in both unlearning effectiveness and efficiency. Figure 3 summarizes results using LLaMA2-7B. Compared to GPT-2 Large, all methods benefit from increased model capacity, yielding higher absolute AUC and ACC. However, the relative ranking of unlearning methods remains consistent. CURE-AT again achieves the strongest performance among non-retraining approaches (AUC = 79.1, ACC = 72.8) while also attaining the lowest divergence (JSD = 1.61). Although RecEraser and SISA partially preserve utility, their substantially longer unlearning times make them impractical for frequent or large-scale unlearning scenarios. Figure 3 presents results on the GoodReads dataset using GPT-2 Large. Compared to MovieLens-1M, GoodReads exhibits a more challenging unlearning setting, with larger variance in JSD across methods. While retraining again yields the highest accuracy (AUC = 74.2, ACC = 70.85), CURE-AT achieves near-retraining performance (AUC = 72.9, ACC = 70.6) with the lowest divergence (JSD = 0.98). CURE-AI follows closely (JSD = 1.03), outperforming RecEraser and SISA in both effectiveness and efficiency. Baseline methods such as ROME and WISE show particularly high divergence (JSD >> 3.7), indicating severe forgetting failure despite their fast execution. PCGrad improves over several baselines but remains inferior to CURE-based approaches in both utility preservation and unlearning completeness. Figure 3 reports corresponding results using LLaMA2-7B. Similar trends persist at larger model scale: CURE-AT achieves the best trade-off between accuracy (AUC = 75.3, ACC = 72.6) and unlearning effectiveness (JSD = 0.91), substantially narrowing the gap to retraining while maintaining low computational cost. Other methods either exhibit higher divergence or incur significantly longer unlearning times, reinforcing the robustness of CURE-AT across datasets and model sizes. Unlearning Time Comparison Table 1 compares the efficiency of different unlearning methods. Overall, methods exhibit large variance in computational efficiency. Retraining is the most time-consuming method, incurring prohibitively expensive cost than other unlearning methods. RecEraser and SISA are consistently the most expensive approaches, requiring up to tens of thousands of seconds on LLaMA2-7B. While these methods partially preserve recommendation utility, their reliance on repeated retraining or multiple model shards makes them impractical for large-scale or frequent unlearning scenarios. In contrast, CURE-AI and CURE-AT consistently achieve strong efficiency across both MovieLens-1M and GoodReads, with unlearning times comparable to or faster than ROME and WISE, while maintaining significantly lower JSD divergence and higher recommendation accuracy. Notably, CURE-AI is 3.5×3.5× faster and CURE-AT is 3.3×3.3× than E2URec. Although slightly slower than WISE and ROME due to their localized updating strategy, CURE achieves significantly better performance. Table 1. Unlearning time comparison (s) ↓ . GoodReads MovieLens-1M Method GPT-2 LLaMA-2 GPT-2 LLaMA-2 Retrain 15,800 128,000 29,300 132,300 E2URec 1,900 13,600 3,200 21,800 RecEraser 4,200 32,500 7,100 49,700 SISA 3,700 29,800 6,800 45,300 ROME 600 3,600 1,700 12,100 WISE 800 7,200 1,200 9,800 PCGrad 2,600 16,700 3,600 23,300 CURE-AI (Ours) 600 3,900 1,500 10,200 CURE-AT (Ours) 600 4,100 1,700 11,300 6.3. Effectiveness in Mitigating Gradient Conflicts(RQ2) CURE greatly reduces the occurence of conflicting gradients As shown in Figure 5, we compare the distribution of cosψ ψ before and after applying CURE on MovieLens-1M (ML-1M). The results show that CURE substantially suppresses severely conflicting gradient pairs (cosψ∈[−1,−0.02) ψ∈[-1,-0.02)), reducing their proportion by at least 55%55\% and up to 76%76\% compared with E2URec. In contrast, PCGrad and ROME yields only marginal reductions, and in some cases even increase the proportion of conflicting gradients. An interesting observation is that WISE also exhibits fewer conflicting gradients, likely due to its restriction to updating later layers, which implicitly avoids interference with shared early representations. 6.4. In-depth Analysis(RQ3) 6.4.1. Ablation Study We examine the effect of ωr _r, which balances the retain and forget losses, as shown in Table 2. The results shows the necessity of jointly optimizing ℒFLF and ℒRLR to achieve a favorable trade-off. When ωr≥0.6 _r≥ 0.6, performance remains stable; however, for ωr≤0.4 _r≤ 0.4, AUC drops sharply. Accordingly, we set ωr=0.6 _r=0.6 in all experiments. 6.4.2. Why does Circuit-aware Unlearning work To validate the of our circuit-aware approach, we compare CURE with two parameter-efficient unlearning baselines, ROME and WISE, by examining the modules they modify. As showen in Figure 4, we observe that WISE primarily edits later layers, while ROME focuses on early layers. However, both methods operate on isolated modules within the LLM and fail to capture several critical components involved in the unlearning procedure. Instead, CURE identifies complete end-to-end circuits spanning all relevant modules, and propagates targeted modifications along the information flow from the input to the logits. Moreover, each module within an identified circuit attends to semantically relevant input tokens. For example, movies sharing the same genre as the target movie (e.g., GoldenEye and Congo) are primarily attended to by mlp8, mlp9, and mlp11, enabling CURE to forget such interactions in a transparent and interpretable manner. Figure 4. Transparent comparison of circuit-aware unlearning with ROME and WISE Figure 5. The distribution of gradient conflicts (cosψ ψ) on GoodReads. The left and right columns use GPT-2 Large and Lamma-2 7B as backone, respectively Table 2. Ablation study of ωr _r on GoodRead across different backbone. Best performance is bold and second best is underlined. ωr _r Llama-2 GPT-2 AUC (↑)( ) ACC (↑)( ) JSD (↓)( ) AUC (↑)( ) ACC (↑)( ) JSD (↓)( ) 0.2 71.3 69.1 0.91 69.1 67.5 0.94 0.4 73.5 71.2 0.90 71.2 69.5 0.98 0.6 75.3 72.1 0.91 72.9 70.6 0.98 0.8 74.9 71.8 0.95 72.9 70.5 1.05 7. Related Work 7.1. Unlearning for LLM-based Recommendation. Traditional recommendation unlearning methods, such as RecEraser (chen2022recommendation) and AltEraser (liu2022forgetting), focus on partitioning collaborative data to safeguard privacy but are ill-suited for the massive parameter space of Large Language Models (LLMs). Conversely, general LLM unlearning often relies on approximate methods like gradient ascent (yao2024large) or in-context label flipping, which frequently trigger catastrophic forgetting and degrade recommendation utility. To mitigate these issues, recent LLMRec-specific frameworks have adopted parameter-efficient fine-tuning (PEFT) (zhang2025parameter). E2URec(wang2025towards) introduces a teacher-student architecture to guide the unlearning process via minimal LoRA updates, while the Adapter Partition and Aggregation (APA) framework employs data sharding and adapter retraining to achieve exact unlearning. However, these methods typically treat the model as a black box and apply uniform updates, leading to gradient conflicts between forgetting and retaining objectives. Our framework, CURE, addresses this by moving beyond uniform updates to a more granular, component-specific optimization strategy. 7.2. Circuit Discovery Circuit discovery is the task of identifying sparse, functional subgraphs within a neural network that are causally responsible for implementing specific capabilities or behaviors (conmy2023towards). This paradigm shifts the focus from global parameter analysis to localizing the "essential computation" for a particular task. Early manual investigations, such as those by (wang2022interpretability) and (hanna2023does), utilized causal mediation analysis and activation patching to uncover circuits for indirect object identification and mathematical reasoning in small-scale models. However, the manual search space grows exponentially with model depth, leading to the development of automated frameworks. ACDC (conmy2023towards) automates circuit identification by recursively pruning edges based on their contribution to model faithfulness. While effective, its reliance on iterative testing renders it too computationally expensive for the scale of modern LLMs. To overcome this, Subnetwork Probing (cao2021low) treats circuit identification as a mask-learning problem, optimizing for both fidelity and sparsity. More recently, Edge Attribution Patching (EAP) (syed2024attribution) has emerged as a high-efficiency alternative, leveraging gradient-based importance scores to approximate the effect of interventions with minimal forward and backward passes. While these techniques have primarily been applied to linguistic benchmarks, we leverage circuit discovery techniques to extract the core circuits underlying item recommendation in LLM based recommendation. This structural decomposition provides the necessary transparency to disentangle the model into forget-specific and retain-specific modules (cheng2023gnndelete; fan2023salun; cheng2023multimodal), bridging the gap between mechanistic interpretability and trustworthy unlearning in recommender systems. 7.3. Gradient Conflicts in Unlearning The optimization of unlearning objectives often mirrors the challenges of multi-task learning, where a model must simultaneously satisfy competing goals. In the context of unlearning, a fundamental tension exists between the forgetting objective (erasing specific data) and the retaining objective (maintaining model utility) (patel2025learning). Previous studies have identified that these dual goals frequently lead to detrimental gradient interference, where the update direction for one task adversely impacts the other (yu2020gradient). While PCGrad (yu2020gradient) introduced a model-agnostic “gradient surgery” approach—projecting conflicting gradients onto their respective normal planes—this method does not account for the unique functional structure of LLMs. Recent analysis in BLUR (reisizadeh2025blur) shows that these gradient conflicts are even more severe in Large Language Models (LLMs). This complexity requires specialized optimization strategies to ensure that forgetting specific data does not accidentally damage the model’s performance on remaining tasks. Our work, CURE, builds on these insights by using circuit discovery to physically isolate the parameters where these conflicts occur, allowing for a more targeted resolution than previous gradient-projection methods. 8. Conclusion In this paper, we propose CURE, a circuit-aware unlearning framework that addresses the challenges of gradient conflicts in LLMRec unlearning. By leveraging mechanistic interpretability to disentangle computational circuits into functionally distinct modules, CURE enables precise, task-specific parameter updates that effectively remove sensitive information while preserving model utility. Our evaluation demonstrates that CURE outperforms state-of-the-art baselines with an 18%18\% improvement in unlearning efficiency and a 6%6\% gain in utility, while achieving a 3.5×3.5× speedup. By shifting from black-box updates to transparent, circuit-level interventions, CURE provides a robust and efficient solution for privacy-preserving recommendation. References