Paper deep dive

COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluation

Ruike Cao, Fugen Yao, Liang Dong, Jian Xu, Guanjun Jiang, Li Xiao

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 101

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/24/2026, 4:16:12 AM

Summary

The paper introduces COPE, a framework for continual personalization of Large Language Models (LLMs) under sparse user feedback. COPE utilizes learnable personalized user embeddings and a self-evaluation mechanism to generate proxy rewards, enabling continuous model updates without relying on constant explicit user feedback. It integrates supervised fine-tuning for preference capture, reinforcement learning for self-evaluation calibration, and reinforcement learning for personalized response optimization within a unified update step.

Entities (6)

Large Language Models · technology · 99%COPE · method · 98%PersonaLens · dataset · 92%Proximal Policy Optimization · algorithm · 90%Retrieval-Augmented Prompting · method · 90%Reinforcement Learning from Human Feedback · method · 88%

Relation Signals (6)

COPE → uses → learnable personalized embeddings

confidence 95% · Our framework assigns learnable personalized embeddings to each user and synergistically integrates preference capture, self-evaluation calibration, and personalized response optimization within a single update step.

COPE → uses → self-evaluation

confidence 95% · A key innovation of our method is the use of self-evaluation to generate proxy rewards, enabling continuous model updates even when explicit user feedback is unavailable.

PersonaLens → usedby → COPE

confidence 92% · we employ the PersonaLens Zhao et al. [38] dataset to instantiate the personalized interaction scenario defined in Section 2.

COPE → outperforms → training-free baselines

confidence 90% · Experiments show that COPE consistently outperforms strong training-free and training-based baselines under sparse feedback

COPE → outperforms → training-based baselines

confidence 90% · Experiments show that COPE consistently outperforms strong training-free and training-based baselines under sparse feedback

COPE → iscomplementaryto → Retrieval-Augmented Prompting

confidence 85% · Experiments show that COPE consistently outperforms strong training-free and training-based baselines under sparse feedback, and remains complementary to Retrieval-Augmented Prompting (RAP).

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:While Large Language Models (LLMs) have achieved remarkable results across various benchmarks, their alignment with normative values often results in homogenized responses that fail to address diverse user preferences. Existing training-free methods often occupy valuable context windows through prompt engineering, while training-based methods typically remain static post-training, failing to support the continual optimization required in real-world settings. To address these challenges, we propose COPE (Continual Optimization with Personalized embedding and self-Evaluation), a novel optimization framework tailored for real-world-motivated interaction settings with sparse user feedback. Our framework assigns learnable personalized embeddings to each user and synergistically integrates preference capture, self-evaluation calibration, and personalized response optimization within a single update step. A key innovation of our method is the use of self-evaluation to generate proxy rewards, enabling continuous model updates even when explicit user feedback is unavailable. Experiments show that COPE consistently outperforms strong training-free and training-based baselines under sparse feedback, and remains complementary to Retrieval-Augmented Prompting (RAP). Further analyses confirm COPE's reliable self-evaluation, meaningful preference patterns, stable general capabilities, and robustness under shifting preferences and alternative evaluators.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

97,348 characters extracted from source content.

Expand or collapse full text

*Corresponding author. COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluation Ruike Cao Affiliation: University of Science and Technology of China Affiliation: Qwen Business Unit of Alibaba Fugen Yao Affiliation: Qwen Business Unit of Alibaba Liang Dong Affiliation: Qwen Business Unit of Alibaba Jian Xu Affiliation: Qwen Business Unit of Alibaba Guanjun Jiang Affiliation: Qwen Business Unit of Alibaba Li Xiao Affiliation: University of Science and Technology of China Abstract While Large Language Models (LLMs) have achieved remarkable results across various benchmarks, their alignment with normative values often results in homogenized responses that fail to address diverse user preferences. Existing training-free methods often occupy valuable context windows through prompt engineering, while training-based methods typically remain static post-training, failing to support the continual optimization required in real-world settings. To address these challenges, we propose COPE (Continual Optimization with Personalized embedding and self-Evaluation), a novel optimization framework tailored for real-world-motivated interaction settings with sparse user feedback. Our framework assigns learnable personalized embeddings to each user and synergistically integrates preference capture, self-evaluation calibration, and personalized response optimization within a single update step. A key innovation of our method is the use of self-evaluation to generate proxy rewards, enabling continuous model updates even when explicit user feedback is unavailable. Experiments show that COPE consistently outperforms strong training-free and training-based baselines under sparse feedback, and remains complementary to Retrieval-Augmented Prompting (RAP). Further analyses confirm COPE’s reliable self-evaluation, meaningful preference patterns, stable general capabilities, and robustness under shifting preferences and alternative evaluators.11 1 Code, data, and prompts are available at https://github.com/Quark-Medical/COPE. 00footnotetext: Corresponding author: Fugen Yao (fugen.yfg@alibaba-inc.com). 1 Introduction In recent years, Large Language Models (LLMs) have achieved rapid advances across challenging benchmarks [34, 31, 7] and are increasingly used as everyday assistants [20]. Beyond general capabilities in knowledge answering and reasoning, user adoption also depends on whether responses reflect individual preferences [10]. However, existing LLMs are commonly aligned through Reinforcement Learning from Human Feedback (RLHF) [23] on carefully curated normative preferences, implicitly assuming a shared value system across users [2, 29]. As a result, they tend to produce homogenized responses that reflect the “average” of group preferences and may underrepresent minority preferences [24, 4]. To address this limitation, prior work has explored both training-free and training-based personalization. Training-free methods mainly rely on in-context signals, such as retrieved dialogue histories in Retrieval-Augmented Prompting (RAP) [26, 30], summarized user profiles in Profile-Augmented Prompting (PAP) [18, 25], or user-specified preferences [15]. Although these methods avoid model updates, they consume valuable context space and may suffer from retrieval noise or information loss during summarization [16]. Training-based approaches learn personalization through optimization, including Multi-Objective Reinforcement Learning (MORL) over predefined preference dimensions [33, 40], user-specific PEFT modules [36, 17], and reinforcement learning based interaction modeling such as RLPA [37]. However, MORL and RLPA rely on predefined preference dimensions or discrete user types, PEFT-style methods face data and scaling constraints, and most remain static after offline training, limiting adaptation to evolving user preferences [16]. To overcome these limitations, we consider a real-world-motivated personalized interaction setting in which a diverse population of users queries an assistant and expects responses aligned with their individual preferences, while providing explicit feedback only sparsely. We propose COPE (Continual Optimization with Personalized embedding and self-Evaluation), which assigns each user a learnable personalized embedding to guide user-specific response generation. COPE optimizes the assistant through a unified update step that integrates three complementary objectives: Supervised Fine-Tuning for Preference Capture, which uses available feedback to encode user preferences into personalized embeddings; Reinforcement Learning for Self-Evaluation Calibration, which aligns the model’s own feedback prediction with real user feedback and learns proxy rewards; and Reinforcement Learning for Personalized Response Optimization, which improves response generation using real feedback when available and proxy rewards otherwise. Executed as the optimization phase of the “Interact-Collect-Optimize” loop, this update enables continuous model optimization under sparse feedback. Appendix A further discusses how COPE relates to and differs from recent user-embedding, prefix-based, and feedback-driven personalization methods [22, 13, 14, 19]. Our contributions are summarized as follows: 1. We establish a real-world-motivated setting for continual personalization under sparse user feedback and instantiate it on PersonaLens with a controlled chronological traversal protocol, bridging static personalization benchmarks with controlled evaluation of continual model optimization. 2. We propose COPE, a continual optimization framework that maintains learnable personalized embeddings as compact user-specific preference representations and updates these embeddings together with the assistant model in the “Interact-Collect-Optimize” loop. Calibrated self-evaluation supplies proxy rewards for interactions without explicit feedback, allowing the optimization loop to keep using collected data when real user feedback is sparse. 3. Experiments show that COPE outperforms several strong training-free and training-based baselines across feedback probabilities and remains complementary to retrieval-based prompting. Analyses further show that the self-evaluation aligns with user feedback, the learned embeddings capture meaningful preference patterns, the optimized model largely preserves general capabilities and remains effective under preference shifts, and the gains hold under alternative evaluators. Figure 1: Illustration of the dynamic interaction scenario and the iterative “Interact-Collect-Optimize” loop. (a) Interaction & Sparse Feedback: During session t, for each user, the assistant πθt _ _t generates a response to the query, guided by the user-specific personalized embedding. If it fails to align with the user’s hidden preferences ηi _i, explicit feedback (textual fitf_i^t and numerical sits_i^t) is provided sparsely with probability p. (b) Data Collection & Optimization: Interaction data 1tD_1^t (without feedback) and 2tD_2^t (with feedback) are collected to iteratively optimize the model. (c) Subsequent Interaction & Improvement: When encountering a query involving the similar preference in subsequent sessions t+kt+k, the updated assistant πθt+k _ _t+k successfully incorporates learned preferences to provide a personalized response. 2 Setup To effectively evaluate and optimize the personalization capabilities of the assistant model, we define a real-world-motivated interaction scenario for continual personalization. As shown in Figure 1, a personalized assistant πθ _θ interacts with a group of N users U=u1,u2,…,uNU=\u_1,u_2,…,u_N\, each with distinct preferences. Each user engages in a total of T dialogue sessions with the assistant, where each session consists of a single-turn interaction. Appendix D details how the same framework can be extended to multi-turn sessions with session-level feedback. Specifically, for the t-th session, we denote the current model parameters as θt _t, and user uiu_i poses a query qitq_i^t to request task completion, where the task associated with the query belongs to a specific domain in the domain set X. The assistant then generates a response ait=πθt(qit)a_i^t= _ _t(q_i^t). A key aspect of this scenario is the modeling of latent user preferences and sparse user feedback. Each user uiu_i is associated with a latent preference profile ηi _i, hidden from the assistant, which can be expressed as a concise description of user preferences across domains. When the response aita_i^t does not fully align with ηi _i, the user experiences a reduced level of satisfaction. To reflect the sparsity of real-world feedback, we introduce a feedback probability p: after each interaction, users provide explicit feedback only with probability p. When provided, feedback consists of textual feedback fitf_i^t, which provides a qualitative evaluation of the response with respect to the user’s preferences, including both positive aspects and deficiencies, and a numerical score sits_i^t indicating the overall degree of satisfaction. The goal of this scenario is to support continuous personalized optimization. At early stages, the assistant lacks knowledge of user preferences and produces weakly personalized responses. By collecting interaction tuples (qit,ait)(q_i^t,a_i^t) along with sparse feedback (fit,sit)(f_i^t,s_i^t) when available, we apply the optimization procedure described in Section 3 to update the assistant from πθt _ _t to πθt+1 _ _t+1. Over time, the assistant is expected to internalize user preferences and generalize them across related tasks. 3 Methodology In this section, we elaborate on the optimization phase within the “Interact-Collect-Optimize” loop, detailing how the interaction data collected in each session is leveraged to continuously optimize the assistant’s personalization performance. Formally, we associate each user uiu_i with a learnable personalized embedding i∈ℝl×dv_i ^l× d, where l corresponds to the number of virtual tokens allocated to capture user-specific preferences and d denotes the dimensionality of the model’s token embeddings. These embeddings constitute a dynamically maintained collection of user-specific parameters iteratively updated through the “Interact-Collect-Optimize” loop, where the set at the beginning of the t-th session is denoted as t=1t,2t,…,NtV_t=\v_1^t,v_2^t,…,v_N^t\, with the initial set 1V_1 randomly initialized prior to the first session. Specifically, for session t, given the current model parameters θt _t and the personalized embeddings tV_t, the assistant interacts with users and generates responses. We denote Ω1t _1^t as users without feedback and Ω2t _2^t as users with feedback, such that |Ω1t|+|Ω2t|=N| _1^t|+| _2^t|=N, yielding datasets 1t=(qit,ait)i∈Ω1tD_1^t=\(q_i^t,a_i^t)\_i∈ _1^t and 2t=(qit,ait,fit,sit)i∈Ω2tD_2^t=\(q_i^t,a_i^t,f_i^t,s_i^t)\_i∈ _2^t. Subsequently, we leverage this collected data to update both the model parameters and the personalized embeddings by simultaneously optimizing three distinct objectives within a unified optimization step. 3.1 Supervised Fine-Tuning for Preference Capture To effectively capture the personalized preferences of user uiu_i while saving context space, we utilize the corresponding personalized embedding itv_i^t to capture the user’s implicit preferences ηi _i. When valid feedback is available (i.e., for dataset 2tD_2^t), we treat the textual feedback fitf_i^t as a glimpse into the user’s implicit preferences ηi _i, manifested as the user’s evaluation of the model’s response aita_i^t to the query qitq_i^t. Formally, given the query qitq_i^t and the response aita_i^t, we prepend the personalized embedding itv_i^t to the sequence of text embeddings and train the personalized embedding to maximize the likelihood of generating the textual feedback fitf_i^t. The goal of this supervised fine-tuning objective is to minimize the following loss: ℒSFT=−i∈Ω2t[ _SFT=-E_i∈ _2^t [ ∑k=1|fit|logπθt(fi,kt∣it,qit, _k=1^|f_i^t| _ _t (f_i,k^t _i^t,q_i^t, . (1) ait,fi,<kt)], .a_i^t,f_i,<k^t ) ], where fi,ktf_i,k^t denotes the k-th token of the textual feedback fitf_i^t. In this objective, we exclusively calculate gradients for the personalized embedding itv_i^t, while treating the model parameters θt _t as frozen. This forces the preference information contained in the textual feedback to flow into the personalized embedding. 3.2 Reinforcement Learning for Self-Evaluation Calibration The primary obstacle in optimizing personalization is the sparsity of user feedback in real-world scenarios. When explicit feedback is absent, the lack of supervisory signals prevents optimizing the model’s personalized responses, resulting in underutilization of valuable interaction data. To address this, we train the assistant model to act as a self-evaluator. Specifically, we perform self-evaluation on the model’s response aita_i^t given qitq_i^t, with the personalized embedding itv_i^t prepended to the text embeddings to guide this process. The model first generates the self-reflection, speculating on the user’s textual feedback, and subsequently predicts the numerical feedback score s^it s_i^t. To facilitate learning, we introduce a teacher-forcing strategy: with a probability q, we include the textual feedback fitf_i^t in the input context. This exposure allows the model to adjust its evaluation logic to align with the user’s actual standards. We design a composite reward function Reval(⋅)R_eval(·) consisting of a sparse binary match reward ri,bintr_i,bin^t and a normalized dense distance-based reward ri,disttr_i,dist^t: Reval(s^it,sit)=ri,bint+ri,distt,R_eval( s_i^t,s_i^t)=r_i,bin^t+r_i,dist^t, (2) where the two components are calculated as: ri,bint=1,if s^it=sit0,otherwise r_i,bin^t= cases1,&if s_i^t=s_i^t\\ 0,&otherwise cases (3) ri,distt=Cmax−|s^it−sit|Cmax. r_i,dist^t= C_ -| s_i^t-s_i^t|C_ . Here, CmaxC_ denotes the maximum possible score value (i.e., the upper bound of the score range). We incorporate ri,disttr_i,dist^t alongside the binary reward because the binary signal alone is too sparse, especially early in training when predictions are inaccurate, providing little learning signal for effective policy optimization. The distance-based term, normalized by CmaxC_ to align its scale with ri,bintr_i,bin^t, supplies a dense reward signal that guides the predicted score toward the true score, while ri,bintr_i,bin^t encourages precise alignment once predictions are sufficiently close. The goal of this reinforcement learning objective is to maximize the expected reward defined as: eval=i∈Ω2t[Reval(s^it,sit)−βKL(πθt||πref)],J_eval=E_i∈ _2^t [R_eval( s_i^t,s_i^t)- ( _ _t|| _ref) ], (4) where πref _ref is the original policy without any optimization (i.e., πθ1 _ _1). In this objective, we compute gradients for both the personalized embeddings and the assistant model: backpropagation refines itv_i^t to better capture user preferences, while simultaneously calibrating the model to score its own responses according to user standards. As a result, for users whose prior feedback has helped calibrate self-evaluation, the model can generate proxy rewards via self-evaluation when explicit feedback is unavailable in subsequent interactions, mitigating feedback sparsity, maximizing the utilization of all interaction data, and improving learning efficiency. 3.3 Reinforcement Learning for Personalized Response Optimization This objective focuses on optimizing the generation of response aita_i^t given the query qitq_i^t under the guidance of personalized embeddings, addressing sparse feedback by leveraging the self-evaluation capabilities established in the preceding “Interact-Collect-Optimize” loop. To prevent the degradation of the model’s general capabilities, we define a composite reward composed of two distinct factors: completeness and personalization. First, to assess whether the assistant’s response fulfills the task requirements specified by the user query, we employ an extra evaluator (implemented via Qwen-Flash Alibaba Cloud [1]). This evaluator provides a binary completeness reward ri,compt∈0,1r_i,comp^t∈\0,1\, where 11 indicates that the response is sufficiently comprehensive to be considered complete and 00 indicates missing key points. This evaluation focuses strictly on task completion and is independent of user preference alignment. Second, for the personalization component, we utilize the hybrid signal derived from real or proxy feedback. We denote this personalization score as ri,perstr_i,pers^t: ri,perst=s^it,if i∈Ω1t(Proxy Feedback)sit,if i∈Ω2t(Real Feedback).r_i,pers^t= cases s_i^t,&if i∈ _1^t (Proxy Feedback)\\ s_i^t,&if i∈ _2^t (Real Feedback) cases. (5) The final reward ritr_i^t is the product: rit=ri,compt⋅ri,perstr_i^t=r_i,comp^t· r_i,pers^t, ensuring the model prioritizes completeness as a prerequisite for personalization, as responses that fail the task receive no policy reward even if they match user preferences, reducing the incentive to trade off task completion for personalized style. We then define the goal of this reinforcement learning objective as maximizing the following expected reward: policy=i∈Ω1t∪Ω2t[rit−βKL(πθt||πref)],J_policy=E_i∈ _1^t∪ _2^t [r_i^t- ( _ _t|| _ref) ], (6) where πref _ref is the original policy without any optimization (i.e., πθ1 _ _1). In this objective, we calculate gradients exclusively for the model parameters θt _t, treating the personalized embedding itv_i^t as frozen. This encourages the model to utilize the preference representations captured in personalized embedding to generate personalized responses. 3.4 Synergistic Updating Having established the three optimization objectives above, to prevent off-policy issues caused by sequential optimization of these objectives, we aggregate the gradients from all three objectives to update the model and personalized embedding parameters simultaneously. Specifically, the gradients for Preference Capture (SFT) are derived via backpropagation from the loss in Eq. 1. Meanwhile, the gradients for Self-Evaluation Calibration (RL) and Personalized Response Optimization (RL) are derived by maximizing the objectives shown in Eq. 4 and Eq. 6, respectively, via the Proximal Policy Optimization (PPO) algorithm Schulman et al. [28] (detailed in Appendix B). Finally, we aggregate the gradients of these three objectives to perform the parameter updates. This ensures that both Self-Evaluation Calibration and Personalized Response Optimization are optimized using data sampled from the current policy (on-policy), thereby stabilizing the learning process. For clarity, the overall optimization process is summarized in Algorithm 1, with a schematic illustration of the personalized embedding integration and objective computation provided in Appendix E. 4 Experiments and Results 4.1 Experimental Setup Considering user privacy protection and the difficulty of collecting real-world interaction data, we employ the PersonaLens Zhao et al. [38] dataset to instantiate the personalized interaction scenario defined in Section 2. PersonaLens is a public personalization benchmark covering 2020 domains (i.e., ||=20|X|=20), where each domain contains 44 to 55 tasks centered on the user’s preferences within that domain, and all preference types can be fully exposed through a Minimal Task Set (MTS) of no more than 33 tasks. PersonaLens also pre-defines a set of interested domains per user (ranging from 99 to 2020), reflecting that users are not necessarily interested in all domains. To balance domain and user diversity, we select 256256 users, each interested in exactly 1717 domains, which together cover all 2020 domains. We further standardize task configurations by fixing the number of tasks per domain to 44, randomly discarding one non-MTS task in 55-task domains to preserve complete preference exposure. The interaction order is strictly controlled: for each user, the assistant traverses interested domains in alphabetical order, and within each domain, tasks are sampled without replacement with MTS tasks prioritized before the remaining ones. This design guarantees that (1) all preference types within a domain are exposed after the first three traversals, and (2) during the fourth traversal, the remaining task involves only previously observed preferences (illustrated in Appendix F). Accordingly, we perform continual optimization during the first three traversals, updating the model after each of the 5151 interactions (33 tasks across 1717 domains), and reserve the fourth traversal for evaluation. This chronological split ensures that test queries involve already exposed preferences but unseen tasks, enabling a controlled assessment of continual personalization. Performance is measured by averaging the product of the personalization score and a binary completeness score over all users, domains, and interactions, reported as mean± over three runs; Appendix G reports the two component scores separately. 4.2 Comparison Methods To validate the effectiveness of our proposed COPE method, we compare it against several representative baselines. We first include Base, where the assistant directly interacts with the user without additional optimization. For training-free personalization, we compare with Profile-Augmented Prompting (PAP) and Retrieval-Augmented Prompting (RAP), including Sparse Retrieval (SR), Dense Retrieval (DR), and a query-expansion variant of dense retrieval (Q-DR). For training-based personalization, we include Training Profile-Augmented Prompting (T-PAP), which serves to isolate the effect of training from the use of learnable personalized embeddings. In addition, to examine whether COPE is complementary to training-free methods, we report the performance of the COPE-optimized model integrated with SR, denoted as COPE+SR. We also integrate T-PAP with the same SR module (T-PAP+SR) to control for the contribution of retrieval. We evaluate all methods under three feedback probability settings (p∈0.25,0.50,0.75p∈\0.25,0.50,0.75\). Implementation details of COPE and the comparison methods are provided in Appendix H. The experimental results are presented in Table 1. Table 1: Scores (personalization score × binary completeness score, out of 1010) of models optimized by different methods under varying user feedback probabilities (mean±stdmean ). † indicates methods that involve training. Methods User Feedback Probability p=0.25p=0.25 p=0.50p=0.50 p=0.75p=0.75 Base 2.36±0.002.36±0.00 2.36±0.002.36±0.00 2.36±0.002.36±0.00 SR 3.08±0.033.08±0.03 3.09±0.013.09±0.01 3.11±0.043.11±0.04 DR 3.07±0.053.07±0.05 3.15±0.033.15±0.03 3.25±0.023.25±0.02 Q-DR 3.15±0.073.15±0.07 3.36±0.063.36±0.06 3.47±0.063.47±0.06 PAP 3.57±0.033.57±0.03 3.65±0.023.65±0.02 3.69±0.013.69±0.01 T-PAP† 4.45±0.044.45±0.04 4.47±0.014.47±0.01 4.69±0.024.69±0.02 COPE† 4.58±0.064.58±0.06 4.69±0.054.69±0.05 4.74±0.024.74±0.02 T-PAP†+SR 4.76±0.02¯ 4.76±0.02 4.83±0.04¯ 4.83±0.04 4.98±0.03¯ 4.98±0.03 COPE†+SR 5.08±0.035.08±0.03 5.23±0.055.23±0.05 5.36±0.025.36±0.02 4.3 Main Results As presented in Table 1, performance generally improves from training-free to training-based personalization, while combining training with retrieval yields the strongest results, indicating that both richer inference-time context and training-based adaptation improve personalization. Among training-free methods, approaches using stronger inference-time signals perform better: DR generally improves over SR, Q-DR further benefits from query expansion, and PAP achieves the best training-free results by summarizing user history into a coherent profile, though at the cost of additional context and latency. COPE substantially improves over these baselines; for example, at p=0.75p=0.75, it reaches 4.744.74 compared with 3.693.69 for PAP. Its advantage over T-PAP, which uses the same reinforcement learning objectives but no learnable personalized embeddings, shows that the gain is not merely due to training but also to the personalized embedding representation. Finally, COPE+SR achieves the best scores in all settings, peaking at 5.365.36 when p=0.75p=0.75, confirming that learned implicit preferences and retrieved explicit histories provide complementary personalization signals. Adding SR improves T-PAP by 0.290.29–0.360.36 points, but improves COPE by a larger 0.500.50–0.620.62 points. This suggests that T-PAP’s textual summary partially overlaps with retrieved histories, whereas COPE’s compact latent preferences are more complementary to explicit retrieved evidence; accordingly, COPE+SR consistently outperforms T-PAP+SR by 0.320.32–0.400.40 points. Figure 2: Analysis of COPE performance. (a-b) Comparison of personalization and completeness scores against Base across 20 domains. (c-d) Training dynamics of personalization and completeness scores over interaction sessions. Figure 3: (a) Visualization of personalized embedding clusters. (b) Impact of personalized embedding length l on COPE. 4.4 Ablation and Analysis Performance Dynamics. Figure 2(a-b) compares COPE with Base on personalization and completeness across all 2020 domains, showing consistent gains and completeness scores near 1.0. Across sessions (Figure 2(c-d)), completeness rises rapidly within the first 2020 sessions and then stabilizes, whereas personalization improves more gradually before tapering off as additional history provides diminishing marginal information. To examine whether the self-evaluation signal aligns with user feedback, we evaluate the assistant-estimated and user-provided personalization scores at two granularities. At the population level, the Pearson correlation between user-averaged trajectories reaches r=0.81r=0.81 during training and r=0.71±0.06r=0.71± 0.06 on three held-out runs. At the instance level, a pairwise ranking analysis on the validation tasks (Appendix J) shows that the proxy ranks consistently with the user on 72.63%72.63\% of pairs where both produce a directional judgment. These results indicate that the self-evaluation signal tracks user feedback at both the population and individual interaction level. Appendix J.1 provides mechanistic intuition and qualitative examples. Embedding Visualization. To interpret the learned embeddings, we cluster 50V_50 using UMAP McInnes et al. [21] and HDBSCAN Campello et al. [3] (Figure 3(a)). Two adjacent clusters, G1G1 and G2G2, illustrate how spatial relation aligns with preference patterns: both contain high-income users who prefer business or first-class travel, yet they separate by finer sub-preferences, with G1G1 favoring modern aesthetics (e.g., electronic music, competitive gaming) and G2G2 leaning toward traditional tastes (e.g., classical music, chess). This example suggests that the embeddings organize users according to preference patterns beyond coarse demographics. Table 2: Ablation of optimization objectives in COPE (p=0.5p=0.5). ℒSFTL_SFT evalJ_eval policyJ_policy Scores ✗ ✗ ✓ 3.36±0.063.36± 0.06 ✗ ✓ ✓ 4.57±0.044.57± 0.04 ✓ ✓ ✓ 4.69±0.054.69± 0.05 Table 3: Ablation of trainable parameter configurations for each objective in COPE (p=0.5p=0.5). ℒSFTL_SFT evalJ_eval policyJ_policy Scores ,θv,θ ,θv,θ θ 4.18±0.074.18± 0.07 v ,θv,θ ,θv,θ 4.45±0.084.45± 0.08 v θ θ 4.49±0.074.49± 0.07 v v θ 4.53±0.054.53± 0.05 v ,θv,θ θ 4.69±0.054.69± 0.05 Embedding Length. The capacity of personalized embeddings is governed by their length l. Figure 3(b) illustrates the performance of COPE (p=0.5p=0.5) across different l values. While performance improves with l, it exhibits diminishing marginal returns beyond l=4l=4. Balancing performance maximization with parameter efficiency, we select l=10l=10 as the final configuration. Under this setting, the user-specific parameters introduce an overhead of only approximately 1.2×10−51.2× 10^-5 of the total LLM (Qwen3-1.7B Yang et al. [34]) parameters θ per user, ensuring the method remains highly efficient for large-scale deployment. Objective Ablation. We analyze the contribution of each optimization objective in Table 2. Removing either ℒSFTL_SFT or evalJ_eval degrades performance, with the omission of evalJ_eval having the most pronounced impact. Without ℒSFTL_SFT and evalJ_eval, the model fails to explicitly leverage textual and numerical feedback, respectively. Consequently, the personalized embeddings fail to effectively capture user preferences, and the model is constrained to learning exclusively from the subset of data containing explicit feedback (2D_2), thereby substantially curtailing learning efficiency. Trainable parameter configuration Ablation. Table 3 explores the impact of updating different parameter sets (personalized embeddings v and LLM weights θ) for each objective. For ℒSFTL_SFT, additionally updating θ degrades performance (4.184.18), since this objective focuses on injecting textual preference knowledge into v, and updating θ likely disrupts the LLM’s general capabilities. For evalJ_eval, jointly updating v and θ is optimal, as this objective must simultaneously compress preferences into v and calibrate the scoring capability within θ, and updating only one leads to suboptimal alignment. For policyJ_policy, training only θ is most effective, since the model here utilizes the captured personalized information to generate responses; forcing updates on v causes overfitting as no new preference information is introduced, leading to a performance drop (4.454.45). 5 Discussion General Capability Retention. Since continual personalization updates the shared model parameters, a natural concern is whether it degrades the model’s general capabilities. To examine this issue, we evaluate the model after the full optimization phase, i.e., after updating the model across the 5151 interactions from the first three traversals (33 tasks across 1717 domains), and compare it with the original Qwen3-1.7B Yang et al. [34]. As shown in Appendix K, the optimized model maintains general capabilities after continual personalization. These results indicate that COPE improves personalization without sacrificing the model’s general capability, suggesting that the proposed continual optimization does not induce catastrophic forgetting over the evaluated training horizon. Adaptation to Preference Shifts. In realistic deployments, user preferences may evolve rather than remain fixed. To examine whether COPE can adapt under such distribution shifts, we conduct two simulated preference-shift experiments in which the personalized embeddings are carried over across the shift rather than re-initialized. The first simulates shifted preferences within previously seen domains, while the second imposes a more stringent shift to non-overlapping new domains. As shown in Figures 4 and 5, COPE maintains or quickly recovers both personalization and completeness under the within-domain preference shift, and it also rapidly adapts under the more challenging new-domain preference shift despite an immediate post-shift drop. Experimental details are provided in Appendix L. (a) Personalization score. (b) Completeness score. Figure 4: Within-domain preference shift evaluation. The vertical dashed line marks the simulated preference shift. Personalized embeddings are carried over rather than re-initialized after the shift. (a) Personalization score. (b) Completeness score. Figure 5: New-domain preference shift evaluation. The vertical dashed line marks the simulated preference shift. Personalized embeddings are carried over rather than re-initialized after the shift. Scaling to Larger Models. To examine whether COPE scales beyond the Qwen3-1.7B backbone used in our main experiments, we conduct additional experiments with Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct Grattafiori et al. [9] under p=0.5p=0.5. Table 4 retains the original Qwen3-1.7B result for reference. COPE consistently improves over Base across all backbones, while its score within the Qwen3 family increases monotonically from 4.694.69 at 1.7B to 5.495.49 at 8B, demonstrating improved performance with greater backbone capacity. The improvement on Llama-3.1-8B-Instruct shows that COPE remains effective with a non-Qwen assistant, thereby alleviating concerns about same-family coupling between the assistant and the Qwen-Flash simulator and completeness evaluator. Table 4: Model-scaling results with p=0.5p=0.5. Base denotes the corresponding unoptimized model, and COPE denotes the model after continual personalization training. Backbone Base COPE Qwen3-1.7B 2.36±0.002.36± 0.00 4.69±0.054.69± 0.05 Qwen3-4B 3.17±0.003.17± 0.00 5.26±0.045.26± 0.04 Qwen3-8B 3.60±0.013.60± 0.01 5.49±0.035.49± 0.03 Llama-3.1-8B-Instruct 3.20±0.043.20± 0.04 5.16±0.065.16± 0.06 Robustness to Evaluator Choice. To examine whether COPE’s gains depend on the Qwen-family pipeline shared by the user simulator and the rubric-based scoring across training and evaluation, we conduct two complementary checks. First, swapping the user simulator with DeepSeek-V3 DeepSeek-AI et al. [6] under p=0.5p=0.5 and q=0.5q=0.5 yields lower absolute scores but preserves the relative ordering, with COPE still achieving the best performance (Appendix M). Second, we conduct a blinded pairwise comparison between COPE and T-PAP on 204204 aligned validation response pairs using a human annotator; the annotator does not receive the simulator’s rubrics or numerical scores. Results show that the human annotator consistently prefers COPE over T-PAP (Figure 9; Appendix N). Both checks indicate that COPE remains effective under different user-model judging styles and that the observed gains are not driven by closed-loop bias. 6 Conclusion In this work, we proposed COPE, a unified framework that enables continual personalization optimization with learnable embeddings and self-evaluation. Experiments demonstrate that COPE improves personalization under sparse feedback, remains complementary to retrieval-based prompting, and learns personalized embeddings that capture meaningful preference patterns. Additional analyses show that COPE’s self-evaluation aligns with user feedback, the optimized model largely preserves general capabilities and recovers from preference shifts within seen domains and across new domains, and the gains remain robust under alternative evaluators, suggesting that COPE is a practical step toward continually adaptive personalized assistants. Thanks to the complementarity between COPE and retrieval-augmented prompting, a deployment with millions of users can update one user subset at a time while supporting the remaining users through retrieval over their interaction histories, without sacrificing personalization support for users awaiting parametric updates. Future work will aim to extend the framework to more complex multi-domain multi-turn tasks and privacy-preserving evaluations in real-user scenarios. Limitations Due to user-privacy considerations and the limited availability of longitudinal real user–assistant interaction data, our current evaluation uses benchmark-derived simulated users from PersonaLens. This design enables controlled and reproducible evaluation under sparse feedback without collecting sensitive real-user interaction traces. Future work can complement this controlled evidence with privacy-preserving real-user evaluation and additional datasets. Our chronological protocol already spans 5151 interactions per user during optimization; evaluating COPE over still longer horizons would provide a valuable further stress test. The current evaluation is also limited to single-turn sessions. Appendix D provides a concrete extension of COPE to multi-turn sessions with session-level feedback, while empirical validation on privacy-preserving multi-turn data remains future work. Acknowledgments This work was supported by Qwen Business Unit through Alibaba Research Intern Program. References [1] Alibaba Cloud (2026) Alibaba cloud model studio: models. Note: https://w.alibabacloud.com/help/en/model-studio/modelsAccessed: 2026-05-15. Qwen-Flash commercial model documentation Cited by: Appendix M, §H.1, §H.4, Appendix I, §3.3. [2] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan (2022) Constitutional ai: harmlessness from ai feedback. External Links: 2212.08073, Link Cited by: §1. [3] R. J. Campello, D. Moulavi, and J. Sander (2013) Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, p. 160–172. Cited by: §4.4. [4] D. Chen, Y. Chen, A. Rege, Z. Wang, and R. K. Vinayak (2025) PAL: sample-efficient personalized reward modeling for pluralistic alignment. In The Thirteenth International Conference on Learning Representations, Cited by: §1. [5] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Appendix K. [6] DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Wang, J. Chen, J. Chen, J. Yuan, J. Qiu, J. Li, J. Song, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Xu, L. Xia, L. Zhao, L. Wang, L. Zhang, M. Li, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, N. Tian, P. Huang, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Chen, Q. Du, R. J. Chen, R. L. Jin, R. Ge, R. Zhang, R. Pan, R. Wang, R. Xu, R. Zhang, R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Ye, S. Ma, S. Wang, S. Zhou, S. Yu, S. Zhou, S. Pan, T. Wang, T. Yun, T. Pei, T. Sun, W. L. Xiao, W. Zeng, W. Zhao, W. An, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, X. Q. Li, X. Jin, X. Wang, X. Bi, X. Liu, X. Wang, X. Shen, X. Chen, X. Zhang, X. Chen, X. Nie, X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yu, X. Song, X. Shan, X. Zhou, X. Yang, X. Li, X. Su, X. Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Y. Zhang, Y. Xu, Y. Xu, Y. Huang, Y. Li, Y. Zhao, Y. Sun, Y. Li, Y. Wang, Y. Yu, Y. Zheng, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Tang, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Wu, Y. Ou, Y. Zhu, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Zha, Y. Xiong, Y. Ma, Y. Yan, Y. Luo, Y. You, Y. Liu, Y. Zhou, Z. F. Wu, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Huang, Z. Zhang, Z. Xie, Z. Zhang, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Xu, Z. Wu, Z. Zhang, Z. Li, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Gao, and Z. Pan (2025) DeepSeek-v3 technical report. External Links: 2412.19437, Link Cited by: Appendix M, §5. [7] DeepSeek-AI, A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Lu, C. Zhao, C. Deng, C. Xu, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, E. Li, F. Zhou, F. Lin, F. Dai, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Li, H. Liang, H. Wei, H. Zhang, H. Luo, H. Ji, H. Ding, H. Tang, H. Cao, H. Gao, H. Qu, H. Zeng, J. Huang, J. Li, J. Xu, J. Hu, J. Chen, J. Xiang, J. Yuan, J. Cheng, J. Zhu, J. Ran, J. Jiang, J. Qiu, J. Li, J. Song, K. Dong, K. Gao, K. Guan, K. Huang, K. Zhou, K. Huang, K. Yu, L. Wang, L. Zhang, L. Wang, L. Zhao, L. Yin, L. Guo, L. Luo, L. Ma, L. Wang, L. Zhang, M. S. Di, M. Y. Xu, M. Zhang, M. Zhang, M. Tang, M. Zhou, P. Huang, P. Cong, P. Wang, Q. Wang, Q. Zhu, Q. Li, Q. Chen, Q. Du, R. Xu, R. Ge, R. Zhang, R. Pan, R. Wang, R. Yin, R. Xu, R. Shen, R. Zhang, S. H. Liu, S. Lu, S. Zhou, S. Chen, S. Cai, S. Chen, S. Hu, S. Liu, S. Hu, S. Ma, S. Wang, S. Yu, S. Zhou, S. Pan, S. Zhou, T. Ni, T. Yun, T. Pei, T. Ye, T. Yue, W. Zeng, W. Liu, W. Liang, W. Pang, W. Luo, W. Gao, W. Zhang, X. Gao, X. Wang, X. Bi, X. Liu, X. Wang, X. Chen, X. Zhang, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yu, X. Li, X. Yang, X. Li, X. Chen, X. Su, X. Pan, X. Lin, X. Fu, Y. Q. Wang, Y. Zhang, Y. Xu, Y. Ma, Y. Li, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Qian, Y. Yu, Y. Zhang, Y. Ding, Y. Shi, Y. Xiong, Y. He, Y. Zhou, Y. Zhong, Y. Piao, Y. Wang, Y. Chen, Y. Tan, Y. Wei, Y. Ma, Y. Liu, Y. Yang, Y. Guo, Y. Wu, Y. Wu, Y. Cheng, Y. Ou, Y. Xu, Y. Wang, Y. Gong, Y. Wu, Y. Zou, Y. Li, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Z. F. Wu, Z. Z. Ren, Z. Zhao, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Huang, Z. Wu, Z. Li, Z. Zhang, Z. Xu, Z. Wang, Z. Gu, Z. Zhu, Z. Li, Z. Zhang, Z. Xie, Z. Gao, Z. Pan, Z. Yao, B. Feng, H. Li, J. L. Cai, J. Ni, L. Xu, M. Li, N. Tian, R. J. Chen, R. L. Jin, S. S. Li, S. Zhou, T. Sun, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Song, X. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Z. Huang, Z. Xu, Z. Zhang, D. Ji, J. Liang, J. Guo, J. Chen, L. Xia, M. Wang, M. Li, P. Zhang, R. Chen, S. Sun, S. Wu, S. Ye, T. Wang, W. L. Xiao, W. An, X. Wang, X. Sun, X. Wang, Y. Tang, Y. Zha, Z. Zhang, Z. Ju, Z. Zhang, and Z. Qu (2025) DeepSeek-v3.2: pushing the frontier of open large language models. External Links: 2512.02556, Link Cited by: §1. [8] L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou (2024) The language model evaluation harness. Zenodo. External Links: Document, Link Cited by: Appendix K. [9] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, G. (. Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Y. (. Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma (2024) The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §5. [10] J. Guan, J. Wu, J. Li, C. Cheng, and W. Wu (2025) A survey on personalized alignment–the missing piece for large language models in real-world applications. arXiv preprint arXiv:2503.17003. Cited by: §1. [11] D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt (2021) Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR). Cited by: Appendix K. [12] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021) Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR). Cited by: Appendix K. [13] B. Huber, G. Fazelnia, A. Damianou, S. Peleato, M. Lefarov, P. Chandar, M. De Nadai, M. Lalmas, and P. N. Bennett (2025) Embedding-to-prefix: continual personalization with large language models. In AI That Keeps Up: NeurIPS 2025 Workshop on Continual and Compatible Foundation Model Updates, Cited by: Appendix A, §1. [14] S. Kim and J. Kim (2026) SPRInG: continual llm personalization via selective parametric adaptation and retrieval-interpolated generation. arXiv preprint arXiv:2601.09974. Cited by: Appendix A, §1. [15] S. Lee, S. H. Park, S. Kim, and M. Seo (2024) Aligning to thousands of preferences via system message generalization. Advances in Neural Information Processing Systems 37, p. 73783–73829. Cited by: §1. [16] J. Liu, Z. Qiu, Z. Li, Q. Dai, W. Yu, J. Zhu, M. Hu, M. Yang, T. Chua, and I. King (2025) A survey of personalized large language models: progress and future directions. arXiv preprint arXiv:2502.11528. Cited by: §1. [17] J. Liu, W. Yu, Q. Dai, Z. Li, J. Zhu, M. Yang, T. Chua, and I. King (2025) Exploring personalization shifts in representation space of llms. In Knowledgeable Foundation Models at ACL 2025, Cited by: §1. [18] Q. Liu, N. Chen, T. Sakai, and X. Wu (2024) Once: boosting content-based recommendation with both open-and closed-source large language models. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, p. 452–461. Cited by: §1. [19] X. Ma, L. Zhao, W. Yao, H. Di, W. Wang, and J. Li (2026) Personalizing llms with binary feedback: a preference-corrected optimization framework. arXiv preprint arXiv:2605.10043. Cited by: Appendix A, §1. [20] B. T. McDaniel, F. F. Nova, and J. A. Pater (2025) Artificial intelligence in everyday family life: issues, applications, and implications. Family Relations 74 (3), p. 1049–1055. Cited by: §1. [21] L. McInnes, J. Healy, and J. Melville (2018) Umap: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426. Cited by: §4.4. [22] L. Ning, L. Liu, J. Wu, N. Wu, D. Berlowitz, S. Prakash, B. Green, S. O’Banion, and J. Xie (2025) User-llm: efficient llm contextualization with user embeddings. In Companion Proceedings of the ACM on Web Conference 2025, p. 1219–1223. Cited by: Appendix A, §1. [23] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 27730–27744. External Links: Document, Link Cited by: §1. [24] S. Poddar, Y. Wan, H. Ivison, A. Gupta, and N. Jaques (2024) Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems 37, p. 52516–52544. Cited by: §1. [25] Y. Qiu, X. Zhao, Y. Zhang, Y. Bai, W. Wang, H. Cheng, F. Feng, and T. Chua (2025) Measuring what makes you unique: difference-aware user modeling for enhancing llm personalization. arXiv preprint arXiv:2503.02450. Cited by: §1. [26] A. Salemi, S. Mysore, M. Bendersky, and H. Zamani (2024) Lamp: when large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7370–7392. Cited by: §1. [27] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2015) High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. Cited by: Appendix B. [28] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: Appendix B, §3.4. [29] A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell (2023) Distributional preference learning: understanding and accounting for hidden context in rlhf. arXiv preprint arXiv:2312.08358. Cited by: §1. [30] M. Tavakoli, A. Salemi, C. Ye, M. Abdalla, H. Zamani, and J. R. Mitchell (2025) Beyond a million tokens: benchmarking and enhancing long-term memory in llms. arXiv preprint arXiv:2510.27246. Cited by: §1. [31] K. Team, Y. Bai, Y. Bao, Y. Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, Z. Chen, J. Cui, H. Ding, M. Dong, A. Du, C. Du, D. Du, Y. Du, Y. Fan, Y. Feng, K. Fu, B. Gao, C. Gao, H. Gao, P. Gao, T. Gao, Y. Ge, S. Geng, Q. Gu, X. Gu, L. Guan, H. Guo, J. Guo, X. Hao, T. He, W. He, W. He, Y. He, C. Hong, H. Hu, Y. Hu, Z. Hu, W. Huang, Z. Huang, Z. Huang, T. Jiang, Z. Jiang, X. Jin, Y. Kang, G. Lai, C. Li, F. Li, H. Li, M. Li, W. Li, Y. Li, Y. Li, Y. Li, Z. Li, Z. Li, H. Lin, X. Lin, Z. Lin, C. Liu, C. Liu, H. Liu, J. Liu, J. Liu, L. Liu, S. Liu, T. Y. Liu, T. Liu, W. Liu, Y. Liu, Y. Liu, Y. Liu, Y. Liu, Z. Liu, E. Lu, H. Lu, L. Lu, Y. Luo, S. Ma, X. Ma, Y. Ma, S. Mao, J. Mei, X. Men, Y. Miao, S. Pan, Y. Peng, R. Qin, Z. Qin, B. Qu, Z. Shang, L. Shi, S. Shi, F. Song, J. Su, Z. Su, L. Sui, X. Sun, F. Sung, Y. Tai, H. Tang, J. Tao, Q. Teng, C. Tian, C. Wang, D. Wang, F. Wang, H. Wang, H. Wang, J. Wang, J. Wang, J. Wang, S. Wang, S. Wang, S. Wang, X. Wang, Y. Wang, Y. Wang, Y. Wang, Y. Wang, Y. Wang, Z. Wang, Z. Wang, Z. Wang, Z. Wang, C. Wei, Q. Wei, H. Wu, W. Wu, X. Wu, Y. Wu, C. Xiao, J. Xie, X. Xie, W. Xiong, B. Xu, J. Xu, L. H. Xu, L. Xu, S. Xu, W. Xu, X. Xu, Y. Xu, Z. Xu, J. Xu, J. Xu, J. Yan, Y. Yan, H. Yang, X. Yang, Y. Yang, Y. Yang, Z. Yang, Z. Yang, Z. Yang, H. Yao, X. Yao, W. Ye, Z. Ye, B. Yin, L. Yu, E. Yuan, H. Yuan, M. Yuan, S. Yuan, H. Zhan, D. Zhang, H. Zhang, W. Zhang, X. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, Z. Zhang, H. Zhao, Y. Zhao, Z. Zhao, H. Zheng, S. Zheng, L. Zhong, J. Zhou, X. Zhou, Z. Zhou, J. Zhu, Z. Zhu, W. Zhuang, and X. Zu (2026) Kimi k2: open agentic intelligence. External Links: 2507.20534, Link Cited by: §1. [32] A. Trotman, A. Puurula, and B. Burgess (2014) Improvements to bm25 and language models examined. In Proceedings of the 19th Australasian Document Computing Symposium, p. 58–65. Cited by: §H.4. [33] Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi (2023) Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems 36, p. 59008–59033. Cited by: §1. [34] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: Appendix K, §H.1, Appendix I, §1, §4.4, §5. [35] Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, and J. Zhou (2025) Qwen3 embedding: advancing text embedding and reranking through foundation models. External Links: 2506.05176, Link Cited by: §H.4. [36] Y. Zhang, J. Wang, L. Yu, D. Xu, and X. Zhang (2024) Personalized lora for human-centered text understanding. Proceedings of the AAAI Conference on Artificial Intelligence 38 (17), p. 19588–19596. Cited by: §1. [37] W. Zhao, X. Sui, Y. Hu, J. Guo, H. Liu, B. Li, Y. Zhao, B. Qin, and T. Liu (2025) Teaching language models to evolve with users: dynamic profile modeling for personalized alignment. arXiv preprint arXiv:2505.15456. Cited by: §H.2, §1. [38] Z. Zhao, C. Vania, S. Kayal, N. Khan, S. B. Cohen, and E. Yilmaz (2025) PersonaLens: a benchmark for personalization evaluation in conversational AI assistants. In Findings of the Association for Computational Linguistics: ACL 2025, p. 18023–18055. Cited by: §H.2, §4.1. [39] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou (2023) Instruction-following evaluation for large language models. External Links: 2311.07911, Link Cited by: Appendix K. [40] Z. Zhou, J. Liu, J. Shao, X. Yue, C. Yang, W. Ouyang, and Y. Qiao (2024) Beyond one-preference-fits-all alignment: multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, p. 10586–10613. Cited by: §1. Appendix A Related Work and Positioning Recent work has explored user embeddings as a compact interface between user histories and LLMs. User-LLM [22] generates user embeddings from interaction histories and integrates them with LLMs through cross-attention, improving efficiency over text-only user-context prompts. Embedding-to-Prefix [13] maps dynamically updated user embeddings into a soft prefix for a frozen LLM, providing a lightweight mechanism for continual personalization without updating the LLM backbone. These methods show that compact user representations can reduce context overhead and provide useful personalization signals. COPE shares this motivation but differs in how the user representation is acquired and used: rather than relying on precomputed user encoders or frozen-backbone prefix injection, COPE learns per-user embeddings from sparse textual and numerical feedback inside the interaction loop and uses them together with policy optimization. Other recent methods study continual or feedback-based personalization. SPRInG [14] combines selective parametric adaptation with retrieval-interpolated generation to handle preference drift and mitigate forgetting in continual personalization. C-BPO [19] studies binary-signal preference optimization by treating target-user histories as positive signals and other users’ histories as implicit negatives under a preference-corrected objective. These directions are complementary to COPE: they focus on drift-aware semi-parametric adaptation or binary-signal preference optimization, while COPE focuses on sparse-feedback user–assistant interactions where missing feedback is addressed through a self-evaluation-based proxy reward and a unified update step for preference capture, self-evaluation calibration, and personalized response optimization. Appendix B Preliminaries on Proximal Policy Optimization (PPO) In this work, we employ Proximal Policy Optimization (PPO) [28], a widely adopted policy gradient method, to optimize both the Self-Evaluation Calibration and Personalized Response Optimization objectives. PPO operates within the Actor-Critic framework, consisting of a policy network (Actor) πθ _θ that generates actions (responses) and a value network (Critic) VϕV_φ that estimates the expected return of a given state (query and context). Importance Sampling. PPO is an on-policy algorithm that alternates between data collection and policy optimization. In each iteration, a batch of trajectories is first collected using the current policy, referred to as the old policy πθold _ _old. To enable multiple epochs of optimization over the same batch while accounting for the discrepancy between the updated policy and the data-collecting policy, PPO employs importance sampling. We define the probability ratio (importance weight) as: rt(θ)=πθ(at|st)πθold(at|st).r_t(θ)= _θ(a_t|s_t) _ _old(a_t|s_t). (7) Generalized Advantage Estimation (GAE). To reduce the variance of policy gradient estimates, PPO uses the Critic VϕV_φ to estimate the advantage using Generalized Advantage Estimation (GAE) [27]. The advantage A^t A_t measures how much better an action is compared to the average expectation. It is calculated as: A^t A_t =∑k=0∞(γλ)kδt+k, = _k=0^∞(γλ)^k _t+k, (8) where δt _t =rt+γVϕ(st+1)−Vϕ(st). =r_t+γ V_φ(s_t+1)-V_φ(s_t). Here, rtr_t denotes the scalar reward at timestep t, γ is the discount factor, λ is the GAE smoothing parameter, and δt _t represents the temporal difference (TD) error. Value Network Training. In our implementation, we maintain two separate critics: VϕevalV_ _eval for Self-Evaluation Calibration and VϕrespV_ _resp for Personalized Response Optimization. They are initialized from the same assistant model but trained independently using the returns of their corresponding PPO objective. For o∈eval,respo∈\eval,resp\, the critic minimizes the value loss ℒVo(ϕo)=12t[(Vϕo(st)−R^to)2],L_V^o( _o)= 12E_t [(V_ _o(s_t)- R_t^o)^2 ], (9) where R^to R_t^o is the return estimated by GAE; in our terminal-reward setting with γ=λ=1γ=λ=1, it equals the corresponding final reward, i.e., Reval(s^it,sit)R_eval( s_i^t,s_i^t) for Self-Evaluation Calibration and ritr_i^t for Personalized Response Optimization. Critic gradients are optimized separately and are not included in the joint update of the assistant model and personalized embeddings. Optimization Objective. The core of PPO is to maximize a surrogate objective while constraining the policy update to avoid excessive deviation from the behavior policy. In addition to the clipping mechanism, and following standard practice in RLHF-style optimization, we further incorporate a KL divergence penalty with respect to a fixed reference policy πref _ref to prevent the learned policy from drifting too far from the pretrained model. The overall objective is defined as: PPO(θ) _PPO(θ) =t[CLIPt(θ) =E_t [J^CLIP_t(θ) (10) −βKL(πθ(⋅|st)∥πref(⋅|st))]. -β\,KL ( _θ(·|s_t)\,\|\, _ref(·|s_t) ) ]. Specifically, the clipped surrogate objective is given by: tCLIP(θ) ^CLIP_t(θ) =min(rt(θ)A^t, = \! (r_t(θ) A_t, (11) OPENclip⁡(rt(θ),1−ϵ,1+ϵ)A^t), \! (r_t(θ),1-ε,1+ε ) A_t ), where ϵε is a hyperparameter (e.g., 0.20.2) controlling the trust region, and β determines the strength of the KL regularization. Appendix C Optimization Algorithm Algorithm 1 summarizes the complete “Interact-Collect-Optimize” loop of COPE, including sparse feedback collection, the computation of the three optimization objectives, and the joint update of model parameters and personalized embeddings. The two PPO objectives use separate critics initialized from the assistant model. Each critic is trained on the returns of its corresponding trajectories and provides the advantages for that objective’s policy update; critic gradients are optimized separately from the joint actor and personalized-embedding gradients. Algorithm 1 “Interact-Collect-Optimize” Loop of COPE 0: User set U, Initial Model Parameters θ1 _1, Number of Sessions T, Feedback Probability p 1: Initialize personalized embeddings 1V_1 randomly 2: Initialize critics VϕevalV_ _eval and VϕrespV_ _resp from the assistant model 3: for session t=1t=1 to T do 4: // Interaction & Collection 5: Collect queries qiti=1N\q_i^t\_i=1^N 6: Generate responses ait∼πθt(⋅|it,qit)a_i^t _ _t(·|v_i^t,q_i^t) for all users 7: Receive sparse feedback: fit,sitf_i^t,s_i^t with probability p for all users 8: Construct datasets 1tD_1^t (i∈Ω1ti∈ _1^t, no feedback) and 2tD_2^t (i∈Ω2ti∈ _2^t, with feedback), such that |Ω1t|+|Ω2t|=N| _1^t|+| _2^t|=N 9: // Unified Optimization Step 10: Step 1: Compute Gradients for Preference Capture (SFT) 11: Calculate ∇ℒSFT _vL_SFT via Eq. 1 (for i∈Ω2ti∈ _2^t) 12: Step 2: Compute Gradients for Self-Evaluation Calibration (RL) 13: Generate self-reflection and score s^it s_i^t using πθt _ _t and itv_i^t for i∈Ω1t∪Ω2ti∈ _1^t∪ _2^t 14: Compute reward Reval(s^it,sit)R_eval( s_i^t,s_i^t) for i∈Ω2ti∈ _2^t 15: Compute returns and advantages using VϕevalV_ _eval; update ϕeval _eval via Eq. 9 16: Calculate ∇θ,eval _θ,vJ_eval via PPO (for i∈Ω2ti∈ _2^t) 17: Step 3: Compute Gradients for Personalized Response Optimization (RL) 18: Evaluate completeness ri,comptr_i,comp^t and personalization ri,perstr_i,pers^t for i∈Ω1t∪Ω2ti∈ _1^t∪ _2^t 19: Compute final reward rit=ri,compt⋅ri,perstr_i^t=r_i,comp^t· r_i,pers^t for i∈Ω1t∪Ω2ti∈ _1^t∪ _2^t 20: Compute returns and advantages using VϕrespV_ _resp; update ϕresp _resp via Eq. 9 21: Calculate ∇θpolicy _θJ_policy via PPO (for i∈Ω1t∪Ω2ti∈ _1^t∪ _2^t) 22: Step 4: Joint Parameter Update 23: Update θt→θt+1 _t→ _t+1 using aggregated gradients ∇θeval+∇θpolicy _θJ_eval+ _θJ_policy 24: Update it→it+1v_i^t _i^t+1 using aggregated gradients ∇ℒSFT+∇eval _vL_SFT+ _vJ_eval 25: end for Appendix D Extension to Multi-Turn Sessions Our experiments use single-turn sessions because longitudinal multi-turn personalization data are difficult to obtain under user-privacy constraints. Nevertheless, COPE can be extended to multi-turn interactions with session-level feedback without changing its three-objective design. For user uiu_i in session t, we replace the single-turn tuple (qit,ait)(q_i^t,a_i^t) with a dialogue trajectory hit=(qi,kt,ai,kt)k=1K,h_i^t=\(q_i,k^t,a_i,k^t)\_k=1^K, (12) where K is the number of turns in the session. At turn k, the assistant generates ai,kt∼πθt(⋅∣it,hi,<kt,qi,kt),a_i,k^t _ _t (· _i^t,h_i,<k^t,q_i,k^t ), (13) where hi,<kth_i,<k^t is the preceding dialogue history. The same personalized embedding itv_i^t is used across all turns to represent preferences accumulated across sessions, while hi,<kth_i,<k^t provides the local context within the current session. After the final turn, the user provides one textual feedback message fitf_i^t and one overall satisfaction score sits_i^t with probability p. The collected datasets become 1t=hiti∈Ω1tD_1^t=\h_i^t\_i∈ _1^t for sessions without feedback and 2t=(hit,fit,sit)i∈Ω2tD_2^t=\(h_i^t,f_i^t,s_i^t)\_i∈ _2^t for sessions with feedback. For preference capture, the complete dialogue replaces the single query–response pair as the context for predicting the session-level textual feedback: ℒSFTmulti= _SFT^multi= −i∈Ω2t[∑j=1|fit|logπθt(fi,jt∣ -E_i∈ _2^t [ _j=1^|f_i^t| _ _t (f_i,j^t (14) it,hit,fi,<jt)]. _i^t,h_i^t,f_i,<j^t ) ]. As in the single-turn formulation, this objective updates only the personalized embedding. For self-evaluation calibration, the model predicts an overall score s^it s_i^t from (it,hit)(v_i^t,h_i^t), optionally conditioned on fitf_i^t under the same teacher-forcing strategy, and is optimized against sits_i^t using the reward in Eq. 2. Thus, the self-evaluator learns to assess the personalization quality of the complete session rather than a single response. For personalized response optimization, an evaluator first assigns a session-level completeness reward ri,comptr_i,comp^t. The personalization reward is the real session score when feedback is available and the predicted session score otherwise: ri,perst=s^it,i∈Ω1t,sit,i∈Ω2t. r_i,pers^t= cases s_i^t,&i∈ _1^t,\\ s_i^t,&i∈ _2^t. cases (15) ri,sesst=ri,compt⋅ri,perst. r_i,sess^t=r_i,comp^t· r_i,pers^t. The complete dialogue is treated as one trajectory, with ri,sesstr_i,sess^t serving as its terminal reward. For brevity, let ci,kt=(it,hi,<kt,qi,kt)c_i,k^t=(v_i^t,h_i,<k^t,q_i,k^t) denote the personalized context at turn k, and define di,kt=KL(πθt(⋅∣ci,kt)∥πref(⋅∣ci,kt)).d_i,k^t=KL\! ( _ _t(· c_i,k^t)\| _ref(· c_i,k^t) ). (16) The corresponding policy objective is policymulti=i∈Ω1t∪Ω2t[ri,sesst−β∑k=1Kdi,kt].J_policy^multi=E_i∈ _1^t∪ _2^t [r_i,sess^t-β _k=1^Kd_i,k^t ]. (17) The three objectives can then be aggregated in the same unified update described in Section 3. Therefore, the main change is the optimization unit: a complete dialogue session replaces a single interaction, while COPE’s personalized embedding, sparse-feedback mechanism, and joint optimization remain unchanged. Figure 6: Schematic illustration of personalized embedding integration and the forward computation and backward optimization of the three objectives in the optimization stage of COPE. Personalized embeddings are implemented as l virtual tokens and prepended to the text embeddings to provide user-specific guidance. For clarity, the assistant model is conceptually decomposed into an embedding layer and a decoder to explicitly visualize the embedding of textual inputs and their concatenation with personalized embeddings, although both components belong to the same assistant model. Two different system prompts are used: a response-generation prompt (pink) and a self-evaluation prompt (green), both of which are provided in our code repository. Trainable parameters under each objective are marked with spark icons, while frozen parameters are indicated by snowflake icons. Appendix E Illustration of Personalized Embedding Integration and Optimization Flow Figure 6 illustrates the optimization stage of COPE, showing how personalized embeddings are prepended to text embeddings and how the three objectives are computed and optimized. Appendix F Chronological Traversal Protocol Figure 7 illustrates the chronological traversal process used in our experimental setup. Figure 7: Chronological traversal and preference coverage. The experiment proceeds through four traversals: Traversals 11-33 continually expose user preferences via MTS tasks across 1717 domains, while Traversal 44 evaluates generalization on a hold-out task. The illustration shows one user for clarity; all users follow the same protocol in parallel. Appendix G Decomposed Main Results Table 5 reports the completeness and personalization components underlying the product scores in Table 1. Table 5: Decomposed completeness and personalization scores under varying user feedback probabilities (mean±stdmean ). † indicates methods that involve training. Methods User Feedback Probability p=0.25p=0.25 p=0.50p=0.50 p=0.75p=0.75 Completeness Base 0.78±0.000.78±0.00 0.78±0.000.78±0.00 0.78±0.000.78±0.00 SR 0.87±0.010.87±0.01 0.87±0.000.87±0.00 0.87±0.000.87±0.00 DR 0.85±0.000.85±0.00 0.86±0.010.86±0.01 0.86±0.000.86±0.00 Q-DR 0.85±0.010.85±0.01 0.85±0.000.85±0.00 0.85±0.010.85±0.01 PAP 0.90±0.010.90±0.01 0.91±0.000.91±0.00 0.91±0.000.91±0.00 T-PAP† 0.95±0.000.95±0.00 0.95±0.000.95±0.00 0.96±0.000.96±0.00 COPE† 0.98±0.010.98±0.01 0.99±0.000.99±0.00 0.99±0.000.99±0.00 COPE†+SR 0.97±0.000.97±0.00 0.98±0.000.98±0.00 0.98±0.000.98±0.00 Personalization Base 2.69±0.012.69±0.01 2.69±0.012.69±0.01 2.69±0.012.69±0.01 SR 3.35±0.043.35±0.04 3.37±0.013.37±0.01 3.40±0.033.40±0.03 DR 3.36±0.033.36±0.03 3.44±0.023.44±0.02 3.55±0.023.55±0.02 Q-DR 3.51±0.063.51±0.06 3.70±0.043.70±0.04 3.82±0.043.82±0.04 PAP 3.85±0.013.85±0.01 3.93±0.023.93±0.02 3.96±0.003.96±0.00 T-PAP† 4.59±0.034.59±0.03 4.62±0.034.62±0.03 4.83±0.014.83±0.01 COPE† 4.62±0.064.62±0.06 4.73±0.054.73±0.05 4.78±0.024.78±0.02 COPE†+SR 5.20±0.025.20±0.02 5.31±0.055.31±0.05 5.43±0.035.43±0.03 The decomposed results show that optimization improves both task completion and preference alignment. Compared with Base, all personalization methods achieve higher completeness and personalization scores, and training-based methods further strengthen both components. COPE obtains the highest completeness across all feedback probabilities, while COPE+SR achieves the strongest personalization scores and remains close to COPE in completeness. These trends indicate that learned user embeddings and retrieved histories provide complementary signals for improving both dimensions of personalized response quality. Appendix H Implementation Details H.1 COPE Implementation Details We use Qwen3-1.7B Yang et al. [34] as the assistant model and Qwen-Flash Alibaba Cloud [1] as the user simulator in the main experiments. We pre-generate the initial query and rubrics for each user-task instance from the user’s true preferences ηi _i; fixing these artifacts before interaction reduces simulator randomness and ensures that both user behavior and feedback criteria consistently reflect the hidden preferences. We manually inspect the pre-generated query-rubric pairs to check that they reflect the underlying user preferences. Thus, during interaction, the user simulator poses the pre-generated query and, with probability p, provides evaluative feedback on the assistant’s response. Appendix H.2 further discusses the rationale for this simulator-based protocol. During evaluation, we use the pre-generated rubrics to assess the assistant’s response. Each rubric item specifies a condition the response should satisfy and an associated score. Since the rubrics are fixed in advance and manually inspected for consistency with the user’s preferences, the user model only needs to judge the response against these rubrics, enabling more stable feedback that remains aligned with the user’s preferences. The simulator evaluates the response against these rubrics to generate textual feedback fitf_i^t, summarizing satisfied or violated criteria, and a numerical score sits_i^t, computed as the sum of scores for all satisfied items. A concrete example of the interaction, feedback, and rubrics, is provided in Appendix I. To reflect the coarse granularity of real-world feedback, we compress the raw integer feedback (00-1010) into a coarser four-point scale (00-33). Specifically, original scores 0,1,2\0,1,2\ are mapped to 00, 3,4,5\3,4,5\ to 11, 6,7,8\6,7,8\ to 22, and 9,10\9,10\ to 33. Consequently, the self-evaluation mechanism predicts these mapped scores to compute matching rewards, and the mapped scores subsequently serve as the reward signal for personalized response optimization. However, when evaluating model performance, we still report results using the original (pre-mapping) scores. Regarding hyperparameters, we set the learning rate to 1×10−31× 10^-3 for personalized embeddings and 1×10−61× 10^-6 for model parameters, with a KL penalty weight β of 0.010.01. Each personalized embedding consists of 1010 learnable virtual tokens (i.e., l=10l=10) and is randomly initialized prior to the training process. The two critics are independently initialized from the assistant model’s weights and warmed up for 55 steps. The teacher-forcing probability q during self-evaluation calibration is set to 0.250.25. Our algorithm is implemented based on the VeRL framework, with corresponding adaptations to enable efficient inference using SGLang with personalized embedding prefixes. Our implementation, associated datasets, and all prompts used in this study are available at https://github.com/Quark-Medical/COPE. H.2 Rationale for Simulator-Based Evaluation Although our experimental setting is motivated by real user–assistant interactions, recruiting a large number of users to participate in the full continual optimization loop would be costly and would require careful privacy protection, especially when persistent user representations are updated over 5151 chronological interactions. For reproducible and efficient algorithm development, we therefore follow the common practice of using LLM-based user simulation. Specifically, we instantiate users from PersonaLens [38], a public benchmark designed for personalization evaluation in task-oriented conversational assistants, with diverse profiles, domain-specific preferences, interaction histories, situational contexts, and validation of profile consistency, preference distributions, lexical diversity, and user-agent quality. Our simulator is also constrained by pre-generated and manually inspected query-rubric pairs. During interaction, the user simulator only presents the fixed query and judges the response against the fixed rubric when feedback is sampled, rather than freely inventing new preferences or criteria. This makes feedback generation a more stable instruction-following and rubric-checking task, while keeping the evaluation criteria aligned with the hidden profile and comparable across methods. Simulator-based user interaction is also common in recent personalized alignment work such as RLPA [37]. We therefore view this setting as a controlled benchmark-derived approximation of sparse personalized interaction, while leaving real-user validation to future privacy-preserving studies. H.3 Artifacts, Licenses, and Compute We use PersonaLens as a public research benchmark for evaluating personalization, rather than as deployed user data. PersonaLens is released under the C-BY-NC-4.0 license, contains English-language task-oriented conversational data, and provides rich user profiles, interaction histories, and 2020 domains with 44 to 55 tasks per domain. We use Qwen3-1.7B under its Apache-2.0 license. Qwen-Flash and DeepSeek-V3 are accessed through their respective API services for simulation and robustness evaluation. All artifacts are used for research and evaluation purposes in accordance with their documented access conditions. The public repository is intended for research-use reproducibility and will follow release terms compatible with the upstream non-commercial artifacts. We do not collect new real-user interaction data or recruit human participants. Our experiments use benchmark-derived user profiles and simulator-generated interactions. We manually inspect the generated query-rubric pairs for consistency with the underlying preferences and for obvious personally identifying information and unsafe or offensive content. No additional anonymization is applied beyond using benchmark-provided user identifiers and simulator-generated interaction artifacts. Our implementation uses VeRL 0.6.0, SGLang 0.4.9, rank-bm25 for BM25 retrieval, and lm-eval v0.4.3 for general capability evaluation. H.4 Comparison Method Implementation Details For PAP, a separate model (Qwen-Flash Alibaba Cloud [1]) first summarizes the user’s interaction history from the first three domain traversals to build a user profile; the profile is then concatenated with the prompt to guide the assistant model in generating the response. RAP enhances personalization by retrieving historical interactions relevant to the user’s current query. In our experiments, we retrieve the top-66 historical interactions. SR uses BM25 Trotman et al. [32] for sparse retrieval, while DR uses Qwen3-Embedding Zhang et al. [35] to encode both the interaction history and the user’s query and then performs retrieval based on embedding similarity. Q-DR is a query-expansion variant of dense retrieval, where the assistant model first generates an auxiliary intermediate response conditioned on the initial query; this intermediate response is then embedded and used to retrieve the top-66 relevant historical interactions. For T-PAP, we follow PAP by incorporating the user profile summarized by Qwen-Flash Alibaba Cloud [1] as part of the prompt. To ensure a fair comparison with COPE, T-PAP employs the same optimization objectives as COPE, namely Self-Evaluation Calibration (RL) and Personalized Response Optimization (RL), both guided by the summarized user profiles. Since T-PAP does not include learnable personalized embeddings, it omits the Preference Capture (SFT) objective. COPE+SR combines the COPE-optimized model with the same SR module described above, allowing the model to use both learned personalized embeddings and retrieved historical interactions during response generation. Appendix I Interaction Case In this section, we provide an example comprising a single interaction turn between the user simulator (Qwen-Flash Alibaba Cloud [1]) and the assistant (Qwen3-1.7B Yang et al. [34]), along with the corresponding pre-generated initial query and rubrics and the user-generated feedback, in Figure 8. Figure 8: An illustrative example of the interaction between the user simulator and the assistant. The figure shows the pre-generated user query, the assistant’s response, the rubrics, and the user feedback. The feedback includes textual feedback summarizing satisfied or violated criteria, and a numerical feedback computed as the sum of scores for all satisfied items. Appendix J Instance-Level Self-Evaluation Reliability The Pearson correlation reported in Section 4 is computed from step-level aggregates and could in principle hide instance-level disagreements. To examine reliability at the level of individual interactions, we conduct an additional pairwise ranking analysis. Specifically, we randomly select a subset of the held-out validation interactions and run the assistant on each twice, yielding 1,1331,133 pairs of responses where the user provides real feedback on both runs. Both sides are compared in the same 00–33 grade space: the user’s raw 00–1010 score is mapped via the same discretization the assistant is trained against. We discard pairs where the user assigns the same grade to both runs, since these pairs have no ground-truth ordering. The remaining user-ranked pairs are classified into three categories: • Concordant: the proxy ranks the two runs in the same direction as the user. • Discordant: the proxy ranks them in the opposite direction. • Proxy silent: the proxy assigns the same grade to both runs (the user can rank, the proxy cannot). Among the 1,1331,133 pairs, 710710 are discarded as user-tied, leaving 423423 user-ranked pairs (207207 concordant, 7878 discordant, 138138 proxy-silent). The proxy gives a directional judgment on 67.4%67.4\% of these pairs; the remaining 32.6%32.6\% are proxy-silent and reflect the inherent granularity loss of the four-grade discretization. Among the 285285 pairs where the proxy does produce a ranking, its direction agrees with the user’s 72.63%72.63\% of the time. This indicates that, in the cases where the proxy is decisive, it carries non-trivial directional signal at the instance level. J.1 Why Self-Evaluation Calibration Works Self-evaluation calibration combines preference representation and score calibration. Preference Capture SFT encodes textual feedback into each personalized embedding, teacher forcing grounds response properties in user judgments, and PPO jointly updates the embedding and LLM against numerical feedback. Thus, the embedding represents what the user values, while the LLM maps response quality under those preferences to a user-specific score. Supporting this interpretation, adding evalJ_eval and then ℒSFTL_SFT improves performance from 3.363.36 to 4.574.57 and 4.694.69 (Table 2); within evalJ_eval, updating only the LLM or embedding yields 4.494.49 and 4.534.53, versus 4.694.69 when both are updated (Table 3). To complement the aggregate results, we analyze two held-out self-evaluation outputs for which the current textual feedback is unavailable. In one shopping interaction, the self-evaluator recognizes that the response covers the user’s preferred product categories and attributes such as recycled materials, ethical production, and cruelty-free products, and assigns the same highest coarse grade as the mapped user feedback. In a merchandise interaction, it detects that the response recommends San Lorenzo products despite the user’s established preference for Club Universidad de Chile, and assigns the same lowest grade as the mapped feedback. These examples illustrate how the self-evaluator identifies both preference matches and conflicts, showing how training-time feedback and accumulated user preferences can support self-evaluation on unseen interactions. Appendix K General Capability Evaluation To evaluate whether continual personalization causes catastrophic forgetting of general capabilities, we assess the optimized model after the 5151-interaction optimization phase described in Section 4. We use lm-eval Gao et al. [8] to evaluate instruction following with IFEval Zhou et al. [39], mathematical reasoning with GSM8K Cobbe et al. [5], and multi-domain knowledge understanding with MMLU Hendrycks et al. [12], Hendrycks et al. [11]. Table 6 compares the optimized model with the original Qwen3-1.7B Yang et al. [34]. Table 6: General capability evaluation before and after continual personalization. Benchmark Metric Qwen3-1.7B Optimized model IFEval inst_level_loose_acc 73.5073.50 74.5874.58 inst_level_strict_acc 70.5070.50 71.1071.10 prompt_level_loose_acc 65.43±2.0565.43±2.05 67.47±2.0267.47±2.02 prompt_level_strict_acc 61.74±2.0961.74±2.09 62.66±2.0862.66±2.08 GSM8K flexible-extract 68.69±1.2868.69±1.28 69.37±1.2769.37±1.27 strict-match 68.31±1.2868.31±1.28 69.14±1.2769.14±1.27 MMLU humanities 52.73±0.7052.73±0.70 52.50±0.7052.50±0.70 other 64.15±0.8464.15±0.84 64.11±0.8464.11±0.84 social sciences 68.28±0.8268.28±0.82 67.99±0.8367.99±0.83 stem 59.21±0.8559.21±0.85 59.02±0.8559.02±0.85 overall 60.12±0.4060.12±0.40 59.93±0.4059.93±0.40 The optimized model shows small numerical increases on IFEval and GSM8K, while the MMLU overall score decreases only marginally from 60.1260.12 to 59.9359.93. However, these changes are all within one standard deviation of the corresponding benchmark results, so we interpret them as comparable performance rather than evidence of broad general-capability improvement or degradation. Overall, COPE’s continual personalization improves personalized behavior while maintaining general instruction-following, reasoning, and knowledge understanding capabilities in this setting, suggesting that it does not induce catastrophic forgetting over the evaluated training horizon. Appendix L Preference Shift Adaptation Evaluation Real-world users may change their preferences over time. We therefore evaluate COPE under two simulated preference-shift settings. Both settings use the same 256256 users as in the main experiments: the within-domain shift splits these users into two groups while keeping the domain coverage fixed, and the new-domain shift samples its two 6464-user groups from non-overlapping domain sets. In both settings, the model continues training after the simulated shift, and the personalized embedding associated with each user slot is carried over and further updated rather than re-initialized. This tests whether the existing user representation can be revised when the underlying preference distribution changes. For the curves, checkpoints before the shift are evaluated on the validation tasks of the pre-shift users, while checkpoints after the shift are evaluated on the validation tasks of the post-shift users. Each checkpoint is evaluated three times, and the plotted point and error bar report the mean and standard deviation across these validation runs. In the within-domain shift setting, we randomly split the users into two groups of 128128 users. During the first phase, COPE is trained for 5151 interactions using tasks from the first user group. After this point, each user slot is switched to a user from the second group to simulate a shift in the underlying preference distribution, and training continues with the same personalized embedding. As shown in Figure 4, the personalization score decreases slightly after the shift, but quickly recovers and continues to improve with further training. In contrast, the completeness score is largely unaffected by the shift and remains at a high level throughout the second phase. These results indicate that COPE can revise the carried-over personalized embeddings without losing task completeness when preferences shift within the seen domain coverage. In the new-domain shift setting, we further stress-test adaptation to new preference domains. We first randomly choose 88 domains and sample 6464 users interested in these domains. We then choose another 88 domains that do not overlap with the first set and sample another 6464 users interested in the new domains. COPE is trained for 2424 interactions on the first group and then continues training on the second group using the carried-over personalized embedding for each user slot. As shown in Figure 5, both the personalization and completeness scores decrease after the shift, which is expected because the model is exposed to a new set of domains. As training proceeds, both scores consistently increase, showing that COPE can adapt not only to preference changes within the same broad environment, but also to shifts involving substantially different domains. The recovery after the shift further suggests that COPE can adapt to new preferences even when the relevant domains change substantially. Appendix M Cross-Family User Model Evaluation To test whether the observed gains are tied to the Qwen-family user simulator, we conduct an additional evaluation by replacing the user model with DeepSeek-V3 DeepSeek-AI et al. [6]. The assistant model, chronological traversal protocol, and comparison methods remain unchanged. We use the setting p=0.5p=0.5 and q=0.5q=0.5, where p is the probability of receiving user feedback and q is the teacher-forcing probability in self-evaluation calibration. Initial queries and rubrics are still pre-generated from user preferences, while DeepSeek-V3 provides user feedback during interaction and evaluates both personalization and task completeness. Table 7: Cross-family user model evaluation with DeepSeek-V3 as the user simulator (p=0.5p=0.5, q=0.5q=0.5). Scores are personalization score × binary completeness score, out of 1010 (mean±stdmean ). † indicates methods that involve training. Methods Scores Base 2.16±0.022.16±0.02 SR 3.00±0.013.00±0.01 DR 3.08±0.033.08±0.03 Q-DR 3.30±0.013.30±0.01 PAP 3.31±0.033.31±0.03 T-PAP† 4.06±0.024.06±0.02 COPE† 4.23±0.064.23±0.06 Compared with the Qwen-Flash Alibaba Cloud [1] user setting, DeepSeek-V3 yields lower absolute scores across methods, which is attributable to its stricter criteria when judging task completeness and preference alignment. Nevertheless, the overall ordering is preserved: retrieval and profile-based prompting improve over the base assistant, training-based personalization further improves performance, and COPE obtains the highest score. These results indicate that COPE remains effective under different user-model judging styles. Appendix N Independent Blinded Pairwise Evaluation To address the concern that COPE’s gains may reflect fitting the Qwen-family simulator and its rubric-based scoring rather than genuine preference adaptation, we conduct an independent, blinded pairwise evaluation of COPE and T-PAP responses (p=0.5p=0.5) using a human annotator. We randomly sample 1212 aligned COPE/T-PAP response pairs from each of the 1717 held-out validation steps in the fourth traversal, yielding 204204 pairs in total. For each pair, we randomize the presentation order of the two responses to mitigate position bias and ask the human annotator to select the response that better matches the user. The annotator sees only the user’s demographics, the affinities relevant to the queried domain, the query, and the two responses; the simulator’s rubrics and numerical scores are withheld. Figure 9: Blinded pairwise preferences between COPE and T-PAP from human evaluation. Percentages are computed over all 204204 randomly sampled validation interactions. Across the 204204 pairs, the human evaluation prefers COPE on 120120 pairs and T-PAP on 3636, with 4848 ties. Thus, COPE wins 76.92%76.92\% of the decisive comparisons. The consistent preference for COPE in the blinded, rubric-free human evaluation demonstrates that COPE adapts to human preferences rather than merely learning the simulator’s scoring behavior.

Instant research discovery