Paper deep dive

Learning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning

Yuting Liu, Wei Wu, Jianzhe Zhao, Guibing Guo

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 85

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/13/2026, 4:17:57 AM

Summary

The paper introduces AlignXada, a training-free meta-learning framework for task-specific preference adaptation in Large Language Model (LLM) personalization. It utilizes verbal reinforcement learning to induce reusable textual refinement policies that compress universal user preference summaries into task-specific representations, thereby improving downstream model performance while reducing context noise. The method achieves significant gains across 13 tasks and three models, outperforming Retrieval-Augmented Generation (RAG) in most scenarios.

Entities (9)

AlignXada · method · 98%RAG · baseline-method · 95%Verbal Reinforcement Learning · technique · 92%PersonaMem-v2 · dataset · 90%Refined Profile · data-object · 90%Universal User Profile · data-object · 90%Meta-learner · component · 88%MemoryCD · dataset · 85%Refiner · component · 85%

Relation Signals (8)

AlignXada → produces → Refined Profile

confidence 95% · The refined profile is then provided to downstream models for personalized inference.

AlignXada → transforms → Universal User Profile

confidence 95% · AlignXada adapts universal preference summaries to task-specific ones.

AlignXada → uses → Verbal Reinforcement Learning

confidence 95% · The refinement policy is iteratively optimized by a meta learner through verbal reinforcement learning.

AlignXada → outperforms → RAG

confidence 92% · outperforming RAG in 36 cells.

Refiner → applies → Refinement Policy

confidence 90% · used by a frozen refiner to adapt the universal user preference

AlignXada → evaluatedon → PersonaMem-v2

confidence 90% · On PersonaMem-v2, a faithfulness audit further shows...

Meta-learner → optimizes → Refinement Policy

confidence 90% · The refinement policy is iteratively optimized by a meta learner

AlignXada → evaluatedon → MemoryCD

confidence 85% · we construct a composite benchmark by integrating PersonaMem-v2 and MemoryCD.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Natural language user preferences provide an interpretable interface for LLM personalization. However, universal preference summaries often contain information irrelevant to a particular downstream task. Directly supplying the full preference summary therefore wastes context capacity and introduces cross-task distraction, while manually designing task-specific preference views is difficult to scale. In this work, we study \emph{task-specific preference adaptation}: given a universal user preference summary and a downstream task, derive a task-conditioned representation that preserves sufficient decision-relevant evidence while removing redundant context. To this end, we propose \textsc{AlignXada}, a training-free meta-learning framework that induces reusable textual refinement policies for adapting universal preference summaries to task-specific ones. The refinement policy is iteratively optimized by a meta learner through verbal reinforcement learning. Across 13 tasks and three downstream models (39 task--model cells), \textsc{AlignXada} achieves an average gain of 3.82 points, improving 33 cells while retaining only 22.8\% of the original profile tokens and outperforming RAG in 36 cells. An extended faithfulness analysis further shows that the refined profiles remain largely grounded in the source preferences while preserving task-relevant personalization signals, suggesting that profile-side adaptation serves as a practical complement to universal memory construction for lifelong personalized agents.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

84,149 characters extracted from source content.

Expand or collapse full text

BBAAD9C2010037A16BA0000FCD90B6403519B27D1D132B20A7D9FE32B1A92BBA9B41B73861D17B0725792708B84C113CD0E926A3E1D03B116B1AC805767E3102841088021B72C89794E38D776BE341D6A973EC0275E3C39112E119B2BEA1C92D993819B8E3 Learning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Yuting Liu 1,2 Wei Wu 2,* Jianzhe Zhao 1 Guibing Guo 1,* 1 Software College, Northeastern University, China 2 Ant International * Corresponding authors: Guibing Guo and Wei Wu. liuyuting@stumail.neu.edu.cn, wuwei19850318@gmail.com, guogb,zhaojz@swc.neu.edu.cn Abstract Natural language user preferences provide an interpretable interface for LLM personalization. However, universal preference summaries often contain information irrelevant to a particular downstream task. Directly supplying the full preference summary therefore wastes context capacity and introduces cross-task distraction, while manually designing task-specific preference views is difficult to scale. In this work, we study task-specific preference adaptation: given a universal user preference summary and a downstream task, derive a task-conditioned representation that preserves sufficient decision-relevant evidence while removing redundant context. To this end, we propose AlignXada, a training-free meta-learning framework that induces reusable textual refinement policies for adapting universal preference summaries to task-specific ones. The refinement policy is iteratively optimized by a meta learner through verbal reinforcement learning. Across 13 tasks and three downstream models (39 task–model cells), AlignXada achieves an average gain of 3.82 points, improving 33 cells while retaining only 22.8% of the original profile tokens and outperforming RAG in 36 cells. An extended faithfulness analysis further shows that the refined profiles remain largely grounded in the source preferences while preserving task-relevant personalization signals, suggesting that profile-side adaptation serves as a practical complement to universal memory construction for lifelong personalized agents. Code is available at https://github.com/AntResearchNLP/AlignX-Family/tree/main/AlignXada. Keywords: large language model personalization, task-specific preference adaptation, verbal reinforcement learning, meta-learning 1 Introduction Personalization, the technique of aligning AI systems with human preferences, has played an important role in the train- ing of large language models since their early development [Ouyang et al., 2022]. Recently, with the proliferation of personal AI agents [OpenClaw, 2026, Hermes, 2026] and the expanding use of LLMs in applications such as search [Baek et al., 2024], recommendation [Geng et al., 2022, Lyu et al., 2024, Peng et al., 2026], and dialogue systems [Otsuka et al., 2024], personalization techniques — particularly as a key component in building LLM memories [Wang et al., 2023, Xu et al., 2025, Chhikara et al., 2025, Yu et al., 2025, Zhong et al., 2024] — have become increasingly critical and continue to drive advances at the research frontier. A central problem in personalization is user preference representation, which serves as the interface through which AI systems perceive user preferences and adapt their behav- iors accordingly. Early studies modeled user preferences through user embeddings or by encoding user information directly into model parameters [Koren et al., 2009, Bao et al., 2023, Liu et al., 2025b, Zhang et al., 2025]. While effective, such black-box representations suffer from limited interpretability and are difficult to update in real time. With the emergence of LLMs, in-context learning has enabled a simple yet effective alternative by explicitly incorporating user behaviors into the model context as demonstrations [Du et al., 2026, Salemi et al., 2024]. However, such approaches are fundamentally constrained by the context capacity of LLMs, making it difficult to comprehensively capture user preferences while remaining highly sensitive to noise in the selected examples. More recently, the advancement of LLM reasoning capabilities has inspired efforts to infer universal user preferences from heterogeneous data sources and summarize them in natural language [Chen et al., 2025, Nam et al., 2025, Liu et al., 2026, Yang et al., 2026]. This paradigm offers significant improvements in interpretability and scalability. More importantly, it lays the foundation for lifelong personal agents, where preference representa- Ant International Research1 arXiv:2608.09507v2 [cs.CL] 12 Aug 2026 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning (a) Given your history with knee pain from a college sports injury, you might focus on activities like swimming, gentle yoga... (b) To stay active without putting too much strain on your legs, you might try swimming, gentle yoga... (c) Given your strong connection to community and rich cultural traditions, you might enjoy activities like Zumba ... (d) Given your love of baking as a creative outlet, you might balance that sedentary hobby with light activities like swimming... What are some good ways to stay active without putting too much strain on the legs? Query (task=Personalemail) Downstream Model: GPT-5 Model picks (c) with the universal profile. Wrong: the dominant cross-scenario signal overwhelms the small wellness clue. ... Her professional life is intensely focused on public policy, specifically civil liberties, with a strong emphasis on data privacy, surveillance technology oversight, and protecting vulnerable communities. ... Her political identity is characterized by a blend of fierce advocacy and a warm, community-oriented approach. ... ...often grounding them in personal anecdotes or cultural metaphors to make them more relatable and impactful. ... She has shared stories about her family, her children, a past college soccer injury, a challenging pregnancy, and the stress of her demanding job. ... She is conscious of her physical and mental well-being. She mentions a past college soccer injury, a history of gestational diabetes, and stable ferritin levels, suggesting a proactive approach to her health. ... Universal Profile 푃 7,675 chars | >= 6 scenarios Only ~1% of characters are task-relevant for this query. ... Often weaves personal narratives into communications to build trust and add emotional weight. Examples include stories about her family (children, partner Daniel), a past college soccer injury, or a challenging pregnancy. ... Wellness: Discusses managing stress, work-life balance, mindfulness, and specific health topics (e.g., past soccer injury, HPA axis). ... Refined Profile ෨ 푃 637 chars | 8 % of raw Model selects (a)with the refined profile. Correct: the decision is anchored on task-relevant evidence. dominant distractor Induced policy 휙: compress profile, drop distractors, keep task evidence... task-relevantcluecompeting distractorneutral context Figure 1: An example for task-specific preference adaptation. Given a query, only a small portion of the universal profile푃is relevant to the task, while the remaining information may act as distractors and lead to suboptimal personalization. AlignXada refines푃by retaining task-relevant evidence and removing distracting context, enabling the downstream model to ground its response in the appropriate user information. tions can continuously evolve as user–agent interactions accumulate over time. In this work, we study LLM personalization from the per- spective of universal user preference interfaces. Rather than pursuing improved methods for constructing universal prefer- ence summaries, we assume a universal user profile is already available and investigate a complementary yet practically important problem: how to effectively adapt the universal profile to specific tasks. Our motivation stems from the observation that universal preference summaries, while com- prehensive, inevitably contain redundant or task-irrelevant information that may act as noise in downstream applica- tions. As illustrated in Figure 1, the user query is strongly associated with her past college injury, whereas information such as “community-oriented approach” is largely irrelevant. Such redundant information misleads the downstream model (i.e., GPT-5), resulting in a suboptimal response. In contrast, the result can be substantially improved by using an adapted profile that is shorter and more focused on task-relevant information. Toward task-specific preference adaptation, we propose a framework for transforming universal preference summaries into task-aware representations. Ideally, the adapted rep- resentation should satisfy two desiderata: (1) Sufficiency, preserving all preference information necessary for down- stream personalization; and (2) Compactness, removing redundant or task-irrelevant information to reduce noise and improve context efficiency. To this end, we propose AlignX- ada, a meta-learning framework for preference adaptation. Rather than directly training a model to refine user profiles, AlignXada employs a meta learner to learn structured refinement policies in natural language, which are subse- quently used by a frozen refiner to adapt the universal user preference. This disentanglement between policy generation and preference refinement makes the adaptation process transparent and controllable, enabling human-in-the-loop diagnosis and refinement. The meta learner is optimized via verbal reinforcement learning, where refinement policies are iteratively improved using natural language feedback derived from task-specific demonstrations of user preferences. By avoiding parameter updates during policy learning, AlignX- ada naturally supports both open-source and proprietary models. We evaluate AlignXada on a composite benchmark span- ning nine conversational tasks and four ranking, rating, and generation tasks with three downstream models. Across 39 task–model cells, AlignXada improves 33 cells by an average of 3.82 points while reducing the profile token ratio to 22.8%. It strictly outperforms RAG in 36 cells, showing that task-oriented preference reorganization provides ben- efits beyond query-level retrieval. On PersonaMem-v2, a faithfulness audit further shows that 97.5% of refined-profile claims are supported by the source profiles, while 83.3% of available gold preference evidence is retained, suggesting that AlignXada mainly performs controlled task-specific compression and reorganization. We summarize our contributions as follows: • We formalize the task-specific preference adaptation problem, which aims to adapt universal user prefer- ences to downstream tasks by removing redundant or task-irrelevant information, thereby paving the way for lifelong personalized agents equipped with memory. •We introduce AlignXada, a meta-learning framework that induces natural language refinement policies from a small set of task-specific demonstrations. The policy is iteratively optimized via verbal reinforcement learning, 2 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning making AlignXada compatible with both open-source and proprietary LLMs. •We conduct extensive evaluations across thirteen tasks and three downstream models. The results show that AlignXada consistently achieves a favorable trade-off between task performance and context-token usage, ow- ing to the faithfulness and compactness of the refined preferences. 2 Related Work 2.1 LLM Personalization As large language models (LLMs) evolve from generalized chatbots into personal AI agents, there has been growing interest in building personalized AI systems in which the behavior of a general-purpose LLM is aligned with indi- vidual preferences. Existing approaches can be broadly categorized into four groups. Retrieval-based methods iden- tify relevant user records or profile elements from external memory or databases [Sun et al., 2025, Du et al., 2026, Liu et al., 2023, Shi et al., 2025], and leverage the retrieved content for downstream personalization. Parametric ap- proaches encode persona information into trainable model parameters, such as LoRA modules, adapters, soft prompts, model-merging weights, and rerankers [Tan et al., 2024, Clarke et al., 2024, Li et al., 2023, Liu et al., 2025a, Jang et al., 2023, Zhuang et al., 2024], thereby adapting a general- purpose LLM toward user-specific behavior. In addition, motivated by the strong in-context learning capability of LLMs, prompt-based approaches append preference signals to the input context and steer response generation through personalized prompts [Dong et al., 2023, Yang et al., 2024b, Cheng et al., 2024, Li et al., 2025a]. More recently, with the emergence of lifelong personal agents and LLM memory systems, several studies have advocated learning universal user profiles as long-term and continuously evolving repre- sentations of user preferences [Li et al., 2025b, Liu et al., 2026, Yang et al., 2026]. AlignXada is motivated by this trend toward lifelong personalization. However, rather than constructing universal profiles themselves, AlignXada as- sumes such profiles are already available and studies how to adapt them to specific downstream tasks in real-world applications. In this sense, AlignXada complements ex- isting efforts by bridging the gap between universal profile construction and task-oriented deployment. 2.2 Textual Optimization With advances in LLM reasoning capabilities, recent work has used strong LLMs to optimize natural-language arti- facts without gradient-based training. OPRO [Yang et al., 2024a] iteratively proposes task-level instructions based on reward trajectories, EvoPrompt [Guo et al., 2024] applies evolutionary operators to candidate prompts, TextGrad [Yuk- sekgonul et al., 2024] propagates verbal “gradients” through prompt computation graphs, and Reflexion [Shinn et al., 2023] generates reflective critiques for per-instance trajec- tories. In these methods, the optimized artifact is typically either a task-level instruction shared across users or a per- instance self-correction text, making it user-independent or instance-local. AlignXada extends this paradigm by for- mulating task-specific profile adaptation as the optimization of a reusable user-conditional rewrite policy that transforms each user’s universal preference into a task-specific repre- sentation, rather than introducing a new general-purpose textual optimizer. 3 Methodology Figure 2 presents an overview of AlignXada. In a nutshell, AlignXada consists of two stages: few-shot policy induction and policy deployment for task-specific personalization. During policy induction, the framework leverages a small support set of task-specific demonstrations, each consisting of a universal preference summary, a user query, and the corresponding user response, and iteratively refines the rewriting policy generated by a meta learner using natural language feedback. Once policy optimization converges, AlignXada enters the deployment stage, where the selected policy is consumed by a refiner to produce a refined profile for task adaptation. The refined profile is then provided to downstream models for personalized inference. Throughout the entire process, all models—including the meta learner, the refiner, and the downstream model—remain frozen. In the following, Section 3.1 formalizes the learning problem, and Section 3.2 presents the policy induction procedure based on verbal reinforcement learning. 3.1 Problem Formalization Let푢denote a user and푃 푢 denote the corresponding uni- versal preference summary, which may be obtained from an external LLM or memory system and is treated as prior knowledge in this work. Given a task휏, the objective is to derive a task-adapted profile ̃ 푃 푢 from푃 푢 such that the performance of a downstream model푀for user푢can be substantially improved on task 휏. A common approach to deriving ̃ 푃 푢 from푃 푢 is to learn a generative model푅as a refiner and define ̃ 푃 푢 = 푅(푃 푢 ,M 휏 ), whereM 휏 specifies the task휏. In this work, we instantiate M 휏 as a support set 푆 (휏) defined as 푆 (휏) =(푃 푢 푖 ,푥 푖 , 푦 푖 ) 푏 푖=1 ,(1) where푃 푢 푖 is the universal profile of user푢 푖 ,푥 푖 denotes an in- put prompt,푦 푖 denotes the corresponding response, and푏is the size of the support set. Although the representational ca- pacity of푆 (휏) may be limited by the selection of(푃 푢 푖 ,푥 푖 , 푦 푖 ) tuples and the budget푏, such a formulation naturally aligns with real-world user-AI interactions: user prompts specify the task, while user responses provide signals of task-specific 3 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Upstream PreferenceSource (out of scope) Dataset Provided Memory System Giventask 휏anddownstream model푀. A. Few-shot policy induction on support set푆 ! . SupportSet 푆 (!) =푃 $ ! ,푥 % ,푦 % %&' ( Refinement Policy 휙 ) -Goal -Preserve -Compress -Avoid -Output Style -Priority Refiner 푅 frozenexecutor Downstream Model푀 frozentarget model 푆푐표푟푒 ! (3푦,푦) Reward 푟 ()) Structured Feedback 퐸 ) Meta Learner 휋 *+), frozen policy generator 휙 )-' t = 1..T rounds errors, transitions,... Best-epoch selection: argmax over휙 . ...휙 / Frozen policy휙 (!) for task 휏 B. Policydeploymentfortask-specificpersonalization. Query Example 푃,푥 + Frozen Policy 휙 (!) Refiner 푅 frozen executor Refined Preference 8 푃 Downstream Model푀 frozen target model Prediction 3푦 All models frozen, only policy text updates. Preference Summarizer ... Figure 2: Overview of AlignXada. For each task, AlignXada induces a task-specific textual refinement policy from a small support set in several rounds and then freezes the selected policy for held-out deployment. All models remain frozen throughout, and only the policy text is updated during induction. preferences and behavioral patterns, thereby alleviating the cold-start problem. Instead of updating the parameters of푅, we keep the model frozen and learn a textual refinement policy휙 (휏) using a meta learner휋 meta . The task-adapted profile is then defined as ̃ 푃 푢 = 푅(푃 푢 , 휙 (휏) ) . The learning objective is to maximize E (푃 푢 푖 ,푥 푖 ,푦 푖 )∼푆 (휏) Score 휏 푀(푥 푖 , ̃ 푃 푢 푖 ), 푦 푖 ,(2) whereScore 휏 (·,·)denotes the evaluation function for task 휏. We propose a verbal reinforcement learning approach to optimize Eq. (2), where the refinement policy휙 (휏) is iteratively estimated from휋 meta using feedback derived from 푆 (휏) . Details are presented in next section. 3.2 Policy Induction Overview. Algorithm 1 presents the task-specific pol- icy induction procedure in AlignXada. Starting from a predefined task-agnostic initial policy휙 0 (detailed in Ap- pendix F.1), AlignXada iteratively induces a refinement policy for task휏through two core flows: rollout and update. At each round푡, the rollout flow evaluates the current policy 휙 푡 on the support set푆 (휏) by applying the refiner푅, query- ing the downstream model푀, and computing task-specific scores. The update flow then summarizes the rollout records into structured feedback퐸 푡 , which is used by the meta learner휋 meta to revise the policy and produce the next policy 휙 푡+1 . After all rounds, AlignXada returns the policy with the highest development-set performance. Rollout. At round푡, AlignXada evaluates the current refinement policy휙 푡 on each support example(푃 푢 푖 ,푥 푖 , 푦 푖 ) ∈ 푆 (휏) . The refiner푅applies휙 푡 to rewrite the universal preference summary into a task-adapted profile: ̃ 푃 푢 푖 ,푡 = 푅(푃 푢 푖 , 휙 푡 ).(3) The detailed prompt template is provided in Appendix E.1. The downstream model then conditions on the input 푥 푖 and the refined profile ̃ 푃 푢 푖 ,푡 to generate a prediction: ˆ푦 푖,푡 = 푀(푥 푖 , ̃ 푃 푢 푖 ,푡 ).(4) This prediction is evaluated against the reference response 푦 푖 using the task-specific evaluation function: 푠 푖,푡 = Score 휏 ( ˆ푦 푖,푡 , 푦 푖 ).(5) The rollout record at round 푡 is then defined as R 푡 =(푃 푢 푖 ,푥 푖 , 푦 푖 , ̃ 푃 푢 푖 ,푡 , ˆ푦 푖,푡 , 푠 푖,푡 ) |푆 (휏) | 푖=1 .(6) To reduce overfitting, AlignXada evaluates each policy on a development set퐷 (휏) disjoint from푆 (휏) . Applying Eqs. 3–5 to its examples yields development scores푠 퐷 푖,푡 , whose average defines the policy utility: 퐽 퐷 (휏) (휙 푡 ) = 1 |퐷 (휏) | |퐷 (휏) | ∑︁ 푖=1 푠 퐷 푖,푡 .(7) The evaluated policy and its development-set utility are stored in the history: H ←H ∪(휙 푡 , 퐽 퐷 (휏) (휙 푡 )).(8) Maintaining this history allows AlignXada to retain all eval- uated policies, as verbal policy updates are not guaranteed to improve monotonically. 4 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Algorithm 1 Task-specific policy induction in AlignXada. Require:Task휏; support set푆 (휏) ; development set퐷 (휏) ; frozen meta learner휋 meta ; frozen refiner푅; frozen downstream model푀; evaluation functionScore 휏 ; ini- tial policy 휙 0 ; number of rounds 푇 Ensure: Task-specific refinement policy 휙 (휏) 1: H ←∅⊲ history of evaluated policies 2: for 푡 = 0, 1,...,푇 do 3: R 푡 ←∅⊲ support-set rollout records 4: for all(푃 푢 푖 ,푥 푖 , 푦 푖 ) ∈ 푆 (휏) do 5: ̃ 푃 푢 푖 ,푡 ← 푅(푃 푢 푖 , 휙 푡 ) 6:ˆ푦 푖,푡 ← 푀(푥 푖 , ̃ 푃 푢 푖 ,푡 ) 7:푠 푖,푡 ← Score 휏 ( ˆ푦 푖,푡 , 푦 푖 ) 8: R 푡 ←R 푡 ∪ (푃 푢 푖 ,푥 푖 , 푦 푖 , ̃ 푃 푢 푖 ,푡 , ˆ푦 푖,푡 , 푠 푖,푡 ) 9: end for 10: 퐽 퐷 (휏) (휙 푡 ) ← 1 |퐷 (휏) | Í |퐷 (휏) | 푖=1 푠 퐷 푖,푡 11: H ←H ∪(휙 푡 , 퐽 퐷 (휏) (휙 푡 )) 12: if 푡 < 푇 then 13:퐸 푡 ← AGG(R 푡 ) 14:휙 푡+1 ← 휋 meta (휙 푡 , 퐸 푡 ) 15: end if 16: end for 17: 휙 (휏) ← arg max (휙 푡 ,퐽 퐷 (휏) (휙 푡 ))∈H 퐽 퐷 (휏) (휙 푡 ) 18: return 휙 (휏) Policy update. After the rollout, AlignXada constructs structured feedback: 퐸 푡 = AGG(R 푡 ),(9) whereAGG(·)converts the rollout records into a textual diagnostic summary. The feedback includes scalar signals, such as the average support-set score; instance-level signals, such as predictions and scores; and representative failure patterns derived from low-scoring examples. The complete feedback format is provided in Appendix E.3. Given the current policy and structured feedback, the frozen meta learner produces a revised policy: 휙 푡+1 = 휋 meta (휙 푡 , 퐸 푡 ).(10) The update prompt (provided in Appendix E.2) instructs 휋 meta to follow the predefined policy schema, make targeted revisions based on the feedback, and avoid example-specific rules that merely memorize support instances. The revised policy may adjust the refinement goal, the types of user evidence to preserve, the compression strategy, the error patterns to avoid, the output style, or the prior- ity order among these instructions. An example policy is provided in Appendix F.2. In this way, the meta learner con- verts rollout diagnostics into a reusable task-level refinement policy rather than producing direct answers or per-example corrections. After푇rounds, AlignXada selects and returns the best evaluated policy as the final task-specific refinement policy: 휙 (휏) = arg max (휙 푡 ,퐽 퐷 (휏) (휙 푡 ))∈H 퐽 퐷 (휏) (휙 푡 ). (11) 4 Experiments 4.1 Experimental Setup Benchmark. To evaluate profile refinement in multi- domain, lifelong personalization settings, we construct a composite benchmark by integrating PersonaMem-v2 [Jiang et al., 2025] and MemoryCD [Zhang et al., 2026]. We first derive task-agnostic user summaries from both datasets and extract semantic signatures capturing stable interests, preferences, aversions, and contextual constraints. We then match users one-to-one based on semantic compatibility, excluding pairs with explicit preference conflicts. For each matched pair, we construct a shared universal profile by interleaving and summarizing their PersonaMem and Mem- oryCD histories. Each composite user is evaluated with the same universal profile across 13 downstream tasks: nine PersonaMem-v2 conversational tasks and four MemoryCD tasks—item ranking, rating prediction, review-title gener- ation, and review generation. For each task, we construct user-disjoint support, development, and evaluation sets, en- suring that users involved in policy induction do not appear in the held-out evaluation set. Detailed dataset statistics are provided in Appendix A. We also evaluate the two original benchmarks separately and report the results in Section 4.3. Evaluation Metrics. We evaluate four-way choice ques- tions with exact accuracy, item ranking with Hit@1, rating prediction with Rating-Score, and both generation tasks with ROUGE-L. User profiles are constructed under the all-history memory setting, which summarizes user behavior from the complete historical context. Given the ground-truth rating푦and model predictionˆ푦 ∈ 1, 2, 3, 4, 5, Rating- Score is defined as S( ˆ푦, 푦) = max 0, 1− | ˆ푦−푦| 4 . Alongside task-level scores, we report context compres- sion using the token ratio (TR), defined as the average refined- profile length divided by the average universal-profile length. Implementation Details. In the main experiments, we use Gemini-2.5-Pro [Comanici et al., 2025] to generate a universal user preference from each user’s raw history provided by the benchmark. Unless otherwise stated, both the meta learner and the preference refiner use Gemini-2.5- Pro. We set the support batch size to푏 = 20and the number of policy-update rounds to푇 = 5, subject to our experimental budget. When more demonstrations are available, we select the support examples using the adaptive sampling method described in Appendix B to mitigate sampling bias. The universal-preference baseline directly passes the universal preference to the downstream model without refinement. The RAG baseline uses BM25 [Robertson and Zaragoza, 2009]. 5 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Table 1: Overall performance (%) on the composite benchmark with different downstream models. Higher metric is better unless otherwise specified. Token ratio (TR) denotes the ratio of refined profile token length to those in the original profile. Task-level gains over Raw are significant under a two-sided exact sign test (푝 = 2.44× 10 −4 ). TaskQwen3-8BDeepSeek-V4-FlashGPT-5-mini RawRAGAlignXada TR(%)↓ RawRAGAlignXada TR(%)↓ RawRAGAlignXada TR(%)↓ Chat Acc. 43.46 40.78 −2.68 45.63 +2.17 18.9 55.34 41.75 −13.59 64.08 +8.74 18.1 43.40 40.78 −2.62 49.51 +6.11 19.6 Creative Acc. 55.68 46.59 −9.09 56.82 +1.14 21.7 59.09 54.55 −4.54 73.86 +14.77 22.5 59.32 61.36 +2.04 65.91 +6.59 28.9 Knowledge Acc. 72.19 61.54 −10.65 75.00 +2.81 14.3 82.90 71.01 −11.89 82.84 −0.06 33.5 75.21 71.01 −4.20 73.63 −1.58 18.7 Personal Acc. 34.52 33.33 −1.19 33.33 −1.19 24.9 45.24 39.29 −5.95 54.76 +9.52 20.7 47.62 41.67 −5.95 50.00 +2.38 17.1 Prof.Email Acc. 39.17 35.00 −4.17 37.67 −1.50 25.1 46.67 35.83 −10.84 59.17 +12.5 25.8 41.50 38.33 −3.17 40.83 −0.67 26.4 Prof.Writing Acc. 31.82 27.27 −4.55 40.00 +8.18 14.6 50.00 40.91 −9.09 57.27 +7.27 27.8 44.55 36.36 −8.19 49.09 +4.54 16.8 Social Acc.(%) 36.36 34.09 −2.27 40.91 +4.55 20.3 56.82 53.41 −3.41 62.50 +5.68 37.9 48.41 48.86 +0.45 51.14 +2.73 19.2 Translation Acc. 38.64 34.23 −4.41 42.24 +3.60 24.4 54.95 45.95 −9.00 63.06 +8.11 18.6 54.95 50.45 −4.50 59.46 +4.51 21.7 Trouble Acc. 49.51 46.60 −2.91 53.40 +3.88 22.3 60.19 55.34 −4.85 62.14 +1.94 24.3 55.19 55.19 +0.00 54.37 −0.82 18.5 Ranking Hit@1 59.61 55.77 −3.84 61.53 +1.92 20.8 69.23 78.85 +9.62 82.69 +13.46 31.9 78.69 76.92 −1.77 80.77 +2.08 26.7 Rating Score 75.84 75.48 −0.36 77.40 +1.56 28.2 74.04 76.44 +2.40 78.37 +4.33 27.2 77.88 80.29 +2.41 79.33 +1.45 26.4 Title ROUGE-L 13.46 12.99 −0.47 15.38 +1.92 13.1 12.78 10.76 −2.02 16.02 +3.24 20.4 13.04 13.17 +0.13 13.80 +0.76 30.3 Review ROUGE-L 13.94 13.90 −0.04 14.37 +0.43 22.2 13.89 13.86 −0.03 15.32 +1.43 22.8 11.61 12.03 +0.42 12.06 +0.45 18.4 Avg. Δ –-3.59+2.2720.8–-4.86+7.0025.5–-1.92+2.1922.2 For each query, we segment the universal profile at structural boundaries and further divide it into overlapping windows of 120 words with a 30-word overlap. We use the downstream task prompt as the retrieval query and rank all profile chunks with BM25 (푘 1 = 1.5,푏 = 0.75). The top eight chunks are restored to their original order and concatenated within an approximate 768-token profile budget (푇푅 ≈ 20%), which is comparable to the length of profiles refined by our method. The downstream models are Qwen3-8B [Yang et al., 2025], DeepSeek-V4-Flash [DeepSeek-AI, 2026], and GPT-5-mini [OpenAI, 2025]. We evaluate generalization with DeepSeek-V4-Flash as the meta model in Appendix C, provide further analyses beyond performance comparisons in Appendix D, and present case studies in Appendix G. Unless otherwise specified, Qwen3-8B is used as the default downstream model for these analyses. 4.2 Main Results We evaluate whether AlignXada improves the performance– context trade-off across heterogeneous personalization tasks and downstream models. An effective preference refinement method should reduce preference context while preserv- ing or improving downstream performance. We compare AlignXada with the raw universal-preference and RAG baselines across 39 task–model cells. The results in Table 1 yield three observations. (1) AlignXada consistently improves the performance– efficiency trade-off across models and tasks. Across the39task–model cells, AlignXada improves33, with an average gain of+3.82points over the raw universal pref- erence. All three downstream models improve on average: +2.27points for Qwen3-8B,+7.00for DeepSeek-V4-Flash, and+2.19for GPT-5-mini. The gains also span task for- mats, covering all12MemoryCD cells. Meanwhile, the refined profiles retain only22.8%of the original tokens, and performance gains are nearly uncorrelated with token ratio (푟 = 0.06). Thus, AlignXada improves the utility of retained evidence rather than relying on longer profiles. (2) The advantage over RAG shows that task-oriented preference adaptation goes beyond query-level retrieval. AlignXada outperforms RAG in36cells, ties in one, and underperforms it in only two, with an average margin of +7.28points. In contrast, RAG reduces the raw-profile score by3.46points on average. For example, on professional email with DeepSeek-V4-Flash, RAG lowers accuracy from 46.67%to35.83%, whereas AlignXada raises it to59.17%. Retrieval may surface locally relevant records but fragment preferences whose relevance is indirect or distributed across interactions. AlignXada instead reorganizes the consol- idated profile into a coherent decision context. (3) The optimal preference representation depends on both the task and the downstream model. For professional email, the changes are−1.50,+12.50, and−0.67points for Qwen3- 8B, DeepSeek-V4-Flash, and GPT-5-mini, respectively; for 6 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Table 2: Task-level performance and token ratios on separate PersonaMem-v2 and MemoryCD (values×100%). TaskRaw RAG AlignXada TR↓ Benchmark=PersonaMem-v2 Chat Message30.11 27.4230.9131.4 Creative Writing31.13 27.1332.7837.0 Knowledge Query49.92 43.9848.7143.3 Personal Email30.93 29.8332.3940.4 Professional Email 31.09 27.7233.8338.7 Professional Writing 28.46 24.7327.6651.6 Social Media Post25.99 29.6629.9439.7 Translation31.80 29.6832.4338.5 Trouble Consult32.77 33.0635.2946.3 Avg. Δ–-2.11+1.3040.8 Benchmark=MemoryCD Item Ranking57.14 54.2561.1462.1 Rating Prediction78.86 75.8180.0059.5 Review Title13.47 14.3214.3644.7 Review Generation 13.72 14.0414.4625.8 Avg. Δ–-1.19+1.6948.0 knowledge query, they are+2.81,−0.06, and−1.58points. These sign reversals indicate that no single compression strategy is optimal for all downstream models, motivating the use of downstream feedback for policy induction. Despite this heterogeneity, all regressions remain within1.58points, whereas the largest gain reaches+14.77 points. 4.3 Results on Source-Native Benchmarks The composite benchmark merges heterogeneous histories from PersonaMem-v2 and MemoryCD into a universal user profile, exposing each downstream task to both relevant and unrelated evidence. Although this setting reflects the noisy histories of lifelong agents, AlignXada may benefit mainly from removing cross-domain noise introduced by benchmark construction. We therefore evaluate AlignXada separately on the two source benchmarks, where profiles are built from more domain-coherent native histories, to test whether it remains effective from a cleaner starting point. The results in Table 2 show two patterns. AlignXada remains effective on cleaner, source-native profiles. It improves the average primary metric by1.30points on PersonaMem-v2, outperforming the raw profile on seven of nine tasks, and by1.69points on MemoryCD, improv- ing all four tasks. These gains indicate that AlignXada goes beyond removing cross-domain noise by using support- set feedback to induce preference representations better aligned with downstream requirements. Feedback-based adaptation provides a better performance–compression trade-off than query-level retrieval. AlignXada outper- forms RAG on every task, whereas RAG reduces the average score by2.11points on PersonaMem-v2 and1.19points on MemoryCD. Meanwhile, AlignXada reduces the profiles to average token ratios of40.8%and48.0%, respectively. 0 5 10 15 20 25 30 Mean profile token ratio (%) Lower is better 51020 Feedback batch size -1 0 1 2 3 4 Primary metric change (p) Higher is better 4/13 improved -0.33 7/13 improved +0.73 11/13 improved +2.58 Task trends (norm.)MeanMedianToken ratio Figure 3: Effect of feedback batch size on the composite bench- mark. Solid and dashed lines show the mean and median primary- metric changes, respectively. Gray lines show normalized task-level trends, bars show the mean profile token ratio, and annotations indicate the number of tasks outperforming the raw-profile baseline. This suggests that support-set feedback reorganizes user evidence into a compact representation that is more useful to downstream models than locally retrieved excerpts. 4.4 Hyperparameter Analysis Impact of Support Batch Size푏. Figure 3 compares feedback batch sizes푏 ∈ 5, 10, 20while keeping the in- duction pool and update-round budget fixed. (1) Larger feedback batches improve the reliability of policy induc- tion. As푏increases from5to20, the mean primary-metric change rises from−0.33to+2.58points, the median from −0.91to+1.94points, and the number of improved tasks from4/13to11/13(7/13at푏 = 10). Smaller batches make each update more sensitive to the sampled examples, whereas larger batches provide broader and more balanced feedback, leading to more stable policies. (2) The gains come from better feedback coverage rather than weaker compression.푏 = 20achieves the highest mean and median gains while producing the lowest mean token ratio (20.5%, compared with21.1%at푏 = 5and21.7%at푏 = 10). Thus, its advantage does not result from retaining more con- text. Instead, broader feedback helps distinguish recurring evidence-loss patterns from isolated failures and prioritize more useful preference evidence. We therefore use푏 = 20 as the default, as it provides the best accuracy–compression trade-off among the evaluated settings. Larger batches are not evaluated because they exceed the context budget. Impact of the Number of Update Rounds푇. Figure 4 shows when the final selected policy first appears in diagnos- tic runs with푇 ∈ 5, 10. (1) Effective refinement policies are typically learned within a few updates. Eight of the thir- teen task-specific policies emerge by the second round, and 7 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning 012345678910 The number of update round of the final selected policy. Chat Message Creative Writing Knowledge Query Personal Email Professional Email Professional Writing Social Media Post Translation Trouble Consult Item Ranking Rating Prediction Review Generation Review Title E5 E2 E1 E1 E1 Initial E1 E3 E5 E2 E9 E5 E2 T=5 budget 12/13 policies learned by E5 At or before epoch 5After epoch 5 Figure 4: Update round of the final selected policy in the푇 ∈ 5, 10diagnostic runs. Blue bars denote policies obtained by round five, red bars denote later policies. Claim Support Non- Hallucination Non- Contradiction Source Coverage Optimized Retention Conditional Retention Non-critical Missing 0 20 40 60 80 100 Score (%) 97.5 97.9 99.8 34.0 30.2 83.3 84.7 Claim faithfulnessDecision evidence Faithfulness Audit Figure 5: Faithfulness audit result on the PersonaMem-v2. twelve within five rounds, indicating that the meta-learner can quickly translate support-set feedback into effective poli- cies. (2) Additional rounds offer task-dependent benefits. Rating prediction is the only task whose final policy first appears after round five, at round nine. Moreover, among the seven tasks for which푇 = 10performs better, six select policies generated within the first five rounds. Thus, most tasks converge early, while additional rounds mainly benefit challenging or unstable tasks. Balancing policy quality and inference cost, we use 푇 = 5 in the main experiments. 4.5 Faithfulness Audit Performance and compression alone do not establish whether a refined preference can serve as a faithful substitute for the user universal preference. A refiner may reduce the context budget while removing the decisive preference evidence or introduce fictional traits. We therefore audit the refined preferences along two dimensions: claim-level faithfulness to the input profile and preference evidence retention. Experimental details are elaborated in Appendix D.2. We have two main observations about AlignXada from Figure 5. (1) AlignXada is highly faithful to the source profile. Across all scenarios,97.5%of extracted claims are supported by the original profile, the non-hallucination rate is97.9%, and the non-contradiction rate reaches99.8%. These results show that the verbal policy does not turn the refiner into an unconstrained generator. Instead, AlignX- ada largely functions as a controlled compression-and- reorganization operator that preserves the factual content of the universal preference representation while adapting it for downstream use. (2) The main bottleneck is incomplete decision-evidence availability rather than profile hallu- cination. Only34.0%of benchmark preference evidence is recoverable from the original source profiles, while re- fined profiles retain30.2%overall. This small gap from the source-evidence ceiling suggests that many errors arise from insufficient evidence in the universal profile itself, rather than from the refiner deleting available evidence. Conditional on the original profile containing the golden preference, AlignXada preserves it in83.3%of cases, with a critical missing rate of15.3%. Therefore, the audit provides a more precise interpretation of AlignXada’s downstream limitations: AlignXada can faithfully compress and adapt the profile it receives, but its performance ceiling depends on whether the universal profile contains the cross-topic evidence needed for the decision. The remaining headroom mainly lies in improving source-profile coverage and making decision-relevant evidence more consistently available to the refiner. 5 Conclusion In this work, we study LLM personalization from the per- spective of universal user preference interfaces. We propose AlignXada, which induces a reusable natural-language refinement policy through verbal reinforcement learning, im- proving the performance–budget trade-off while preserving source-supported claims and critical evidence. By enabling effective use of universal preferences, our work offers a new perspective on memory adaptation for lifelong personalized agents. Limitations Despite its advantages, AlignXada has two main limitations. First, although our evaluation spans PersonaMem-v2 and MemoryCD across multiple task formats, both benchmarks are constructed or curated evaluation settings and may not fully capture the evolving, noisy interactions of real-world lifelong agents. Future work should develop broader datasets that provide both universal user preferences and diverse downstream task formats, enabling more comprehensive evaluation of task-specific preference adaptation. Second, AlignXada assumes that the support set used for policy induction is representative of the target task distribution. When the support set is biased or insufficiently diverse, the induced refinement policy may not generalize well. More effective sampling and support-set construction strategies 8 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning remain important directions for future work. Ethics Statement All experimental datasets used in this study are derived from previously published work and obtained either through official APIs or by synthetic construction based on these sources. All user-related information in the datasets has been anonymized, and no personally identifiable information is involved. We do not use any non-open-source data. All data are used solely for scientific research, rather than for commercial purposes or for profiling or decision-making about individuals, and their acquisition and use comply with relevant ethical guidelines and standards of academic integrity. All existing resources used in our experiments, including datasets, pretrained models, and APIs, are accessed and used in accordance with their original licenses and terms of use. The datasets and models we rely on were filtered and processed by their original authors prior to public release to mitigate potential ethical risks. References Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Allen Herring, and Sujay Kumar Jauhar. Knowledge- augmented large language models for personalized con- textual query suggestion. In Proceedings of the ACM Web Conference 2024, pages 3355–3366, 2024. doi: 10.1145/3589334.3645404. URL https://doi.org/10.114 5/3589334.3645404. Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. TALLRec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1007–1014, 2023. doi: 10.1145/3604915.3608857. URL https: //doi.org/10.1145/3604915.3608857. Yizhuo Chen, Xin Liu, Ruijie Wang, Zheng Li, Pei Chen, Changlong Yu, Qingyu Yin, Priyanka Nigam, Meng Jiang, and Bing Yin. POPI: Personalizing LLMs via optimized natural language preference inference. arXiv preprint arXiv:2510.17881, 2025. URL https://arxiv.org/abs/2510 .17881. Chuanqi Cheng, Quan Tu, Wei Wu, Shuo Shang, Cunli Mao, Zhengtao Yu, and Rui Yan. “in-dialogues we learn”: To- wards personalized dialogue without pre-defined profiles through in-dialogue learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10408–10422, 2024. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025. Christopher Clarke, Yuzhao Heng, Lingjia Tang, and Jason Mars. PEFT-U: Parameter-efficient fine-tuning for user personalization. arXiv preprint arXiv:2407.18078, 2024. URL https://arxiv.org/abs/2407.18078. Gheorghe Comanici et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261. DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URL https: //huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/ main/DeepSeek_V4.pdf. Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xi- anchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023. Linfeng Du, Ye Yuan, Zichen Zhao, Fuyuan Lyu, Emil- iano Penaloza, Xiuying Chen, Zipeng Sun, Jikun Kang, Laurent Charlin, Xue Liu, and Haolun Wu. Optimiz- ing user profiles via contextual bandits for retrieval- augmented LLM personalization, 2026. URL https: //arxiv.org/abs/2601.12078. Accepted to ACL 2026. Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt and predict paradigm (P5). In Proceedings of the 16th ACM Conference on Recommender Systems, pages 299– 315, 2022. doi: 10.1145/3523227.3546767. URL https://doi.org/10.1145/3523227.3546767. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Evo- Prompt: Connecting LLMs with evolutionary algorithms yields powerful prompt optimizers. In International Con- ference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2309.08532. Hermes. Hermes agent: The agent that grows with you, 2026. URL https://hermes-agent.nousresearch.com/. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personal- ized soups: Personalized large language model align- ment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023. URL https://arxiv.org/abs/2310 .11564. Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, Ziyi Liu, Anvesh Rao Vijjini, Jiashu He, Hanchao Yu, Radha Poovendran, Gre- gory Wornell, Lyle Ungar, Dan Roth, Sihao Chen, and 9 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Camillo Jose Taylor. PersonaMem-v2: Towards personal- ized intelligence via learning implicit user personas and agentic memory. arXiv preprint arXiv:2512.06688, 2025. URL https://arxiv.org/abs/2512.06688. Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Com- puter, 42(8):30–37, 2009. doi: 10.1109/MC.2009.263. URL https://doi.org/10.1109/MC.2009.263. Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, and Rui Yan. From 1,000,000 users to every user: Scaling up personalized preference for user-level alignment. ArXiv, abs/2503.15463, 2025a. URL https://api.semanticscholar. org/CorpusID:277113478. Jia-Nan Li, Jian Guan, Wei Wu, and Rui Yan. Extended inductive reasoning for personalized preference inference from behavioral signals. ArXiv, abs/2505.18071, 2025b. URL https://api.semanticscholar.org/CorpusID:278886 858. Lei Li, Yongfeng Zhang, and Li Chen. Personalized prompt learning for explainable recommendation. ACM Transactions on Information Systems, 41(4), 2023. doi: 10.1145/3580488. URL https://doi.org/10.1145/3580488. Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. LLMs + persona-plug = personalized LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9373–9385, 2025a. doi: 10.18653/v1/2025.acl-lon g.461. URL https://aclanthology.org/2025.acl-long.461/. Shuai Liu, Hyundong Cho, Marjorie Freedman, Xuezhe Ma, and Jonathan May. RECAP: Retrieval-enhanced context-aware prefix encoder for personalized dialogue response generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8404–8419, 2023. doi: 10.18653/v1/2023.acl-long.468. URL https://aclantholo gy.org/2023.acl-long.468/. Yuting Liu, Jinghao Zhang, Yizhou Dang, Yuliang Liang, Qiang Liu, Guibing Guo, Jianzhe Zhao, and Xingwei Wang. CoRA: Collaborative information perception by large language model’s weights for recommenda- tion. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 39, pages 12246–12254, 2025b. doi: 10.1609/aaai.v39i12.33334. URL https://doi.org/10.1609/aaai.v39i12.33334. Yuting Liu, Jian Guan, Jia-Nan Li, Wei Wu, Jiang-Ming Yang, Jianzhe Zhao, and Guibing Guo. Text as a universal interface for transferable personalization. arXiv preprint arXiv:2601.04963, 2026. URL https://arxiv.org/abs/2601 .04963. Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Chris Leung, Jiajie Tang, and Jiebo Luo. LLM-rec: Personalized recommendation via prompting large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 583–612. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-naacl .39. URL https://aclanthology.org/2024.findings-naacl.3 9/. Hyunji Nam, Yanming Wan, Mickel Liu, Peter Ahnn, Jianxun Lian, and Natasha Jaques. Learning to summarize user information for personalized reinforcement learning from human feedback. arXiv preprint arXiv:2507.13579, 2025. URL https://arxiv.org/abs/2507.13579. OpenAI. GPT-5 system card, 2025. URL https://openai.c om/index/gpt-5-system-card/. OpenClaw. Openclaw: The ai that actually does things, 2026. URL https://openclaw.ai/. Atsushi Otsuka, Kazuya Matsuo, Ryo Ishii, Narichika Nomoto, and Hiroaki Sugiyama. User-specific dia- logue generation with user profile-aware pre-training model and parameter-efficient fine-tuning. arXiv preprint arXiv:2409.00887, 2024. URL https://arxiv.org/abs/2409 .00887. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Ja- cob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Sys- tems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. URL https://proceedings.neurips.c/paper_fil es/paper/2022/file/b1efde53be364a73914f58805a0017 31-Paper-Conference.pdf. Zhiyuan Peng, Xuyang Wu, Huaixiao Tou, Yi Fang, and Yu Gong. Memrerank: Preference memory for personal- ized product reranking, 2026. URL https://arxiv.org/abs/ 2603.29247. Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, page 333–389, 2009. Alireza Salemi, Surya Kallumadi, and Hamed Zamani. Optimization methods for personalizing large language models through retrieval augmentation. arXiv preprint arXiv:2404.05970, 2024. URL https://arxiv.org/abs/2404 .05970. 10 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Teng Shi, Jun Xu, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Yang Song, and Han Li. Retrieval augmented generation with collaborative filtering for personalized text generation. arXiv preprint arXiv:2504.05731, 2025. URL https: //arxiv.org/abs/2504.05731. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 8634–8652, 2023. URL https://papers.nips.c/paper_fil es/paper/2023/hash/1b44b878b782e6954cd888628510 e90-Abstract-Conference.html. Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi Fung, Hou Pong Chan, Kevin Small, ChengXiang Zhai, and Heng Ji. Persona-DB: Efficient large language model personalization for response prediction with collaborative data refinement. In Proceedings of the 31st International Conference on Computational Linguistics, pages 281–296, 2025. URL https://aclanthology.org/2025.coling-main. 20/. Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democratizing large language models via personalized parameter-efficient fine-tuning. arXiv preprint arXiv:2402.04401, 2024. URL https: //arxiv.org/abs/2402.04401. Accepted to EMNLP 2024 Main. Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2306.07174. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: Agentic memory for LLM agents. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL https://arxiv.org/abs/25 02.12110. arXiv:2502.12110. An Yang et al. Qwen3 technical report, 2025. URL https: //arxiv.org/abs/2505.09388. Bufang Yang, Lilin Xu, Yixuan Li, Kaiwei Liu, Xi- aofan Jiang, and Zhenyu Yan. Sensorpersona: An llm-empowered system for continual persona extraction from longitudinal mobile sensor streams, 2026. URL https://arxiv.org/abs/2604.06204. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Confer- ence on Learning Representations (ICLR), 2024a. URL https://arxiv.org/abs/2309.03409. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning, pages 56276–56297, 2024b. Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. Memagent: Reshap- ing long-context llm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.02259, 2025. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. TextGrad: Automatic “differentiation” via text. arXiv preprint arXiv:2406.07496, 2024. URL https://arxiv.org/ abs/2406.07496. Jinghao Zhang, Yuting Liu, Wenjie Wang, Qiang Liu, Shu Wu, Liang Wang, and Tat-Seng Chua. Personalized text generation with contrastive activation steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7128–7141, 2025. doi: 10.18653/v1/2025.acl-lon g.353. URL https://aclanthology.org/2025.acl-long.353/. Weizhi Zhang, Xiaokai Wei, Wei-Chieh Huang, Zheng Hui, Chen Wang, Michelle Gong, and Philip S Yu. Memorycd: Benchmarking long-context user memory of llm agents for lifelong cross-domain personalization. arXiv preprint arXiv:2603.25973, 2026. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. MemoryBank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731, 2024. doi: 10.1609/aaai.v38i17.29946. URL https://doi.org/10.1609/aaai.v38i17.29946. Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. HYDRA: Model fac- torization framework for black-box LLM personalization. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2406.02888. arXiv:2406.02888. 11 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Table 3: Statistics of the constructed composite benchmark. TaskSupport Query Total Personal Email4084210 Professional Email40120256 Professional Writing40110269 Creative Writing4088239 Translation40111264 Trouble Consult40103243 Chat Message40103226 Social Media Post4088217 Knowledge Query40169416 Item Ranking4052131 Rating Prediction4052131 Review Title Generation4052131 Review Generation4052131 Total520 1,184 2,864 Table 4: Statistics of the PersonaMem-v2 benchmark. TaskSupport Query Total Personal Email49424473 Professional Email50485535 Professional Writing63453516 Creative Writing52457509 Translation52478530 Trouble Consult49452501 Chat Message49444493 Social Media Post56426482 Knowledge Query61661722 Total481 4,280 4,761 A Benchmark Statistics Tables 3–5 summarize the task distributions used in our main and source-native evaluations. Since a user may contribute multiple targets and PersonaMem-v2 users do not necessar- ily have an example in every conversational scenario, the number of available records varies across tasks. In Table 3, Support denotes the fixed induction budget used for each task, Query denotes the held-out instances used for final evaluation, and Total denotes all constructed task records before runtime support and development subsampling; the latter therefore also includes candidate induction and devel- opment records not displayed in separate columns. Overall, the composite benchmark contains2,864records across13 tasks. For the source-native evaluations, PersonaMem-v2 contributes4,761examples across nine conversational tasks, while MemoryCD contributes2,240examples across four recommendation and generation tasks. All experiments use user-disjoint induction and evaluation splits to prevent policy Table 5: Statistics of the MemoryCD benchmark. TaskSupport Query Total Item Ranking160400560 Rating Prediction160400560 Review Title Generation160400560 Review Generation160400560 Total6401600 2240 learning from exploiting evaluation-user histories. B Adaptive Sampling for Policy Learning The meta learner has a limited context budget, so each policy update can use at most푏support examples even when a larger candidate pool is available. This makes the choice of support examples important. Fixed or randomly sampled batches may over-represent easy successes, persistent failures, or isolated regressions, leading to updates that are either too conservative or too reactive. To obtain a more balanced diagnostic view, we use adaptive diagnostic sampling, which selects support examples from multiple outcome-transition states. In the main experiments, we reserve a candidate pool C (휏) =(푃 푢 푖 ,푥 푖 , 푦 푖 ) 푚 푖=1 for adaptive support selection and a development set퐷 (휏) for final policy selection. Here,푚 denotes the number of candidate examples available in the experimental split rather than a method-level budget. At each update, the sampler selects푏examples fromC (휏) to form the support set푆 (휏) , where푏 ≤ 푚. This allows each update to remain within the context budget while preserving broader coverage across users, topics, and difficulty levels. In deployment, the candidate-pool size is determined by the available task-specific interactions. For each support example, we compare two binary out- comes: the outcome under the raw universal preference and the most recent outcome under the current refinement policy. This comparison assigns each example to one of four transition states: •improved: the raw profile is incorrect, but the refined profile is correct, indicating useful rewritten-profile rep- resentations; • regressed: the raw profile is correct, but the refined profile is incorrect, exposing evidence that may have been removed or made less usable; •stable-success: both the raw and refined profiles are correct, identifying safe compression behavior; •persistent-failure: both the raw and refined profiles are incorrect, revealing evidence requirements that neither profile makes accessible to the downstream model. At each update round, the sampler partitions the candidate pool by these transition states and allocates the support budget uniformly across them. If푏is not divisible by 12 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning 6065707580 Context token reduction (%) - higher is better -4 -2 0 2 4 6 8 Primary metric change (p) - higher is better Chat Message Creative Writing Knowledge Query Personal Email Professional Email Professional Writing Social Media Post Translation Trouble Consult Item Ranking Rating Prediction Review Title Review Generation FixedAdaptive Figure 6: Fixed versus adaptive support sampling on the composite benchmark. Each arrow connects the same task under fixed sampling (circle) and adaptive sampling (square). The horizontal axis reports context-token reduction and the vertical axis reports the change in the task-specific primary metric; higher is better on both axes. four, the remaining slots are assigned deterministically, with priority given to improved and regressed examples. If a transition state has too few examples, its unused slots are redistributed to the remaining states. This strategy provides the meta learner with both corrective signals from failures and stabilizing signals from successes in each update. We compare fixed and adaptive sampling in Figure 6 and make two observations. (1) Adaptive sampling improves the overall performance–compression trade-off rather than trading additional context for better predictions. The mean task-level gain in the primary metric increases from+1.22points under fixed sampling to+2.58points under adaptive sampling, while the number of tasks that improve over the raw-profile baseline rises from7/13to 11/13. At the same time, the mean context-token reduction increases from68.3%to72.7%. Ten of the thirteen arrows move upward, ten move rightward, and eight move in both directions. Adaptive sampling therefore yields a broadly more favorable shift in performance–compression space by improving the selection of feedback examples used for policy induction, rather than by retaining longer refined profiles. (2) The primary benefit is the recovery of tasks for which fixed sampling induces unstable or incomplete policies. Adaptive sampling changes the gains for creative writing, knowledge query, review title generation, and review gener- ation from negative to positive. It also increases the gain for rating prediction from+0.48to+5.85points, professional Table 6: Performance and token ratio with DeepSeek-V4-Flash as the meta model, rewrite model, and the target model (values ×100%). TaskRaw RAG AlignXada-D TR↓ Chat Message55.34 41.7560.2416.2 Creative Writing59.09 54.5568.7123.1 Knowledge Query82.90 71.0179.1630.7 Personal Email45.24 39.2948.8116.9 Professional Email 46.67 35.8356.6724.6 Professional Writing 50.00 40.9152.1326.5 Social Media Post56.82 53.4161.0631.8 Translation54.95 45.9561.4820.4 Trouble Consult60.19 55.3460.9122.3 Item Ranking69.23 78.8582.4233.7 Rating Prediction74.04 76.4477.8326.2 Review Title12.78 10.7615.3520.5 Review Generation 13.89 13.8614.5018.4 writing from+5.45to+8.18points, and trouble consultation from+0.97to+3.88points. These tasks benefit from repeat- edly exposing the meta-learner to both regressions, which reveal decision-relevant evidence discarded by the current policy, and persistent failures, which reveal evidence that the policy still fails to surface. The effect is not uniform: fixed sampling remains stronger for chat message, personal email, and translation, while professional email remains below the raw-profile baseline. Adaptive sampling should therefore be interpreted as a more reliable strategy for allocating feedback across heterogeneous tasks, rather than as a guarantee of improvement on every task. C Generalization Across Different Meta Mod- els To assess whether AlignXada generalizes beyond its de- fault model configuration, we replace Gemini-2.5-Pro with DeepSeek-V4-Flash for policy induction and profile rewrit- ing. We also use DeepSeek-V4-Flash as the downstream model, yielding an all-DeepSeek variant, AlignXada-D, while keeping the learning procedure and experimental protocol unchanged. AlignXada remains effective when all model roles use DeepSeek-V4-Flash, indicating that its gains are not specific to Gemini-2.5-Pro. AlignXada-D improves12 of the13tasks over the raw universal preference, with an average gain of+4.47points. It improves eight of the nine PersonaMem-v2 tasks, averaging+4.22points, and all four MemoryCD tasks, averaging+5.04points. The refined profiles retain only23.95%of the original tokens. These results demonstrate that AlignXada transfers across model families and task formats while maintaining a favorable performance–efficiency trade-off. Its consistent advantage over RAG suggests that this transfer stems from preference-level adaptation rather 13 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning than model-specific retrieval behavior. AlignXada-D outperforms RAG on all13tasks by9.33points on aver- age, whereas RAG falls4.86points below the raw-profile baseline. Its largest gains over the raw profile occur on item ranking (+13.19), professional email (+10.00), and creative writing (+9.62), spanning ranking, classification, and open-ended generation. Thus, feedback-guided pref- erence reorganization remains effective even when policy induction, rewriting, and downstream inference use the same model family. Different policy-induction and rewriting models never- theless produce distinct performance–compression trade- offs. For the same DeepSeek-V4-Flash downstream model, the default Gemini-based configuration achieves an average gain of+7.00points with a token ratio of25.5%, compared with+4.47points and23.95%for AlignXada-D. AlignX- ada-D therefore produces slightly more compact profiles but smaller performance gains, with knowledge query as its only regression (−3.74points). These results distinguish frame- work generality from model interchangeability: AlignXada transfers across model families, but the choice of policy- induction and rewriting models still affects how effectively decision-relevant evidence is identified and retained. D Diagnostic Analysis on PersonaMem-v2 D.1 Preference Source Robustness To examine whether AlignXada’s behavior depends on the quality and format of the universal preference represen- tation, we construct three sources of universal preference information on PersonaMem-v2. Source A summarizes a user’s identity, background, interests, communication style, and other attributes using GPT-5 [OpenAI, 2025], which is provided by the benchmark. We use A to test whether AlignXada can still improve a compact, curated persona preference that has already removed much of the conversational detail. Source B uses the persona’s raw structured JSON record from the benchmark. This record is the generator-side persona artifact used to create the bench- mark’s preferences, conversation snippets, and answers. To avoid direct conversation leakage, we remove the benchmark conversation branch while retaining the structured persona and preference fields. We use B as a high-information structured-source stress test to measure whether a verbal refiner can replace hand-designed field selection when ex- plicit preferences are available. Source C summarizes each user’s raw chat history into a comprehensive natural lan- guage preference description using Gemini-2.5-Pro. Unlike A and B, C is therefore a history-derived preference rather than a persona-generation artifact. We use C as the default source in the main experiments because it best matches our target setting: a system first summarizes long user–assistant interactions into a rich but task-agnostic preference summary, and AlignXada then adapts that profile into a task-specific preference representation. The results are shown in Table 7, from which we draw the following observations. (1) AlignXada works best as a task-specific refiner for natural-language preference pro- files, rather than as a replacement for upstream profile construction. Both Source A and Source C improve after refinement, with absolute gains of+0.61%and+1.30%, respectively, whereas Source B starts from the strongest raw baseline but decreases after refinement. This pattern suggests that AlignXada benefits from sources whose evi- dence is already expressed in natural language, while raw structured records may require a more field-aware transfor- mation. (2) The gains on A and C come from reorganizing available preference evidence into a more task-oriented form. Source A has the lowest raw accuracy (25.67%), because the dataset-provided expanded persona is compact and often lacks the full cross-topic evidence required by downstream tasks. AlignXada still improves it to26.28%, showing that even a limited persona can benefit from task- specific reorganization. Source C starts from a stronger raw baseline (32.47%), as the comprehensive profile contains richer user history, and AlignXada further improves it to 33.77%. This makes Source C the best match for our target setting: it is sufficiently evidence-rich for personalization while remaining in natural language, allowing the refiner to reliably compress and reorganize it. (3) The drop on B suggests a format mismatch rather than a lack of useful information. Source B’s high raw accuracy (32.91%) is expected, because it is the structured persona used to synthe- size the benchmark and often contains explicit preference aligned with the query-required evidence. However, after refinement, accuracy drops to32.07%. One likely reason is that many useful signals are stored as fine-grained structured fields or leaf values; rewriting them into a compact natural- language profile can remove exact decision evidence that the downstream model can directly read. From the compression perspective, this result also shows that strong compression alone is insufficient. The best trade-off is achieved by Source C, where AlignXada obtains a positive gain while reducing the profile to 40.8% of its original token length. D.2 Faithfulness Audit Details In Section 4.5, we audit refined preferences along two dimensions: claim-level faithfulness to the input profile and preference evidence retention. The experimental setup and implementation details are as follows: Claim-level faithfulness audit. For each unique raw– refined preference pair, we use DeepSeek-V4-Pro [DeepSeek- AI, 2026] as the judge model to decompose the refined preference into atomic claims and label each claim with respect to the universal preference as supported, contradicted, not found, or too vague. Let푁 푐 denote the total number of atomic claims across all audited pairs. The three claim- level metrics—Claim Support, Non-Hallucination, and Non- 14 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Table 7: Ablation on the source of the universal preference (values×100%). Source ASource BSource C TaskRaw AlignXada TR(↓) Raw AlignXada TR(↓) Raw AlignXada TR(↓) Chat Message28.3728.6436.8 30.1929.7316.4 30.1130.9131.4 Creative Writing24.5824.1349.7 33.4231.8931.2 31.1332.7837.0 Knowledge Query33.2137.8435.3 52.7650.1823.6 49.9248.7143.3 Personal Email22.7323.4653.2 31.9530.2728.7 30.9332.3940.4 Professional Email 23.6421.3745.9 29.5828.4227.3 31.0933.8338.7 Professional Writing 24.8225.1645.6 26.9427.3126.5 28.4627.6651.6 Social Media Post26.3526.7845.2 30.6729.1423.8 25.9929.9439.7 Translation25.9127.3241.5 34.2834.7621.4 31.8032.4338.5 Trouble Consult21.4621.8334.7 26.3726.9230.1 32.7735.2946.3 Avg. Δ–+0.6143.1–-0.8425.4–+1.3040.8 Contradiction—are defined as follows: Claim Support = #supp. 푁 푐 , Non-Hallucination = 1− #notf.+ #cont. 푁 푐 , Non-Contradiction = 1− #cont. 푁 푐 . (12) Here, supported indicates that a claim is explicitly stated or semantically entailed by the universal preference, not found indicates unsupported new information, and contradicted indicates inconsistency with the source. Claims labeled too vague are included in푁 푐 but are not counted as supported. We define the judge prompt as follows: Claim-Level Faithfulness Audit Audit the refined profile against the original profile. Labels: - supported: the claim is explicitly stated or semantically entailed by the original profile. - contradicted: the claim conflicts with the original profile. - not_found: the claim adds specific information not present in the original profile. - too_vague: the claim is so generic that it cannot be checked as a concrete profile claim. Return exactly one JSON object with this schema: "claims": [ "claim": "atomic claim from the refined profile", "label": "supported | contradicted | not_found | too_vague", "evidence": "short supporting or contradicting evidence from the original profile, or empty string", "rationale": "brief reason" ] Split the refined profile into concise atomic claims. Do not invent claims that are not in the refined profile. <OriginalProfile> original_profile </OriginalProfile> <RefinedProfile> refined_profile </RefinedProfile> Preference-evidence audit. PersonaMem-v2 provides a “preference” field for each benchmark query, which we treat as ground-truth decision evidence. For each audited query, the judge determines whether this evidence is preserved in the universal preference and in the refined preference. Each preference is labeled as exact/specific, semantically generalized, missing, or contradicted, where the first two labels are counted as evidence retained. Let푁 푒 denote the number of audited query examples. We define four evidence-level metrics: Source Coverage, Refined Retention, Conditional Retention, and Non-Critical Missing: Source Coverage = #source ret. 푁 푒 , Refined Retention = #refined ret. 푁 푒 , Conditional Retention = #source ret. & refined ret. #source ret. , Non-Critical Missing = 1− #critical missing #source ret. . (13) 15 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning Here, source ret. indicates that the universal preference contains the ground-truth evidence, and refined ret. indi- cates that the refined preference preserves it. A critical missing case occurs when the universal preference retains the evidence but the refined preference is labeled missing. The judge prompt is defined as follows: Preference-Evidence Audit Audit whether each profile preserves the benchmark preference. Labels: - exact_or_specific: the profile explicitly preserves the preference or a highly specific equivalent. - semantic_generalized: the profile preserves the preference at a broader but still useful level. - missing: the profile does not contain usable evidence for the preference. - contradicted: the profile states the opposite or says the preference was retracted/should not be used. Return exactly one JSON object with this schema: "original": "label": "exact_or_specific | semantic_generalized | missing | contradicted", "evidence": "short evidence span from the original profile, or empty string", "rationale": "brief reason" , "optimized": "label": "exact_or_specific | semantic_generalized | missing | contradicted", "evidence": "short evidence span from the refined profile, or empty string", "rationale": "brief reason" <BenchmarkPreference> benchmark_preference </BenchmarkPreference> <OriginalProfile> original_profile </OriginalProfile> <RefinedProfile> refined_profile </RefinedProfile> E Prompt Templates E.1 Rewrite Prompt Template The refiner receives the current policy휙 푡 , the downstream task family description, and the original profile 푃 푢 . Prompt Template for Preference Generation Current rewrite pattern phi_t: current_policy Downstream task family: task_description Original profile P_u: original_profile Rewrite the profile strictly according to phi_t. Requirements: 1. Execute phi_t exactly, including its preservation, compression, abstraction, structure, and style rules. 2. Preserve factual faithfulness to P_u; do not add unsupported facts, preferences, or assumptions. 3. Keep the rewrite query-agnostic and do not optimize it for a specific prompt, option, answer, or evaluation instance. 4. Output only the rewritten profile P*_u. E.2 Meta-Learner Prompt Template The meta learner receives the current policy and mini-batch feedback, then emits only the next policy. Prompt Template for Policy Update Meta prompt version: prompt_version Downstream task family: task_description Primary optimization metric: primary_metric_name Current pattern phi_t: current_policy Support-set error information (JSON): feedback_payload Optimization objective: - Improve future rewritten-profile performance for the same task family while updating only the rewrite policy, not the downstream answer behavior. - Preserve or improve the primary metric first; reduce redundant context only after retaining decision evidence. - Keep the policy query-agnostic so it can serve unseen examples rather than the current support prompts. 16 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning - Treat concrete user facts, constraints, sensitive or anti-stereotypical preferences, and privacy behavior as potential decision evidence unless the feedback shows they are harmful. Use feedback: - Prioritize regressions as evidence of omitted, distorted, or under-weighted signals; preserve behaviors that explain improvements and stable successes. - Separate evidence loss from downstream confusion, ambiguity, or source-profile insufficiency before changing the policy. - Balance latent evidence with ordinary concrete evidence such as roles, places, objects, language/register, named interests, and everyday preferences. Constraints: - Make the smallest useful update to phi_t and avoid overfitting to one example, user, topic, or surface form. - next_pattern must describe how to rewrite profiles, not how to answer the downstream task. - Return only the final JSON. E.3 Structured Feedback Template The structured feedback record퐸 푡 is serialized as a JSON list, where each entry corresponds to one support example in the mini-batch. By default, the feedback does not include the full raw preference; instead, the meta learner observes the rewritten profile, paired outcomes, scores, and task metadata. Feedback Template [ "idx": int, "prompt": string, "reference": string, "optimized_profile": string, "original_prediction": string, "optimized_prediction": string, "original_primary_score": float, "optimized_primary_score": float, "score_delta": float, "original_metrics": object, "optimized_metrics": object, "meta_summary": "transition": "improved | regressed | persistent_failure | stable_success", "scenario": string, "pref_type": string, "topic_query": string, "reference_letter": string, "gold_signal": string, "original_prediction": string, "original_signal": string, "optimized_prediction": string, "predicted_signal": string, "prediction_changed": bool , "original_profile_words": int, "optimized_profile_words": int, "word_reduction": int ] F Policy Examples F.1 Initial Policy All experiments start from the same task-agnostic initial policy: Initial Policy Rewrite the profile into concise, stable preference rules that preserve relevant user signals while reducing unnecessary detail. F.2 An example of Induced Policies Example Induced Policy Goal: Produce a compact, faithful refined profile that lets a Qwen3-8B downstream answer a 4-way personal-email choice question grounded in the user’s relationship context, tone, and recurring email phrasing, while preserving cross-scenario evidence that disambiguates the choice. Preserve: - Relationship roles and names of recurring email recipients. - Tone, greetings, signoffs, and recurring phrasing by relationship. - Lifestyle constraints and cross-scenario preferences that affect email decisions. Compress: - Narrative chit-chat from chat history. - Duplicated preferences or long descriptions that do not change the email decision. Avoid: - Removing cross-scenario evidence or confusing the user’s preferences with those of family members. - Copying exact answer-text phrasings from chat history. Output Style: 17 ANT INTERNATIONAL RESEARCHLearning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning - Bulleted, with explicit field labels and relationship/scenario tags. - No prose narrative. Priority: When in doubt, preserve cross-scenario preference evidence over local email-style fluff. G Case Study We select a representative user case, together with its raw universal preference and refined preference, as a case study. The raw universal preference leads the downstream model to an incorrect answer, whereas the AlignXada-refined profile leads it to the correct one. The user asks for calming activities or audio options after a long day. Among the four candidate responses, the gold answer recommends a tranquil Bach harpsichord suite or softly recorded Gregorian chant. With the raw universal preference, the downstream model selects a response centered on Japanese shakuhachi and koto music; after refinement, it selects the gold response. The boxes below show lightly formatted excerpts copied from the corresponding JSON outputs, with omitted material marked by ellipses. Raw Universal Preference Excerpt This user is a highly educated, articulate, and reflective individual, likely an academic in the humanities, with a professional focus on history, theology, and liturgy, particularly within the Anglican tradition. Their name is Elaine H. Nakamura. They reside in or near Portland, Oregon, and are of East Asian, specifically Japanese, descent. Elaine is deeply engaged in both her professional life as a university lecturer and her personal life within a parish community, where she is a lay leader and choir member. . . . The rhythms of the church calendar (Advent, feast days) are significant markers in her life. Her aesthetic sensibilities are refined and consistent. She appreciates: - Music: She is a choir member, rehearsing and performing works by composers like Tallis. She has a sophisticated ap- preciation for classical music (Bach’s Jesu, meine Freude), choral traditions, and the emotional impact of music in different settings. She also enjoys karaoke with friends. - Literature: She is an avid reader, enjoying 19th-century novels like Middlemarch, mystery novels (P.D. James, Sarah Waters), and poetry. Reading is a cherished ritual, often done in the evening with tea and a cozy blanket. - Visual Arts & Design: She is drawn to art galleries, the- ater (both scripted and improvisational), and the aesthetics of interior design . . . AlignXada-Refined Profile Excerpt Tone & Style - Formality: Prefers a polite, formal, and articulate com- munication style, even in personal emails. - Tone: Warm, reflective, and slightly literary. - Vocabulary: Uses precise, rich, sensory vocabulary (e.g., “faint patter of rain,” “ancient cadences”). . . . Content & Interests - Professional Identity: University lecturer in Church History. This is a central part of her identity and informs her worldview. - Core Academic Interests: Liturgical practices and history (Anglican tradition, medieval mystery plays, fu- neral liturgies); theology and its intersection with culture; 19th-century parish library development. - Cultural & Aesthetic Preferences: - Music: Classical (Bach), choral traditions (Tallis). Participates in a parish choir. Also enjoys karaoke with friends. - Literature: 19th-century novels (Middlemarch), mys- tery novels (P.D. James, Sarah Waters), poetry. Reading is a cherished ritual. - Arts: Enjoys art galleries, theater (scripted and im- prov), and appreciates handmade crafts like greeting cards. This example shows that AlignXada can improve down- stream behavior without introducing new user facts. The raw preference already contains the relevant evidence, but it presents decision-relevant musical and liturgical prefer- ences alongside many other salient identity and cultural details. In this case, the raw model appears to overweight the user’s Japanese heritage and selects the candidate men- tioning shakuhachi and koto. The refined profile reorganizes the same source-supported information into task-relevant clusters: classical music, choral practice, Anglican parish life, 19th-century reading, and quiet evening rituals. This makes the Bach/Gregorian-chant candidate more directly supported than the Japanese-instrument distractor, while reducing the profile by 46.6%. 18

Instant research discovery